Foundation Models in Deep Learning

Veranstaltungsart: Vorlesung + Übung (Master)
Modulsignatur: INF-0093
Credits: 2 + 2 SWS, 5 LP
Turnus: summer term 
Empfohlenes Semester:
ab 1. Semester
Prüfung: oral exam, each term
Sprache: English


In this lecture, we will cover the most recent approaches and principles at the frontier of learning and applying of foundation models in language, vision (image generation, image segmentation, image understanding) as well as multimodal (=multimedia) data processing, including

  1. learning vision foundation models from natural language supervision, with applications to open-vocabulary image classification and retrieval, object detection, segmentation, and multimodal understanding;
  2. learning vision foundation models via masked image modeling, with its extensions to multimodal pre-training
  3. vision foundation model architecture design with transformer and beyond.
The lecture will be regularly updated to cover the latest developments in the field.



There will be exercises each week. The solutions are discussed in the exercise lecture in the following week. The exercises are not mandatory (but essential for passing the exam).



The lecture is offered each summer term.