Inverse Probleme

  1. Universität
  2. Fakultäten
  3. Mathematisch-Naturwissenschaftlich-Technische Fakultät
  4. Institut für Mathematik
  5. Lehrstühle
  6. Inverse Probleme
  7. Aktuelles

Kommender Vortrag: „Concentration phenomena of self-attention dynamics“

Am 15.07.2026 um 16:00 Uhr im Raum 2004 des L-Gebäudes hält Prof. Leon Bungert einen Vortrag zum Thema "Concentration phenomena of self-attention dynamics".
 

Abstract:

In this talk I will speak about concentration phenomena of self-attention transformers in the regimes of infinitely many layers and tokens.The dynamics are described by the Fokker--Planck equation
involves the inverse heat parameter $\beta>0$.The matrices $V,B\in\mathbb{R}^{d\times d}$ contain learned parameters and are assumed to be constant in time.
It is known that for $\beta\to\infty$ solutions of \eqref{eq:Fokker-Planck} converge to solutions of a linear PDE, the solutions of which concentrate as $T\to\infty$ on the dominating eigendirections of the matrix $VB^\top$.
In our work we will quantify these results by exploiting a striking similarity between \eqref{eq:Fokker-Planck} and the so-called polarized consensus-based optimization (CBO) method for global optimization.
Using a CBO-inspired analysis we give explicit bounds for the Wasserstein-2 distance of the solution of \eqref{eq:Fokker-Planck} and a suitable target measure.
The proof relies on an application of a quantitative Laplace principle to \eqref{eq:consensus-point} as well as a Lyapunov-type analysis for the time asymptotics.
Our result sheds more light on the interior dynamics of self-attention transformers and might help identify reduced effective models. This is joint work with Albert Alcalde, Konstantin Riedl, and Tim Roith.

Suche