On the spectral theory of groups of affine transformations of compact nilmanifolds

Bachir Bekka (1) , Yves Guivarc'H (1)

1 IRMAR - Institut de Recherche Mathématique de Rennes

Let $N$ be a connected and simply connected nilpotent Lie group, $\La$ a lattice in $N$, and $\nil$ the corresponding nilmanifold. Let $\Affnil$ be the group of affine transformations of $\nil$. We characterize the countable subgroups $H$ of $\Affnil$ for which the action of $H$ on $\nil$ has a spectral gap, that is, such that the associated unitary representation $U^0$ of $H$ on the space of functions from $L^2(\nil)$ with zero mean does not weakly contain the trivial representation. Denote by $T$ the maximal torus factor associated to $\nil$. We show that the action of $H$ on $\nil$ has a spectral gap if and only if there exists no proper $H$-invariant subtorus $S$ of $T$ such that the projection of $H$ on $\Aut (T/S)$ has an abelian subgroup of finite index. We first establish the result in the case where $\nil$ is a torus. In the case of a general nilmanifold, we study the asymptotic behaviour of matrix coefficients of $U^0$ using decay properties of metaplectic representations of symplectic groups. The result shows that the existence of a spectral gap for subgroups of $\Affnil$ is equivalent to strong ergodicity in the sense of K.~Schmidt. Moreover, we show that the action of $H$ on $\nil$ is ergodic (or strongly mixing) if and only if the corresponding action of $H$ on $T$ is ergodic (or strongly mixing).

Mots clés: spectre d'opérateur de Markov; nilvariétés; théorie ergodique; groupes de Lie nilpotent

Domaines: Systèmes dynamiques [math.DS]

Bachir Bekka, Yves Guivarc'H. On the spectral theory of groups of affine transformations of compact nilmanifolds. Annales Scientifiques de l'École Normale Supérieure, 2015, 48 (3), pp.607-645. ⟨hal-00602648⟩