Skip to Main content Skip to Navigation
New interface
Journal articles

Asymptotic study of stochastic adaptive algorithm in non-convex landscape

Abstract : This paper studies some asymptotic properties of adaptive algorithms widely used in optimization and machine learning, and among them Adagrad and Rmsprop, which are involved in most of the blackbox deep learning algorithms. Our setup is the non-convex landscape optimization point of view, we consider a one time scale parametrization and we consider the situation where these algorithms may be used or not with mini-batches. We adopt the point of view of stochastic algorithms and establish the almost sure convergence of these methods when using a decreasing step-size towards the set of critical points of the target function. With a mild extra assumption on the noise, we also obtain the convergence towards the set of minimizers of the function. Along our study, we also obtain a \convergence rate" of the methods, in the vein of the works of [GL13].
Document type :
Journal articles
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03857182
Contributor : Service Publications DSR-SCD-UT1C Connect in order to contact the contributor
Submitted on : Thursday, November 17, 2022 - 10:40:53 AM
Last modification on : Friday, November 25, 2022 - 9:10:15 AM

File

wp_tse_1175.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03857182, version 1

Citation

Sébastien Gadat, Ioana Gavra. Asymptotic study of stochastic adaptive algorithm in non-convex landscape. Journal of Machine Learning Research, 2022, 23 (228), pp.1-54. ⟨hal-03857182⟩

Share

Metrics

Record views

0

Files downloads

0