Houssem Ouertatani's thesis

Confiance.AI is part of the France2030 program.

The defense will take place on Friday December 6 at 10am at Inria Lille - Euratechnologies (170 Av. de Bretagne, 59000 Lille), meeting room on the first floor. The presentation will be in English.

Neural network architectures are central to the immense success of deep learning in many tasks. The search for neural network architectures (NNAs) is a critical task in the development of efficient deep learning models, enabling new efficient components or configurations to be discovered, or existing architectures to be optimally adapted to the hardware constraints of deployment, for example via hardware-adaptive NNAs.

Although black-box optimization algorithms are well suited to these problems, the high cost of evaluating individual solutions puts the emphasis on high-sampling-efficiency methods like Bayesian Optimization (BO).
In this work, we start from the fundamental principles underlying the sampling efficiency of Bayesian Optimization, and leverage certain attributes of NAS and the inherent flexibility and predictive performance of Deep Ensembles to significantly reduce the search time and resources needed to efficiently explore search spaces.

On NAS benchmarks, this search strategy achieves a 100x speed-up compared with random search, and up to 50% reduction in search time compared with methods based on Bayesian Optimization or local search.

Designing neural network architectures is a complex problem that can quickly lead to combinatorial explosion. A judicious design of the search space is an important prerequisite for quickly finding high-performance models.

The design of neural network architectures is a complex problem that can rapidly lead to combinatorial explosion.

We demonstrate the versatility and effectiveness of this search approach on several search spaces of different types and degrees of complexity. We focus on finding and improving efficient vision model architectures. We demonstrate that it is possible to find new high-performance models while limiting the computational costs required.

• Amir Nakib LISSI, Université Paris-Est Créteil Rapporteur
• Saïd Mahmoudi ILIA, Université de Mons Rapporteur
• Carola Doerr LIP6, CNRS, Sorbonne Université Examiner
• Mohamed Daoudi CRIStAL, ULille & IMT Nord Europe Examiner

• El-Ghazali Talbi CRIStAL, Université de Lille Thesis co-director
• Smail Niar LAMIH, UPHF Thesis co-director
• Cristian Maxim IRT SystemX Supervisor and referent at IRT SystemX
  .