PhD defence of Mido Assran – “Algorithmic Advances Towards Efficient Learning Machines”

Abstract

Efficient learning is a hallmark of human intelligence; from infancy, we have the remarkable ability to learn novel concepts from very few examples, using a brain that runs on the energy equivalent to an electric razor. Replicating such behaviour in computers is a long-standing challenge of machine learning research with the potential to yield material benefits for society.

This thesis improves the efficiency of learning by producing algorithmic advances in several directions. The first part of this thesis presents theoretical and empirical advances in numerical optimization that enable more efficient training of large-scale machine learning models on distributed computing devices, while the second part of this thesis presents theoretical and empirical advances in representation learning that improve the label-efficiency of learning. Indeed, the ability of humans to quickly acquire new concepts from few examples depends greatly on the many previously constructed abstractions and prior experiences, and one way for an agent to encode prior knowledge and experience is by learning to represent data in ways that facilitate processing.

Together, Parts I and II provide progress towards learning methods that can more efficiently utilize distributed training hardware and training data, so as to build more efficient learning machines. The development of more efficient learning frameworks presents the potential to democratize the practice of machine learning by reducing the computational burden of model training and enabling more effective learning in low-resource settings. Fundamental advances in learning efficiency may also turn out to be chiefly important for longer term goals towards advancing machine intelligence.