Stanford CS330 I Unsupervised Pre-training for Few-shot Learning l 2022 I Lecture 8

Unsupervised pre-training for few-shot learning, vol. 2: reconstruction-based methods For more information about Stanford's Artificial Intelligence programs visit: https://stanford.io/ai To follow along with the course, visit: https://cs330.stanford.edu/ To view all online courses and programs offered by Stanford, visit: http://online.stanford.edu​ Chelsea Finn Computer Science, PhD Plan for Today Recap Problem formulation Contrastive learning Reconstruction-based unsupervised pre-training Why reconstruction? Autoencoders Masked autoencoders: BERT, MAE Autoregressive models: GPT, Flamingo Goals for by the end of lecture: Familiarize you with widely-used methods for unsupervised pre-training Introduce methods for efficient fine-tuning of pre-trained models Prepare you for HW3