Why LLMs Live In 12,288 Dimensions

Modern AI systems represent words, images, proteins and entire documents as points in spaces with hundreds, thousands or even tens of thousands of dimensions. But geometry behaves very differently in these spaces. In high dimensions: • Unit spheres disappear • Almost all points lie near the boundary of a cube • Random vectors become nearly orthogonal • Distances become surprisingly predictable • Entire functions become almost constant These phenomena are examples of one of the most remarkable ideas in modern mathematics: concentration of measure. In this video, we explore the strange geometry of high-dimensional spaces, prove several fundamental concentration results, and discuss how these ideas connect to modern machine learning, embeddings, transformers and deep learning. Along the way we'll encounter: • The disappearing sphere • Boundary concentration • Distance concentration • Random orthogonality • Equator concentration • Lévy's concentration theorem • The curse and blessing of dimensionality • Modern AI and representation learning High-dimensional geometry is often counterintuitive, sometimes shocking, and increasingly important for understanding how modern machine learning systems operate. Timestamps 00:00 Introduction 01:03 The Disappearing Sphere 03:37 Distance Concentration 05:31 Random Orthogonality 08:45 Boundary Concentration 10:01 Equator Concentration 13:44 Lévy's Concentration Theorem 17:23 Machine Learning & AI 30:25 Open Questions #MachineLearning #ArtificialIntelligence #DeepLearning #llm #Transformers #NeuralNetworks #ConcentrationOfMeasure #HighDimensionalGeometry #AI