なぜAIに「単語」は存在しないのか －トークナイザー－

AI reads text not by letters or words, but by units called "tokens." This video explains how to create these tokens (BPE) and why AI struggles with character counting, using a real tokenizer. The AI ​​mechanism is explained with relatable examples and diagrams. From intuition to mechanism, formulas, and implementation, each video gently delves into one topic. 📖 Timestamps 0:00 Opening — The Mystery of Strawberry 1:03 Intuition: "Tokens" that are neither letters nor words 1:55 Concrete Example: Separating with a Real Tokenizer 4:31 Mechanism: BPE → byte-level → vocabulary → ID 8:21 Summary and Next Episode Preview ▶ Next Episode: What Does It Mean for Machines to Have "Meaning"? – Embedding – 🔔 Subscribe to the channel so you don't miss the next episode → https://www.youtube.com/@zunda-tsumug... ── Cast ── 🟢 Zundamon: The Explainer. Likes to understand things in terms of structure. 🌶️ Kasukabe Tsumugi: The Listener. This video addresses all of the viewers' common questions. 【For those who:】 ・Use ChatGPT or Claude but don't know the inner workings ・Want to understand the mechanism rather than just the superficial usage 【Primary sources referenced】 Sennrich, Haddow, Birch 2016 "Neural Machine Translation of Rare Words with Subword Units" (BPE) https://aclanthology.org/P16-1162.pdf Radford et al. 2019 "Language Models are Unsupervised Multitask Learners" (GPT-2, byte-level BPE) https://cdn.openai.com/better-languag... OpenAI tiktoken https://github.com/openai/tiktoken Kudo & Richardson 2018 "SentencePiece" https://aclanthology.org/D18-2012/ [BGM/SE] BGM: Amacha Music Studio https://amachamusic.chagasi.com/ Sound Effect Lab https://soundeffect-lab.info/ [Voice] VOICEVOX: Zundamon, Kasukabe Tsumugi https://voicevox.hiroshiba.jp/ #AIexplanation #LLM #ChatGPT #LargeScaleLanguageModel #Zundamon