The BEST Way to Chunk Text for RAG

To try everything Brilliant has to offer—free—for a full 30 days, visit https://brilliant.org/AdamLucek/ You’ll also get 20% off an annual premium subscription! Resources: Chunking Notebook: https://github.com/ALucek/chunking-st... ChromaDB Technical Report: https://research.trychroma.com/evalua... ChromaDB Report Repo: https://github.com/brandonstarxel/chu... OpenAI Token Visualizer: https://platform.openai.com/tokenizer Greg Kamradt 5 Levels of Text Splitting: https://github.com/FullStackRetrieval... Jaccard Index: https://en.wikipedia.org/wiki/Jaccard... Chapters: 00:00 - Background on Text Chunking 02:28 - Brilliant! 03:47 - Character Text Splitting 06:28 - Token Text Splitting 10:26 - Recursive Character/Token Splitting 16:07 - Kamradt & Modified Semantic Chunking 20:43 - Cluster Semantic Chunking 24:46 - LLM Semantic Chunking 27:56 - Chunking Metrics & Comparison 30:00 - Overall Findings #ai #programming #datascience This video is sponsored by Brilliant

Join Today
Improving RAG Retrieval by 60% with Fine-Tuned Embeddings
▶︎

Improving RAG Retrieval by 60% with Fine-Tuned Embeddings

GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem
▶︎

GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem

RAG vs. CAG: Solving Knowledge Gaps in AI Models
▶︎

RAG vs. CAG: Solving Knowledge Gaps in AI Models

Knowledge Graph or Vector Database… Which is Better?
▶︎

Knowledge Graph or Vector Database… Which is Better?

The 5 Levels Of Text Splitting For Retrieval
▶︎

The 5 Levels Of Text Splitting For Retrieval

Does Fine Tuning Embedding Models Improve RAG?
▶︎

Does Fine Tuning Embedding Models Improve RAG?

Don't learn AI Agents without Learning these Fundamentals
▶︎

Don't learn AI Agents without Learning these Fundamentals

Find the BEST RAG Strategy with Domain Specific Evals
▶︎

Find the BEST RAG Strategy with Domain Specific Evals

Is RAG Still Needed? Choosing the Best Approach for LLMs
▶︎

Is RAG Still Needed? Choosing the Best Approach for LLMs

AI Agents for Beginners – Part 1 (Free Labs)
▶︎

AI Agents for Beginners – Part 1 (Free Labs)

How AI agents & Claude skills work (Clearly Explained)
▶︎

How AI agents & Claude skills work (Clearly Explained)

Do Reranking Models Actually Improve RAG?
▶︎

Do Reranking Models Actually Improve RAG?

Advanced RAG with LlamaIndex - Metadata Extraction [2025]
▶︎

Advanced RAG with LlamaIndex - Metadata Extraction [2025]

Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker
▶︎

Turing Award Winner: Disagreeing with Google, Postgres, Future Problems | Mike Stonebraker

Building Production RAG Over Complex Documents
▶︎

Building Production RAG Over Complex Documents

Chunking Best Practices for RAG Applications
▶︎

Chunking Best Practices for RAG Applications

OWASP's Top 10 Ways to Attack LLMs: AI Vulnerabilities Exposed
▶︎

OWASP's Top 10 Ways to Attack LLMs: AI Vulnerabilities Exposed

RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models
▶︎

RAG vs Fine-Tuning vs Prompt Engineering: Optimizing AI Models

Practical RAG - Choosing the Right Embedding Model, Chunking Strategy, and More
▶︎

Practical RAG - Choosing the Right Embedding Model, Chunking Strategy, and More

The AI Scaling Problem
▶︎

The AI Scaling Problem