Triton Kernels Actually Work - Here's Proof

In this episode of Bielik Anatomy, it’s finally moment of truth time! We are putting together every single custom OpenAI Triton kernel we’ve built so far—RMSNorm, MatMul, RoPE, Flash Attention, and SwiGLU-into a fully functional Bielik 1.5B Instruct model architecture. Watch step-by-step as we construct the complete decoder layer, solve the unexpected challenge of handling bias in linear and activation layers, load the official pretrained weights, and spin up an interactive chat interface to talk with Bielik. We also dive deep into the engineering numbers: I benchmark our custom Triton implementation against native PyTorch on an RTX 4060 Ti. While we achieved a massive 28% speedup in Time To First Token (TTFT), we also hit a major memory bandwidth bottleneck during autoregressive generation. Let’s look under the hood and break down why! Whether you’re into GPU kernel optimization (Triton, CUDA, Thunder Kittens), system programming, or deep learning infrastructure, this episode connects all the dots. What you’ll learn in this episode: Bielik 1.5B Instruct architecture overview - Why biases changed our kernel strategy. AHA! Moment #1: Writing a custom 2D-grid Embedding kernel vs. native PyTorch. Kernel Fusion in action: Adding Bias to MatMul and SwiGLU with zero branching overhead using compile-time flags (`tl.constexpr`). Assembling the Decoder Layer (GQA, RoPE, and Flash Attention integration). Loading HuggingFace `safetensors` weights and implementing proper parameter mapping. The Ultimate Benchmark: Latency, TTFT wins, and why the lack of a KV Cache limits overall throughput. Project Repository & Code: https://github.com/qooba/bielik-anato... Chapters / Timestamps 0:00 - Intro: Stitching our Triton kernels together! 0:27 - Bielik 1.5B Instruct architecture & the bias problem 1:18 - AHA! Moment #1: Custom Embedding kernel (Triton vs PyTorch) 2:43 - AHA! Moment #2: Zero-cost MatMul Bias fusion via tl.constexpr 4:00 - AHA! Moment #3: Fused SwiGLU with Bias in GPU registers 4:50 - Assembling the Decoder Layer (GQA, RoPE, and Flash Attention) 6:15 - Building the full model & autoregressive sampling mechanics 7:07 - How to load safetensors weights from HuggingFace 7:18 - IT'S ALIVE! Interactive chat with our scratch-built Bielik 7:44 - Benchmarks: Triton vs PyTorch latency on RTX 4060 Ti 8:50 - The Big Bottleneck: Why missing KV Cache swallows our gains 9:48 - Outro & what’s coming up next! #triton #gpu #llm #pytorch #bielik