Setting up context in a local LLM

Setting up the right context window size is important when configuring a local LLM. Too high, and you will exceed your VRAM, which has performance implications. Too low, and your LLM will not be able to hold enough information at a time to work through problems. Let's talk about how to pick the right number, and how to test the results of our efforts. 00:00 Introduction 01:29 Tokenization 02:43 Measuring VRAM usage 04:09 Setting max context 05:35 Dialing it back 06:34 Real world test: English text 08:11 Real world test: Java code 09:32 Trying with max size 11:30 Summarizing the results 12:02 Next steps 12:26 Outro