Inside MAX Serve: From Prompt to Response

MAX serve is Modular's open source inference server. In this interview, AI Performance Engineer Kyle Caverly walks through what happens from the moment a request arrives to the moment text streams back to the client. You can explore Kyle's diagram here: https://drive.google.com/file/d/1Zigs... All of the code discussed is open source. Start with the MAX serve repository: https://github.com/modular/modular/tr... 0:00 Intro 0:35 MAX serve architecture 3:27 API server receives request 5:48 Server creates TextContext object 9:23 Request reaches the model worker 12:25 Construct the batch via TextBatchConstructor 15:48 Prefix caching and chunked prefill 21:23 Pipeline execution 25:24 Consuming completed tokens 27:43 Post-process output and prepare response 29:39 Client receives response 30:51 Multimodality 33:28 Open source code