How a Shell Reads Commands | Building a Tokenizer in Assembly

Follow along with me in the x64Playground: https://x64.halb.it (Select GNU as the Assembler!) Watch the course from the beginning:    • Learn Assembly For Beginners | Introductio...   Next Video in Course:    • How Strings Are Compared in Assembly | Beg...   Previous Video in Course:    • Arrays in Assembly | Beginner Tutorial | L...   00:00 How Shells Read Commands 03:10 How Tokenizing Works 06:15 How to View Memory on x64Playground 09:50 Writing the Tokenize Function 13:05 Finding the End of Words 17:26 Building the Array of Tokens 18:05 A rogue function that corrupts our memory! 19:40 Skipping Empty Space Function 23:18 Stopping the rogue black hole function 28:20 Printing a String Array (Printing Tokens) 32:42 Println Function in Assembly (Inefficient But Works!) 33:40 More Testing & Some Edge Cases ;## Our functions to copy and paste. Below that is the FULL PROGRAM if you want to test it out without writing it yourself. exit: xor rdi, rdi mov rax, 60 syscall print: push rdi call slen pop rsi mov rdx, rax mov rax, 1 mov rdi, 1 syscall ret slen: xor rcx, rcx slen.loop: mov al, [rdi + rcx] cmp al, 0 je slen.ret inc rcx jmp slen.loop slen.ret: mov rax, rcx ret readline: mov rdx, rsi mov rsi, rdi mov rax, 0 mov rdi, 0 syscall ret ;###### ;### FULL PROGRAM BELOW ###### .intel_syntax noprefix .global _start .text _start: lea rdi, [buf] lea rsi, [bufsize - 1] call readline cmp rax, 0 je nodata lea rdi, [buf] mov byte ptr [rdi + rax], 0 call memview lea rdi, [buf] lea rsi, [tokens] call tokenize int3 call print_tokens call memview call exit nodata: lea rdi, [s_nodata] call print call exit print_tokens: lea r15, [tokens] pt.loop: mov rdi, [r15] cmp rdi, 0 je pt.end call println add r15, 8 jmp pt.loop pt.end: ret tokenize: mov r15, rsi tokenize.loop: call skipwhite cmp byte ptr [rdi], 0x0 je tokenize.end mov [r15], rdi add r15, 8 call memview call findend cmp byte ptr [rdi], 0 je tokenize.end mov byte ptr [rdi], 0 inc rdi call memview jmp tokenize.loop tokenize.end: ret findend: cmp byte ptr [rdi], ' ' je findend.ret cmp byte ptr [rdi], 0 je findend.ret inc rdi jmp findend findend.ret: ret skipwhite: cmp byte ptr [rdi], ' ' je skipwhite.advance ret skipwhite.advance: inc rdi jmp skipwhite exit: xor rdi, rdi mov rax, 60 syscall print: push rdi call slen pop rsi mov rdx, rax mov rax, 1 mov rdi, 1 syscall ret println: push rdi call slen pop rsi mov rdx, rax mov rax, 1 mov rdi, 1 syscall mov rax, 1 mov rdi, 1 lea rsi, [newline] mov rdx, 1 syscall ret slen: xor rcx, rcx slen.loop: mov al, [rdi + rcx] cmp al, 0 je slen.ret inc rcx jmp slen.loop slen.ret: mov rax, rcx ret readline: mov rdx, rsi mov rsi, rdi mov rax, 0 mov rdi, 0 syscall ret memview: mov r14, rsp lea rsp, [buf] int3 mov rsp, r14 ret .data buf: .skip 128 ;# 4096 bufsize = . - buf tokens: .skip 128 newline: .byte '\n' s_nodata: .asciz "\nNo data entered!\n" :#######