dev-willbird1936/gguf-rwkv-trie-oob-poc

llamacpp-05 PoC: RWKV tokenizer trie-walk heap out-of-bounds read

llamacpp05_rwkv_trie_oob.gguf is a minimal GGUF file (640 bytes) that selects the RWKV tokenizer (tokenizer.ggml.model = "rwkv") and defines a 3-token vocabulary where one token's escaped text decodes to bytes containing an embedded NUL (0x00). Building the tokenizer's naive_trie from that token plants a NUL edge inside the trie.

llama_tokenize's RWKV session tokenize loop advances the scan position with a pre-increment, text[++position], with no bounds check against text.size(). When the walk reaches the input's real end, the C-string terminator at text[size()] matches the trie's NUL edge and the loop does not stop: the next pre-increment reads text[size()+1], one byte past the heap-allocated std::string buffer.

Confirmed with AddressSanitizer: heap-buffer-overflow READ of size 1 at llama-vocab.cpp:1323, reached through the real, public llama_tokenize API, no internal-only code path involved.

Load this file with any llama.cpp build (or any consumer that calls llama_tokenize on text long enough to exceed the small-string optimization threshold) to trigger the read.