Exact parallel and streaming tokenization for LLM tokenizers: decides from a tokenizer's definition where its input can be cut, then encodes in parallel or as a stream with token ids byte-identical to sequential encoding. GPT-2, cl100k, o200k and Llama 3, in C++23.
unicode tokenizer parallel-computing tokenization automata-theory finite-automata bpe byte-pair-encoding gpt-2 cpp23 llm tiktoken pretokenization cl100k certified-splitting exact-tokenization parallel-tokenization tokenizer-validation
-
Updated
Nov 4, 2026 - C++