cl100k

Here are 4 public repositories matching this topic...

NeaByteLab / BPE-Tokenizer

Byte pair encoding tokenizer for OpenAI models that encodes text into token IDs, decodes them back, and counts tokens.

Updated Sep 28, 2026
TypeScript

Exact parallel and streaming tokenization for LLM tokenizers: decides from a tokenizer's definition where its input can be cut, then encodes in parallel or as a stream with token ids byte-identical to sequential encoding. GPT-2, cl100k, o200k and Llama 3, in C++23.

unicode tokenizer parallel-computing tokenization automata-theory finite-automata bpe byte-pair-encoding gpt-2 cpp23 llm tiktoken pretokenization cl100k certified-splitting exact-tokenization parallel-tokenization tokenizer-validation

Updated Nov 4, 2026
C++

adam2go / puretiktoken

Star

tiktoken in pure Python: byte-for-byte compatible BPE tokenizer for GPT-4/4o, no Rust extension, zero dependencies

tokenizer wasm openai pure-python gpt bpe pyodide llm tiktoken token-counting cl100k o200k

Updated Jun 19, 2026
Python

enquannacc / tokenizer-cost-benchmark

Star

Measured chars/4 against cl100k on 10,000 prompts: 40.5% of rows off by more than 20%, and the error flips sign by content type.

benchmark tokenizer dataset openai cost-estimation llm tiktoken token-count cl100k o200k

Updated Nov 3, 2026
Python

Add this topic to your repo

To associate your repository with the cl100k topic, visit your repo's landing page and select "manage topics."

Learn more

Navigation Menu