Pinned Loading

1. munch munch Public

   Runtime lexers for real languages, and exact decisions about where a scan may be cut: certified boundaries, exact parallel tokenization for BPE, WordPiece and Unigram, and an auditor for flex, re2c…

   C++ 3

2. hopper hopper Public

   A C++23 toolkit for recursive-descent parsers over munch lexers: one-token lookahead, source spans, structured errors and certified lexical recovery, with an RFC 8259 JSON parser whose documents re…

   C++ 2

3. matisse matisse Public

   Exact parallel and streaming tokenization for LLM tokenizers: decides from a tokenizer's definition where its input can be cut, then encodes in parallel or as a stream with token ids byte-identical…

   C++ 1