GPT-style language model with Byte Pair Encoding tokenizer, built from scratch in PyTorch.
Java library implementing Byte-Pair Encoding Tokenization
A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.
This project implements a tokenizer based on the Byte Pair Encoding (BPE) algorithm, with additional custom tokenizers, including one similar to the GPT-4 tokenizer.
[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust
From-scratch Marathi BPE tokenizer with Flask API and web interface for encoding/decoding Marathi text using Byte Pair Encoding algorithm.
A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.
MayaTok is a Byte Pair Encoding based Tokenizer.
A web app to compare pre-built or self-built tokenizers
Strings Tokenization with Byte Pair Encoding (BPE).
A pure Python implementation of Byte Pair Encoding (BPE) tokenizer. Train on any text, encode/decode with saved models, and explore BPE tokenization fundamentals.
implementation of BPE algorithm and training of the tokens generated
Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)
A byte-level BPE tokenizer built from scratch in Python with Unicode support, vocabulary learning, encode/decode, serialization, tests, and compression evaluation.
self made byte-pair-encoding tokenizer
This repository is reimplementation of Transformer model which was introduced in 2017 NeurIPS paper "Attention is all you need"
Natural Language Processing course assignments @ University of Tehran
Build a byte-level BPE tokenizer from scratch in Python, exploring tokenization, Unicode, vocabulary design, encoding/decoding, and token efficiency through hands-on experiments.
To associate your repository with the bytepairencoding topic, visit your repo's landing page and select "manage topics."