Play with a simple tokenizer

Type up to 50 characters and watch your text break into colorful tokens — just for fun.

Enter text (50 char max)

BPE-like (GPT-style)

Greedy subword merges; splits common suffixes/prefixes.

Unigram-like (SentencePiece-ish)

Probabilistic subwords; often prefers shorter, frequent units.

Token Count: 0