Byte Pair Encoding Merge Process

Color Key

Character vocabulary

Early subword merges

Later subword merges

Whole words as tokens

Merge / frequency step

Hover over a node to see how each Byte Pair Encoding step works.