Tokenizer MicroSim

To use this MicroSim in your course, just add the following HTML line to your website:

<iframe src="https://dmccreary.github.io/tracking-ai-course/sims/tokenizer/main.html"  height="470px" scrolling="no"
  style="overflow: hidden;"></iframe>

Run the Tokenizer MicroSim

Edit the Tokenizer MicroSim

References

• OpenAI Tokenizer Demo - uses the text background color to show individual tokens.

• HuggingFace Xenova Tokenizer Playground - Allows you to compare 14 different tokenizers!. This one also color codes each word.