Show HN: Skeletoken, a Python package for editing model tokenizers
stephantul Friday, February 06, 2026
Hello!
I just released a new version of Skeletoken, a package for editing tokenizers. New in this version is the ability to automatically adapt a model to an edited tokenizer.
For example, you can a new token to your tokenizer, and then ask skeletoken to add new token indices in the correct embedding tables.
Let me know what you think!
Summary
The article introduces Skeletoken, a Python library for generating skeleton-like tokens from text data. It describes the library's capabilities, such as handling different types of text and providing options for customizing the token output.
1
0
Summary
github.com