The article investigates the presence of untrained tokens in various tokenizers, with a focus on single-byte tokens and their implications for model performance. Common issues arise due to the inclusion of fallback bytes and the duplication of ASCII characters as tokens. Although identification of untrained tokens is complex and varies by model, the study highlights the effectiveness of indicators in detecting these tokens, as well as notable patterns that emerge across different model architectures. The findings suggest a need for improved tokenizer design to enhance model training efficiency.
Many tokenizers include every byte as a token, but duplicate tokens for standard ASCII characters, resulting in a complex landscape of trained and untrained tokens.
Untrained single byte tokens are usually classified as 'partial UTF-8 sequences' or 'unreachable', which our indicators effectively identify through rigorous analysis.
Collection
[
|
...
]