Chemistry SMILES demand different tokenization than natural language—and the choice matters

The field of chemical machine learning has borrowed byte-pair encoding (BPE) from natural language processing without asking whether it's optimal for molecules. This paper shows that BPE and its underexplored alternative, Unigram-LM, produce fundamentally different tokenization schemes for chemistry, suggesting that picking a tokenizer is a real modeling decision, not just an implementation detail inherited by convention.

The core finding: BPE and Unigram-LM barely overlap

When the author compared BPE and Unigram-LM head-to-head on SMILES (the standard string notation for molecules), they discovered the two algorithms build almost completely different vocabularies. Across 22 matched experimental conditions—varying corpus type (diverse, drug-like, natural-products), pre-tokenization rules, and vocabulary size—the Jaccard overlap (a measure of how many learned tokens the two share) never exceeded 0.161. When weighted toward frequent tokens that a model actually trains on, overlap dropped to just 5%.

This is striking. In natural language, BPE and Unigram-LM are known to differ, but chemistry appears to amplify that divergence sharply.

The two algorithms also segment molecules differently in quantity: Unigram-LM breaks held-out molecules into 29–41% more tokens than BPE. Yet they largely agree on where to cut; the difference is how deeply. On 80–99% of molecules, BPE's splits form a strict coarsening of Unigram-LM's finer segmentation. In other words, BPE makes bigger chunks, but from the same breakpoints.

How the study works (and what it doesn't do)

The author performed a controlled comparison using a fixed 165-token chemistry base vocabulary across three corpus typologies and two boundary-handling policies. Vocabulary sizes stayed small (where token embeddings remain learnable without excessive parameter counts). Crucially, the study trained no language models—it focused purely on how the two tokenization algorithms construct their subword inventories, avoiding confounds from downstream task performance.

This methodological restraint is useful: it isolates the algorithmic choice from model training dynamics, clarifying that the divergence is inherent to how BPE and Unigram-LM score and rank candidate merges, not an artifact of downstream learning.

Why this matters

Tokenization is often treated as a solved preprocessing step in chemistry, with practitioners defaulting to BPE because it works in NLP. This paper demonstrates that the choice is not neutral. Because the two methods carve vocabulary space so differently—sharing almost no learned tokens, and segmenting molecules at different granularities—they likely encode molecules into embedding space in structurally incompatible ways. A model trained on BPE tokens learns a different inductive bias than one trained on Unigram-LM tokens.

The robustness of the separation across corpus type, pre-tokenization boundary policy, and even eightfold vocabulary scaling suggests the effect is fundamental, not a quirk of one experimental setup. For practitioners building chemical language models, this implies that tokenization is a hyperparameter worth tuning, not a default to inherit. Future work could evaluate whether one algorithm better preserves chemical structure or improves downstream task performance, but this paper establishes that they are not interchangeable.