Paper 01 · Natural language processing

Reading a character by its parts

A drop-in structural embedding that lets a language model see inside a Chinese character, using decompositions Unicode already publishes.

Under review at ACL Rolling Review 2027 Repository →

Authors. Liqian Yan and Thomas Sing-wing Wu, equal contribution Co-first author

Venue. ACL Rolling Review 2027 Under review

Method. A recursive Tree-MLP over canonicalised IDS parse trees, fused at the embedding interface. 2.5M params · +5% train

What a tokenizer throws away

Read 樹 and you will notice 木 on the left, the tree radical. Even on first meeting the character, you can place it: something wooden, something growing. A tokenizer notices none of this. It looks the character up in a table, hands the model an integer, and the anatomy is gone before the first layer runs.

For a common character that loss is affordable, because context will eventually teach the model what it means. A rare one appears too seldom to accumulate those updates. A character outside the vocabulary altogether reaches the model as [UNK], an empty token. Scale leaves this untouched, because pre-training only ever sees what the input interface has kept.

Rare forms become compositions
Character	IDS decomposition	Meaning
语	讠吾	Language
湖	氵胡	Lake
森	木林	Forest

What the model is given

Unicode already publishes the decomposition. An Ideographic Description Sequence writes 辯 as ((立,十), (亠,二,口), (立,十)): a left-middle-right operator over three branches, every one of which splits again. CNM-BERT parses that string into a tree, walks it from the leaves upward with a small recursive network, and folds the resulting vector into the standard embedding before the first transformer layer sees anything.

The character 辯 at the root of a tree. A left-middle-right operator splits it into three branches; the outer two split again under top-bottom operators into 立 and 十, and the middle branch splits under a top-middle-bottom operator into 亠, 二 and 口 — seven atomic components in all. — **Fig. 1.** Canonical IDS parse tree for 辯 (U+8FAF). Internal nodes are layout operators; the seven leaves are atomic Unicode components, and the Tree-MLP composes the tree from the bottom up. With scripts on, type or pick a character to see its own tree beside what the tokenizer sees; decompositions come from the BabelStone IDS database the paper trains on.

Everything downstream is left alone: same tokenizer, same 21,128-token vocabulary, same output head, same twelve or twenty-four transformer layers. The structural pathway adds about 2.5 million parameters against BERT-base's 110 million, and roughly 5% to training time, because the tree is evaluated once per distinct character in a batch rather than once per token position. It drops into an existing fine-tuning pipeline and leaves it as it was.

A character tokenizer feeds two parallel streams: the unchanged token embedding, and a Tree-MLP encoder over the character's IDS parse tree. A fusion layer combines them and passes the result to an unmodified transformer backbone. — **Fig. 2.** A drop-in rather than a replacement. Only the embedding interface changes. The backbone, vocabulary and output space are identical to the baseline.

What happened

On ordinary text the gain is modest: 1.3 points of Structure F₁ over the strongest baseline. On the rarest tier of characters it widens to 4.0. On characters outside the tokenizer's vocabulary it reaches 9.8, and by that point the models that lack a structural pathway have collapsed to near chance, scoring 14.0 against CNM-BERT's 76.0.

The pattern matters more than any single number in it. A margin that widens as the data gets harder is what you would expect if the structural pathway is doing real work. Whether it is, or whether it is propagating a resource the benchmark also inherits, is the question the Limitations section of the manuscript takes up.

Line chart of Structure F1 across three splits. All four models score around 90 on the IID split. On the long-tail split they separate. On the out-of-vocabulary slice the token-only baseline falls to 14.0 while CNM-BERT holds 76.0, 9.8 points above ChineseBERT. — **Fig. 3.** The gap widens with difficulty. Structure macro-F₁ on the CCD benchmark of Wu et al. (2025), across three splits of increasing rarity. Figures as reported in the submitted manuscript.

Table I — out-of-vocabulary slice, all four metrics
Model	Structure ↑	Radical ↑	Stroke MAE ↓	Stroke-type ↑
BERT-wwm-ext	12.4	6.1	2.10	10.3
RoBERTa-wwm-ext	13.1	6.8	2.02	11.0
MacBERT	14.0	7.4	1.98	11.8
SubChar-Pinyin	32.7	21.5	1.46	30.1
SubChar-Wubi	65.2	45.0	1.10	50.8
ChineseBERT	66.2	48.4	0.75	61.2
CNM-BERT	76.0	56.1	1.05	52.4

Baselines re-run from official checkpoints under one protocol: five seeds, one grid, the same hardware; the full diagnostic is Table 1 of the manuscript. ChineseBERT keeps the stroke columns, which suits a model that sees rendered pixels. The two approaches capture different signals.