Teaching machines to read Chinese the way it is built.

CNM-BERT

Read 樹 and you will notice 木 on the left — the tree radical. You may never have met the character before, and you can still place it: something wooden, something growing. A tokenizer notices none of this. It looks the character up in a table, hands the model an integer, and the anatomy is gone before the first layer runs.

For a common character that loss is affordable, because context will eventually teach the model what it means. For a rare one there is no eventually — it appears too seldom to accumulate the updates. And for a character outside the vocabulary altogether, the model receives [UNK] and has nothing whatsoever to work with. No amount of scale repairs this. Pre-training cannot recover information that the input interface has already discarded.

What the model is given

Unicode already publishes the decomposition. An Ideographic Description Sequence writes 辯 as ⿲(⿱(立,十), ⿳(亠,二,口), ⿱(立,十)) — a left-middle-right operator over three branches, every one of which splits again. CNM-BERT parses that string into a tree, walks it from the leaves upward with a small recursive network, and folds the resulting vector into the standard embedding before the first transformer layer sees anything.

Everything downstream is left alone. Same tokenizer, same 21,128-token vocabulary, same output head, same twelve or twenty-four transformer layers. The structural pathway adds about 2.5 million parameters against BERT-base's 110 million, and roughly 5% to training time — the tree is evaluated once per distinct character in a batch, not once per token position. It drops into an existing fine-tuning pipeline without touching it.

What happened

On ordinary text the gain is modest — 1.3 points of Structure F1 over the strongest baseline. On the rarest tier of characters it widens to 4.0. On characters the tokenizer cannot represent at all it reaches 9.8, and by that point the models without a structural pathway have collapsed to near chance: 14.0, against CNM-BERT's 76.0.

The shape of that sequence carries more weight than any single number in it. A margin that widens as the data gets harder is what you would expect if the structural pathway is carrying real weight. Whether it is doing that, or propagating a resource the benchmark also inherits, is the question the next section takes up.

Baselines re-run from official checkpoints under one protocol — five seeds, one grid, the same hardware. ChineseBERT keeps the stroke columns, which suits a model that sees rendered pixels; the two approaches capture different signals.

What the result does and doesn't establish

Two caveats travel with that 9.8. The IDS database CNM-BERT trains on and the benchmark it is evaluated against both descend from compositional decompositions of Han characters. Where those resources overlap, the result shows the model propagating structural information it was given, which is a weaker claim than discovering an independent capability. Settling it needs a probe built from a disjoint taxonomy, and that probe does not yet exist.

On general benchmarks the margins are small. CNM-BERT posts the best average on CLUE at both scales, but by 0.18 and 0.19 points; reading comprehension and named-entity recognition follow the same pattern, with the clearest wins on noisy, rare-character text such as Weibo. The claim being made is parity, not dominance: structure can be added without costing anything elsewhere.