CNM-BERT: A Drop-In Structural Embedding for Chinese Characters via Ideographic Description Sequences

Abstract

1Introduction

Modern Transformer language models rely on tokenization, which enables stable training but imposes a rigid information bottleneck. Once text is segmented, any internal linguistic structure within a token becomes opaque. While largely benign for alphabetic languages, this abstraction is fundamentally lossy for logographic scripts like Chinese, where characters are recursive compositions of semantic and phonetic components — for example 辯 = ⿲(⿱(立, 十), ⿳(亠, 二, 口), ⿱(立, 十)).

Mapping characters to atomic IDs discards this structure, forcing models to learn character semantics indirectly through contextual co-occurrence. The cost of this indirection falls heavily on the long tail: rare characters receive few gradient updates, and standard embedding tables cannot systematically share parameters across orthographically related characters. Crucially, this limitation is architectural, not scale-bound — increasing parameter count or pre-training data cannot recover information the input interface has already discarded. An out-of-vocabulary ([UNK]) or under-trained character remains opaque regardless of model size.

Contribution

We propose the Compositional Network Model (CNM), a lightweight, drop-in augmentation that injects discrete sub-character structure into Transformer encoders without modifying the backbone, vocabulary, or output space. CNM canonicalizes Ideographic Description Sequences (IDS) into rooted parse trees, encodes each character with a recursive Tree-MLP, and fuses the resulting structural embedding into the standard BERT embedding layer. Because structure is symbolic and computed only once per unique character per batch, CNM is highly efficient, adding minimal parameters and incurring just ≈5% training overhead.

Positioning

We evaluate CNM under a dual empirical framework. Our goal is to demonstrate that explicit structural modeling closes a specific representation gap that token-only models cannot resolve via scaling, while strictly preserving general NLU performance.

• Primary: structural probing. On the Chinese Character Dataset (CCD) of Wu et al. (2025) — an external benchmark testing sub-character understanding such as layout and radical decomposition — CNM-BERT improves Structure accuracy on the OOV slice by +9.8 points and Radical F₁ by +7.7 points over the strongest visual baseline. This confirms that symbolic decomposition recovers structural signals that scale-only approaches miss.
• Secondary: general NLU. On standard CLUE, MRC and NER benchmarks, CNM-BERT consistently matches or exceeds the best Chinese PLMs of equivalent scale. This establishes explicit structural injection as a strict refinement of the token interface, conferring specialized OOV robustness without downstream regression.

2Related Work

Chinese pre-trained encoders

BERT (Devlin et al., 2019) established character-level MLM as the standard for Chinese, and subsequent work has primarily refined the masking strategy: BERT-wwm (Cui et al., 2020a) and ERNIE (Zhang et al., 2019b) introduce whole-word and entity-level masking; MacBERT (Cui et al., 2020a) replaces masked tokens with synonyms; ZEN (Diao et al., 2020) and NEZHA (Wei et al., 2019) inject N-gram information or relative positional encodings. CNM is orthogonal to all of these — we adopt whole-word masking as best practice but modify the embedding interface rather than the masking objective.

Visual and phonological augmentation

Glyce (Meng et al., 2019), ChineseBERT (Sun et al., 2021) and MECT (Wu et al., 2021) extract sub-character signals from rendered glyphs via CNNs. These methods rely on implicit feature extraction from continuous pixel statistics, which introduces font sensitivity and substantial compute overhead. Pinyin- and Bopomofo-based variants (Zhang et al., 2021; Tan et al., 2022; Si et al., 2023) replace orthography with pronunciation but suffer from severe homophone ambiguity (Du and Way, 2017), since pronunciation correlates weakly with character semantics compared to compositional structure.

Compositional and structural modeling

Static embeddings such as cw2vec (Cao et al., 2018) and radical-level embeddings (Yin et al., 2016; Sun et al., 2014) established that compositional decomposition yields richer character semantics. Sub-character tokenization (Si et al., 2023; Zhang et al., 2019a) flattens components into the vocabulary, but this alters the output space and risks generating invalid characters (Nikolov et al., 2018); Si et al. (2023) explicitly note that their method is unsuitable for open-ended generation. CNM differs by treating each character as a recursive symbolic function: rather than pixels or flattened sequences, we encode the IDS tree with a recursive Tree-MLP, preserving the standard tokenizer interface while exposing discrete compositional structure.

Sequence-level structural integration

Lattice-BERT (Lai et al., 2021) integrates word-level lattice information to disambiguate Chinese segmentation by exposing multiple candidate tokenizations to the encoder. CNM is complementary rather than competitive: Lattice-BERT refines the sequence of tokens, while CNM refines the representation of each token via sub-character composition. The two operate at orthogonal granularities and could in principle be combined.

3Model

We propose the Compositional Network Model (CNM), a lightweight architectural augmentation for Chinese Transformer encoders. CNM preserves the entire Transformer backbone — all self-attention and feed-forward layers remain identical to a baseline BERT encoder — and modifies only the input embedding interface. Specifically, CNM introduces a structure encoder that computes a per-character structural embedding from a deterministic canonicalization of Ideographic Description Sequences. The structural embedding is then fused with the standard BERT embedding at the input layer and propagated through the unchanged encoder stack.

CNM is designed to satisfy three practical constraints: (i) drop-in compatibility with standard BERT fine-tuning pipelines, at the same sequence length and vocabulary interface; (ii) explicit discrete structure derived from symbolic character composition rather than font-dependent rasterized glyphs; and (iii) efficiency, by caching and vectorizing structure computation over unique characters per batch.

3.1 · Inputs and tokenization

We adopt character-level tokenization for Chinese, consistent with common Chinese BERT practice: each CJK Unified Ideograph occupies one token position.¹ Let a batch have size B and padded sequence length T. CNM consumes the standard BERT inputs input_ids ∈ ℕ^B×T — together with attention_mask and token_type_ids as usual — plus a structural index tensor struct_idx ∈ ℕ^B×T.

Structural indexing. struct_idx_b,t is computed directly from the raw Unicode string before mapping to token IDs. It indexes a precomputed table of canonical IDS parses for the original character at position (b, t), even when the corresponding input_ids entry maps to [UNK] under the baseline vocabulary.

3.2 · Structural representation from IDS

CNM represents each Han character using an IDS parse tree that encodes its spatial composition. IDS strings are formed from Ideographic Description Characters (IDCs), which specify binary or ternary layout operators, and component leaves given as Unicode codepoints. Let V_char be the set of Unicode Han characters observed in pre-training and downstream data. For each character x ∈ V_char we construct a rooted ordered tree T_x whose internal nodes are IDCs and whose leaves are component codepoints. We denote the set of unique leaf codepoints by V_cmp and the set of IDC operators by V_op, restricted to a fixed set of standard Unicode binary and ternary layout operators.

Canonicalization. IDS decompositions are not guaranteed to be unique. CNM applies a deterministic canonicalization that maps each character x to exactly one tree T_x. We (i) discard candidates with private-use codepoints or non-standard markers, (ii) resolve intermediate aliases until all leaves are Unicode codepoints in V_cmp, and (iii) restrict internal nodes to binary or ternary IDCs in V_op. Among remaining candidates we select by lexicographic order over (a) tree depth, (b) the predicate that every operator lies in the high-frequency subset {⿰, ⿱}, (c) total node count, and (d) a stable lexicographic operator hash for tie-breaking. The full procedure is given in Appendix B. Characters without a valid IDS entry — about 3% of the vocabulary — map to a learnable structural embedding s_unk.

3.3 · Recursive Tree-MLP structure encoder

To map the discrete tree T_x into a dense structural embedding s_x ∈ ℝ^d_s, we introduce a Tree-MLP encoder computed bottom-up. Let E_cmp ∈ ℝ^|V_cmp|×d_s and E_op ∈ ℝ^|V_op|×d_s be learnable embedding matrices for components and operators. For any node n in T_x we compute a hidden state h_n ∈ ℝ^d_s.

Leaf nodes. If n is a leaf corresponding to component c ∈ V_cmp,

Internal nodes. If n is an internal node with operator o ∈ V_op and ordered children c₁, …, c_k where k ∈ {2, 3}, we apply an operator-conditioned MLP with a bounded residual path:

where W₂ ∈ ℝ^d_s×3d_s, W₃ ∈ ℝ^d_s×4d_s and b_k ∈ ℝ^d_s. The structural embedding for character x is the root hidden state:

Batch efficiency. Computing s_x independently at each token position would repeat work across identical characters. During training and inference, CNM therefore computes structure embeddings only for the set of unique Han characters appearing in the current batch, denoted V_batch. We precompile each T_x into a topologically sorted instruction buffer — a post-order traversal — so that the Tree-MLP can be vectorized across nodes, then gather the corresponding s_x back to token positions via struct_idx. This reduces per-step structural computation from O(BT) tree evaluations to O(|V_batch|) character evaluations, with small constant factors.

3.4 · Fusion at the embedding interface

CNM injects structural information at the embedding interface while leaving the Transformer encoder layers unchanged. Let e_tok,i ∈ ℝ^d denote the baseline token embedding at position i, and p_i, g_i ∈ ℝ^d the baseline position and segment embeddings. CNM computes a structure vector s_xi ∈ ℝ^d_s from the original character x_i; for positions without a defined IDS — [CLS], [SEP], punctuation, Latin characters — we use a learnable null structural embedding s_∅, and for characters missing from the IDS table we use s_unk.

We fuse the token and structure streams by concatenation followed by a linear projection to the model hidden size:

where W_f ∈ ℝ^d×(d+d_s) and b_f ∈ ℝ^d. The final input embedding to the Transformer is then formed as in BERT:

followed by the baseline embedding LayerNorm and dropout. The fused embeddings are fed to the unchanged Transformer encoder layers.

Parameters and overhead. CNM introduces parameters in E_cmp, E_op, (W₂, W₃) and W_f, plus s_∅ and s_unk. Counts and throughput impact are reported in §5.5.

3.5 · Pre-training objective

We train CNM-BERT end-to-end with whole-word-masked MLM at 15% corruption with the standard 80/10/10 replacement strategy, applying the same corruption pattern to the aligned structural indices to prevent gold-structure leakage. We additionally use an auxiliary component-prediction loss: for each masked position m with gold leaf components {c₁, …, c_Lm}, we predict each c_l from a target structural embedding computed only from the gold tree — that is, not visible to the Transformer input. The total loss is 𝓛 = 𝓛_MLM + λ𝓛_aux with λ = 0.1. The full derivation is in Appendix D.

4Experiment Setup

4.1 · Baselines

We compare CNM-BERT against a broad set of Chinese PLMs of comparable scale. Masking variants: BERT (Devlin et al., 2019), BERT-wwm and BERT-wwm-ext (Cui et al., 2019), RoBERTa-wwm-ext, MacBERT (Cui et al., 2019), ERNIE (Zhang et al., 2019b). Visual and phonological augmentation: ChineseBERT (Sun et al., 2021). Sub-character tokenization: SubChar-Wubi and SubChar-Pinyin (Si et al., 2023), which replace the vocabulary with Wubi keystrokes or Pinyin syllables. We were unable to obtain Lattice-BERT (Lai et al., 2021) because its released code repository is no longer accessible.

Controlled re-training. To attribute gains cleanly, we additionally re-trained BERT and MacBERT under an identical recipe to CNM-BERT — same corpus, vocabulary, optimizer, schedule, batch size, update count, FP16 and gradient-clipping settings, and fine-tuning protocol — so that controlled comparisons differ only in the presence of the CNM structural pathway, and in whole-word masking, which is absent from the BERT baseline. All other models use their released checkpoints under our common fine-tuning protocol.

4.2 · Pre-training data

Following Sun et al. (2021), we pre-train on Chinese Wikipedia plus filtered CommonCrawl (≈4B tokens), with HTML, deduplication and non-Chinese filtering, Jieba segmentation for whole-word masking, and IDS structural decompositions from the BabelStone database, which covers 97,680 CJK Unified Ideographs. Trees use 12 IDS operators with maximum depth 6 over a component vocabulary of ≈5,000 atomic radicals; the ~3% of characters lacking a valid IDS entry fall back to a learnable embedding s_unk.

4.3 · Evaluation data

Primary: structural probing (CCD). The Chinese Character Dataset of Wu et al. (2025) is an external diagnostic benchmark designed specifically to test sub-character understanding. Each example is a single character formatted as [CLS] x [SEP], with annotations for (i) top-level layout structure (8-way macro-F₁), (ii) radical decomposition (set F₁), (iii) stroke count (MAE) and (iv) stroke-type sequence (F₁). We evaluate on three splits: an IID character split; a long-tail tier split, training on high- and mid-frequency characters and testing on the lowest-frequency tier; and an OOV slice of the long-tail test set restricted to characters that map to [UNK] under the model's tokenizer. The OOV slice isolates the regime where token-only models structurally collapse.

Secondary: general NLU. We evaluate on twelve standard Chinese NLU tasks: CLUE (TNEWS, IFLYTEK, AFQMC, CMNLI, CSL, CLUEWSC2020), machine reading comprehension (CMRC 2018, DRCD, C3) and named entity recognition (MSRA, OntoNotes 4.0, Weibo). We use the official CLUE splits and canonical MRC/NER splits.

4.4 · Hyper-parameters

We train two scales following standard BERT configurations: base (12 layers, hidden 768) initialized from bert-base-chinese, and large (24 layers, hidden 1,024) initialized from hfl/chinese-roberta-wwm-ext-large. Structural components use d_s = 256, hidden 512, max tree depth 6, and a ~5K-component, 16-operator vocabulary. The fusion layer is initialized with Identity+Zero weights to preserve pre-trained representations. We pre-train for 1M steps with effective batch size 256 (base) / 512 (large), the LAMB optimizer, peak learning rate 1×10⁻⁴, 10K warmup steps, FP16 on 8×A100 80GB GPUs (≈7 and ≈14 days). Fine-tuning uses a learning-rate grid {1, 2, 3, 5}×10⁻⁵ with five seeds and early stopping on dev. Full detail is in Appendix E.

5Experiment Results

We organize results around the dual evaluation contract introduced in §1: primary structural-probing evidence on CCD (§5.1), secondary general-NLU evidence on CLUE, MRC and NER (§5.2–5.3), an ablation isolating which components produce which gains (§5.4), and a brief efficiency note (§5.5).

Reproduction methodology. Published Chinese-PLM results vary substantially across hyper-parameter grids, fine-tuning scripts, seeds and library versions, which makes canonical comparisons fragile. We therefore re-run all baselines from official HuggingFace checkpoints under a single identical protocol — five seeds, same grid, same hardware — and our reproduced baseline numbers fall within ±0.1 of published values on most tasks.

5.1 · Primary result: structural probing on CCD

CCD is the most direct test of the claim CNM-BERT actually makes: that explicit symbolic decomposition recovers sub-character information that token-only models cannot represent. Table 1 reports Structure macro-F₁, Radical F₁, stroke-count MAE and stroke-type F₁ across three splits of increasing difficulty.

Stroke metrics are omitted here for width; ChineseBERT leads on both, at 0.75 MAE and 61.2 stroke-type F₁ on the OOV slice against CNM-BERT's 1.05 and 52.4. The full table is in the PDF.

Findings

Three observations characterize the CCD results. First, on the OOV slice the gap between token-only baselines (Structure ≤ 14.0, Radical ≤ 7.4) and structurally-aware models is enormous: when the tokenizer fails, the encoder has no fallback and the model collapses to chance. CNM-BERT lifts OOV Structure to 76.0 and Radical to 56.1, exceeding the strongest visual baseline by +9.8 and +7.7 respectively, and the strongest sub-character-tokenization baseline by +10.8 and +11.1.

Second, the gain grows with distributional shift: from +1.3 Structure on IID, to +4.0 on long-tail, to +9.8 on OOV. This is the qualitative signature of an architectural prior rather than a corpus artifact. Third, on visual stroke metrics ChineseBERT remains best, which is consistent with its design — it sees pixels. CNM-BERT and ChineseBERT thus capture genuinely complementary signals, symbolic composition and visual rendering, and the appropriate evaluation question is which is more transferable downstream.

5.2 · Secondary result: the CLUE benchmark

CNM-BERT achieves the best average on CLUE at both base (71.03) and large (73.56) scales, surpassing every reproduced baseline. The margin over the strongest external baseline is small — +0.18 base, +0.19 large — but statistically reliable across five fine-tuning seeds and consistent at both scales. Against our matched-recipe MacBERT and BERT baselines the gains are +0.50 and +2.69 average respectively.

The relevant claim is parity, not state of the art. Structural injection is a strict refinement of the standard token interface and does not regress general NLU.

The two SubChar baselines reveal a typical sub-character trade-off: they gain on TNEWS and CSL by 5–6 points but lose about 6 points on AFQMC and IFLYTEK, because vocabulary substitution destroys standard token semantics. CNM-BERT shows no such trade-off, because it augments rather than replaces the token interface.

5.3 · Secondary result: MRC and NER

Reading comprehension and NER show the same pattern as CLUE: CNM-BERT is competitive everywhere and best on roughly half the metrics, with the cleanest gains appearing on noisy or rare-character tasks — Weibo NER +1.13 at base, C3 Test +1.0. On clean span extraction (CMRC, DRCD) CNM-BERT and ChineseBERT trade leads within a fraction of a point.

NER results for noisy social-media text (Weibo) show the largest CNM-BERT gains, consistent with the CCD long-tail finding: structural priors help most when surface statistics are unreliable.

5.4 · Ablation study

To isolate which component of CNM produces which gain — the Tree-MLP fusion pathway, the auxiliary component-prediction loss, or the recursive hierarchy itself — we ablate each independently on a single A100 80GB GPU.

The ablation supports three conclusions. (i) Structural injection is what matters: the auxiliary loss without Tree-MLP fusion (row 4) lifts CCD-OOV by only +7.6 against +41.8 for the full model, ruling out multi-task regularization as the source of gain. (ii) Hierarchy carries real signal: flattening to a bag-of-components (row 5) loses 17.7 points on CCD-OOV, from 76.0 to 58.3, so the IDS tree structure — not just the components — is informationally load-bearing. (iii) The auxiliary loss is a small but reliable addition: rows (2) against (3) show λ = 0.1 adds +0.22 CLUE-Avg and +1.5 CCD-OOV over λ = 0, consistent across five seeds, so we keep it as a regularizer rather than as a load-bearing component.

5.5 · Efficiency

CNM-BERT adds only ≈2.5M parameters over BERT-base for the structural pathway — the component table (≈1.28M), operator table (4K), operator-conditioned MLPs (≈460K) and fusion projection (≈790K). This is ≈18× less than the ≈45M ChineseBERT adds for its glyph CNN and Pinyin embeddings. Training is correspondingly cheap: ≈5% slowdown over vanilla BERT (142 → 135 samples/sec at batch size 32, sequence length 512, single A100), substantially faster than ChineseBERT at 98 samples/sec, a 30% penalty. The overhead is bounded by our caching strategy: the Tree-MLP is evaluated only on the set of unique characters in each batch, not at every token position.

6Discussion and Conclusion

We presented the Compositional Network Model (CNM), a lightweight, drop-in augmentation that exposes discrete sub-character structure to a Transformer encoder via deterministic IDS canonicalization and a recursive Tree-MLP. The contribution rests on two empirical claims.

First, in the regime that token-only models cannot represent — rare and OOV characters — explicit symbolic decomposition produces large, qualitatively different gains: +9.8 Structure and +7.7 Radical points over the strongest visual baseline on the CCD OOV slice. Second, on broad NLU, CNM-BERT achieves the highest average on CLUE, MRC and NER at both scales, with margins that are small in absolute terms (≈0.2–0.5 average) but statistically reliable across five seeds and consistent at both scales. Ablations attribute the bulk of the gain to the hierarchical Tree-MLP fusion.

Together these results show that an architectural prior targeting sub-character structure can deliver real downstream value and close the OOV structural gap without trading off against general NLU — a dual property that, to our knowledge, no prior method achieves. The broader implication is that the sub-character information gap is architectural: it cannot be closed by scale alone, because no amount of contextual co-occurrence recovers structure the input interface has discarded, and even much larger generative models that share the character-as-atom assumption inherit this limitation.

Limitations

External resource dependence. CNM depends on the coverage and quality of an external IDS database; characters lacking a valid IDS entry fall back to a learnable embedding — about 3% of our vocabulary, mostly rare variants — so the structural pathway provides no benefit for those characters.

Scope. This work is restricted to Chinese. The methodology is in principle transferable to other Han-derived scripts (Japanese Kanji, Korean Hanja, Vietnamese Chữ Nôm) and to morphologically rich non-logographic languages, but we leave empirical verification to future work.

Resource overlap. The paper's strongest empirical claim depends heavily on CCD. This is appropriate given the structural-probing goal, but we want to be explicit about the relationship between CCD labels and the IDS source used by CNM: both ultimately derive from compositional decomposition resources for Han characters. If the two share substantial overlap, the +9.8 OOV gain is best interpreted as evidence of structured-resource transfer — CNM correctly propagates the structural information available in its training-time database to the evaluation-time probing task — rather than as evidence of an independent emergent capability. We believe both readings are scientifically valuable, but the more conservative one should be preferred until cross-resource probes, such as CCD-style labels derived from a disjoint compositional taxonomy, are available.

Effect size on general NLU. Our CLUE, MRC and NER results are best characterized as small but consistent improvements over the strongest existing Chinese PLM baselines. We report that CNM-BERT achieves the highest average score on every general-NLU table, by margins that are statistically reliable across five fine-tuning seeds but practically small — typically 0.2 to 0.5 average points over MacBERT and ChineseBERT. The contribution is not to dominate these benchmarks but to demonstrate that an architectural prior targeting sub-character structure can deliver this small but real improvement on general NLU while also closing a large gap on out-of-vocabulary structural probes.

Acknowledgments

This work was conducted entirely independently by the listed authors. No mentorship, assistance, or guidance of any kind contributed to any aspect of this paper.

We thank Alibaba Cloud for generously providing the computational resources used in this work, which enabled our experiments and analyses.

References

1. Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. 2018. cw2vec: Learning Chinese word embeddings with stroke n-gram information. AAAI, 5053–5061.
2. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020a. Revisiting pre-trained models for Chinese natural language processing. Findings of EMNLP 2020, 657–668.
3. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for Chinese BERT. arXiv:1906.08101.
4. Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2020b. A sentence cloze dataset for Chinese machine reading comprehension. COLING 2020, 6717–6723.
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 2019, 4171–4186.
6. Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. Findings of EMNLP 2020, 4729–4740.
7. Jinhua Du and Andy Way. 2017. Pinyin as subword unit for Chinese-sourced neural machine translation. AICS.
8. Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2021. Lattice-BERT: Leveraging multi-granularity representations in Chinese pre-trained language models. NAACL-HLT 2021, 1716–1731.
9. Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. SIGHAN, 108–117.
10. Yuxian Meng, Wei Wu, Fei Wang, Xin Li, Ping Nie, Fan Yin, Ming Li, Qinghong Han, Xiaoyan Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for Chinese character representations. NeurIPS.
11. Nikola I. Nikolov, Yuhuang Hu, Mi Xue Tan, and Richard H. R. Hahnloser. 2018. Character-level Chinese-English translation through ASCII encoding. WMT, 10–16.
12. Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. EMNLP 2015, 548–554.
13. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. CoNLL, 1–27.
14. Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2019. DRCD: A Chinese machine reading comprehension dataset. arXiv:1806.00920.
15. Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2023. Sub-character tokenization for Chinese pretrained language models. TACL, 11:469–487.
16. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging Chinese machine reading comprehension. TACL, 8:141–155.
17. Yaming Sun, Lei Lin, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2014. Radical-enhanced Chinese character embedding. ICONIP, 279–286.
18. Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li. 2021. ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin information. ACL-IJCNLP 2021, 2065–2075.
19. Minghuan Tan, Yong Dai, Duyu Tang, Zhangyin Feng, Guoping Huang, Jing Jiang, Jiwei Li, and Shuming Shi. 2022. Exploring and adapting Chinese GPT to Pinyin input method. ACL 2022, 1899–1909.
20. Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. NEZHA: Neural contextualized representation for Chinese language understanding. arXiv:1909.00204.
21. Shuang Wu, Xiaoning Song, and Zhenhua Feng. 2021. MECT: Multi-metadata embedding based cross-transformer for Chinese named entity recognition. ACL-IJCNLP 2021, 1529–1539.
22. Xiaofeng Wu, Karl Stratos, and Wei Xu. 2025. The impact of visual information in Chinese characters: Evaluating large models' ability to recognize and utilize radicals. NAACL 2025, 331–350.
23. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, and others. 2020. CLUE: A Chinese language understanding evaluation benchmark. COLING 2020, 4762–4772.
24. Rongchao Yin, Quan Wang, Peng Li, Rui Li, and Bin Wang. 2016. Multi-granularity Chinese word embedding. EMNLP 2016, 981–986.
25. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. arXiv:1904.00962.
26. Ruiqing Zhang, Chao Pang, Chuanqiang Zhang, Shuohuan Wang, Zhongjun He, Yu Sun, Hua Wu, and Haifeng Wang. 2021. Correcting Chinese spelling errors with phonetic pre-training. Findings of ACL-IJCNLP 2021, 2250–2261.
27. Wei Zhang, Feifei Lin, Xiaodong Wang, Zhenshuang Liang, and Zhen Huang. 2019a. Sub-character Chinese–English neural machine translation with Wubi encoding. arXiv:1911.02737.
28. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019b. ERNIE: Enhanced language representation with informative entities. ACL 2019, 1441–1451.

Appendices

A · Dataset statistics

B · IDS canonicalization algorithm

The BabelStone IDS database is multi-source and admits multiple decompositions for a single character; typical sources include Unicode CJK-Unihan, Adobe-Japan1, Twitter Han and GTBJ. Naive ingestion produces non-deterministic trees that destabilize training. We therefore apply a deterministic canonicalization procedure that maps each character x to exactly one tree.

Filter set. Let 𝒞_x = {T₁, …, T_n} be the candidate parses for character x across BabelStone sources. We define a filtering predicate φ(T) = φ_pua(T) ∧ φ_ops(T) ∧ φ_cycle(T) ∧ φ_leaves(T):

• φ_pua — T contains no Private Use Area codepoints (U+E000–U+F8FF, U+F0000–U+FFFFD, U+100000–U+10FFFD).
• φ_ops — every internal node label belongs to the standard binary/ternary IDC set {⿰, ⿱, ⿲, ⿳, ⿴, ⿵, ⿶, ⿷, ⿸, ⿹, ⿺, ⿻}, the 12 standard IDCs in our notation; the implementation also reserves a learnable [UNK_OP] slot, yielding |V_op| = 16 after special tokens.
• φ_cycle — alias resolution of T terminates without entering a cycle.
• φ_leaves — every leaf is either a single Unicode codepoint or resolves through alias substitution to one.

Trees that fail φ are discarded; if all candidates fail we set T_x = ⊥ and the structure encoder routes x to the learnable s_unk embedding.

Selection rule. Among the surviving candidates we apply a strict lexicographic selection over four scores:

where d(T) is tree depth, smaller preferred; std(T) is the predicate that every internal operator lies in the high-frequency subset {⿰, ⿱}; |T| is total node count; and π(T) is the lexicographic operator sequence of the post-order traversal, used purely for tie-breaking determinism.

Component recovery for OOV characters. A character that is OOV with respect to the BERT vocabulary is not necessarily OOV with respect to the tree space: even when it maps to [UNK] for the token pathway, its IDS decomposition may still be available in BabelStone, and its leaf components are typically common radicals shared with high-frequency characters. The structure pathway therefore retains a meaningful, gradient-trained embedding for it via the recursive composition of E_cmp vectors, even though the token pathway has collapsed. This is the mechanism behind the OOV gains in Table 1.

C · Tree-MLP as a compositional operator

Recursion as fold. The Tree-MLP is a parameterized fold over rooted ordered trees. Let Σ_k = V_op × (ℝ^d_s)^k for k ∈ {2, 3}. The encoder is a pair of functions f_leaf : V_cmp → ℝ^d_s and f_k : Σ_k → ℝ^d_s, extended to trees by the catamorphism:

The structural embedding of character x is s_x = Enc(T_x). This formulation is purely compositional: for any sub-tree shared between two characters, the encoder's intermediate state is identical, and gradient signal back-propagates equally to both characters' losses. This is the formal basis for the parameter-sharing claim in §1 — orthographically related characters share structure-encoder parameters by construction.

Parameter count, additional beyond the BERT backbone: component table |V_cmp|·d_s ≈ 5,000 × 256 ≈ 1.28M; operator table 16 × 256 = 4,096; binary MLP 3d_s·d_s + d_s = 196,864; ternary MLP 4d_s·d_s + d_s = 262,400; fusion projection W_f ≈ 0.79M (base) / 1.31M (large); special embeddings 2 × 256 = 512. Total ≈2.5M base and ≈3.0M large, dominated by the component table. ChineseBERT, by comparison, adds ≈45M parameters for its glyph CNN and Pinyin embeddings.

Why hierarchy matters. The flat-fusion ablation (Table 4, row 5) replaces the recursive encoder with a mean over leaf component embeddings. This destroys two pieces of information: operator identity, which captures spatial layout, and component ordering — the difference between 杲 = ⿱(日, 木) and 杳 = ⿱(木, 日) is undetectable to a bag-of-components encoder. Empirically this loses 17.7 points on CCD-OOV.

D · Pre-training objective: full derivation

WWM corruption. Whole-word masking produces a corrupted sequence X̃ from X by selecting a 15% mask budget over Jieba word boundaries, then applying the 80/10/10 strategy at each masked position m: x̃_m = [MASK] with probability 0.8, a random vocabulary token with probability 0.1, or x_m unchanged with probability 0.1. Let ℳ denote the set of masked positions.

Aligned structural corruption. For each masked position m we additionally corrupt the structural index, so that it indexes the structure of the corrupted token rather than the gold one. This ensures the encoder cannot use gold structure as a leak channel for the gold character.

Auxiliary component-prediction loss. For each m ∈ ℳ, let {c₁^(m), …, c_Lm^(m)} be the canonical leaf components of the gold character x_m. We compute a target structural embedding from the gold tree, s_xm^tgt = Enc(T_xm), and pass it through an auxiliary head g_ψ : ℝ^d_s → ℝ^|V_cmp|:

Information-leak prevention. Crucially, s^tgt is supplied only to the auxiliary head; the Transformer input at position m uses the corrupted structural embedding. This forces the MLM prediction to recover the gold character from non-gold structural context, preserving the difficulty of the MLM task while still using structural supervision to shape the structural embedding space.

E · Detailed hyper-parameters

F · Auxiliary figure recipes

For reproducibility, the manuscript provides rendering recipes for three further analyses recommended for an extended version: a UMAP projection of the learned structural embeddings coloured by top-level IDS operator, with an expected cluster purity ≥ 0.90; a per-operator gain radar over the eight standard operators, expected to dominate ChineseBERT on every axis with the largest gains on the ternary operators ⿲ and ⿳; and pre-training loss curves, expected to hold MLM perplexity within 5% of BERT throughout while auxiliary component accuracy rises from chance to ≈70% by 200K steps. Full parameters are in the PDF.

¹ Non-Han characters — Latin letters, digits and punctuation — are tokenized by the baseline WordPiece rules; CNM attaches a null structural embedding to these positions (§3.4).