Page 188 - Read Online

Hei et al. J. Mater. Inf. 2026, 6, 15 Page 13 of 22

Table 5. Entity extraction performance of the proposed SciBERT-based model
1 2 3 4 Random
Entity
F1 P R F1 P R F1 P R F1 P R F1 P R
MAT 0.93 0.89 0.98 0.97 0.98 0.96 0.87 0.96 0.94 0.93 1 0.88 0.94 0.97 0.91
PRO 0.93 0.91 0.95 0.88 0.89 0.88 0.93 0.96 0.89 0.89 0.93 0.96 0.96 0.97 0.95
PRO V 0.91 0.86 0.95 0.97 0.99 0.95 0.96 0.96 0.95 0.98 0.99 0.86 0.98 0.96 1
CON 1 1 1 0.5 1 0.33 0.8 1 0.67 1 1 1 0 0 0
CON V 0.92 0.88 0.95 0.94 0.95 0.93 0.93 0.95 0.91 0.95 0.98 0.93 0.96 0.95 0.97
TOTAL 0.92 0.88 0.95 0.94 0.95 0.93 0.93 0.95 0.91 0.95 0.98 0.93 0.96 0.95 0.97
MAT, PRO, PRO V, CON, and CON V represent MATERIAL, PROPERTY, PROPERTY VALUE, CONDITION, and CONDITION VALUE, respectively.

Table 6. Performance comparison of BERT-based model variants for entity extraction: BERT-base, SciBERT, and MatSciBERT
1 2 3 4 Random
Entity
F1 P R F1 P R F1 P R F1 P R F1 P R
BERT-base 0.78 0.81 0.74 0.82 0.89 0.75 0.78 0.86 0.72 0.85 0.90 0.80 0.80 0.84 0.77
SciBERT 0.92 0.88 0.95 0.94 0.95 0.93 0.91 0.87 0.96 0.95 0.98 0.93 0.96 0.95 0.97
MatSciBERT 0.98 0.97 0.98 0.96 0.98 0.95 0.92 0.88 0.97 0.97 0.95 0.99 0.98 0.99 0.97
MatSciBERT with the highest F1 score highlighted in bold. BERT: Bidirectional encoder representations from transformers.

exhibits high variance, ranging from 0.00 to 1.00. For CONDITION VALUE entities, MatSciBERT obtains 0.91-1.00, SciBERT 0.92-0.96, and BERT-base 0.00-0.89. The
total performance metrics (as shown in Table 3) corroborate these findings. Overall F1 ranges from 0.92-0.98 for MatSciBERT, 0.91-0.96 for SciBERT, and 0.78-0.85 for
BERT-base, with MatSciBERT remaining stable across single-tuple through multi-tuple, and random test sets. SciBERT is steady but consistently behind, while
BERT-base degrades in multi-tuple settings, sometimes dropping to 0.78.

These patterns align with expectations for domain-adaptive pre-training in symbol-heavy technical text. PROPERTY VALUE benefits from surface regularity and unit
markers. MATERIAL is supported by repeated alloy formula patterns and consistent naming. PROPERTY varies more in phrasing as the tuple count increases, which
explains the mild decline. Rare categories amplify noise in totals; presenting disaggregated and aggregated metrics together gives a more accurate account of extraction
quality under realistic class imbalance.