Page 192 - Read Online

Hei et al. J. Mater. Inf. 2026, 6, 15 Page 17 of 22

Table 8. Results of the ablation study
Our model Without allocation Without intra Without inter
Dataset
F1 P R F1 P R F1 P R F1 P R
2 0.95 0.95 0.95 0.56 0.40 0.95 0.87 0.87 0.87 0.86 0.86 0.86
3 0.85 0.89 0.81 0.45 0.31 0.83 0.81 0.85 0.77 0.79 0.84 0.75
4 0.75 0.75 0.75 0.34 0.21 0.85 0.73 0.73 0.73 0.69 0.69 0.69
Random 0.85 0.83 0.88 0.56 0.41 0.90 0.84 0.81 0.86 0.82 0.79 0.84
Study with the highest F1 score highlighted in bold.

relationships is more important than capturing relationships within entities of the same type for accurate tuple extraction. Specifically, removing inter-entity attention
disrupts semantic alignment across entities, increasing mismatches between properties and their appropriate units or scales, such as matching “yield strength” with
“Hv” units instead of “MPa”. This mechanism helps maintain value-unit associations and handle modifiers within entity boundaries, creating a false sense of
correctness. In contrast, removing intra-entity attention weakens internal consistency within the same entity. The resulting outputs may appear syntactically valid but
violate domain constraints. Without it, the model frequently outputs incomplete or fragmented value-unit pairs, which distorts quantitative meaning in materials
design tasks.

Regardless of the model settings, the F1 score decreases as the number of tuples to be extracted in the test set increases. Nevertheless, the complete model retains the
highest scores at each level of complexity. Notably, the complete model also achieves the highest recall (0.95) on dataset 2, indicating that it successfully identifies all
correct tuples during the entity allocation phase without introducing incorrect ones.

Overall, the performance drops observed in the ablation study directly translate into an increased likelihood of scientifically invalid extractions. Without these
components, users are more likely to encounter cases where extracted tuples appear syntactically correct but violate domain knowledge, requiring additional
verification steps and ultimately reducing trust in automated materials data extraction for practical applications.

Error analyses
To pinpoint the factors behind the observed decline as tuple density increases, we performed fine-grained error analyses across all datasets and examined
mispredictions. Four error classes account for most of the degradation, often interacting within the same sentence. Table 9 presents a quantitative breakdown of error
causes across datasets.