Page 191 - Read Online

Page 16 of 22 Hei et al. J. Mater. Inf. 2026, 6, 15

Figure 4. F1 scores of the proposed and baseline models. Four baseline models are employed, all of which are strong LLMs. Among all
models, the proposed model achieves the best performance. LLMs: Large language models.

Ablation studies
Since our work uses several novel components, especially attention mechanisms, for the task of entity
allocation, we conducted a series of ablation experiments to verify the importance of these components for
structural information extraction in materials science literature. In particular, after removing selected
components from the model, we retrain it and evaluate its performance on the test datasets. The examined
components include entity allocation, intra-entity attention, and inter-entity attention (These settings are
denoted as without allocation, without intra, and without inter in Table 8). We start the ablation experiments
from dataset 2, as dataset 1 does not involve entity allocation. The overall results are shown in Table 8. The
best F1, precision (P), and recall (R) values for each test set are highlighted in bold.

To enhance interpretability, the results are visualized in Figure 5. Overall, the proposed model with all
components achieves the best F1 scores across all datasets, indicating that entity allocation, intra-entity
attention, and inter-entity attention all contribute positively to model performance. Our ablation results
show that removing any of these three components degrades semantic accuracy in distinct ways, with direct
implications for user trust.

The model without entity allocation consistently shows the lowest F1 and precision scores, but the highest
recall. The F1 score drops to 0.56 on the random dataset and as low as 0.34 for datasets with higher tuple
density, with precision decreasing substantially while recall remains unchanged. This pattern arises because
the model without entity allocation generates all possible combinations of entities as output tuples. As a
result, the model performs poorly in balancing precision and recall, resulting in the lowest F1 scores.
Moreover, removing entity allocation disrupts cross-mention coherence when multiple materials appear in
the same passage. The model then mixes entities across materials, yielding tuples that pair a property value
from one alloy with the name of another. Such cross-assignment produces scientifically invalid combinations
that would mislead downstream analyses. This underscores the significance of entity allocation in our work.

When entity allocation is retained, the F1 scores (0.86, 0.79, 0.69, and 0.82, respectively) of the model without
inter-attention are consistently lower than those (0.868, 0.81, 0.73, and 0.84, respectively) of the model
without intra-entity attention, proving that inter-entity attention is more important than intra-entity
attention for extracting structured information. This suggests that correctly modeling cross-type