Page 190 - Read Online

Hei et al. J. Mater. Inf. 2026, 6, 15 Page 15 of 22

Table 7. Performance of the proposed and baseline models
1 2 3 4 Random
Entity
F1 P R F1 P R F1 P R F1 P R F1 P R
Claude 3 0.93 0.93 0.93 0.51 0.52 0.50 0.52 0.50 0.55 0.62 0.86 0.48 0.67 0.76 0.61
GPT-4o 0.88 0.88 0.88 0.51 0.54 0.48 0.39 0.49 0.32 0.66 0.88 0.53 0.66 0.73 0.59
Gemini 1.5 0.93 0.93 0.93 0.51 0.52 0.50 0.51 0.47 0.56 0.71 0.90 0.59 0.67 0.76 0.61
Llama 3.1 0.90 0.90 0.90 0.50 0.50 0.50 0.47 0.43 0.53 0.62 0.91 0.47 0.64 0.71 0.57
BERT base 0.78 0.81 0.74 0.34 0.37 0.32 0.32 0.41 0.26 0.37 0.41 0.33 0.34 0.40 0.30
SciBERT 0.93 0.89 0.98 0.71 0.71 0.70 0.70 0.71 0.69 0.68 0.69 0.67 0.73 0.71 0.74
Ours 0.96 0.95 0.98 0.95 0.95 0.95 0.85 0.89 0.81 0.75 0.75 0.75 0.85 0.83 0.88
Model with the highest F1 score highlighted in bold. BERT: Bidirectional encoder representations from transformers.

The observed decline as tuple density increases reflects the growing difficulty of maintaining correct matches as more tuples appear within a sentence, compounded by
syntactic complexity and semantic overlap. Authors also tend to use abbreviations or omit contextual information in multituple settings, reducing the available signal.
Annotation sparsity for certain entity types and upstream extraction errors further reduce endtoend tuple accuracy. Detailed analyses follow. Across encoders,
MatSciBERT remains the most stable as tuple density increases, SciBERT performs consistently but at a slightly lower level, and BERTbase degrades in multituple
settings. These results are consistent with those observed in the extraction stage and support the view that domain-adaptive pre-training mitigates distributional
mismatch in symbol-heavy, convention-rich materials science text, thereby benefiting entity allocation.

On the other hand, the performance of LLMs remains relatively consistent across datasets. Notably, Gemini 1.5 demonstrates superior efficacy, slightly trailing Claude 3
on the third dataset (∆ = -0.008) while significantly outpacing it on the fourth dataset (∆ = +0.091). For the remaining datasets, both models achieve identical F1 scores.
At k = 1, F1 scores for the large models consistently exceed 0.9, indicating their proficiency in the entity extraction task. However, in contrast to the proposed model,
LLMs exhibit diminished performance on the second and third datasets (average F1 = 0.507 for k = 2 and average F1 = 0.473 for k = 3). Although their performance
improves on the fourth dataset (average F1 = 0.653 for k = 4), it remains inferior to that of the proposed model.

Our analysis suggests that large models are prone to extract excessive irrelevant content as entities, often misidentifying the concentration of a chemical element within
an alloy or fabrication methods as CONDITION entities. Additionally, these models tend to generate entities containing extraneous information rather than extracting
precise entities based on the input sentence, resulting in reduced performance. Conversely, on the fourth test set, where sentences focus more on tuple-related content
with minimal extraneous information, the precision of LLMs improves significantly (average P = 0.89 for k = 4), even surpassing that of the proposed model (P = 0.75
for k = 4). However, their recall remains lower and the overall F1 score still lags behind.