Page 189 - Read Online

Page 14 of 22 Hei et al. J. Mater. Inf. 2026, 6, 15

Performance of entity allocation
The innovative aspect of the entity allocation task presented in this paper lies in its direct prediction of which
entities are assigned to the same tuple, thereby circumventing the need to predict relationships between
entities. This approach not only reduces potential errors but also simplifies the process. It is evident that the
complexity of the entity allocation task varies across datasets, particularly in sentences containing a diverse
number of tuples. When evaluating model performance, we argue that only complete and accurate tuples can
contribute to an increase in the total correct count, irrespective of the efficacy of entity extraction. Therefore,
the metrics we employ to evaluate entity allocation performance are calculated in a manner similar to those
used for entity extraction.

Table 7 shows that entity allocation separates the three encoders more sharply than extraction. MatSciBERT
yields F1 scores between 0.75 and 0.96 across tuple complexities, SciBERT ranges from 0.68 to 0.93, and
BERT-base from 0.32 to 0.78. The widening performance gaps in the allocation task underscore the benefit of
materials-specific pre-training when inference depends on multiple interacting entities.

To achieve a comprehensive comparison between our proposed task-specific approach and task-agnostic
LLMs, we also employ a pre-training and prompting strategy utilizing four prominent models - Claude 3
Haiku , GPT-4o , Gemini 1.5 , and Llama 3.1 (70 B) - as benchmarks. The overall results for both the
[31]
[22]
[32]
[33]
proposed model and the reference models are presented in Table 7 (for brevity, these models are denoted as
Claude 3, GPT-4o, Gemini 1.5, and Llama 3.1, and the highest scores are highlighted in bold for easy
identification of the top-performing model), while the F1 scores for all models are illustrated in Figure 4. Our
model surpasses all baseline models across all test sets in terms of F1 score. Furthermore, to mitigate the
potential issue of hallucination associated with LLMs to some extent, we restrict the entity “CONDITION” to
temperature within the prompt. The prompt is defined as “Extract as many tuples of the form
[“MATERIAL”, “PROPERTY”, “PROPERTY VALUE”, “TEMPERATURE”, “TEMPERATURE VALUE”]
from the input sentences as possible”.

Overall, the performance of the proposed model decreases as the number of tuples contained within a single
sentence (denoted as k) increases. Specifically, when there is only one tuple (k = 1), the model is exclusively
tasked with entity extraction, and it achieves its optimal performance with an F1 score of 0.963. At k = 2, the
model’s precision and recall are equal, as the number of tuples generated by the model matches that of the
gold standard labels. Additionally, the number of extracted entities aligns with the true labels. Therefore, the
results for k = 1 and k = 2 both underscore the efficacy of the MatSciBERT + Pointer Network approach. The
slight decline in F1 to 0.947 at k = 2 suggests the emergence of errors due to confusion during entity
allocation.

Moreover, with each increment of k by 1 starting from k = 2, the F1 score decreases by approximately 0.1.
Notably, at k = 4, the model’s performance deteriorates significantly to an F1 of 0.753. This decline is partly
attributable to the fact that each sentence in the dataset contains, on average, four tuples, resulting in more
complex sentence structures that are challenging for the model. Furthermore, as the content volume
increases, authors are prone to employ omissions and abbreviations that hinder the model’s learning and
prediction capabilities. Additionally, the limited number of sentences containing four tuples and variability
in writing styles may further complicate model training.

In the last dataset (with an average k of 2.17), the F1 score is 0.854, slightly higher than that for k = 3 (∆ =
+0.006), yet markedly lower than that for k = 2 (∆ = -0.093). Given that a prediction is considered correct
only when all five entities are accurately extracted and allocated, an F1 score of 0.854 is regarded as a strong
result.