Page 85 - Read Online

Shu et al. J. Mater. Inf. 2025, 5, 36 https://dx.doi.org/10.20517/jmi.2025.13 Page 9 of 31

sampling with replacement from the dataset. The bootstrap method assesses statistical properties such as
variance, confidence intervals, and bias. It is especially useful for small and imbalanced datasets and can be
[26]
combined with ML algorithms to provide more accurate predictions. Holdout method : The Holdout
method randomly divides the dataset into training, validation, and test sets. The model training on the
training set, hyperparameters are tuned on the validation set, and its performance is finally evaluated on the
test set. The holdout method is simple and intuitive but requires a sufficiently large dataset to avoid random
evaluation results.

The performance metrics used in different ML models (e.g., classification, regression) vary. For
classification models, performance metrics include accuracy, precision, recall, F1 score, receiver operating
characteristic (ROC) curve, and area under the curve (AUC) [105,106] . The confusion matrix is also commonly
[107]
used to evaluate classification models, comparing actual and predicted classes . For regression models,
performance metrics include root mean square error (RMSE), mean absolute error (MAE), mean squared
2 [33]
error (MSE), and the coefficient of determination (R ) . These metrics measure the difference between
predicted and actual values, assessing the model accuracy and reliability. Ultimately, model performance
should be evaluated using the test set. The test set must be entirely independent of the training and
validation sets to ensure genuine model performance on unseen data. The test set selection should avoid
overlapping with training data to prevent the model from encountering learned samples during testing,
which could result in falsely high performance. Additionally, the test set should be sufficiently large and
randomly selected to ensure the evaluation results are representative and reliable [108,109] .

In addition to testing on an independent dataset, model performance can be further enhanced through
hyperparameter tuning. Hyperparameters (such as the learning rate, regularization strength, number of
hidden layers in neural networks, and the number of trees in ensemble methods) , unlike model
[110]
parameters, are not learned during training but rather set prior to the learning process, significantly
impacting the model accuracy and generalization. Hyperparameter tuning involves systematically searching
for the optimal combination of hyperparameters to maximize model performance. Common methods for
hyperparameter tuning include Grid Search , an exhaustive approach that evaluates every possible
[111]
combination within a predefined set of hyperparameter values. Although computationally intensive, grid
[79]
search can be effective for smaller search spaces. Random Search , rather than evaluating all combinations,
random search randomly selects combinations, making it more efficient for large search spaces and often
yielding satisfactory results with fewer trials. An enhancement to this approach is Hyperband , which
[112]
dynamically allocates resources to promising configurations while employing early stopping for less
promising ones. This combination effectively allows Hyperband to explore large search spaces, often
achieving competitive results with fewer evaluations than traditional random search methods. Bayesian
Optimization (BO) uses probabilistic models to predict promising hyperparameter combinations,
[113]
iteratively refining the search based on previous evaluations. BO is efficient and often yields better results
than exhaustive methods for complex models.

By fine-tuning hyperparameters, models can balance bias and variance optimally, minimizing overfitting or
underfitting and improving training and test data performance. The final model, selected after
hyperparameter tuning, is evaluated on the independent test set to confirm its predictive power and
generalizability.