Each MatBench submission contains three principal components: a metadata file (info.json) describing the algorithm and its software requirements, source code for training and evaluating the model, and a compressed result file (results.json.gz) containing the submitted predictions and benchmark metrics [Figure 1]. We evaluated 28 submitted algorithms using their publicly available metadata and source code, while treating the submitted result files as the reference for numerical comparison.

The assessment comprised three sequential levels. First, environment reconstruction tested whether the dependency information supplied with a submission was sufficient to establish an import-valid Python environment. An environment was considered successfully reconstructed when, under at least one tested Python version and installation route, dependency installation completed and all imports identified in the submitted execution scripts could be loaded without error. This criterion evaluates environment availability and import compatibility but does not establish that the complete benchmark workflow is executable.

Second, code executability tested whether the submitted source code could complete a selected representative MatBench task and generate benchmark outputs in the reconstructed environment. Third, numerical result reproducibility quantified the difference between the regenerated benchmark metric and the value recorded in the original submission. These three levels were evaluated separately because successful environment reconstruction does not necessarily guarantee successful code execution, and successful execution does not necessarily guarantee numerical agreement with the reported result. The overall workflow is illustrated in Figure 2.

Environment reconstruction was conducted in clean, isolated Linux runners using GitHub Actions. For each submission, the dependency specifications in info.json were tested across Python versions 3.6-3.12 using pip and conda-based installation routes. After installation, the pipeline automatically scanned the source code to collect all import statements and performed an import-based sanity check. The reconstruction procedure comprised three stages of progressively increasing intervention.

In Stage 1, dependencies were installed using the author-provided package names and version constraints without modification. Submissions that completed installation and passed the import check were assigned to Category 1 (Cat. 1).

Submissions that failed Stage 1 proceeded to Stage 2, in which predefined error-guided rules were used to relax unavailable or conflicting version constraints. The rules operated only on dependency specifications and did not modify the submitted model or evaluation code. Submissions that passed the import check after this automated remediation were assigned to Cat. 2. The error-classification rules, retry procedure, and workflow pseudocode are provided in Supplementary Method 1.

Submissions that remained unresolved proceeded to Stage 3. At this stage, the metadata and execution scripts were reviewed manually to identify omitted dependencies and potential incompatibilities. Package versions, installation order, and the use of pip, conda, or a combination of the two were adjusted where necessary. A large language model (LLM) was used as a human-supervised troubleshooting aid for selected difficult cases, but all suggested changes were reviewed, implemented, and validated by the authors. No submitted model or evaluation code was modified during environment reconstruction. The complete intervention protocol, stopping criteria, and LLM-assisted procedure are described in Supplementary Method 2.

The four environment categories were assigned according to the minimum level of intervention required to satisfy the environment-reconstruction criterion:

These categories describe the reconstruction pathway and the minimum intervention required; they do not indicate whether the source code subsequently executed successfully or whether the regenerated numerical results agreed with the original submission. Cat. 4 likewise denotes failure under the predefined testing protocol and should not be interpreted as proof that reconstruction is impossible under all configurations.

Submissions in Cat. 1-3 were advanced to code re-execution. Their reconstructed environments were installed on an High-Performance Computing (HPC) system running Rocky Linux 9.4 (Blue Onyx), equipped with an NVIDIA A100-SXM4-80GB Graphics Processing Unit (GPU) and CUDA 12.8. Environments that passed the GitHub Actions validation could also be installed on this system.

To limit the computational cost while covering both MatBench input classes, we selected the smallest representative task from each class: steels for composition-based models and jdft2d for structure-based models. Each submission was evaluated on the applicable task according to its input type. Any submission-specific exceptions and the task selected for each algorithm are provided in Supplementary Table 1, available as a separate Excel file. Code re-execution was considered successful when the submitted workflow completed the selected task and generated predictions and benchmark metrics in the expected MatBench output format. Runtime failures and their causes were recorded separately from failures of environment reconstruction.

For submissions that completed re-execution, numerical agreement was quantified using the relative difference between the regenerated and originally reported five-fold mean root mean square error (RMSE):

(1) $$ \Delta_{\text {RMSE }}(\%)=\frac{\text { RMSE }_{\text {re-executed }}-\text { RMSE }_{\text {reported }}}{\text { RMSE }_{\text {reported }}} \times 100 . $$

Here, RMSE_reported is the mean RMSE across the five predefined cross-validation folds recorded in the original submission, and RMSE_re-executed is the corresponding value obtained through re-execution. Positive values indicate higher error in the re-executed result, whereas negative values indicate lower error. We report the signed and absolute differences rather than imposing an arbitrary binary threshold for numerical reproducibility.

Environment instantiation was also identified as a nontrivial issue in a parallel reproducibility effort we conducted on the JARVIS-Leaderboard benchmark set^[19]. Submissions to this platform contain model predictions on the platform benchmarks, an executable run.sh script (which in most cases executes a python script), and a metadata.json file which plays a similar role as the info.json file used in MatBench submissions where the most relevant information for the environment is in the software_used field.

The reproducibility effort for JARVIS-Leaderboard models aimed not only to set up a valid environment in which to run the models but to do so using a purely functional Nix packaging ecosystem^[20]. The functional packaging paradigm models the software environment as the output of a pure function whose inputs are the pure functions that similarly output the environment’s direct dependencies^[21]. This functional relation continues recursively such that an environment closure precisely specifies dependencies as a directed acyclic graph with nodes ranging from the model benchmark script, through the Python interpreter and modules, to low-level transitive dependencies such as the C standard library. The functional approach then makes full or partial updates to the environment straightforward, while only rebuilding portions of the dependency graph downstream of any changes. This is in contrast with the more common imperative model of instantiating software environments, where a series of steps are performed, each mutating the filesystem of some OS image to add, remove, or update packages to arrive at a given environment. Stronger reproducibility guarantees can then be made in the functional model without the dependence on the details of an entire OS image^[22-24]. See Figure 5 for an example graph of the package dependencies which can readily be constructed from the Nix function describing an environment.

To narrow the initial scope of this work, we focus on packaging just the models which provide predictions for the exfoliation energy benchmark using Nix. This was chosen to be generally cheaper to test as the prediction output is a single scalar, and the training dataset is smaller than for other benchmarks. This allows efforts to be directed toward reproducing usable environments for the model to run rather than other factors. The general approach was to start by writing a Nix function for each model’s environment containing only the software in JARVIS-Leaderboard entry. This function is modified by adding packages, constraining package versions, and/or adding patches to the source script until a trial training and inference run succeeds. Source scripts were also patched to enable Command-Line Interface (CLI) flags for running only the desired benchmark (some scripts ran more than just exfoliation energy) and running in a “test mode” to reduce the number of training iterations.

Nix packages for non-Python dependencies and the Python interpreter itself were obtained from the nixpkgs repository^[25]. While nixpkgs also contains a number of Python packages, it does not contain all of the Python Package Index (PyPI) or some other packages that are more easily handled by Python-specific tooling. For Python-specific environments we prepared a pyproject.toml file^[26] for each module compatible with the PDM^[27] Python dependency management tool. This was then incorporated into the Nix function via dream2nix^[28]. No attempt was made to constrain dependency versions to those used by the original authors. The JARVIS-Leaderboard project only required the software_used field of metadata.json to contain a string of comma-separated software projects, so such constraints were rarely provided. For models where version constraints were provided, they were utilized only on select packages when unconstrained dependency versions resulted in errors.

In addition to just identifying dependencies and versions, it also became necessary to patch the source files of some benchmark scripts. Some patches were needed due to API changes in dependencies (similar to those mentioned in Section “Reproducing the benchmark results”). Other patches were needed for Nix-specific reasons, mostly to avoid hardcoding filesystem paths for output data. In particular, mutating the source directory at runtime breaks the purely functional model. Patches were also added to create a uniform CLI interface to select the benchmark set to run and a-test flag which runs a computationally cheaper version of the model to test the environment (e.g., with fewer training iterations). Making edits across multiple contribution scripts from different authors to achieve similar behavior modification was facilitated by LLM models, though the patches were sufficiently contained in scope to be reviewed and judged to be very unlikely to have an impact on results.

Some models (namely, those from the kgcnn group) contained a conda environment.yaml file with specific versions; while not used directly (as conda was not used in our packaging approach), this was the most complete of the environments documented (indeed it seems it may be overcomplete, likely containing some packages which are not used). Most models had incomplete descriptions of the software used. In this case, Python reflection capabilities were used to obtain a list of required modules which could then be used to identify packages needed in the environment. One unfortunate aspect of the Python ecosystem is that module names and package names need not be the same; indeed, the appropriate “jarvis” module itself is provided by the package “jarvis-tools”. Even with reflection tools, this task could only be partially automated. In addition to identifying the required packages, the package versions were also constrained where needed. Additional patching (not related to the Nix filepath constraints or added CLI flags) was then also applied where needed. Some remaining errors were a result of changes in the PyPI, such as the package “sklearn” being renamed to “scikit-learn”. The matminer-lgbm model also had an error related to the fact that the JARVIS-Leaderboard repo had changed a directory name where benchmark data was stored and the script was not kept in sync with this change.

After these modifications, still 3 out of the 14 models require a more substantial effort to properly package their environments. The alignn model depends on the deep graph library (DGL) which is not present in the PyPI and requires additional packaging work to be amenable for use with the Nix approach. The provided scripts for the cgcnn model download data and code which are then executed all at runtime breaking the separation between environment instantiation and model training/inference execution, the untangling of which was not perused. Finally, the matformer model run script appears to activate a conda environment presumably present on the contributor’s machine, but which is not present in the leaderboard contribution.

The central issue of insufficiently specified environments is just as present in JARVIS-Leaderboard as in MatBench and the Nix approach to defining and testing environments taken comes with a distinct set of tradeoffs. Writing Nix functions for model environments is likely less familiar to most materials data science researchers. However, this barrier can likely be lowered significantly by LLMs, especially since Nix environments are fully represented in text. The Nix approach enables sandboxed environments to be built and run locally while iterating to debug issues without needing to rebuild portions of the environment which remain unchanged between iterations. GPU acceleration is crucial to the field, but was not fully addressed at this stage of the work. This “hardware dependent software” has some necessary implications on reproducibility and requires some special treatment within the functional packing model. Multiple, evolving approaches currently exist within the ecosystem which will be utilized in subsequent work^[29,30].

Based on the discussion above, we conclude that the main reproducibility challenges on the MatBench and JARVIS-Leaderboard platform stem from the difficulty of accurately reconstructing the Python environments originally used by the authors. Once a runnable environment can be successfully established, the reported results are generally highly reproducible [Figure 6]. The limitations of the currently provided metadata can be summarized as follows: (i) the list of required packages is sometimes incomplete, leading to import errors before execution can begin; (ii) specific package versions may exhibit dependency conflicts, whereas substituting alternative versions can render the code incompatible; and (iii) as the package ecosystem evolves, some previously specified dependencies may become unavailable, causing environment reconstruction to fail. Therefore, additional measures are needed to improve the success rate of environment reconstruction and the executability of the submitted code, thereby strengthening the reproducibility of benchmark results.

Based on these observations, we propose several recommendations for benchmark platform maintainers to improve the reproducibility of submitted algorithms as shown in Figure 7. These recommendations can be broadly divided into two categories: submission-time validation and preservation of validated environments. First, when a new algorithm is submitted to a benchmark platform, a continuous integration workflow could be used to automatically validate the corresponding Python environment. Submissions could be accepted only after they successfully pass environment reconstruction and import checks. To further strengthen this validation process, the workflow could include a lightweight smoke test that executes a minimal benchmark run within the same gated submission pipeline, ensuring that the submitted code is not only installable but also executable. Once a valid environment has been established, it should be preserved to support future re-execution. For example, authors or platform maintainers could export an environment backup file, such as environment.yml, which provides a user-friendly starting point for environment reconstruction. Third, as package ecosystems evolve, an environment backup file may become insufficiently robust for resolving the intended dependency set. In such cases, tools such as conda-lock can post-process environment.yml file by generating a fully resolved lockfile that explicitly specifies the concrete dependency versions to be installed. As a result, even if upstream package repositories change in the future, environment resolution remains unaffected. Fourth, in some cases, the target package is no longer available (e.g., matbench v0.1.0), making it impossible for even a lockfile to retrieve the required artifacts. In such situations, another practical option is to use a containerization tool (e.g., conda-pack) to package the entire environment into a relocatable archive that users can activate and run directly.

Notably, these improvements can be achieved without changing the current submission format. An environment backup file can be automatically exported, and the installed environment can be post-processed using package locker or containerization tools by configuring a continuous integration workflow. As a result, maintainers would not need to invest substantial additional manual effort for future submissions, because continuous integration workflows can perform validation checks while improving environment reproducibility. A working example of the code and end-to-end execution workflow has been uploaded to our repository.

Our recommendations primarily emphasize the reconstruction of executable software environments. However, the reproducibility of benchmark results may also be affected by differences in operating systems, hardware driver versions, hardware configurations, and nondeterministic behavior in numerical computation. Because it is impractical to require all training runs to be performed on identical machines, several additional strategies could be considered to improve the executability, accessibility, and robustness of algorithms hosted on benchmark platforms. For instance, environment reconstruction and execution workflows could be provided through a hosted cloud service, allowing users to test both environment configuration and code execution on a widely accessible managed cloud environment. Alternatively, callable model interfaces, associated metadata, and execution specifications could be published through Machine Learning Operations (MLOps) platforms such as Garden-AI, which may improve model discoverability and simplify remote invocation, particularly for standardized machine-learning workflows. Another option is to distribute each algorithm through a container (e.g., as a Docker image or Dockerfile), which can preserve much of the user-space software stack and reduce dependency drift caused by changes in the Python package ecosystem. Nevertheless, containerized execution does not eliminate all sources of variation, as host drivers, GPU hardware, system architecture, and registry maintenance may still affect execution. These approaches can improve different aspects of reproducibility and usability, but they also impose additional burdens on method developers, algorithm contributors, and platform maintainers in terms of packaging, validation, infrastructure, storage, and long-term maintenance. These approaches address different aspects of environment preservation and workflow accessibility, and no single strategy eliminates all sources of computational variation. The trade-off between these operational costs and the resulting gains in reproducibility should be carefully considered. Their respective capabilities, limitations, and maintenance requirements are summarized in Table 5.