What is InfiniteScienceGym?
InfiniteScienceGym (ISG) is a benchmark for evaluating large language models as scientific assistants on repository-grounded empirical data reasoning, introduced in our paper InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis .
The benchmark procedurally generates seed-indexed scientific repositories, each with a realistic file-system structure, tabular data, and a verifiable question-answer task. It includes both answerable and unanswerable questions spanning repository metadata, file metadata, directory traversal, univariate statistics, bivariate statistics, and hypothesis testing.
Unlike prior benchmarks built only from published studies or fixed datasets, InfiniteScienceGym is generated on demand and comes with exact ground truth by construction. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool use at effectively unbounded scale, without distributing a large static corpus.
From Random Seeds to Scientific Repositories
Every repository begins with a random integer seed. Repository #1 has the random seed 1, repository #2 has the random seed 2, and so on. Given the random seed, we first sample a field, domain, and subdomain from a taxonomy of scientific areas. Next, we sample a specific project idea, defined by a title, description and abstract, from an LLM, conditioned on the scientific context. The video above shows this workflow.
From the instantiated project, we then build out a plausible directory structure and file schema from the independent, dependent, and other variables relevant to the project. All independent variables are defined either as a distribution over a set of choices (categorical) or as a named, parameterized distribution, such as a Gaussian or Binomial distribution (continuous). All dependent variables are defined as a Python function of the independent variables, with some added random noise. The generating LLM is used to determine all the necessary metadata for the repository's variables.
With all the necessary metadata defined, we can now confirm whether any path is valid for a given repository, as well as generating the contents of a specific file on-the-fly. To do this, we hash the file's path into an integer value, and then use that value to set the random seed. From there, we simply sample the independent variables and use their values to calculated the dependent variables' values via their Python functions.
QA Task
Given a system that produces repositories, we use this setup to evaluate LLMs' abilities to answer the types of questions a scientist might ask about their data. Crucially, by simulating the scientific data, we have access to the ground truth process that creates the data, and can therefore ask questions that are unsanswerable by construction, such as:
For the experimental runs with a coupling strength of zero and a trial number of one, and considering only the measurement points where the time to amplitude death is no greater than 269.3 seconds and the phase coherence is at least 0.01655 radians, what is the variance of the phase coherence across those points?
This question is unanswerable because, after performing the required filtering on experimental results, there isn't actually any data left to calculate the variance from. For other questions this unswerability can come from requesting a continuous metric (e.g. average, variance, etc.) about a non-numeric, categorical variables. In other questions, the unsanswerability may relate to a technicality, like trying to calculate the Pearson's coefficient coefficient on a single data point (at least two points are needed to perform a linear regression). Refer to the paper for more details about our operational definition of unanswerability, and for the full description of the question types included in the benchmark.