Omnibenchmark
About this Project
In data-rich scientific fields, there are often competing computational tools to analyze data, and it is important, both for users and method developers, to know their relative merits (e. g., performance, tradeoffs, user friendliness).
By introducing broader patterns of open science practices, including open data and especially open research code, we aim to make method performance assessments (i. e., benchmarks) more accessible and impactful to the community. Our initiative, Omnibenchmark, is poised to improve not only the efficiency and transparency of (computational) methods research, but also the robustness and usability of state-of-the-art methods for various end-users.
Scientific Summary
Omnibenchmark provides a system for extensible benchmarks that are open to contributions of the community. Omnibenchmark defines, runs and versions benchmark execution pipelines by leveraging a formal benchmark specification and free and open source (reusable) benchmarking modules (software to test). Each benchmarking module implements a data processing step (typically a preprocessing step, method, or metric) and is stored as an independent git repository.
Anyone can start (or contribute to) a benchmark. A contribution can be a relevant ground truth dataset, a newly developed method or metric, or update to existing modules.
Omnibenchmark relies on widely-used, free and open software solutions, including: git (version control system), GitLab (DevOps), Snakemake (workflow management system), easybuild (reproducible software environments), and apptainer (containerization).
Challenges and Goals
- future-proof scientific benchmark design based on formal specifications
- reproducible software environments
- automated updates and continuous deployment
- re-use of open reference datasets
- user friendliness
- community engagement
Results and Output
Omnibenchmark is still in active development at https://github.com/omnibenchmark/. An initial working prototype is planned for the end of 2024.
Impact on Open Science Practices
Omnibenchmark fosters modern open research data, open code and overall open science practices by providing a systematic approach to evaluate the performance of existing computational approaches, including reproducible and reusable software environments, transparency on how references datasets are selected and processed, and details of how methods are applied as well as performance.
