HPCTESTS 2025
Third International Workshop on HPC Testing and Evaluation of Systems, Tools, and Software
https://olcf.github.io/hpc-system-test-wg/hpctests/hpctests2025
Details
- When: 08:30 - 12:00 CT on Friday, November 21, 2025, held in conjunction with SC25.
- When: Held in conjunction with SC25 in St. Louis, Mo, USA.
Description
The HPCTESTS workshop brings together HPC researchers, operators, and vendors from around the globe to present and discuss state-of-the-art HPC system testing methodologies, tools, benchmarks, procedures, and best practices. The increasing complexity of HPC architectures and growing need to leverage HPC for integrated workflows more extensive testing than ever in order to thoroughly evaluate the status of the system after installation or a software upgrade and ensure proper operation before it is transitioned to production. Different methodologies are used to evaluate systems during their lifetime, not only at the beginning during the installation, but also during maintenance windows and alongside regular operations. This workshop provides a venue to present and discuss the latest HPC system testing technologies and methodologies.
The event will include an opening talk focused on current HPC system testing topics, followed by a series of paper presentations from peer-reviewed accepted submissions and concluding with a panel discussion.
Call for Papers
The Third International Workshop on HPC Testing and Evaluation of Systems, Tools, and Software (HPCTESTS) will bring together experts from high performance computing (HPC) centers and vendors around the globe to present and discuss state-of-the-art HPC system testing methodologies, tools, and best practices. The workshop will encourage submissions that highlight current benchmarks, tests, and procedures utilized in today’s HPC systems. This event will provide an avenue to showcase newly developed tools and methodologies, and those that are being actively designed to allow authors to gather feedback from the community that could help guide their project. As machine learning (ML), deep learning (DL), and integrated workflows become more prevalent workloads, HPC centers must provide a wider range of services and more robust and resilient resources in order to support both traditional HPC and emerging workloads. The workshop will also invite submissions that are looking ahead at the post-exascale future of HPC system testing to initiate investigation of alternate mechanisms that could be used to adapt to the evolving and emerging workloads. The event will invite and welcome international participation from HPC centers, academic institutions, and representatives from vendors in the supercomputing space.
In addition to discussing procedures and tools utilized, submissions can describe challenges, lessons learned, and best practices used for regression testing, acceptance testing, and hardware evaluations. Furthermore, the workshop aims to encourage submissions that explore testbed evaluations as a means to gather preliminary results on system readiness to assist system design and deployment efforts.
This half-day workshop will kick off with an opening presentation focused on HPC system testing at scale. The session will be followed by a series of paper talks selected from the open call for submissions that will be held. The workshop will conclude with a panel discussing key topics impacting the HPC system testing community.
The workshop description, program committee, Call for Papers (CFP), papers, and presentations will be archived in the HPC System Test Working Group website (https://olcf.github.io/hpc-system-test-wg/) under a dedicated section for the HPCTESTS workshop.
Due to the wide range of systems currently in production and the rapidly evolving landscape of supercomputer architectures, the community should leverage individual center efforts to develop a collaborative set of best practices that will evolve from the publication of current tools, tests, and benchmarks. The diversity of architectures, compounded with the distinct user workloads supported at each HPC center has resulted in each center developing their own procedures, tests, benchmarks, and tools to conduct acceptance and regression testing. However, as we have learned through our events and workshops, these mechanisms are conceptually similar, and the tests utilized could be leveraged by multiple centers. In the workshop, we hope to attract stakeholders from vendors, HPC center staff, students, and faculty, interested in exploring HPC system testing topics more in depth.
Topics of interest include, but are not limited to:
- Testing methodologies and procedures.
- Tools for regression testing, frameworks.
- Automation of testing and continuous regression testing and performance monitoring.
- Selection and development of proxy-applications, benchmarks, synthetic vs. real applications.
- Efforts to improve reproducibility, sustainability, and availability of tests that can be leveraged by the community.
- Hardware and component focused testing (compute, memory, network, storage) at all scales from a single server (CPU/GPU) to clusters and cloud environments.
- System software, programming languages, and library testing.
- Monitoring and analysis of test results.
- Best practices and lessons.
- Early detection of failures using novel approaches (e.g., leveraging ML).
- Benchmark and testing for emerging technologies (e.g., quantum computing, ML/AI).
- Testing methodologies for integrated workflows and ecosystems.
Paper Submissions
The workshop will publish its proceedings with the SC25 conference. Authors must follow the formatting guidelines from SC25 Papers which are available here. Submissions can be 5-10 two-column pages (U.S. letter – 8.5 inches x 11 inches), excluding the bibliography, using the ACM proceedings template. Latex users, please use the “sigconf” option (use of the “review” option is recommended but not required). Word authors can use the “Interim Layout”. See the templates here.
Please also note the reproducibility initiative, which we follow for HPCTESTS.
Submissions will be accepted through the SC25 Submissions site: https://submissions.supercomputing.org/. After you create an account, you will be able to submit to the HPCTESTS 2025 form.
Workshop Deadlines
- Paper Submission Deadline: August 1, 2025 AoE (UTC-12)
- Author Notification: September 5, 2025 AoE
- Camera-ready: September 29, 2025 AoE
Organizing Committees
HPCTESTS 2025 General Chairs
- Verónica G. Melesse Vergara (Oak Ridge National Laboratory, USA)
- Bilel Hadri (King Abdullah University of Science and Technology, Saudi Arabia)
- Vasileios Karakasis (NVIDIA, Switzerland)
HPCTESTS 2025 Program Chairs
- Nick Hagerty (Oak Ridge National Laboratory, USA)
- Andreas Herten (Jülich Supercomputing Centre, Germany)
HPCTESTS 2025 Steering Committee
TBC
HPCTESTS 2025 Program Committee
TBC