The Hermes team is excited to announce that the Hermes 0.1.0-beta release is now available from GitHub for testing (https://github.com/HDFGroup/hermes).
Hermes is a distributed I/O buffering system for deep distributed storage hierarchies, which are commonly found on modern HPC systems (learn more here). Hermes is an application extension and does not require any privileged access. Currently, Hermes can be used by applications in three different ways:
- LD_PRELOAD-able Hermes adapters for UNIX STDIO, POSIX, and MPI-IO
- HDF5 extensions such as the HDF5 Hermes Virtual File Driver
- Hermes-native API providing maximum control over I/O buffering
For options 1 and 2, no or only minimal code changes are required. Users who want to get the most out of Hermes might consider option 3, introducing a new dependence and requiring code changes or new development.
Please check out our Getting Started with Hermes guide, which includes a step-by-step walkthrough of how to use Hermes with your applications.
The intended audience for this Beta release includes developers who are exploring innovative ways to accelerate I/O operations in their awesome workflows. Early adopters are welcome to try our Beta software to evaluate how Hermes can be integrated into their applications and systems.
This version of Hermes offers a lot. (See the Hermes core library schematic in the Wiki.) Several components, such as the buffer organizer or prefetcher, are working behind the scenes, but they are not yet customizable to the degree we want them to be. Stay tuned for the next release iteration, where we will have more to say on that. Also, consider registering for our upcoming webinar on December 1st, 2021.
Please note that Hermes is under active development. Most of the action takes place on GitHub. If you are familiar with the Git workflow and you would like to get involved, create an issue or feature request, submit a pull request, etc.
As a buffering system, Hermes has a configurable but finite buffering capacity. These are the resources from the user’s storage hierarchy to set aside for I/O buffering purposes. Since that total capacity is the aggregate of a heterogeneous mixture, the configuration cannot be boiled down to a single number. Guidance on configuration defaults and how to configure Hermes can be found in the Hermes GitHub Wiki.
One of the easier ways to get started with Hermes is to use one of our Docker images:
- hermes-deps - contains just the dependencies necessary to build Hermes
- hermes - has a ready-to-run Hermes environment
- hermes-dev - is a full-blown Hermes development environment
Thanks to all our supporters and contributors. Happy testing!
Special thanks to our sponsor, the National Science Foundation, where discoveries begin!
Don't forget to tune into Call the Doctor for free HDF5 help each Tuesday
Every Tuesday at 1:00 p.m. CDT, The HDF Group hosts Call the Doctor, a session to answer attendee questions and, as time allows, provide helpful tips and tricks for using HDF5 or go over the previous week’s HDF Forum posts.
The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.
To participate, please submit questions and topics in this google doc. One-time registration required. If you’ve already registered, no need to register again but consider this your reminder for tomorrow. You can also catch up on all past Call the Doctor sessions on youtube.