Currently, to access official binaries for non-latest versions (which is needed, for example, for CI purposes, to test how using the older versions will affect the users), one needs to “log in to see the downloads”. The download link then has a temporary download in it.
What’s the reason behind hiding those binaries? It would be nice if there were stable official URLs with previously released binaries (this is most important for windows builds really, for other platforms users would typically install them using the respective package managers).
I could find some binaries here - https://support.hdfgroup.org/ftp/HDF5/prev-releases/ - but there’s only 1.8.21 and 1.10.0 with ‘bin’ folders for whatever reason, and all other versions only contain the source.
That would certainly be nice. Posting the binaries as releases the GitHub repo would be a good start. For our own CI testing I posted the binaries to our own webserver which solved the problem for us but clearly not for anyone else.
Another thought, if the binaries are accessible via http, it would also be nice to have structured machine-readable URLs, like /1.10/1.10.6/win64-msvc14/hdf5-1.10.6-win64-msvc14.zip etc.
Thank you for your suggestions. I created an issue HDFV-11125 and we will think what we can do especially when we move to Github (as soon as we release HDF5 1.10.7, currently end of August)
All,
Here is some rational behind not distributing binaries for all platforms:
As many of you know HDF5 is free but there is a cost to maintain it and to provide the services like the requested one.
Organizations and groups you represent could help us in many ways, for example, testing and building software and providing binaries for the platforms you are interested in, fixing bugs and contributing features, or getting a support agreement with us so we can provide (and continue to provide) such services to the community.
Binaries we post on our Website are paid by the support grants we receive. Those come from a tiny fraction of organizations that use and depend on HDF5. If you are interested in better support and well-maintained HDF5, please contribute to HDF5 maintenance, advocate for HDF5 maintenance, convince your superiors that without contribution it would be hard to get a well-maintained software your organization depends on. Your help would be highly appreciated.
If you want to learn more about our support agreements, please contact us at info@hdfgroup.org.
@epourmal Thanks a lot for taking time to reply to this.
As a disclaimer, I’m not interested in this commercially, but rather as a developer of an open-source library (hdf5-rust) which I maintain for free and in my own free time so as to allow other developers to use HDF5 in their Rust projects. Although the project is in early stage, I think it’s already used in production in a few places. The reason for this request is being able to run CI tests over a variety of different HDF5 versions, including the official binaries, and to be able to discover faults and bugs early, both on our side and in the core libhdf5.
As for the price of hosting the binaries, if you were to upload them to S3, and assuming the binaries for all possible versions take 10GB, it would cost you $2.76 per year, i.e. almost zero. Assuming you have those binaries built already (because they are downloadable via auth-requiring links), it’s just a matter of uploading them, naming things properly and consistently and posting the links on your website, after which there’s no maintenance required whatsoever. Just a thought.
I would like to address this issue from the context of another small business, mine. BlueQuartz Software produces a purely open source application that relies on service contracts and DoD funding for its development. Some years are better than others. Some years have seen my business contract to just myself and a single engineer. We none-the-less have our codes on GitHub and post binaries (both nightly and official) on our website free for anyone to download. We have built up a build bot farm over the years in order to build and test the codes on a reasonable set of configurations. We have recently figured out Azure where we have our own on-prem build machines acting as clients to the Azure CI system. All of this takes resources and at times it can consume more time than I would wish but for our community it is worth the cost. There are no hurdles for our users to download and use our codes. We have had feedback from users/clients who have thanked us for that openness so that their own projects can progress forward without undue hindrances.
Placing unneeded hurdles in front of developers who are trying to use HDF5 in their products doesn’t really help the adoption of HDF5, it hinders it and at times stops it dead. I have been an avid user and quiet advocate for HDF5 since the mid 2000s. I advocate for its use in materials science and engineering disciplines where I can. We made HDF5 the “native” file format for our open source data analysis package because I believed in the project. That advocacy is starting to wane a bit recently because of moves like placing the binaries behind a wall. In talking with other quiet advocates about these moves we have all admitted that maybe it is time to move onto another format. We have at least started the due diligence of taking a look at what else is around.
GitHub will happily host the binaries for each release. This is “free” as in money. What isn’t free is the time it takes to compile and upload those binaries. I fully understand that The HDFGroup is still on the hook to pay an employee to generate the binaries and upload the results. The revenue to pay those employees must come from somewhere. The marketing and community outreach will hopefully provide enough leads and funding resources to pay those bills. Again, I am personally faced with those same problems for our own product. And there are lots of other companies in the same situation. How do we make open source software but still pay the bills. There is no silver bullet answer to that question. In my strong opinion, putting up barriers to adoption isn’t an answer or even a step in the right direction. In these tough times we are all looking for funding sources to keep the doors open and the lights on. Having advocates in the community voicing their support is what you want. You want them advocating for HDF5 as a standard file format.
I think a better approach would be to post here on the list on a regular basis a gentle reminder that HDF5 is run by a non-profit organization that needs continual funding to make the updates/bug fixes and feature changes that we as a community want. Maybe a bulleted list that says the things that are being worked on, links to someone at The HDFGroup to get in contact with for service help, or just how to advocate for HDF5 funding. A few prepared slides that someone could show a manager should they come into a stream of funding.
Not much to add here, I pretty much concur with everything Mike said as my experience is quite similar.
On the topic, if there’s any particular help needed, in storing / organizing the builds, if the HDF Group voiced what exactly they need help with, I believe the community would gladly help.
