Any users of the multi-file driver?

Hdf-forum archives
1

Hello,

We have been entertaining an idea of removing multi-file driver in the future releases of HDF5 and leaving the split driver only (it will need reimplementation since the split driver is a wrapper of the multi-file driver). Before we make any decision, we would like to know if anyone uses and what is your experience with the

  - <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplMulti> multi-file drive<https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplMulti>r
  - split driver <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplSplit>

Thank you!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2

Hi Elena,

To the best of my knowledge, we do not use either driver at LCLS at SLAC and would not notice their removal. I have a sense that the new virtual object layer feature is a better way to split large hdf5 files.

best,

David

···

On 01/18/16 08:32, Elena Pourmal wrote:

Hello,

We have been entertaining an idea of removing multi-file driver in the future releases of HDF5 and leaving the split driver only (it will need reimplementation since the split driver is a wrapper of the multi-file driver). Before we make any decision, we would like to know if anyone uses and what is your experience with the

  - multi-file drive <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplMulti>r
  - split driver <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplSplit>

Thank you!

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

3

Hi David,

Thank you for your response!

···

On Jan 19, 2016, at 11:03 AM, David A. Schneider <davidsch@slac.stanford.edu<mailto:davidsch@slac.stanford.edu>> wrote:

Hi Elena,

To the best of my knowledge, we do not use either driver at LCLS at SLAC and would not notice their removal. I have a sense that the new virtual object layer feature is a better way to split large hdf5 files.

Hmm.. I am not sure about VOL, but it is probably possible. Are you thinking about VDS<https://www.hdfgroup.org/HDF5/docNewFeatures/NewFeaturesVirtualDatasetDocs.html> ?

Elena
best,

David

On 01/18/16 08:32, Elena Pourmal wrote:
Hello,

We have been entertaining an idea of removing multi-file driver in the future releases of HDF5 and leaving the split driver only (it will need reimplementation since the split driver is a wrapper of the multi-file driver). Before we make any decision, we would like to know if anyone uses and what is your experience with the

- multi-file drive <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplMulti>r
- split driver <https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplSplit>

Thank you!

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

4

Hi Elena,

I've used the split (raw/meta) on several occasions for special purposes and when trying to decouple meta-data performance from raw data performance.

But, I've never used the multi-way driver. Indeed it does seem a bit like overkill to support both as I think the raw/meta split is the primary multi-case.

One thing, making HDF5 'smart' about H5open()'ing files created by split (or multi) driver. My recollection is that the lib is not smart enough to know
that the split driver is needed when opening a file so created and that caller is obligated to set file access props appropriately. In fact, I don't even think
the lib returns an error from H5open() that might suggest to caller that a split driver should be used. It'd be need if that was improved :wink:

Mark'

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, January 18, 2016 8:32 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Any users of the multi-file driver?

Hello,

We have been entertaining an idea of removing multi-file driver in the future releases of HDF5 and leaving the split driver only (it will need reimplementation since the split driver is a wrapper of the multi-file driver). Before we make any decision, we would like to know if anyone uses and what is your experience with the

  - <https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplMulti> multi-file drive<https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplMulti>r
  - split driver <https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplSplit>

Thank you!

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://secure-web.cisco.com/159O6bSmWH0BjUfC8NZuNitBamQuSZ-Pb8yfxF-sRwLXF_HwkK3cyby4pCDE4EPHlvaj_uJWiVdLX9NEfJvaIUJW1W8fo7WVGMCLvO3YiZOKaqSoxi0zv-bqoiO_go0-44PlU3p617adjsvVRRR8fpOI8qXEE1nmjoMzPpY2J14CcL3YIoCiFXmJsbzqQsio5qFSg8lQYJjlMqWIVV5RWz4a4_hcYQxStS0X8C0RzJhqrX3u2OAwjPXjuS-wFrwU1PDSvwICM47tl5nQLCIAJSC-HknmcFnaKYAHkyOGdP8SqfzIpqrDOrGiJ6KzWd3e145SGvgjKvW3umH6jWez7vaCgGMn8xPgu_wGuMmJAWNI/http%3A%2F%2Fhdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

5

Hi Mark,

Thank you for feedback!

···

On Jan 19, 2016, at 12:13 PM, Miller, Mark C. <miller86@llnl.gov<mailto:miller86@llnl.gov>> wrote:

Hi Elena,

I've used the split (raw/meta) on several occasions for special purposes and when trying to decouple meta-data performance from raw data performance.

Yes, the split driver was created to address performance issues with metadata writes. The feature goes back to HDF5 1.2.0!

But, I've never used the multi-way driver. Indeed it does seem a bit like overkill to support both as I think the raw/meta split is the primary multi-case.

Good to know. We really want to drop it because maintenance of this feature is the issue.

One thing, making HDF5 'smart' about H5open()'ing files created by split (or multi) driver. My recollection is that the lib is not smart enough to know
that the split driver is needed when opening a file so created and that caller is obligated to set file access props appropriately.
You are correct.

In fact, I don't even think
the lib returns an error from H5open() that might suggest to caller that a split driver should be used. It'd be need if that was improved :wink:

The enhancement is on our to-do list. One shouldn’t know the driver in order to open a valid HDF5 file. We even started working on the rework, but VDS and SWMR features pushed the priorities. Hopefully, we will get back to it after 1.10.0.

Best,

Elena
Mark'

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, January 18, 2016 8:32 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Any users of the multi-file driver?

Hello,

We have been entertaining an idea of removing multi-file driver in the future releases of HDF5 and leaving the split driver only (it will need reimplementation since the split driver is a wrapper of the multi-file driver). Before we make any decision, we would like to know if anyone uses and what is your experience with the

  - <https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplMulti> multi-file drive<https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplMulti>r
  - split driver <https://secure-web.cisco.com/1nGpvY3IzOhBKqzh6GJsP6NMRrou6rHtKKqwvx_rDxGZoQlvumycbs9O7fXKsDr9j1rI_-s2WlBgkdNHM6HwpqK8jvq-8h_VPBr1xSARvadM-jpIg67aEuOxTsgsRXj6MM8uTczI8bYtqZMFrj8OGW3vJlPh10WJ7qj_qihUhpW1hS_HOhcE-OzAXPTwwa7MfyknwiXMJha4bswES3PGF04vpQySKws_rng4VXeSnOJ8gXeGeBtJC8ooI-oSBS1LNM5mw-GnszqgwKhaGrvyc81OpCW5EEzD84G565QKG1_c1lcIDqlvY_PAUao8FaIbICzYUwmDrlFQs1B0qNXMdStJY6L0ckzkQmHHsSLBDTcM/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FRM%2FRM_H5P.html#Property-SetFaplSplit>

Thank you!

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://secure-web.cisco.com/159O6bSmWH0BjUfC8NZuNitBamQuSZ-Pb8yfxF-sRwLXF_HwkK3cyby4pCDE4EPHlvaj_uJWiVdLX9NEfJvaIUJW1W8fo7WVGMCLvO3YiZOKaqSoxi0zv-bqoiO_go0-44PlU3p617adjsvVRRR8fpOI8qXEE1nmjoMzPpY2J14CcL3YIoCiFXmJsbzqQsio5qFSg8lQYJjlMqWIVV5RWz4a4_hcYQxStS0X8C0RzJhqrX3u2OAwjPXjuS-wFrwU1PDSvwICM47tl5nQLCIAJSC-HknmcFnaKYAHkyOGdP8SqfzIpqrDOrGiJ6KzWd3e145SGvgjKvW3umH6jWez7vaCgGMn8xPgu_wGuMmJAWNI/http%3A%2F%2Fhdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5