Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
Yes, h5repack is the only way to “un-virtualize” VDS without writing a code. Unfortunately there is a known issue with h5repack and VDS as you have discovered.
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 5, 2016, at 5:06 PM, Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>> wrote:
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
I honestly don't know. But, if you have a small file with VDS datasetes in it, maybe give it a try and see what happens.
Mark
···
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Nelson, Jarom" <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, April 5, 2016 3:06 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] h5repack on files with VDS
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
I tried a few different tests:
1. Adding some filters (SHUF, GZIP) looks to have failed silently.
$ h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.shuf.gzip1.h5
No error reported, but the data was not accessible. H5dump output in attached.
2. I attempted to change the layout to CONTI (contiguous), but it also fails silently.
$ h5repack -l CONTI h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.conti.h5
Same as test#1: No error reported, but the data was not accessible. H5dump output in attached.
3. I attempted to change the layout to CHUNK=100, but it looks like the resulting file is identical. H5dump doesn't indicate that the layout is now chunked, and the dataset is still virtual:
$ h5repack -l CHUNK=100 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.h5
No errors, but the h5dump output is identical to the original.
4. Same result with CHUNK=100 and SHUF, GZIP filters.
$ h5repack -l CHUNK=100 -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.shuf.gzip.h5
Same result as test #3: No errors, but the h5dump output is identical to the original.
5. I also (unwisely) attempted to change layout to CONTI and add SHUF,GZIP filters, but realized that you can't add these filters with contiguous layout. I suppose this error is expected.
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
#000: H5Pdcpl.c line 2009 in H5Pset_chunk(): chunk dimensionality must be positive
major: Invalid arguments to routine
minor: Out of range
h5repack error: <h5g_output_parallel.global.100.10.h5>: Could not copy data to: h5g_output_parallel.conti.shuf.gzip1.h5
Resulting files and "h5dump -p" output in the attached (with exception of CONTI/SHUF/GZIP test).
Original file was created in HDF5 1.10.0 with single VDS with source datasets spread across several files (attached).
$ h5repack --version
h5repack: Version 1.10.0
$ h5dump --version
h5dump: Version 1.10.0
My suspicion is that what I'm attempting to do is not (yet?) supported.
Jarom
h5repack.vds.tar.gz (2.86 KB)
···
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, April 05, 2016 3:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
I honestly don't know. But, if you have a small file with VDS datasetes in it, maybe give it a try and see what happens.
Mark
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Nelson, Jarom" <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, April 5, 2016 3:06 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] h5repack on files with VDS
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
Hi Jarom,
This is a known problem with h5repack and VDS (see Known Problems section https://www.hdfgroup.org/HDF5/release/obtain5110.html and our release announcement to this FORUM). The info didn’t make into the RELEASE.txt file because the problem was discovered just after the source for the release was out.
As a quick fix that will allow you to repack VDS data, you may try to edit tools/h5repack/h5repack_copy.c , line 1037. Replace
if (nelmts > 0 && space_status != H5D_SPACE_STATUS_NOT_ALLOCATED) {
with
if (nelmts > 0) {
We are working on the patch and will issue it as soon as it becomes available. "space_status != H5D_SPACE_STATUS_NOT_ALLOCATED" condition was there to avoid issuing H5Dwrite call when there was no data to write. Removing the condition, we do address VDS case, since the space is not allocated for VDS, but we are changing previous behavior. I.e., this is really a workaround.
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 5, 2016, at 6:26 PM, Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>> wrote:
I tried a few different tests:
1. Adding some filters (SHUF, GZIP) looks to have failed silently.
$ h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.shuf.gzip1.h5
No error reported, but the data was not accessible. H5dump output in attached.
2. I attempted to change the layout to CONTI (contiguous), but it also fails silently.
$ h5repack -l CONTI h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.conti.h5
Same as test#1: No error reported, but the data was not accessible. H5dump output in attached.
3. I attempted to change the layout to CHUNK=100, but it looks like the resulting file is identical. H5dump doesn’t indicate that the layout is now chunked, and the dataset is still virtual:
$ h5repack -l CHUNK=100 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.h5
No errors, but the h5dump output is identical to the original.
4. Same result with CHUNK=100 and SHUF, GZIP filters.
$ h5repack -l CHUNK=100 -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.shuf.gzip.h5
Same result as test #3: No errors, but the h5dump output is identical to the original.
5. I also (unwisely) attempted to change layout to CONTI and add SHUF,GZIP filters, but realized that you can’t add these filters with contiguous layout. I suppose this error is expected.
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
#000: H5Pdcpl.c line 2009 in H5Pset_chunk(): chunk dimensionality must be positive
major: Invalid arguments to routine
minor: Out of range
h5repack error: <h5g_output_parallel.global.100.10.h5>: Could not copy data to: h5g_output_parallel.conti.shuf.gzip1.h5
Resulting files and “h5dump -p” output in the attached (with exception of CONTI/SHUF/GZIP test).
Original file was created in HDF5 1.10.0 with single VDS with source datasets spread across several files (attached).
$ h5repack --version
h5repack: Version 1.10.0
$ h5dump --version
h5dump: Version 1.10.0
My suspicion is that what I’m attempting to do is not (yet?) supported.
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, April 05, 2016 3:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
I honestly don't know. But, if you have a small file with VDS datasetes in it, maybe give it a try and see what happens.
Mark
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Nelson, Jarom" <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, April 5, 2016 3:06 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] h5repack on files with VDS
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
<h5repack.vds.tar.gz>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Thank you. The workaround patch does as described. I'm able to repack into a file and it contains all the content of the VDS.
I think I found a possibly related bug. After patching h5repack as described, I wanted to check to make sure the resulting files are identical in content, so I tried h5diff and got this error:
···
$ ~/cots/hdf5-1.10.0/tools/h5repack/h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
$ h5diff h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
--------------------------------
Some objects are not comparable
--------------------------------
Use -c for a list of objects.
$ h5diff -c h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
Not comparable: </common huge dataset> or </common huge dataset> is an empty dataset
Comparing the h5dump output of each file matches, however.
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Wednesday, April 06, 2016 6:22 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
Hi Jarom,
This is a known problem with h5repack and VDS (see Known Problems section https://secure-web.cisco.com/1f7EOVN4DFDFhrLtlLYigQ_7ppWqlLNwXyuNCq_yAgg3X5ArEtrkG2QpJ0jY-RkMmVBkZ2O6t6vomPY8u73k8qUnzR5q1X59mUhMpl26PcqzcIo53vBzZfOOZeXy2YXrzOqVsNNs-li6N264vH1cWXfIhVbPk8Olv7W8Qf-yqfSOewdVfq4X_U5IDaw7iyqbo6C8FU41nRfTxBFSDaALWw69lHXU-ULZVqNOx7f0qmOwsMOdNt-1eJmFZfwls_8_BcYXyW0R4RwCDpGgz8Nxnx7HWtdmdc7QSFn4kZDPc29KA3gCkJ35FPeWSQmBf16-Yqh1TlyzJgT0D5pAxCHzt3fipa_jOCY5pQj5oE_0j8TQ/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Frelease%2Fobtain5110.html and our release announcement to this FORUM). The info didn't make into the RELEASE.txt file because the problem was discovered just after the source for the release was out.
As a quick fix that will allow you to repack VDS data, you may try to edit tools/h5repack/h5repack_copy.c , line 1037. Replace
if (nelmts > 0 && space_status != H5D_SPACE_STATUS_NOT_ALLOCATED) {
with
if (nelmts > 0) {
We are working on the patch and will issue it as soon as it becomes available. "space_status != H5D_SPACE_STATUS_NOT_ALLOCATED" condition was there to avoid issuing H5Dwrite call when there was no data to write. Removing the condition, we do address VDS case, since the space is not allocated for VDS, but we are changing previous behavior. I.e., this is really a workaround.
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://secure-web.cisco.com/1HvSvHJAWEx-Bp9_sRioNfTmllRdUrk-ADyNaLrE3jYpCXllUtIDFN_Vw6EXU6JeZwCxYzHCpgd6x-sWGLJvKV8dCtuyw6rPYflhbJdMyC00OLPv7n8U6-QX9h3q2kh7l1hFIEAjQ7YpEnpCTB28KJ2NiFUwk9fcgUMYff9-BffeOZ5H2PszEWHGDoFliM4hyf6NYPpCMmQLDczUUV05Tp0UNi_ZWXy8ddvpRDy0F5TZ5PDI117uJVWIMwkPvlid1eLFtR_mlvO_DXuFWJoaVZnf58zzyO8AWwFg9XYp-y1CrD0s1DWLkVonMHfEiTNt19kqImZiU7stW_o6qDJPvbVKOUcf5_jj8g4fJVXxzBl8/http%3A%2F%2Fhdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 5, 2016, at 6:26 PM, Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>> wrote:
I tried a few different tests:
1. Adding some filters (SHUF, GZIP) looks to have failed silently.
$ h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.shuf.gzip1.h5
No error reported, but the data was not accessible. H5dump output in attached.
2. I attempted to change the layout to CONTI (contiguous), but it also fails silently.
$ h5repack -l CONTI h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.conti.h5
Same as test#1: No error reported, but the data was not accessible. H5dump output in attached.
3. I attempted to change the layout to CHUNK=100, but it looks like the resulting file is identical. H5dump doesn't indicate that the layout is now chunked, and the dataset is still virtual:
$ h5repack -l CHUNK=100 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.h5
No errors, but the h5dump output is identical to the original.
4. Same result with CHUNK=100 and SHUF, GZIP filters.
$ h5repack -l CHUNK=100 -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.shuf.gzip.h5
Same result as test #3: No errors, but the h5dump output is identical to the original.
5. I also (unwisely) attempted to change layout to CONTI and add SHUF,GZIP filters, but realized that you can't add these filters with contiguous layout. I suppose this error is expected.
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
#000: H5Pdcpl.c line 2009 in H5Pset_chunk(): chunk dimensionality must be positive
major: Invalid arguments to routine
minor: Out of range
h5repack error: <h5g_output_parallel.global.100.10.h5>: Could not copy data to: h5g_output_parallel.conti.shuf.gzip1.h5
Resulting files and "h5dump -p" output in the attached (with exception of CONTI/SHUF/GZIP test).
Original file was created in HDF5 1.10.0 with single VDS with source datasets spread across several files (attached).
$ h5repack --version
h5repack: Version 1.10.0
$ h5dump --version
h5dump: Version 1.10.0
My suspicion is that what I'm attempting to do is not (yet?) supported.
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, April 05, 2016 3:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
I honestly don't know. But, if you have a small file with VDS datasetes in it, maybe give it a try and see what happens.
Mark
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Nelson, Jarom" <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, April 5, 2016 3:06 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] h5repack on files with VDS
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
<h5repack.vds.tar.gz>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1D93fpHEQg441X7FiEikFo5lm3FesRe87j3d5ya1Hsp_UBJkiPWgvnNQPpb-Y28nmOuiELhzXuzRWzgKVNqN6MijHH5NUqaV9YjaxblhWqLgeilj7jr6bfZTqXDFrnmIjYR0d-gavu04418NDLz_QSFtYMZCwwKOI4YjezOlRBzD1clD8VAObp21PiyB8YxTEQ3_SUOLImbtJdG7JaBI4TOC1OYSsRLlN50TX7sP96RG1gPq6wauEiLaEwXNhDltLFji7mR_qZxXWvw1Y3TVjDHoyyrAPpcvMTnGd61npjvxhe8dWO5X3EerqylMt3O5faWV27l9Q0ae0LFxmvoK3F8P3MpMXFhjKSmbi0Nj_oZk/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Thank you for reporting!
h5diff definitely needs some improvements to handle VDS layout. I entered an issue into JIRA (HDFFV-9756 for your reference) and scheduled it for the HDF5 1.10.1 release.
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 6, 2016, at 11:00 AM, Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>> wrote:
Thank you. The workaround patch does as described. I’m able to repack into a file and it contains all the content of the VDS.
I think I found a possibly related bug. After patching h5repack as described, I wanted to check to make sure the resulting files are identical in content, so I tried h5diff and got this error:
$ ~/cots/hdf5-1.10.0/tools/h5repack/h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
$ h5diff h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
--------------------------------
Some objects are not comparable
--------------------------------
Use -c for a list of objects.
$ h5diff -c h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.fixed.shuf.gzip1.h5
Not comparable: </common huge dataset> or </common huge dataset> is an empty dataset
Comparing the h5dump output of each file matches, however.
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Wednesday, April 06, 2016 6:22 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
Hi Jarom,
This is a known problem with h5repack and VDS (see Known Problems section https://secure-web.cisco.com/1f7EOVN4DFDFhrLtlLYigQ_7ppWqlLNwXyuNCq_yAgg3X5ArEtrkG2QpJ0jY-RkMmVBkZ2O6t6vomPY8u73k8qUnzR5q1X59mUhMpl26PcqzcIo53vBzZfOOZeXy2YXrzOqVsNNs-li6N264vH1cWXfIhVbPk8Olv7W8Qf-yqfSOewdVfq4X_U5IDaw7iyqbo6C8FU41nRfTxBFSDaALWw69lHXU-ULZVqNOx7f0qmOwsMOdNt-1eJmFZfwls_8_BcYXyW0R4RwCDpGgz8Nxnx7HWtdmdc7QSFn4kZDPc29KA3gCkJ35FPeWSQmBf16-Yqh1TlyzJgT0D5pAxCHzt3fipa_jOCY5pQj5oE_0j8TQ/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Frelease%2Fobtain5110.html and our release announcement to this FORUM). The info didn’t make into the RELEASE.txt file because the problem was discovered just after the source for the release was out.
As a quick fix that will allow you to repack VDS data, you may try to edit tools/h5repack/h5repack_copy.c , line 1037. Replace
if (nelmts > 0 && space_status != H5D_SPACE_STATUS_NOT_ALLOCATED) {
with
if (nelmts > 0) {
We are working on the patch and will issue it as soon as it becomes available. "space_status != H5D_SPACE_STATUS_NOT_ALLOCATED" condition was there to avoid issuing H5Dwrite call when there was no data to write. Removing the condition, we do address VDS case, since the space is not allocated for VDS, but we are changing previous behavior. I.e., this is really a workaround.
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://secure-web.cisco.com/1HvSvHJAWEx-Bp9_sRioNfTmllRdUrk-ADyNaLrE3jYpCXllUtIDFN_Vw6EXU6JeZwCxYzHCpgd6x-sWGLJvKV8dCtuyw6rPYflhbJdMyC00OLPv7n8U6-QX9h3q2kh7l1hFIEAjQ7YpEnpCTB28KJ2NiFUwk9fcgUMYff9-BffeOZ5H2PszEWHGDoFliM4hyf6NYPpCMmQLDczUUV05Tp0UNi_ZWXy8ddvpRDy0F5TZ5PDI117uJVWIMwkPvlid1eLFtR_mlvO_DXuFWJoaVZnf58zzyO8AWwFg9XYp-y1CrD0s1DWLkVonMHfEiTNt19kqImZiU7stW_o6qDJPvbVKOUcf5_jj8g4fJVXxzBl8/http%3A%2F%2Fhdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Apr 5, 2016, at 6:26 PM, Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>> wrote:
I tried a few different tests:
1. Adding some filters (SHUF, GZIP) looks to have failed silently.
$ h5repack -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.shuf.gzip1.h5
No error reported, but the data was not accessible. H5dump output in attached.
2. I attempted to change the layout to CONTI (contiguous), but it also fails silently.
$ h5repack -l CONTI h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.conti.h5
Same as test#1: No error reported, but the data was not accessible. H5dump output in attached.
3. I attempted to change the layout to CHUNK=100, but it looks like the resulting file is identical. H5dump doesn’t indicate that the layout is now chunked, and the dataset is still virtual:
$ h5repack -l CHUNK=100 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.h5
No errors, but the h5dump output is identical to the original.
4. Same result with CHUNK=100 and SHUF, GZIP filters.
$ h5repack -l CHUNK=100 -f SHUF -f GZIP=1 h5g_output_parallel.global.100.10.h5 h5g_output_parallel.global.100.10.chunk100.shuf.gzip.h5
Same result as test #3: No errors, but the h5dump output is identical to the original.
5. I also (unwisely) attempted to change layout to CONTI and add SHUF,GZIP filters, but realized that you can’t add these filters with contiguous layout. I suppose this error is expected.
HDF5-DIAG: Error detected in HDF5 (1.10.0) thread 0:
#000: H5Pdcpl.c line 2009 in H5Pset_chunk(): chunk dimensionality must be positive
major: Invalid arguments to routine
minor: Out of range
h5repack error: <h5g_output_parallel.global.100.10.h5>: Could not copy data to: h5g_output_parallel.conti.shuf.gzip1.h5
Resulting files and “h5dump -p” output in the attached (with exception of CONTI/SHUF/GZIP test).
Original file was created in HDF5 1.10.0 with single VDS with source datasets spread across several files (attached).
$ h5repack --version
h5repack: Version 1.10.0
$ h5dump --version
h5dump: Version 1.10.0
My suspicion is that what I’m attempting to do is not (yet?) supported.
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, April 05, 2016 3:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] h5repack on files with VDS
I honestly don't know. But, if you have a small file with VDS datasetes in it, maybe give it a try and see what happens.
Mark
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Nelson, Jarom" <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, April 5, 2016 3:06 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] h5repack on files with VDS
Can h5repack be used to un-virtualize a VDS?
Jarom Nelson
Lawrence Livermore National Lab
<h5repack.vds.tar.gz>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1D93fpHEQg441X7FiEikFo5lm3FesRe87j3d5ya1Hsp_UBJkiPWgvnNQPpb-Y28nmOuiELhzXuzRWzgKVNqN6MijHH5NUqaV9YjaxblhWqLgeilj7jr6bfZTqXDFrnmIjYR0d-gavu04418NDLz_QSFtYMZCwwKOI4YjezOlRBzD1clD8VAObp21PiyB8YxTEQ3_SUOLImbtJdG7JaBI4TOC1OYSsRLlN50TX7sP96RG1gPq6wauEiLaEwXNhDltLFji7mR_qZxXWvw1Y3TVjDHoyyrAPpcvMTnGd61npjvxhe8dWO5X3EerqylMt3O5faWV27l9Q0ae0LFxmvoK3F8P3MpMXFhjKSmbi0Nj_oZk/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5