Problem with Combining Split File and File Family

Hdf-forum archives
1

Hi,

I'm combining the Split File and File Family drivers in the following way:
- Split File is applied in order to separate the data into two files
(metadata and raw data, i.e. data.meta and data.raw)

- then additionally, for the raw data storage the File Family driver is
applied

This results in having one metadata file (data.meta) and multiple raw data
files according to the File Family member size setting (data0.raw, data1.raw
etc.).

For the most part this combination works as one would expect, but there is
one circumstance in which it doesn't seem to work properly.

Let's say that the file family member size is set 100 MB when creating the
HDF5 file. If during the first write to this file the amount of data that
ends up being written is more than 100 MB, for example 120 MB, then
everything works as expected - the result is one data.meta file and two raw
files; data0.raw which is 100 MB and data1.raw which is 20 MB, and appending
even more data always results in *.raw files of 100MB (except for the last
raw file which may not be full).

However, if the first writing to this file is less than 100MB (less than the
file family member size), for example 50 MB, and the file is closed, then
this will create 2 files; data.meta and data0.raw of size 50 MB. This is all
still as expected. If however, we then write more data to that file, then
things work as if the file family member size setting was set to 50 MB
rather than 100 MB, i.e. if we write 120 more MB to this file after closing
it, this will result in data.meta, data0.raw of 50 MB, data1.raw of 50 MB,
data2.raw 50 MB, and data3.raw of 20 MB.

It's as if the setting for the file family member size is 50 MB rather than
100 MB, i.e. the file family member size becomes the size of the first raw
file that is written, rather than the one that is specified in the creation
properties.

The HDF5 version is 1.8.7, operating system is Windows 7 64 bit.

1. Is the described file driver combination fully supported?

2. Is there a way to get around this behavior, so that the raw files end up
respecting the file family member size creation setting, rather than
inheriting the size of the first raw file after the first write session?

Thank you.

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Problem-with-Combining-Split-File-and-File-Family-tp3558487p3558487.html
Sent from the hdf-forum mailing list archive at Nabble.com.

2

Hi,

Hi,

I'm combining the Split File and File Family drivers in the following way:
- Split File is applied in order to separate the data into two files
(metadata and raw data, i.e. data.meta and data.raw)

- then additionally, for the raw data storage the File Family driver is
applied

This results in having one metadata file (data.meta) and multiple raw data
files according to the File Family member size setting (data0.raw, data1.raw
etc.).

For the most part this combination works as one would expect, but there is
one circumstance in which it doesn't seem to work properly.

Let's say that the file family member size is set 100 MB when creating the
HDF5 file. If during the first write to this file the amount of data that
ends up being written is more than 100 MB, for example 120 MB, then
everything works as expected - the result is one data.meta file and two raw
files; data0.raw which is 100 MB and data1.raw which is 20 MB, and appending
even more data always results in *.raw files of 100MB (except for the last
raw file which may not be full).

However, if the first writing to this file is less than 100MB (less than the
file family member size), for example 50 MB, and the file is closed, then
this will create 2 files; data.meta and data0.raw of size 50 MB. This is all
still as expected. If however, we then write more data to that file, then
things work as if the file family member size setting was set to 50 MB
rather than 100 MB, i.e. if we write 120 more MB to this file after closing
it, this will result in data.meta, data0.raw of 50 MB, data1.raw of 50 MB,
data2.raw 50 MB, and data3.raw of 20 MB.

It's as if the setting for the file family member size is 50 MB rather than
100 MB, i.e. the file family member size becomes the size of the first raw
file that is written, rather than the one that is specified in the creation
properties.

The HDF5 version is 1.8.7, operating system is Windows 7 64 bit.

1. Is the described file driver combination fully supported?

I would describe this as "Allowed behavior, but not explicitly
recommended and certainly not tested by us." That said, I would
expect this setup to work correctly.

2. Is there a way to get around this behavior, so that the raw files end up
respecting the file family member size creation setting, rather than
inheriting the size of the first raw file after the first write session?

I'm not sure of a work-around that would fix your problem.

I've entered a bug report concerning this behavior since this sounds
like a bug in the driver. I'm not sure how high of a priority it will
have since this is a more unusual way to use the virtual file drivers,
but I'll bring it up at our release planning meeting.

Thanks for reporting the issue!

Dana

···

On Sat, Dec 3, 2011 at 9:56 PM, Charles Darwin <trokabg@yahoo.com> wrote:

Thank you.

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Problem-with-Combining-Split-File-and-File-Family-tp3558487p3558487.html
Sent from the hdf-forum mailing list archive at Nabble.com.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org