anyone have experience with H5Pset_small_data_block_size

Hi All,

I am seeing some behavior in some tests that I am doing that suggests
that on the sec2 vfd (or any vfd I've tried) the small data block size
property (H5Pset_small_data_block_size) is having no effect in changing
I/O requests to underlying system I/O interface.

Does anyone have experience using this file access property such that
they KNOW FOR CERTAIN it is taking effect and indeed resulting in
different (lower) I/O request frequency to underlying I/O interface?

Mark

···

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

Hi Mark,

···

On Mar 3, 2010, at 12:51 PM, Mark Miller wrote:

Hi All,

I am seeing some behavior in some tests that I am doing that suggests
that on the sec2 vfd (or any vfd I've tried) the small data block size
property (H5Pset_small_data_block_size) is having no effect in changing
I/O requests to underlying system I/O interface.

Does anyone have experience using this file access property such that
they KNOW FOR CERTAIN it is taking effect and indeed resulting in
different (lower) I/O request frequency to underlying I/O interface?

  This API routine is only used to "push" allocation of the space for storing small dataset raw data together in the file. It has no effect on I/O requests to the underlying system. H5Pset_sieve_buf_size may help you out though...

  Quincey

Am I missing something here?

I thought the intent was to perform sub-allocations out of a larger
contiguous block so that you could write all the data associated with
the datasets that went into a 'small block' in a single I/O request.

Mark

···

On Wed, 2010-03-03 at 13:04, Quincey Koziol wrote:

Hi Mark,

On Mar 3, 2010, at 12:51 PM, Mark Miller wrote:

> Hi All,
>
> I am seeing some behavior in some tests that I am doing that suggests
> that on the sec2 vfd (or any vfd I've tried) the small data block size
> property (H5Pset_small_data_block_size) is having no effect in changing
> I/O requests to underlying system I/O interface.
>
> Does anyone have experience using this file access property such that
> they KNOW FOR CERTAIN it is taking effect and indeed resulting in
> different (lower) I/O request frequency to underlying I/O interface?

  This API routine is only used to "push" allocation of the space for storing small dataset raw data together in the file. It has no effect on I/O requests to the underlying system. H5Pset_sieve_buf_size may help you out though...

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

Hi Mark,

Am I missing something here?

I thought the intent was to perform sub-allocations out of a larger
contiguous block so that you could write all the data associated with
the datasets that went into a 'small block' in a single I/O request.

  No, that's not what occurs currently. (Although it might be a logical outgrowth of the current feature) Currently, it is only trying to reduce fragmentation of space allocation in the file.

  Quincey

···

On Mar 3, 2010, at 3:47 PM, Mark Miller wrote:

Mark

On Wed, 2010-03-03 at 13:04, Quincey Koziol wrote:

Hi Mark,

On Mar 3, 2010, at 12:51 PM, Mark Miller wrote:

Hi All,

I am seeing some behavior in some tests that I am doing that suggests
that on the sec2 vfd (or any vfd I've tried) the small data block size
property (H5Pset_small_data_block_size) is having no effect in changing
I/O requests to underlying system I/O interface.

Does anyone have experience using this file access property such that
they KNOW FOR CERTAIN it is taking effect and indeed resulting in
different (lower) I/O request frequency to underlying I/O interface?

  This API routine is only used to "push" allocation of the space for storing small dataset raw data together in the file. It has no effect on I/O requests to the underlying system. H5Pset_sieve_buf_size may help you out though...

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I tried sieve buf setting. It has no effect either. But, thats kinda
what I expected as docs indicate that really has to do with chunking for
chunked datasets. Mine are contiguous.

Mark

···

On Wed, 2010-03-03 at 13:04, Quincey Koziol wrote:

Hi Mark,

On Mar 3, 2010, at 12:51 PM, Mark Miller wrote:

> Hi All,
>
> I am seeing some behavior in some tests that I am doing that suggests
> that on the sec2 vfd (or any vfd I've tried) the small data block size
> property (H5Pset_small_data_block_size) is having no effect in changing
> I/O requests to underlying system I/O interface.
>
> Does anyone have experience using this file access property such that
> they KNOW FOR CERTAIN it is taking effect and indeed resulting in
> different (lower) I/O request frequency to underlying I/O interface?

  This API routine is only used to "push" allocation of the space for storing small dataset raw data together in the file. It has no effect on I/O requests to the underlying system. H5Pset_sieve_buf_size may help you out though...

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851