Does Parellel HDF5 support derived data write?

Hi Quincey,

     I would like to do some additional work if can achieve this feature in my program. But I don't know how to skip the GAP between two buffer when select.
Because select(generally hyperslab) through setting start, block, stride, count parameters to describe the selection range. when the gap between the buffers is not just the multiple of the unit, how to select?
    For example: fbuf1[2] = {1.0, 2.0}, fbuf2[3] = {3.0,4.0}
    Assume sizeof(float) = 4, and the address of fbuf1 is 1020, the adress of fbuf2 is 1010, the gap between this two buffer is 1012 - 1010 = 2, which is
not sizeof(float) multiple. How should I do select elements using start/count/stride/block parameters?
    May be you have other methods? (If I define the datatype to be char, which only occupy 1 bytes, it may ok. But I only want to see the float data type in my datasets)

Thanks.
Tony

···

------------------------------------------------------------------------------

From: "Quincey Koziol"
To: hdf-forum@hdfgroup.org
Date: 2009-09-03 02:10:24
Subject:
Hi Tony,

On Sep 2, 2009, at 10:13 AM, tony wrote:

Hi Quincey,

    Thanks for your answer.
    I understand what you said. But it seemed to be not a good ideas.
    I was reading the HDF5 user guide. I feel that use H5Tcreate, H5Tinsert to create a new compound datatype may solve this problems. May be I was wrong and I still don't know how to use HOFFSET in the situation.

HOFFSET is designed to describe the offset of fields in structs, which won't work correctly in this situation. (Unless you are planning on doing something I don't expect! :slight_smile:

    Or you have any other suggestions about this questions. I do want to write mass of data to a dataset in parellel HDF5 and I do not want to allocate a continous large memory to store the data.

The memory doesn't have t _actually_ be contiguously allocated, your application can just describe it to HDF5 as if it were. As long as you don't set up your selection to read from (or write to!) the "gaps" between your buffers, there shouldn't be any problem.

Quincey

Tony
Thanks.
    
------------------------------------------------------------------------------

From: "Quincey Koziol"
To: hdf-forum@hdfgroup.org
Date: 2009-09-02 22:12:19
Subject:
Hi Tony,

On Sep 2, 2009, at 8:01 AM, tony wrote:

Hi all,

    I want to write the data to a datasets from different memory

address in parellel HDF5,

    For example:

        buf1[4] = {1,2,3,4}, buf2[3] = {5, 6, 7}, buf3[2]=[8,9}.

   I want to write these datas(memory address not continous) to a

dataset like this:

       {1,2,3,4,5,6,7,8,9}

   I want to write these data by only one H5Dwrite calls, not

through 3 H5Dwrite calls.

  If you are careful, you could get this to work, but you'll have to

determine which buffer has the lowest address in memory which buffer

has the highest address in memory and define a dataspace that spans

that whole range. Then select the elements within that "spanning

buffer" that you'd like to perform I/O to a single dataset on.

  Quincey

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@hdfgroup.org

http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Tony,

Hi Quincey,

     I would like to do some additional work if can achieve this feature in my program. But I don't know how to skip the GAP between two buffer when select.
Because select(generally hyperslab) through setting start, block, stride, count parameters to describe the selection range. when the gap between the buffers is not just the multiple of the unit, how to select?
    For example: fbuf1[2] = {1.0, 2.0}, fbuf2[3] = {3.0,4.0}
    Assume sizeof(float) = 4, and the address of fbuf1 is 1020, the adress of fbuf2 is 1010, the gap between this two buffer is 1012 - 1010 = 2, which is
not sizeof(float) multiple. How should I do select elements using start/count/stride/block parameters?

  Ah, yes, this could occur, in which case my "hack" won't work. :-/ If that's the case, you'll have to make multiple calls to H5Dread/H5Dwrite.

  Quincey

···

On Sep 2, 2009, at 8:54 PM, tony wrote:

    May be you have other methods? (If I define the datatype to be char, which only occupy 1 bytes, it may ok. But I only want to see the float data type in my datasets)

Thanks.
Tony

------------------------------------------------------------------------------

From: "Quincey Koziol"
To: hdf-forum@hdfgroup.org
Date: 2009-09-03 02:10:24
Subject:

Hi Tony,

On Sep 2, 2009, at 10:13 AM, tony wrote:

Hi Quincey,

    Thanks for your answer.
    I understand what you said. But it seemed to be not a good ideas.
    I was reading the HDF5 user guide. I feel that use H5Tcreate, H5Tinsert to create a new compound datatype may solve this problems. May be I was wrong and I still don't know how to use HOFFSET in the situation.

HOFFSET is designed to describe the offset of fields in structs, which won't work correctly in this situation. (Unless you are planning on doing something I don't expect! :slight_smile:

    Or you have any other suggestions about this questions. I do want to write mass of data to a dataset in parellel HDF5 and I do not want to allocate a continous large memory to store the data.

The memory doesn't have t _actually_ be contiguously allocated, your application can just describe it to HDF5 as if it were. As long as you don't set up your selection to read from (or write to!) the "gaps" between your buffers, there shouldn't be any problem.

Quincey

Tony
Thanks.

------------------------------------------------------------------------------

From: "Quincey Koziol"
To: hdf-forum@hdfgroup.org
Date: 2009-09-02 22:12:19
Subject:

Hi Tony, On Sep 2, 2009, at 8:01 AM, tony wrote: > Hi all, > > I want to write the data to a datasets from different memory > address in parellel HDF5, > For example: > buf1[4] = {1,2,3,4}, buf2[3] = {5, 6, 7}, buf3[2]=[8,9}. > I want to write these datas(memory address not continous) to a > dataset like this: > {1,2,3,4,5,6,7,8,9} > I want to write these data by only one H5Dwrite calls, not > through 3 H5Dwrite calls. If you are careful, you could get this to work, but you'll have to determine which buffer has the lowest address in memory which buffer has the highest address in memory and define a dataspace that spans that whole range. Then select the elements within that "spanning buffer" that you'd like to perform I/O to a single dataset on. Quincey _______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@hdfgroup.org http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org