Hi All
I am trying to save image data to the dataset, the image data is
allocated by other library, so the image's scanline length is not equal to
the number
of pixels, and will have the padding bytes. I want save it into hdf5 as a
matrix, which don't have those padding bytes, as the image can be float,
unsigned short, and come from other data, the scanline length(/ step )
between each row may not be the multiples of the pixels. for example, the
image has 9 unsigned short pixels, and the sacn line length is defined by
allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the interger
times the elements length. so it can not work. And I don't want use a
temporary buffer to change the memory layout. I also want all other hdf5
application can directly read the image out as matrix, so I don't want use
raw data format. Is there a way which can directly use hdf5's api and it
will automatically strip the padding bytes for image data.
Thanks in advance.
Hi All
I am trying to save image data to the dataset, the image data is allocated by other library, so the image's scanline length is not equal to the number
of pixels, and will have the padding bytes. I want save it into hdf5 as a matrix, which don't have those padding bytes, as the image can be float, unsigned short, and come from other data, the scanline length(/ step ) between each row may not be the multiples of the pixels. for example, the
image has 9 unsigned short pixels, and the sacn line length is defined by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the interger times the elements length. so it can not work. And I don't want use a temporary buffer to change the memory layout. I also want all other hdf5 application can directly read the image out as matrix, so I don't want use raw data format. Is there a way which can directly use hdf5's api and it will automatically strip the padding bytes for image data.
Hi
The problem is the hyperslab's strid is meaused by the number of pixels,
not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not 4bits.
He Yang
···
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org>wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is
allocated by other library, so the image's scanline length is not equal to
the number
> of pixels, and will have the padding bytes. I want save it into hdf5 as
a matrix, which don't have those padding bytes, as the image can be
float, unsigned short, and come from other data, the scanline length(/ step
) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined
by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the
interger times the elements length. so it can not work. And I don't want
use a temporary buffer to change the memory layout. I also want all other
hdf5 application can directly read the image out as matrix, so I don't want
use raw data format. Is there a way which can directly use hdf5's api and
it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to achieve
what you want.
Hi He,
You can build up this sort of selection with the H5Sselect_hyperslab() routine by using the H5S_SELECT_OR operator (or another convenient operator).
Hi
The problem is the hyperslab's strid is meaused by the number of pixels, not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not 4bits.
He Yang
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is allocated by other library, so the image's scanline length is not equal to the number
> of pixels, and will have the padding bytes. I want save it into hdf5 as a matrix, which don't have those padding bytes, as the image can be float, unsigned short, and come from other data, the scanline length(/ step ) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the interger times the elements length. so it can not work. And I don't want use a temporary buffer to change the memory layout. I also want all other hdf5 application can directly read the image out as matrix, so I don't want use raw data format. Is there a way which can directly use hdf5's api and it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to achieve what you want.
Hi Quincey
I have tried this, the problem is the padding part is only partial of
the element size, so the stride can only be set to be interger, which means
if i have scan line like 00000000, 11111111, 00 here the element size
is 8bit, but the padding bits is 2bit, if i select hyperslab, the stride
can be only the interger, which means the padding bits at least 8bit. So
it will cause problem.
Thanks
He Yang
···
On Tue, Nov 13, 2012 at 4:45 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
Hi He,
You can build up this sort of selection with the H5Sselect_hyperslab()
routine by using the H5S_SELECT_OR operator (or another convenient
operator).
Hi
The problem is the hyperslab's strid is meaused by the number of
pixels, not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not 4bits.
He Yang
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org>wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is
allocated by other library, so the image's scanline length is not equal to
the number
> of pixels, and will have the padding bytes. I want save it into hdf5
as a matrix, which don't have those padding bytes, as the image can be
float, unsigned short, and come from other data, the scanline length(/ step
) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined
by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the
interger times the elements length. so it can not work. And I don't want
use a temporary buffer to change the memory layout. I also want all other
hdf5 application can directly read the image out as matrix, so I don't want
use raw data format. Is there a way which can directly use hdf5's api and
it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to
achieve what you want.
Hi Quincey
I have tried this, the problem is the padding part is only partial of the element size, so the stride can only be set to be interger, which means
if i have scan line like 00000000, 11111111, 00 here the element size is 8bit, but the padding bits is 2bit, if i select hyperslab, the stride can be only the interger, which means the padding bits at least 8bit. So it will cause problem.
*blah* What an awkward data layout I suppose you could create the packed, matrix-like dataset in the file and then write small pieces of your buffer in memory to regions of the dataset in the file, with multiple I/O calls. Not the best performance possibly, but if you use chunked dataset stored for the dataset in the file, the chunk caching in the HDF5 library should mitigate that performance issue significantly.
On Tue, Nov 13, 2012 at 4:45 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
Hi He,
You can build up this sort of selection with the H5Sselect_hyperslab() routine by using the H5S_SELECT_OR operator (or another convenient operator).
Hi
The problem is the hyperslab's strid is meaused by the number of pixels, not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not 4bits.
He Yang
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is allocated by other library, so the image's scanline length is not equal to the number
> of pixels, and will have the padding bytes. I want save it into hdf5 as a matrix, which don't have those padding bytes, as the image can be float, unsigned short, and come from other data, the scanline length(/ step ) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the interger times the elements length. so it can not work. And I don't want use a temporary buffer to change the memory layout. I also want all other hdf5 application can directly read the image out as matrix, so I don't want use raw data format. Is there a way which can directly use hdf5's api and it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to achieve what you want.
Hi Quincey
The problem is the image is not allocated by me, it is from another
library, for some hardware reason, the image data always add some padding
data, and so it can be processed faster, for example, the opengl has 1 or
,2, 4 bytes alignment, and the intel ipp library has its own malloc
fuction, in those image class/structure, there always is a step member
which is the length in bytes to show the scanline length. For most case, we
can assume it will align perfectly, but there may have some special case,
the padding bits is not integer multiply element size. so I want to know if
hdf5 api directly support the image scanline length, as most image
processing library support it. But It seems I have to use a buffer to strip
out it myself.
HDF5 have been used a lot in remote sensing and satellite image
processing application. the tiff format should have this kind of problem.
Hope in the future it will support those steps.
Thanks for answer my problem.
He Yang
···
On Tue, Nov 13, 2012 at 5:26 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
Hi Quincey
I have tried this, the problem is the padding part is only partial of
the element size, so the stride can only be set to be interger, which means
if i have scan line like 00000000, 11111111, 00 here the element size
is 8bit, but the padding bits is 2bit, if i select hyperslab, the stride
can be only the interger, which means the padding bits at least 8bit. So
it will cause problem.
*blah* What an awkward data layout I suppose you could create the
packed, matrix-like dataset in the file and then write small pieces of your
buffer in memory to regions of the dataset in the file, with multiple I/O
calls. Not the best performance possibly, but if you use chunked dataset
stored for the dataset in the file, the chunk caching in the HDF5 library
should mitigate that performance issue significantly.
Hi He,
You can build up this sort of selection with the H5Sselect_hyperslab()
routine by using the H5S_SELECT_OR operator (or another convenient
operator).
Hi
The problem is the hyperslab's strid is meaused by the number of
pixels, not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not
4bits.
He Yang
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org>wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is
allocated by other library, so the image's scanline length is not equal to
the number
> of pixels, and will have the padding bytes. I want save it into hdf5
as a matrix, which don't have those padding bytes, as the image can be
float, unsigned short, and come from other data, the scanline length(/ step
) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined
by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the
interger times the elements length. so it can not work. And I don't want
use a temporary buffer to change the memory layout. I also want all other
hdf5 application can directly read the image out as matrix, so I don't want
use raw data format. Is there a way which can directly use hdf5's api and
it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to
achieve what you want.
Hi Quincey
The problem is the image is not allocated by me, it is from another library, for some hardware reason, the image data always add some padding data, and so it can be processed faster, for example, the opengl has 1 or ,2, 4 bytes alignment, and the intel ipp library has its own malloc fuction, in those image class/structure, there always is a step member which is the length in bytes to show the scanline length. For most case, we can assume it will align perfectly, but there may have some special case, the padding bits is not integer multiply element size. so I want to know if hdf5 api directly support the image scanline length, as most image processing library support it. But It seems I have to use a buffer to strip out it myself.
HDF5 have been used a lot in remote sensing and satellite image processing application. the tiff format should have this kind of problem. Hope in the future it will support those steps.
Thanks for answer my problem.
Hmm, I think you may have misunderstood my previous response - you can make multiple H5Dwrite calls to perform this I/O, by setting the memory buffer pointer to H5Dwrite to different locations within the awkwardly structured buffer you receive from the other library, and selecting a small number of elements to write each time. It'll just take a larger number of H5Dwrite calls than otherwise...
We'll definitely keep the use case in mind, in the future.
Hi Quincey
I have tried this, the problem is the padding part is only partial of the element size, so the stride can only be set to be interger, which means
if i have scan line like 00000000, 11111111, 00 here the element size is 8bit, but the padding bits is 2bit, if i select hyperslab, the stride can be only the interger, which means the padding bits at least 8bit. So it will cause problem.
*blah* What an awkward data layout I suppose you could create the packed, matrix-like dataset in the file and then write small pieces of your buffer in memory to regions of the dataset in the file, with multiple I/O calls. Not the best performance possibly, but if you use chunked dataset stored for the dataset in the file, the chunk caching in the HDF5 library should mitigate that performance issue significantly.
Quincey
Thanks
He Yang
On Tue, Nov 13, 2012 at 4:45 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
Hi He,
You can build up this sort of selection with the H5Sselect_hyperslab() routine by using the H5S_SELECT_OR operator (or another convenient operator).
Hi
The problem is the hyperslab's strid is meaused by the number of pixels, not bytes, so I i have a 8 16bit pixels, and a 4 padding bits,
the stride will not work. as if the stride is 1 it will be 16bit not 4bits.
He Yang
On Tue, Nov 13, 2012 at 12:24 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
> Hi All
> I am trying to save image data to the dataset, the image data is allocated by other library, so the image's scanline length is not equal to the number
> of pixels, and will have the padding bytes. I want save it into hdf5 as a matrix, which don't have those padding bytes, as the image can be float, unsigned short, and come from other data, the scanline length(/ step ) between each row may not be the multiples of the pixels. for example, the
> image has 9 unsigned short pixels, and the sacn line length is defined by allocate 9*(16/2)+4=76 bytes, the stride in hdf5 can only be the interger times the elements length. so it can not work. And I don't want use a temporary buffer to change the memory layout. I also want all other hdf5 application can directly read the image out as matrix, so I don't want use raw data format. Is there a way which can directly use hdf5's api and it will automatically strip the padding bytes for image data.
You should be able to use hyperslab or point selections to achieve what you want.
I am trying to compile and link a fortran program under Visual Studio 2010
with Intel Fortran Compiler 13. The program compiles without errors,
whereas, during linking process it gives two unresolved symbol errors:
Error 1 error LNK2019: unresolved external symbol
_H5LIB_PROVISIONAL_mp_H5OFFSETOF referenced in function _H5WRITE h5write.obj
Error 2 error LNK2019: unresolved external symbol
_H5D_PROVISIONAL_mp_H5DWRITE_PTR referenced in function _H5WRITE h5write.obj
My Linker command line configuration is as follows:
I already set the preprocessor definitions
"HDF5F90_WINDOWS;_HDF5USEHLDLL_". THe HDF5 version I use is 1.8.10, which
was compiled from source with Visual Studio 2010 and Intel Fortran Compiler
13. I used Cmake and enabled Fortran2003. I tried many different options to
no avail. Could you make any suggestions, please?
I suppose you could create the packed, matrix-like dataset in the file and then write small pieces of your buffer in memory to regions of the dataset in the file, with multiple I/O calls. Not the best performance possibly, but if you use chunked dataset stored for the dataset in the file, the chunk caching in the HDF5 library should mitigate that performance issue significantly.