Add data to a file image in memory

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and
transfer them then through TCP/IP, so I read them into memory as
`std::string`, and then read them as HDF5 using:

fileHandle=H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

*My question is:* Since I'm opening this as RW, how can I write an
attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function
that can write an attribute to an object by its ID. Here it's:

voidWriteStringAttribute(hid_tdataset_id,conststd::string&name,conststd::string&value)
{
hid_tAttributeType=H5Tcopy(H5T_C_S1);
H5Tset_size(AttributeType,value.length());
hid_tdataspaceHandle=H5Screate(H5S_SCALAR);
hid_tAttrHandle=H5Acreate(dataset_id,name.c_str(),AttributeType,dataspaceHandle,H5P_DEFAULT,H5P_DEFAULT);
H5Awrite(AttrHandle,AttributeType,&value.front());

H5Aclose(AttrHandle);
H5Sclose(dataspaceHandle);
}

What I don't understand is that after writing the attribute using this
function (Is it right to do this?

WriteStringAttribute(fileHandle,"MyAttr","MyAttrValue");

), the image size should've been changed. How can I get the new correct
image size and write it back to a file (or do anything else with it, for
that matter)?

Thank you.

Best,
Samer Afach

Hello Samer,

I think I get the gist of what you're asking. If you are writing to the
object and then want to read from the updated file, you will need to call
H5Fflush() in order to write your data buffers to the file on your disk.
Modifying objects on your file does not actually immediately change the
file itself. The changes are only made when the object and/or file is
closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

···

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and
transfer them then through TCP/IP, so I read them into memory as
`std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.
size(),H5LT_FILE_IMAGE_OPEN_RW);
*My question is:* Since I'm opening this as RW, how can I write an
attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function that
can write an attribute to an object by its ID. Here it's:

void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{ hid_t AttributeType = H5Tcopy(H5T_C_S1); H5Tset_size(AttributeType, value.length()); hid_t dataspaceHandle = H5Screate(H5S_SCALAR); hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT); H5Awrite(AttrHandle, AttributeType, &value.front());

    H5Aclose (AttrHandle); H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this
function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct
image size and write it back to a file (or do anything else with it, for
that matter)?

Thank you.

Best,
Samer Afach

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects,
but /file images/, which is basically the whole file stored in some
buffer in memory. Imagine you use std::fstream::read() or the classic C
FILE's fread() to read the whole file into memory, and then wanting to
open it with the HDF5 library. This is why I'm using
"H5LTopen_file_image()", which doesn't take a file name or file path,
but takes a void* buffer, which is a pointer to the array where the file
is stored /in memory/.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5
file object (or file handle), the one you're talking about. My question
is: Now after we do the changes to the file under that object, and after
we H5Fflush(), that buffer may be changed. My question is: How do we
keep track of that buffer (and its size) to write it back to a file
eventually (using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach

···

On 27.12.2016 12:19 AM, Landon Clipp wrote:

Hello Samer,

I think I get the gist of what you're asking. If you are writing to
the object and then want to read from the updated file, you will need
to call H5Fflush() in order to write your data buffers to the file on
your disk. Modifying objects on your file does not actually
immediately change the file itself. The changes are only made when the
object and/or file is closed or flushed. I believe that should solve
your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de > <mailto:samer@afach.de>> wrote:

    Hello guys:

    In a program, I use some relatively small HDF5 files multiple
    times and transfer them then through TCP/IP, so I read them into
    memory as `std::string`, and then read them as HDF5 using:

    fileHandle=H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

    *My question is:* Since I'm opening this as RW, how can I write an
    attribute to it and retrieve the image back?

    The motivation behind my question is simple. I have a simple
    function that can write an attribute to an object by its ID. Here
    it's:

    voidWriteStringAttribute(hid_tdataset_id,conststd::string&name,conststd::string&value)
    {
    hid_tAttributeType=H5Tcopy(H5T_C_S1);
    H5Tset_size(AttributeType,value.length());
    hid_tdataspaceHandle=H5Screate(H5S_SCALAR);
    hid_tAttrHandle=H5Acreate(dataset_id,name.c_str(),AttributeType,dataspaceHandle,H5P_DEFAULT,H5P_DEFAULT);
    H5Awrite(AttrHandle,AttributeType,&value.front());

    H5Aclose(AttrHandle);
    H5Sclose(dataspaceHandle);
    }

    What I don't understand is that after writing the attribute using
    this function (Is it right to do this?

    WriteStringAttribute(fileHandle,"MyAttr","MyAttrValue");

    ), the image size should've been changed. How can I get the new
    correct image size and write it back to a file (or do anything
    else with it, for that matter)?

    Thank you.

    Best, Samer Afach

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library
makes a copy of the original buffer and operates on that. So once you've
created the attribute you wanted and done a flush, you can get the file
image back from two calls to H5LTopen_file_image as usual (first to get the
size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely need
to look into H5Pset_file_image_callbacks to handle when the file image is
resized and hence when the buffer changes.

Cheers,
Martijn

···

On 27 December 2016 at 10:47, Samer Afach <samer@afach.de> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but *file
images*, which is basically the whole file stored in some buffer in
memory. Imagine you use std::fstream::read() or the classic C FILE's
fread() to read the whole file into memory, and then wanting to open it
with the HDF5 library. This is why I'm using "H5LTopen_file_image()",
which doesn't take a file name or file path, but takes a void* buffer,
which is a pointer to the array where the file is stored *in memory*.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5
file object (or file handle), the one you're talking about. My question is:
Now after we do the changes to the file under that object, and after we
H5Fflush(), that buffer may be changed. My question is: How do we keep
track of that buffer (and its size) to write it back to a file eventually
(using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach

On 27.12.2016 12:19 AM, Landon Clipp wrote:

Hello Samer,

I think I get the gist of what you're asking. If you are writing to the
object and then want to read from the updated file, you will need to call
H5Fflush() in order to write your data buffers to the file on your disk.
Modifying objects on your file does not actually immediately change the
file itself. The changes are only made when the object and/or file is
closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and
transfer them then through TCP/IP, so I read them into memory as
`std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&FileImage.front(),FileImage.size
(),H5LT_FILE_IMAGE_OPEN_RW);
*My question is:* Since I'm opening this as RW, how can I write an
attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function
that can write an attribute to an object by its ID. Here it's:

void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{ hid_t AttributeType = H5Tcopy(H5T_C_S1); H5Tset_size(AttributeType, value.length()); hid_t dataspaceHandle = H5Screate(H5S_SCALAR); hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT); H5Awrite(AttrHandle, AttributeType, &value.front());

    H5Aclose (AttrHandle); H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this
function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct
image size and write it back to a file (or do anything else with it, for
that matter)?

Thank you.

Best, Samer Afach
_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Martijn:

Thank you very much for your response. Since you split your answer into
two parts, I also have to do the same and ask two questions about these
parts, if you don't mind.

1. You said "you can get the file image back from two calls to
H5LTopen_file_image" But I don't get what two calls you meant. Are you
talking about the calls of "H5Pset_file_image_callbacks"?

2. I would love to do this as "in place", but there's a memory leak I
reported in the HDF5 library that happens when opening images as
in-place (ref. HDFFV-10019), and I'm waiting for feedback about that.
But ignoring that memory leak, are you saying this is different from the
first case (when it's not in-place)? And are you saying that to get the
size of the image, the only way is to re-implement *all* the callback
functions of H5Pset_file_image_callbacks?

Honestly that's a little scary that I'd love to see an example about this.

Thank you for all this information! You helped a lot :slight_smile:

Best,
Samer

···

On 01.01.2017 11:16 PM, Martijn Jasperse wrote:

Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the
library makes a copy of the original buffer and operates on that. So
once you've created the attribute you wanted and done a flush, you can
get the file image back from two calls to H5LTopen_file_image as usual
(first to get the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely
need to look into |H5Pset_file_image_callbacks|* *to handle when the
file image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <samer@afach.de > <mailto:samer@afach.de>> wrote:

    Hi Landon:

    Thanks for the response. Actually I'm not talking about file
    objects, but /file images/, which is basically the whole file
    stored in some buffer in memory. Imagine you use
    std::fstream::read() or the classic C FILE's fread() to read the
    whole file into memory, and then wanting to open it with the HDF5
    library. This is why I'm using "H5LTopen_file_image()", which
    doesn't take a file name or file path, but takes a void* buffer,
    which is a pointer to the array where the file is stored /in memory/.

    Now after opening that using "H5LTopen_file_image()", we'll get an
    HDF5 file object (or file handle), the one you're talking about.
    My question is: Now after we do the changes to the file under that
    object, and after we H5Fflush(), that buffer may be changed. My
    question is: How do we keep track of that buffer (and its size) to
    write it back to a file eventually (using a std::fstream::write()
    or the classic C FILE's fwrite())?

    I would appreciate the help of anyone who knows about this.

    Best,
    Samer Afach

    On 27.12.2016 12:19 AM, Landon Clipp wrote:

    Hello Samer,

    I think I get the gist of what you're asking. If you are writing
    to the object and then want to read from the updated file, you
    will need to call H5Fflush() in order to write your data buffers
    to the file on your disk. Modifying objects on your file does not
    actually immediately change the file itself. The changes are only
    made when the object and/or file is closed or flushed. I believe
    that should solve your problem.

    Regards,
    Landon Clipp

    On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de >> <mailto:samer@afach.de>> wrote:

        Hello guys:

        In a program, I use some relatively small HDF5 files multiple
        times and transfer them then through TCP/IP, so I read them
        into memory as `std::string`, and then read them as HDF5 using:

        fileHandle=H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

        *My question is:* Since I'm opening this as RW, how can I
        write an attribute to it and retrieve the image back?

        The motivation behind my question is simple. I have a simple
        function that can write an attribute to an object by its ID.
        Here it's:

        voidWriteStringAttribute(hid_tdataset_id,conststd::string&name,conststd::string&value)
        {
        hid_tAttributeType=H5Tcopy(H5T_C_S1);
        H5Tset_size(AttributeType,value.length());
        hid_tdataspaceHandle=H5Screate(H5S_SCALAR);
        hid_tAttrHandle=H5Acreate(dataset_id,name.c_str(),AttributeType,dataspaceHandle,H5P_DEFAULT,H5P_DEFAULT);
        H5Awrite(AttrHandle,AttributeType,&value.front());

        H5Aclose(AttrHandle);
        H5Sclose(dataspaceHandle);
        }

        What I don't understand is that after writing the attribute
        using this function (Is it right to do this?

        WriteStringAttribute(fileHandle,"MyAttr","MyAttrValue");

        ), the image size should've been changed. How can I get the
        new correct image size and write it back to a file (or do
        anything else with it, for that matter)?

        Thank you.

        Best, Samer Afach

        _______________________________________________ Hdf-forum is
        for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Samer,
Sorry I had a copy-paste malfunction there. I actually meant to use
H5Fget_file_image
<https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-GetFileImage> twice
instead. First using a NULL pointer to retrieve the "file size", then
allocating some memory and calling it again to get the actual data.

I think the purpose of setting your own callbacks is that you can handle
the *realloc *call so you know when the pointer gets changed. Because even
if you use the "in-place" flag, adding attributes will likely cause the
image to be expanded and hence require a new buffer. But having your own
realloc lets you catch that. According to the manual, callback functions
are allowed to be NULL to indicate the defaults should be used.

I did some work with file images a while back, but admittedly we didn't
bother with in-place for our application as the copy overhead was
relatively small. But maybe I can modify some old code to make an example.

Cheers,
Martijn

···

On 2 January 2017 at 13:28, Samer Afach <samer@afach.de> wrote:

Hi Martijn:

Thank you very much for your response. Since you split your answer into
two parts, I also have to do the same and ask two questions about these
parts, if you don't mind.

1. You said "you can get the file image back from two calls to
H5LTopen_file_image" But I don't get what two calls you meant. Are you
talking about the calls of "H5Pset_file_image_callbacks"?

2. I would love to do this as "in place", but there's a memory leak I
reported in the HDF5 library that happens when opening images as in-place
(ref. HDFFV-10019), and I'm waiting for feedback about that. But ignoring
that memory leak, are you saying this is different from the first case
(when it's not in-place)? And are you saying that to get the size of the
image, the only way is to re-implement *all* the callback functions of
H5Pset_file_image_callbacks?

Honestly that's a little scary that I'd love to see an example about this.

Thank you for all this information! You helped a lot :slight_smile:

Best,
Samer

On 01.01.2017 11:16 PM, Martijn Jasperse wrote:

Hi Samer,
Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the library
makes a copy of the original buffer and operates on that. So once you've
created the attribute you wanted and done a flush, you can get the file
image back from two calls to H5LTopen_file_image as usual (first to get
the size and then to get the actual data).

If you wanted to do it "in place" without making a copies you'd likely
need to look into H5Pset_file_image_callbacks to handle when the file
image is resized and hence when the buffer changes.

Cheers,
Martijn

On 27 December 2016 at 10:47, Samer Afach <samer@afach.de> wrote:

Hi Landon:

Thanks for the response. Actually I'm not talking about file objects, but *file
images*, which is basically the whole file stored in some buffer in
memory. Imagine you use std::fstream::read() or the classic C FILE's
fread() to read the whole file into memory, and then wanting to open it
with the HDF5 library. This is why I'm using "H5LTopen_file_image()",
which doesn't take a file name or file path, but takes a void* buffer,
which is a pointer to the array where the file is stored *in memory*.

Now after opening that using "H5LTopen_file_image()", we'll get an HDF5
file object (or file handle), the one you're talking about. My question is:
Now after we do the changes to the file under that object, and after we
H5Fflush(), that buffer may be changed. My question is: How do we keep
track of that buffer (and its size) to write it back to a file eventually
(using a std::fstream::write() or the classic C FILE's fwrite())?

I would appreciate the help of anyone who knows about this.

Best,
Samer Afach

On 27.12.2016 12:19 AM, Landon Clipp wrote:

Hello Samer,

I think I get the gist of what you're asking. If you are writing to the
object and then want to read from the updated file, you will need to call
H5Fflush() in order to write your data buffers to the file on your disk.
Modifying objects on your file does not actually immediately change the
file itself. The changes are only made when the object and/or file is
closed or flushed. I believe that should solve your problem.

Regards,
Landon Clipp

On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de> wrote:

Hello guys:

In a program, I use some relatively small HDF5 files multiple times and
transfer them then through TCP/IP, so I read them into memory as
`std::string`, and then read them as HDF5 using:

fileHandle = H5LTopen_file_image((void*)&Fi
leImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);
*My question is:* Since I'm opening this as RW, how can I write an
attribute to it and retrieve the image back?

The motivation behind my question is simple. I have a simple function
that can write an attribute to an object by its ID. Here it's:

void WriteStringAttribute(hid_t dataset_id, const std::string& name, const std::string& value)
{ hid_t AttributeType = H5Tcopy(H5T_C_S1); H5Tset_size(AttributeType, value.length()); hid_t dataspaceHandle = H5Screate(H5S_SCALAR); hid_t AttrHandle = H5Acreate(dataset_id, name.c_str(), AttributeType, dataspaceHandle, H5P_DEFAULT,H5P_DEFAULT); H5Awrite(AttrHandle, AttributeType, &value.front());

    H5Aclose (AttrHandle); H5Sclose (dataspaceHandle);
}

What I don't understand is that after writing the attribute using this
function (Is it right to do this?

WriteStringAttribute(fileHandle, "MyAttr", "MyAttrValue");

), the image size should've been changed. How can I get the new correct
image size and write it back to a file (or do anything else with it, for
that matter)?

Thank you.

Best, Samer Afach
_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Thanks for explaining! This was very helpful :slight_smile:

Best,
Samer

···

On 02.01.2017 4:02 AM, Martijn Jasperse wrote:

Hi Samer,
Sorry I had a copy-paste malfunction there. I actually meant to
use H5Fget_file_image
<https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-GetFileImage> twice
instead. First using a NULL pointer to retrieve the "file size", then
allocating some memory and calling it again to get the actual data.

I think the purpose of setting your own callbacks is that you can
handle the /realloc /call so you know when the pointer gets changed.
Because even if you use the "in-place" flag, adding attributes will
likely cause the image to be expanded and hence require a new buffer.
But having your own realloc lets you catch that. According to the
manual, callback functions are allowed to be NULL to indicate the
defaults should be used.

I did some work with file images a while back, but admittedly we
didn't bother with in-place for our application as the copy overhead
was relatively small. But maybe I can modify some old code to make an
example.

Cheers,
Martijn

On 2 January 2017 at 13:28, Samer Afach <samer@afach.de > <mailto:samer@afach.de>> wrote:

    Hi Martijn:

    Thank you very much for your response. Since you split your answer
    into two parts, I also have to do the same and ask two questions
    about these parts, if you don't mind.

    1. You said "you can get the file image back from two calls to
    H5LTopen_file_image" But I don't get what two calls you meant. Are
    you talking about the calls of "H5Pset_file_image_callbacks"?

    2. I would love to do this as "in place", but there's a memory
    leak I reported in the HDF5 library that happens when opening
    images as in-place (ref. HDFFV-10019), and I'm waiting for
    feedback about that. But ignoring that memory leak, are you saying
    this is different from the first case (when it's not in-place)?
    And are you saying that to get the size of the image, the only way
    is to re-implement *all* the callback functions of
    H5Pset_file_image_callbacks?

    Honestly that's a little scary that I'd love to see an example
    about this.

    Thank you for all this information! You helped a lot :slight_smile:

    Best,
    Samer

    On 01.01.2017 11:16 PM, Martijn Jasperse wrote:

    Hi Samer,
    Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the
    library makes a copy of the original buffer and operates on that.
    So once you've created the attribute you wanted and done a flush,
    you can get the file image back from two calls
    to H5LTopen_file_image as usual (first to get the size and then
    to get the actual data).

    If you wanted to do it "in place" without making a copies you'd
    likely need to look into |H5Pset_file_image_callbacks|* *to
    handle when the file image is resized and hence when the buffer
    changes.

    Cheers,
    Martijn

    On 27 December 2016 at 10:47, Samer Afach <samer@afach.de >> <mailto:samer@afach.de>> wrote:

        Hi Landon:

        Thanks for the response. Actually I'm not talking about file
        objects, but /file images/, which is basically the whole file
        stored in some buffer in memory. Imagine you use
        std::fstream::read() or the classic C FILE's fread() to read
        the whole file into memory, and then wanting to open it with
        the HDF5 library. This is why I'm using
        "H5LTopen_file_image()", which doesn't take a file name or
        file path, but takes a void* buffer, which is a pointer to
        the array where the file is stored /in memory/.

        Now after opening that using "H5LTopen_file_image()", we'll
        get an HDF5 file object (or file handle), the one you're
        talking about. My question is: Now after we do the changes to
        the file under that object, and after we H5Fflush(), that
        buffer may be changed. My question is: How do we keep track
        of that buffer (and its size) to write it back to a file
        eventually (using a std::fstream::write() or the classic C
        FILE's fwrite())?

        I would appreciate the help of anyone who knows about this.

        Best,
        Samer Afach

        On 27.12.2016 12:19 AM, Landon Clipp wrote:

        Hello Samer,

        I think I get the gist of what you're asking. If you are
        writing to the object and then want to read from the updated
        file, you will need to call H5Fflush() in order to write
        your data buffers to the file on your disk. Modifying
        objects on your file does not actually immediately change
        the file itself. The changes are only made when the object
        and/or file is closed or flushed. I believe that should
        solve your problem.

        Regards,
        Landon Clipp

        On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <samer@afach.de >>> <mailto:samer@afach.de>> wrote:

            Hello guys:

            In a program, I use some relatively small HDF5 files
            multiple times and transfer them then through TCP/IP, so
            I read them into memory as `std::string`, and then read
            them as HDF5 using:

            fileHandle=H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW);

            *My question is:* Since I'm opening this as RW, how can
            I write an attribute to it and retrieve the image back?

            The motivation behind my question is simple. I have a
            simple function that can write an attribute to an object
            by its ID. Here it's:

            voidWriteStringAttribute(hid_tdataset_id,conststd::string&name,conststd::string&value)
            {
            hid_tAttributeType=H5Tcopy(H5T_C_S1);
            H5Tset_size(AttributeType,value.length());
            hid_tdataspaceHandle=H5Screate(H5S_SCALAR);
            hid_tAttrHandle=H5Acreate(dataset_id,name.c_str(),AttributeType,dataspaceHandle,H5P_DEFAULT,H5P_DEFAULT);
            H5Awrite(AttrHandle,AttributeType,&value.front());

            H5Aclose(AttrHandle);
            H5Sclose(dataspaceHandle);
            }

            What I don't understand is that after writing the
            attribute using this function (Is it right to do this?

            WriteStringAttribute(fileHandle,"MyAttr","MyAttrValue");

            ), the image size should've been changed. How can I get
            the new correct image size and write it back to a file
            (or do anything else with it, for that matter)?

            Thank you.

            Best, Samer Afach

            _______________________________________________
            Hdf-forum is for HDF software users discussion.
            Hdf-forum@lists.hdfgroup.org
            <mailto:Hdf-forum@lists.hdfgroup.org>
            http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
            <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
            Twitter: https://twitter.com/hdf5

        _______________________________________________
        Hdf-forum is for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

        _______________________________________________ Hdf-forum is
        for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5