Problem with getting hdf5 file size

Hi Quincey,

Thanks for your answer.
I noticed that if I fill a dataset into the file, than the
H5Fget_filesize function returns the correct size.
For the motivation to use the H5Fget_vfd_handle, I can explain:
I try to somehow serialize a HDF5 file in order to manage multiple
access of relatively small sized files using a relational database,
inserting HDF5 file pieces as binaries (BLOB, byta, etc…). This would
be useful for getting a data access taking advantage of concurrent
access on a relational database (for small to medium file sizes), and
keeping the flexibility of HDF5 format to manipulate data. To simplify,
I would need a function of the type H5Fencode and H5Fdecode !
You seem to have some idea using the CORE VFD. I am curious to
understand more about…

···

Valore legale alle tue mail
InterfreePEC - la tua Posta Elettronica Certificata
http://pec.interfree.it

Hi Daniel,

···

On May 4, 2011, at 1:02 PM, dsentenac@interfree.it wrote:

Hi Quincey,

Thanks for your answer.
I noticed that if I fill a dataset into the file, than the
H5Fget_filesize function returns the correct size.
For the motivation to use the H5Fget_vfd_handle, I can explain:
I try to somehow serialize a HDF5 file in order to manage multiple
access of relatively small sized files using a relational database,
inserting HDF5 file pieces as binaries (BLOB, byta, etc..). This would
be useful for getting a data access taking advantage of concurrent
access on a relational database (for small to medium file sizes), and
keeping the flexibility of HDF5 format to manipulate data. To simplify,
I would need a function of the type H5Fencode and H5Fdecode !
You seem to have some idea using the CORE VFD. I am curious to
understand more about....

  Yes, it sounds like you want to use the core VFD. Here's the link to the reference manual:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

  If you use the core VFD, it'll create a file in memory and then you can call H5Fflush to make certain the file's memory image is consistent, then you can store or transmit the data. There's no way to "open" a memory image of an HDF5 file yet, but we're working on it and expect to have it done within a month (to be included in the 1.8.8 release).

  Quincey

I am happy to see that you will produce soon some functions to encode/decode functions at file level.

I tried to implement what you say about the use of H5Fflush but didn't succeed. Here is the piece of code I tried:

  int *fhandle = NULL;
  hsize_t size = 0;

  /* DEFINE FILE ACCESS METHOD CORE*/
   hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
   size_t inc_size = 2048;
   H5Pset_fapl_core(fapl, inc_size, false);

  /* CREATE FILE AND FILL IT WITH FAKE DATA*/
   hid_t file_id = H5Fcreate("data_element.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
   hid_t grp_id = H5Gcreate2(file_id, "DAQ1", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
    float data[1000];
     for (int i = 0 ; i < 1000;i++)
        data[i] = i / 1000;
     status = H5LTset_attribute_int(file_id, "DAQ1", "Time1", &gps,1);
     status = H5LTset_attribute_int(file_id, "DAQ1", "Time2", &gps,1);
     status = H5LTmake_dataset_float ( grp_id, "fdata",1, &dims, data ) ;

     /* ACCESS FILE HANDLE AFTER FLUSHING FILE*/
     H5Fflush(file_id,H5F_SCOPE_GLOBAL );
     H5Fget_vfd_handle(file_id, fapl, (void**)(&fhandle) );
     H5Fget_filesize( file_id, &size );

    /* ALLOCATE BUFFER AND TRY TO FILL IT FROM FILE HANDLE (FAILS)*/
    char *buf;
    buf = (char*)malloc(sizeof(char) * (size));
    lseek((*fhandle),0,SEEK_SET) // (returns -1 No such file or directory)
    read(*fhandle,buf,size); // (returns -1)

Any idea with what's going wrong with the use of functions read, lseek with the file handle ?
Or should I understand fhandle differently ?

Many Thanks,

Daniel

···

On 05/04/2011 09:06 PM, Quincey Koziol wrote:

Hi Daniel,

On May 4, 2011, at 1:02 PM, dsentenac@interfree.it wrote:

Hi Quincey,

Thanks for your answer.
I noticed that if I fill a dataset into the file, than the
H5Fget_filesize function returns the correct size.
For the motivation to use the H5Fget_vfd_handle, I can explain:
I try to somehow serialize a HDF5 file in order to manage multiple
access of relatively small sized files using a relational database,
inserting HDF5 file pieces as binaries (BLOB, byta, etc..). This would
be useful for getting a data access taking advantage of concurrent
access on a relational database (for small to medium file sizes), and
keeping the flexibility of HDF5 format to manipulate data. To simplify,
I would need a function of the type H5Fencode and H5Fdecode !
You seem to have some idea using the CORE VFD. I am curious to
understand more about....

  Yes, it sounds like you want to use the core VFD. Here's the link to the reference manual:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

  If you use the core VFD, it'll create a file in memory and then you can call H5Fflush to make certain the file's memory image is consistent, then you can store or transmit the data. There's no way to "open" a memory image of an HDF5 file yet, but we're working on it and expect to have it done within a month (to be included in the 1.8.8 release).

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Daniel,

···

On May 5, 2011, at 9:13 AM, Sentenac Daniel wrote:

On 05/04/2011 09:06 PM, Quincey Koziol wrote:

Hi Daniel,

On May 4, 2011, at 1:02 PM, dsentenac@interfree.it wrote:

Hi Quincey,

Thanks for your answer.
I noticed that if I fill a dataset into the file, than the
H5Fget_filesize function returns the correct size.
For the motivation to use the H5Fget_vfd_handle, I can explain:
I try to somehow serialize a HDF5 file in order to manage multiple
access of relatively small sized files using a relational database,
inserting HDF5 file pieces as binaries (BLOB, byta, etc..). This would
be useful for getting a data access taking advantage of concurrent
access on a relational database (for small to medium file sizes), and
keeping the flexibility of HDF5 format to manipulate data. To simplify,
I would need a function of the type H5Fencode and H5Fdecode !
You seem to have some idea using the CORE VFD. I am curious to
understand more about....

  Yes, it sounds like you want to use the core VFD. Here's the link to the reference manual:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

  If you use the core VFD, it'll create a file in memory and then you can call H5Fflush to make certain the file's memory image is consistent, then you can store or transmit the data. There's no way to "open" a memory image of an HDF5 file yet, but we're working on it and expect to have it done within a month (to be included in the 1.8.8 release).

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I am happy to see that you will produce soon some functions to encode/decode functions at file level.

I tried to implement what you say about the use of H5Fflush but didn't succeed. Here is the piece of code I tried:

int *fhandle = NULL;
hsize_t size = 0;

/* DEFINE FILE ACCESS METHOD CORE*/
hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
size_t inc_size = 2048;
H5Pset_fapl_core(fapl, inc_size, false);

/* CREATE FILE AND FILL IT WITH FAKE DATA*/
hid_t file_id = H5Fcreate("data_element.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
hid_t grp_id = H5Gcreate2(file_id, "DAQ1", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
  float data[1000];
   for (int i = 0 ; i < 1000;i++)
      data[i] = i / 1000;
   status = H5LTset_attribute_int(file_id, "DAQ1", "Time1", &gps,1);
   status = H5LTset_attribute_int(file_id, "DAQ1", "Time2", &gps,1);
   status = H5LTmake_dataset_float ( grp_id, "fdata",1, &dims, data ) ;

   /* ACCESS FILE HANDLE AFTER FLUSHING FILE*/
   H5Fflush(file_id,H5F_SCOPE_GLOBAL );
   H5Fget_vfd_handle(file_id, fapl, (void**)(&fhandle) );
   H5Fget_filesize( file_id, &size );

  /* ALLOCATE BUFFER AND TRY TO FILL IT FROM FILE HANDLE (FAILS)*/
  char *buf;
  buf = (char*)malloc(sizeof(char) * (size));
  lseek((*fhandle),0,SEEK_SET) // (returns -1 No such file or directory)
  read(*fhandle,buf,size); // (returns -1)

Any idea with what's going wrong with the use of functions read, lseek with the file handle ?
Or should I understand fhandle differently ?

  When you use the core VFD, the "handle" returned with H5Fget_vfd_handle will be a "void *", pointing to the beginning of the in-memory form of the file. (Your code looks like it would work correctly if you used the default (sec2) VFD)

  Quincey