HDF5 Performance

[leon_vernikov]

I’m using HDF5 Version 1.8.12
I created HDF5 group and use the following sequence of calls to create new group’s attribute
H5Acreate();
H5Awrite();
H5Aclose();
Each group’s attribute has 1024 char string with a protocol’s packet.
Let say I need to store 1M protocol packets in HDF5 database.
HDF5 database performance to store the first 1K packets is outstanding.
I see significant performance (drops 7 times) degradation for 10K stored packets.
HDF5 database performance drops 100 times while the number of attributes reaches 100K.

Would you advice the steps to modify HDF5 model to improve the attribute write performance?
Thank you

[gheber]

1. There’s a low-level “fix,” which is to use a newer version of the file format:
   Create a file access property list and use H5Pset_libver_bounds with H5F_LIBVER_LATEST for low and high. You will see better performance, but it won’t do miracles.
2. Can you tell us a little bit about what made you choose HDF5 attributes to represent the data you’re acquiring? There are probably more efficient ways to represent your data (in HDF5), but we could use a little background.
3. Calling HDF5 a “data base” is OK as long as you don’t expect it to behave like many other things that go under the heading “database.”

G.

[leon_vernikov]

Dear Gerd,

I really appreciate you look at HDF5 performance degradation problem.

1. There’s a low-level “fix,” which is to use a newer version of the file format:

Create a file access property list and use H5Pset_libver_bounds with H5F_LIBVER_LATEST for low and high. You will see better performance, but it won’t do miracles.

=LV= I would really appreciate a small code sample to understand your idea.

2. Can you tell us a little bit about what made you choose HDF5 attributes to represent the data you’re acquiring?

=LV=

• HDF5 attribute base-element is simplest and basic property of HDF5 database to implement.

• HDF5 attribute base-element/API allows to store both network packets and information related to network traffic

(type of network traffic, the traffic property related to all network packets).

• HDF5 attribute base-element/API doesn’t require a separate table to be inserted under HDF5 group.

• Most important: HDFView v2.14.0 has memory problem to open HDF5 group’s table with large number of records. Basically, HDFView is dead at opening.

There are probably more efficient ways to represent your data (in HDF5), but we could use a little background.

=LV= Are you referring H5PT base-element?

3. Calling HDF5 a “data base” is OK as long as you don’t expect it to behave like many other things that go under the heading “database.”

=LV= Please refer Wikipedia for database definition

Database

“In computing, a database is an organized collection of data stored and accessed electronically from a computer system.”

Thank you,

Leon

[gheber]

I don’t know which language you are using. This can all be had a lot easier in Python, Julia, R, H5CPP, etc.
Here is the verbose way w/ all boilerplate. The call to H5Pset_libver_bounds and passing fapl to H5F[create,open] is all that counts. Your code should go to the comment // do something useful w/ FILE.

#include "hdf5.h"

#include <stdio.h>
#include <stdlib.h>

int main()
{
  __label__ fail_fapl, fail_file;
  int ret_val = EXIT_SUCCESS;
  hid_t fapl, file;

  {
    unsigned maj, min, rel;
    if (H5get_libversion(&maj, &min, &rel) < 0) {
      ret_val = EXIT_FAILURE;
      goto fail_fapl;
    }
    printf("Welcome to HDF5 %d.%d.%d!\n", maj, min, rel);
  }

  if ((fapl = H5Pcreate(H5P_FILE_ACCESS)) < 0) {
    ret_val = EXIT_FAILURE;
    goto fail_fapl;
  }

  // bracket the range of LIBRARY VERSIONS for object creation and access,
  // e.g., min. vers. 1.8, max. version current
  if (H5Pset_libver_bounds(fapl, H5F_LIBVER_V18, H5F_LIBVER_LATEST) < 0) {
    ret_val = EXIT_FAILURE;
    goto fail_file;
  }

  if ((file = H5Fcreate("my.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl)) < 0) {
    ret_val = EXIT_FAILURE;
    goto fail_file;
  }

  // do something useful w/ FILE

  H5Fclose(file);

 fail_file:
  H5Pclose(fapl);
 fail_fapl:;

  return ret_val;
}

OK?

G.

[gheber]

@leon_vernikov

• HDF5 attribute base-element is simplest and basic property of HDF5 database to implement.

• HDF5 attribute base-element/API allows to store both network packets and information related to network traffic

(type of network traffic, the traffic property related to all network packets).

• HDF5 attribute base-element/API doesn’t require a separate table to be inserted under HDF5 group.

• Most important: HDFView v2.14.0 has memory problem to open HDF5 group’s table with large number of records. Basically, HDFView is dead at opening.

OK, I don’t fully understand what you are saying, but my guess is that you are acquiring some kind of packet stream. The choices you’ve made are:

1. The stream is represented as an HDF5 group.
2. Stream properties/packet invariants are represented as HDF5 attributes
3. Individual packets are represented as HDF5 ???

What about 3.? How do you represent packets?

I don’t understand your comment on large numbers of records in HDF5 group tables. Groups contain links, not records. If you mean ‘attributes’ by records, then what you are seeing in HDFView is just a replay of the performance problem you are reporting. If you mean ‘links’ by records, then, again, there are low-level tricks to massage the underlying B-tree and heap, but it won’t do miracles for huge (100K, 1M,…) numbers of links.

You haven’t told us how you are reading/accessing your stream. How do you maintain/represent packet order?

I believe you told us that the packet type is a fixed-size string, right?

@leon_vernikov

=LV= Are you referring H5PT base-element?

Not as an implementation, because that one’s done poorly, but as a general idea. For a proper implementation, take a look at @steven 's packet table implementation in H5CPP (example).

G.

[leon_vernikov]

Hello Gerd,

Thank you very much for the code sample.

The code sample in C language is perfect solution to express your ideas.

I integrated your idea into my code-sample (Sent to HDF5 Techsupport on 6/16/2021.

Would you contact Binh-Minh for details?) and executed tests.

I see the write performance degradation 100 times for 5K network packets collected in HDF5 database.

Let me answer your questions

"OK, I don’t fully understand what you are saying, but my guess is that you are acquiring some kind of packet stream. The choices you’ve made are:

1. The stream is represented as an HDF5 group.
2. Stream properties/packet invariants are represented as HDF5 attributes
3. Individual packets are represented as HDF5 ???
   What about 3.? How do you represent packets?"

=LV=

I wish you had an access to my code-sample to address your questions.

1. A network stream is stored under HDF5 group. Each network packet has length 1024 KB

2. Each individual packet is stored as name-value pair in HDF5 attribute.

3. Each new HDF5 attribute has the following parameters:

• a network packet number is HDF5 attribute’s name and

• 1024 KB packet content is HDF5 attribute’s string

“You haven’t told us how you are reading/accessing your stream. How do you maintain/represent packet order?”

=LV= I don’t see HDF5 performance degradation for search and read operations for HDF5 attribute.

This is a list of HDF5 calls for read operation where sAttributeName is the network packet number (see above)

h5_ADattr = H5Aopen_name(H5Group, sAttributeName);

nReturn = H5Aread(h5_ADattr, h5_ADtype, &stringRead);

nReturn = H5Aclose(h5_ADattr);

“I believe you told us that the packet type is a fixed-size string, right?”

=LV= Correct

“I don’t understand your comment on large numbers of records in HDF5 group tables. Groups contain links, not records. If you mean ‘attributes’ by records, then what you are seeing in HDFView is just a replay of the performance problem you are reporting.”

=LV= Created a code-sample where H5Group has H5PT inserted. Stored 1M entries in H5PT where each entry is ~1024 KB string. Open HDF5 database with HDFView v2.14.0 utility. Select H5PT table from H5Group. Observe memory error dialog.

Thank you very much for your help to address HDF5 performance degradation problem during H5Awrite operations,

Leon

gheber
Gerd Heber
The HDF Group Staff

    June 23

leon_vernikov:

• HDF5 attribute base-element is simplest and basic property of HDF5 database to implement.

• HDF5 attribute base-element/API allows to store both network packets and information related to network traffic

(type of network traffic, the traffic property related to all network packets).

• HDF5 attribute base-element/API doesn’t require a separate table to be inserted under HDF5 group.

• Most important: HDFView v2.14.0 has memory problem to open HDF5 group’s table with large number of records. Basically, HDFView is dead at opening.

OK, I don’t fully understand what you are saying, but my guess is that you are acquiring some kind of packet stream. The choices you’ve made are:

1. The stream is represented as an HDF5 group.
2. Stream properties/packet invariants are represented as HDF5 attributes
3. Individual packets are represented as HDF5 ???
   What about 3.? How do you represent packets?

I don’t understand your comment on large numbers of records in HDF5 group tables. Groups contain links, not records. If you mean ‘attributes’ by records, then what you are seeing in HDFView is just a replay of the performance problem you are reporting. If you mean ‘links’ by records, then, again, there are low-level tricks to massage the underlying B-tree and heap, but it won’t do miracles for huge (100K, 1M,…) numbers of links.

You haven’t told us how you are reading/accessing your stream. How do you maintain/represent packet order?

I believe you told us that the packet type is a fixed-size string, right?

leon_vernikov:
=LV= Are you referring H5PT base-element?

Not as an implementation, because that one’s done poorly, but as a general idea. For a proper implementation, take a look at @steven 's packet table implementation in H5CPP (example).

G.

[leon_vernikov]

Hello Gerd,

Let me add few more updates on performance results:

1. Appended the attached diff into existing code to handle HDF5 attribute

Please note: H5Pset_libver_bounds() call parameter is changed due to version 1.8.12 has the following H5F_libver_t enumerator definition

include/H5Fpublic.h

/* Library’s file format versions */

typedef enum H5F_libver_t {

H5F_LIBVER_EARLIEST, /* Use the earliest possible format for storing objects */

H5F_LIBVER_LATEST /* Use the latest possible format available for storing objects*/

} H5F_libver_t;

You are right, the HDF5 attribute write performance is improving.

I started observing “File accessibilty” new problem when about 30K HDF5 attributes were created.

(HDF5 original spelling)

I would expect HDF5 skips creating HDF5 attribute for this particular network packet.

HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:

#000: line 1708 in (): ▒▒߈le to flush mounted file hierarchy

major: File accessibilty

minor: Unable to flush data from cache

#4294967291: line 699 in ▒▒▒hflush_mounts(): ▒▒▒le to flush mounted file hierarchy

major: File accessibilty

minor: Unable to flush data from cache

#4294967292: ▒▒▒0./src/H5Fmount.c line 660 in ▒▒▒flush_mounts_recurse(): ▒▒▒8le to flush file’s cached information

major: File accessibilty

minor: Unable to flush data from cache

#4294967293: ▒▒▒ line 1763 in ▒▒8flush(): ▒▒▒8le to flush metadata cache

major: Object cache

minor: Unable to flush data from cache

#4294967294: ▒▒▒p./src/H5AC.c line 851 in ▒▒t@_flush(): ▒▒t flush cache.

major: Object cache

minor: Unable to flush data from cache

#4294967295: ▒▒▒▒./src/H5C.c line 1868 in ▒▒▒ flush_cache(): ▒▒▒e has protected items

major: Object cache

minor: Unable to flush data from cache

2. Implemented the code samples demonstrating H5PT performance.

The code creates 1M unique H5PT records in a loop using H5PTappend() call.

Each unique H5PT record is 1024 KB string length.

• Write 1M packets to HDF5 database: ~13 sec total.

• Read 1M records from HDF5 database: ~90 sec total.

HDF5 database performance is sufficient to support real-time applications.

• HDF5 database size increases 2GB (uncompressed).

Thank you,

Leon

[gheber]

@leon_vernikov

I started observing “File accessibilty” new problem when about 30K HDF5 attributes were created.

Can you give us an MWE?

@leon_vernikov

Implemented the code samples demonstrating H5PT performance.

1. What chunk_size did you use?
2. I’m not sure I follow the arithmetic. Each packet is 1024 KB (~ 1 MB) and you are writing 1 million packets w/o compression to the tune of 2 GB. How is that possible?

G.

[leon_vernikov]

Hello Gerd,

Let me answer your questions:

1. Can you give us an MWE?
   The original code was submitted to HDF5 Techsupport on 6/16/2021.
   Added small diff yesterday, you recommended

• if ((fapl = H5Pcreate(H5P_FILE_ACCESS)) < 0) {

•    printf("Failed to execute H5Pcreate");
  

• }
• if (H5Pset_libver_bounds(fapl, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST) < 0) {

•    printf("Failed to execute H5Pset_libver_bounds");
  

• }

2. What chunk_size did you use? 2KB
   I’m not sure I follow the arithmetic. Each packet is 1024 KB (~ 1 MB) and you are writing 1 million packets w/o compression to the tune of 2 GB. How is that possible?
   =LD= corrected calculation
   2KB x 1M packets = 2 GB

3. What is HDF5 API to remove H5 Packet Table from HDF5 database Group?

Thank you for your help and support,

Leon

[gheber]

@leon_vernikov

What chunk_size did you use? 2KB

That’s a rather small chunk size. Remember that chunk_size is measured in the number of elements (not bytes). Aim for a 1 MB chunk size. If one packet is 2 KB, you should try putting 512 of them onto a chunk.

@leon_vernikov

What is HDF5 API to remove H5 Packet Table from HDF5 database Group?

The packet table itself is just a decorated HDF5 dataset. HDF5 objects (including datasets) are deleted by unlinking them. Use H5Ldelete to do that. For example, if your packet table is at /my/great/packet table and file is an HDF5 filehandle, call

if(H5Ldelete(file, "/my/great/packet table", H5P_DEFAULT) < 0) {
  perror("Something bad happened!\n");
}

G.

[leon_vernikov]

Hi Gerd,

Right, I had the same impression initially. I thought, H5PT has H5L_TYPE_SOFT type.
HDF5 lists H5PT as H5L_TYPE_HARD type instead and fails to delete with H5Ldelete() API.
What is an alternative solution to remove H5PT ?

Thank you,

Leon

H5L_info_t _linkinfo;
if ( H5Lget_info( H5Group, “12345”, &_linkinfo, H5P_DEFAULT ) >= 0 ) {
printf(“EXISTS\n”);
if ( _linkinfo.type == H5L_TYPE_SOFT ) {
printf(“H5L_TYPE_SOFT\n”);
}
if ( _linkinfo.type == H5L_TYPE_HARD ) {
printf(“H5L_TYPE_HARD\n”);
if(H5Ldelete(H5Group, “12345”, H5P_DEFAULT) < 0) {
printf (“####### Failed to remove 12345\n”);
}
}

if ( _linkinfo.type == H5L_TYPE_EXTERNAL ) {
printf(“H5L_TYPE_EXTERNAL\n”);
}
if ( _linkinfo.type == H5L_TYPE_ERROR ) {
printf(“H5L_TYPE_ERROR\n”);
}
} else
printf(“DOESNT EXIST\n”);

What is HDF5 API to remove H5 Packet Table from HDF5 database Group?

The packet table itself is just a decorated HDF5 dataset. HDF5 objects (including datasets) are deleted by unlinking them. Use H5Ldelete to do that. For example, if your packet table is at /my/great/packet table and file is an HDF5 filehandle, call

  if(H5Ldelete(file, "/my/great/packet table", H5P_DEFAULT) < 0) { perror("Something bad happened!\n"); }

G.

[gheber]

@leon_vernikov

HDF5 lists H5PT as H5L_TYPE_HARD type instead and fails to delete with H5Ldelete() API.

The hard link is expected. There needs to be at least one. What’s the error message from H5Ldelete?
There are alternatives (e.g., manually dropping the reference count to zero), but let’s not go there before we understand what’s going on.

G.

[leon_vernikov]

Hi Gerd,

1. “What’s the error message from H5Ldelete?”
   =LV= H5Ldelete() trace is below.

2. “That’s a rather small chunk size. Remember that chunk_size is measured in the number of elements (not bytes). Aim for a 1 MB chunk size. If one packet is 2 KB, you should try putting 512 of them onto a chunk.”

=LV= This is a code segment to create H5PT. Would you confirm?
herr_t nStatus;
h5_PacketTable = H5I_INVALID_HID;
hid_t h5_PTDataType; // Fixed Length Packet Table data type

      // Fixed length Packet Table DataType
      h5_PTDataType = H5Tcopy (H5T_C_S1);
      CHECK(h5_PTDataType, FAIL, "H5Tcopy");
      nStatus = H5Tset_size (h5_PTDataType, 2048);
      CHECK(nStatus, FAIL, "H5Tset_size");
      h5_PacketTable = H5PTcreate_fl(H5Group, "12345", h5_PTDataType, 512, -1);

3. H5PTcreate_fl() documentation states
   “Compression level, a value of 0 through 9. Level 0 is faster but offers the least compression;
   level 9 is slower but offers maximum compression. A setting of -1 indicates that no compression is desired.”
   I need to select an optimal H5PT performance and reduce HDF5 database file size.
   Would you send me a chart/graph to compare

• H5PT performance vs. compression level;
• HDF5 database file size vs. compression level?

Thank you,

Leon

HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
#000: …/…/src/H5L.c line 609 in H5Ldelete(): unable to delete link
major: Links
minor: Can’t delete message
#001: …/…/src/H5L.c line 2344 in H5L_delete(): can’t unlink object
major: Symbol table
minor: Unable to remove object
#002: …/…/src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#003: …/…/src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#004: …/…/src/H5L.c line 2297 in H5L_delete_cb(): can’t delete self
major: Symbol table
minor: Can’t delete message
####### Failed to remove 12345
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
#000: …/…/src/H5L.c line 868 in H5Lget_info(): unable to get link info
major: Symbol table
minor: Object not found
#001: …/…/src/H5L.c line 2840 in H5L_get_info(): name doesn’t exist
major: Symbol table
minor: Object already exists
#002: …/…/src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#003: …/…/src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#004: …/…/src/H5L.c line 2799 in H5L_get_info_cb(): name doesn’t exist
major: Symbol table
minor: Object not found

gheber
Gerd Heber
The HDF Group Staff

    June 29

leon_vernikov:
HDF5 lists H5PT as H5L_TYPE_HARD type instead and fails to delete with H5Ldelete() API.

The hard link is expected. There needs to be at least one. What’s the error message from H5Ldelete?

There are alternatives (e.g., manually dropping the reference count to zero), but let’s not go there before we understand what’s going on.

G.

[gheber]

@leon_vernikov

=LV= H5Ldelete() trace is below.

Have you closed the packet table (H5PTclose) before trying to delete it?

@leon_vernikov

I need to select an optimal H5PT performance and reduce HDF5 database file size.
Would you send me a chart/graph to compare

You’ll have to create that yourself with your data because the compression ratio will be all over the place for different data. A packet table with all identical packets will compress perfectly at low compression levels. A packet table with random packets will potentially be much larger than its nominal size, at higher compression levels.

G.

[leon_vernikov]

“Have you closed the packet table (H5PTclose) before trying to delete it?”
=LV= Sure, I closed H5PT before delete it
“There are alternatives (e.g., manually dropping the reference count to zero)”
=LV= Would you provide the code samples?

Thank you,

Leon

[gheber]

@leon_vernikov

“There are alternatives (e.g., manually dropping the reference count to zero)”
=LV= Would you provide the code samples?

See the example under DELETE . Again, this is NOT the recommended way of going about it.

G.

[leon_vernikov]

“There are alternatives (e.g., manually dropping the reference count to zero)”
=LV= Would you provide the code samples for version 1.8.12 instead?
I’m using HDF5 Version 1.8.12
You are referring to code samples for version 1.13.0
H5O_info2_t structure doesn’t exist in v 1.8.12
I’ll remove H5Oget_info() call

Thank you,

Leon