[H5PT] - Force chunk memory free()


I am using HDF5 to store a large number (approximately 50000) of
compound datasets composed of a timestamp and a double value (I need a
separate dataset for each item since the timestamps are not
synchronized). Each data is arriving at approximately 100 Hz. I am
running on a 8-core Xeon with 12 Gb of RAM.

Since the data is arriving "on-the-fly", I am using the H5PT
interface. After running for a while, I ran out of memory in my "real"
application so I created a small test program (see at the end of my
message). With this test program, I can easily end up having an
important memory usage (> 4Gb) by adjusting NUM_APPENDS or

After some digging, I realized that while my test program is running,
I get a lot of H5D_chunk_alloc() calls, but no H5D_chunk_xfree() calls
during execution. H5D_chunk_xfree() only seems to get called when I
H5PTclose() the dataset at the end of the program.

I tried using H5Fflush() and played around a bit with various file
properties (sieve buffers, etc.), but I didn't find a way to force
HDF5 to free() the chunk buffers.
I also did "convert" the sample application to using H5D with
extensible datasets and hyperslabs, but I got the same results (no
H5D_chunk_xfree() during runtime).

Any hints on how to make my application run for a long time (without
H5Dwrite() complaining about memory starvation) would be very much



====== Example code ======

#include "hdf5.h"
#include "hdf5_hl.h"
#include <stdlib.h>

#include <iostream>
using namespace std;

#include <boost/thread.hpp>
#include <boost/bind.hpp>
#include <boost/format.hpp>
using boost::format;

// Number of tables to create
#define NUM_PACKET_TABLES 10000
// Chunk size
#define CHUNK_SIZE 4096
// Compression level
// Number of H5PTappend() loops (simulating the arrival of data)
#define NUM_APPENDS 100
// Number of elements to pass to H5PTappend() at each loop
#define APPEND_SIZE 128

// Defintion of the compound data type
typedef struct
  double data;
         long long timestamp;
} data_t;

int main(void)
  hid_t fid;
   hid_t ptable[NUM_PACKET_TABLES];
  // Data buffer
        data_t data[APPEND_SIZE];
   // Open file
  fid = H5Fcreate("packet.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
  // Create compound data type
  hid_t datatype = H5Tcreate(H5T_COMPOUND, sizeof(data_t));
  H5Tinsert(datatype, "Data", HOFFSET(data_t, data), H5T_NATIVE_DOUBLE);
  H5Tinsert(datatype, "Timestamp", HOFFSET(data_t, timestamp), H5T_NATIVE_LLONG);
  // Create packet tables
  for (int i = 0; i < NUM_PACKET_TABLES; i++)
     ptable[i] = H5PTcreate_fl(fid, str(format("/data.%1%") %
i).c_str(), datatype, (hsize_t) CHUNK_SIZE, COMPRESSION_LEVEL);
  // Close data type handle

   for (int i = 0; i < NUM_APPENDS; i++)
    for (int j = 0; j < NUM_PACKET_TABLES; j++)
      // Random data append - we do not care about the values in the buffer
      H5PTappend(ptable[j], (hsize_t) APPEND_SIZE, data);

  for (int i = 0; i < NUM_PACKET_TABLES; i++)
  // Close file
      return 0;