From hdf-forum-bounces@hdfgroup.org Thu Feb 10 16:48:04 2011
From: Rhys Ulerich <rhys.ulerich@gmail.com>
Date: Thu, 10 Feb 2011 16:48:22 -0600
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: [Hdf-forum] Question re: Howison et al Lustre mdc_config tuning recommendationsGood day,
I decided to try some metadata caching parameters related to the
discussion on pages 3-4 of Howison et al
(http://www.hdfgroup.org/pubs/papers/howison_hdf5_lustre_iasds2010.pdf).
The paper gives sample code as follows in Figure 5:H5AC_cache_config_t mdc_config;
hid_t file_id;
file_id = H5Fopen("file.h5", H5ACC_RDWR, H5P_DEFAULT);
mdc_config.version = H5AC__CURR_CACHE_CONFIG_VERSION;
H5Pget_mdc_config(file_id, &mdc_config)
mdc_config.evictions_enabled = 0 /* FALSE */;
mdc_config.incr_mode = H5C_incr__off;
mdc_config.decr_mode = H5C_decr__off;
H5Pset_mdc_config(file_id, &mdc_config);Attempting to directly implement this fails. Modifying the above so
the H5Pget_mdc_config/H5Pset_mdc_config operates on a file access
property list succeeds. However, I saw runtime errors likeHDF5-DIAG: Error detected in HDF5 (1.8.5-patch1) MPI-process 2:
#000: H5Pfapl.c line 1354 in H5Pset_mdc_config(): invalid metadata
cache configuration
major: Invalid arguments to routine
minor: Bad value
#001: H5AC.c line 2665 in H5AC_validate_config(): Can't disable
evictions while auto-resize is enabled.
major: Invalid arguments to routine
minor: Bad valuewhich I've remedied by glancing at H5AC.c:2665 and then adding the statement
mdc_config.flash_incr_mode = H5C_flash_incr__off;
to the code block above. Is this modification in the spirit of what
Howison et al. suggests? Or is using H5C_flash_incr__off asking for
trouble in ways that the paper does not discuss?Thanks for your time,
Rhys_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Rhys,
You did exactly as you should have.
The metadata cache configuration in the paper worked due to an error
on my part when I implemented the flash cache size increment code. I have
since noticed and correct the error -- hence the failure you encountered
when you tried to duplicate the code from the paper.
To give you some background:
In its default configuration, the metadata cache in HDF5 will attempt to
automatically adapt to the current metadata working set size in real time.
While there is no fundamental reason why this feature can't be active when
evictions are disabled, I can't think of any circumstances in which it would
be useful. Further, writing the test code required to verify proper behavior
under these circumstances would require significant effort.
Hence the decision to require that adaptive metadata cache resizing be
disabled when evictions are disabled. Needless to say, this decision will
be re-visited if anyone comes up with a plausible reason to do so.
Best regards,
John Mainzer