Hi all,
I've post a new RFC describing a feature we are planning to add to the HDF5 library that will cache open files from external links:
http://www.hdfgroup.uiuc.edu/RFC/HDF5/ExternalLinkFileAccessProperty/CacheExternalLinkFileOpens.pdf
If you are using external links and would like to comment on this performance enhancement, please do so.
Thanks,
Quincey
On the topic of caching opened files (external or otherwise), I have a
question.
In VisIt, we have many different 'flavors' of HDF5 files we read. A
'flavor' is a particular structure (groups/dataset/attribute names) of
HDF5 file. We have a separate reader plugin for each flavor.
We don't always know ahead of time which plugin should be used to open a
given HDF5 file. So, we can wind up iterating over several, each
attempting some HDF5 operations on the file and, perhaps, throwing
exceptions (VisIt exceptions not HDF5 exceptions) and then moving onto
the next HDF5 reader plugin to try.
Here is my question. If one reader plugin calls H5Fopen and that
succeeds but later in the plugin's attempt to read the file, it throws a
VisIt exception to move onto the next plugin, what if it also does NOT
call H5Fclose? Will that potentially negatively affect the next reader
plugin which will also use H5Fopen to open that same file? Also, how
does fact that these plugins are operating as shared libs effect things,
in particular, any HDF5 library wide global data such as the list of
cached files it has open?
Mark
On Mon, 2010-11-08 at 07:05, Quincey Koziol wrote:
Hi all,
I've post a new RFC describing a feature we are planning to add to the HDF5 library that will cache open files from external links:If you are using external links and would like to comment on this performance enhancement, please do so.
Thanks,
Quincey_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Hi Mark,
On the topic of caching opened files (external or otherwise), I have a
question.In VisIt, we have many different 'flavors' of HDF5 files we read. A
'flavor' is a particular structure (groups/dataset/attribute names) of
HDF5 file. We have a separate reader plugin for each flavor.We don't always know ahead of time which plugin should be used to open a
given HDF5 file. So, we can wind up iterating over several, each
attempting some HDF5 operations on the file and, perhaps, throwing
exceptions (VisIt exceptions not HDF5 exceptions) and then moving onto
the next HDF5 reader plugin to try.Here is my question. If one reader plugin calls H5Fopen and that
succeeds but later in the plugin's attempt to read the file, it throws a
VisIt exception to move onto the next plugin, what if it also does NOT
call H5Fclose? Will that potentially negatively affect the next reader
plugin which will also use H5Fopen to open that same file? Also, how
does fact that these plugins are operating as shared libs effect things,
in particular, any HDF5 library wide global data such as the list of
cached files it has open?
Well, this document describes a method of caching files opened through external links, not files opened directly with H5Fopen(). Also, by default, the caching is disabled, so applications won't get a surprise. And, it's enabled on a per-file basis, not a global basis, again to avoid surprises for applications. So, I don't think it'll affect the situation you describe above with VisIt. Of course, you should close the files you open with H5Fopen() or else they will remain open and chew up resources, etc. 
I don't think the shared library aspect will have any effect.
Quincey
On Nov 8, 2010, at 9:12 AM, Mark Miller wrote:
Mark
On Mon, 2010-11-08 at 07:05, Quincey Koziol wrote:
Hi all,
I've post a new RFC describing a feature we are planning to add to the HDF5 library that will cache open files from external links:If you are using external links and would like to comment on this performance enhancement, please do so.
Thanks,
Quincey_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Quincey,
Thanks for the follow-up. One question; what are the
functions/properties by which a plugin might wind up ENABLING caching of
opened file handles?
Mark
On Tue, 2010-11-09 at 05:06 -0800, Quincey Koziol wrote:
Hi Mark,
On Nov 8, 2010, at 9:12 AM, Mark Miller wrote:
> On the topic of caching opened files (external or otherwise), I have a
> question.
>
> In VisIt, we have many different 'flavors' of HDF5 files we read. A
> 'flavor' is a particular structure (groups/dataset/attribute names) of
> HDF5 file. We have a separate reader plugin for each flavor.
>
> We don't always know ahead of time which plugin should be used to open a
> given HDF5 file. So, we can wind up iterating over several, each
> attempting some HDF5 operations on the file and, perhaps, throwing
> exceptions (VisIt exceptions not HDF5 exceptions) and then moving onto
> the next HDF5 reader plugin to try.
>
> Here is my question. If one reader plugin calls H5Fopen and that
> succeeds but later in the plugin's attempt to read the file, it throws a
> VisIt exception to move onto the next plugin, what if it also does NOT
> call H5Fclose? Will that potentially negatively affect the next reader
> plugin which will also use H5Fopen to open that same file? Also, how
> does fact that these plugins are operating as shared libs effect things,
> in particular, any HDF5 library wide global data such as the list of
> cached files it has open?Well, this document describes a method of caching files opened through external links, not files opened directly with H5Fopen(). Also, by default, the caching is disabled, so applications won't get a surprise. And, it's enabled on a per-file basis, not a global basis, again to avoid surprises for applications. So, I don't think it'll affect the situation you describe above with VisIt. Of course, you should close the files you open with H5Fopen() or else they will remain open and chew up resources, etc.
Quincey
> Mark
>
> On Mon, 2010-11-08 at 07:05, Quincey Koziol wrote:
>> Hi all,
>> I've post a new RFC describing a feature we are planning to add to the HDF5 library that will cache open files from external links:
>>
>> http://BLOCKEDBLOCKEDwww.BLOCKEDBLOCKEDhdfgroup.uiuc.edu/RFC/HDF5/ExternalLinkFileAccessProperty/CacheExternalLinkFileOpens.pdf
>>
>> If you are using external links and would like to comment on this performance enhancement, please do so.
>>
>> Thanks,
>> Quincey
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://BLOCKEDBLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
> --
> Mark C. Miller, Lawrence Livermore National Laboratory
> ================!!LLNL BUSINESS ONLY!!================
> miller86@llnl.gov urgent: miller86@pager.llnl.gov
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Hi Mark,
Hi Quincey,
Thanks for the follow-up. One question; what are the
functions/properties by which a plugin might wind up ENABLING caching of
opened file handles?
There's only the new one in the RFC: H5Pset_elink_file_cache_size() And that routine is only [currently] planned to enable caching of files opened through external links.
Quincey
On Nov 9, 2010, at 11:01 AM, Mark Miller wrote:
Mark
On Tue, 2010-11-09 at 05:06 -0800, Quincey Koziol wrote:
Hi Mark,
On Nov 8, 2010, at 9:12 AM, Mark Miller wrote:
On the topic of caching opened files (external or otherwise), I have a
question.In VisIt, we have many different 'flavors' of HDF5 files we read. A
'flavor' is a particular structure (groups/dataset/attribute names) of
HDF5 file. We have a separate reader plugin for each flavor.We don't always know ahead of time which plugin should be used to open a
given HDF5 file. So, we can wind up iterating over several, each
attempting some HDF5 operations on the file and, perhaps, throwing
exceptions (VisIt exceptions not HDF5 exceptions) and then moving onto
the next HDF5 reader plugin to try.Here is my question. If one reader plugin calls H5Fopen and that
succeeds but later in the plugin's attempt to read the file, it throws a
VisIt exception to move onto the next plugin, what if it also does NOT
call H5Fclose? Will that potentially negatively affect the next reader
plugin which will also use H5Fopen to open that same file? Also, how
does fact that these plugins are operating as shared libs effect things,
in particular, any HDF5 library wide global data such as the list of
cached files it has open?Well, this document describes a method of caching files opened through external links, not files opened directly with H5Fopen(). Also, by default, the caching is disabled, so applications won't get a surprise. And, it's enabled on a per-file basis, not a global basis, again to avoid surprises for applications. So, I don't think it'll affect the situation you describe above with VisIt. Of course, you should close the files you open with H5Fopen() or else they will remain open and chew up resources, etc.
Quincey
Mark
On Mon, 2010-11-08 at 07:05, Quincey Koziol wrote:
Hi all,
I've post a new RFC describing a feature we are planning to add to the HDF5 library that will cache open files from external links:If you are using external links and would like to comment on this performance enhancement, please do so.
Thanks,
Quincey_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDBLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://BLOCKEDmail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
