I'm using temporary HDF5 files to store intermediate processing results. When finished, I'd like to just discard the file, i.e. close the file without flushing (flushing takes some time).
Also, the file can be closed indirectly when the last object handle is released. It's unclear how to tell HDF5 library that no flushing should occur in this case.
But maybe there is still a way to close HDF5 file without flushing?
Best wishes,
Andrey Paramonov
···
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
I'm using temporary HDF5 files to store intermediate processing results.
When finished, I'd like to just discard the file, i.e. close the file
without flushing (flushing takes some time).
Also, the file can be closed indirectly when the last object handle is
released. It's unclear how to tell HDF5 library that no flushing should
occur in this case.
But maybe there is still a way to close HDF5 file without flushing?
Best wishes,
Andrey Paramonov
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Did you try storing everything under a single group and unlinking it from
the root before closing? This *might* prevent HDF5 from wanting to flush
anything.
I don't know if 1M..1G qualifies as big or small
I just feel that several-second delay when closing temp. data files is very stupid.
Did you try storing everything under a single group and unlinking it
from the root before closing? This *might* prevent HDF5 from wanting to
flush anything.
"Might" is not what I'm expecting from the software
So, I guess there is no way to discard file, deterministically?
Best wishes,
Andrey Paramonov
···
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Big is what will not fit in RAM in an ordinary use case. If it would easily
fit in memory use a memory file.
Not sure what you're expecting from the software, I expect developers to
try
Note, HDF5 is normally used for permanent storage not ephemeral. Another
way would be to write a special terminal VFD driver that does exactly what
you want.
Big is what will not fit in RAM in an ordinary use case. If it would
easily fit in memory use a memory file.
Right, but in my case the data won't often fit in memory.
Not sure what you're expecting from the software, I expect developers to
try
Note, HDF5 is normally used for permanent storage not ephemeral. Another
way would be to write a special terminal VFD driver that does exactly
what you want.
I do if fact have one. Is it possible to implement discard feature with VFD hooks?
Best wishes,
Andrey Paramonov
···
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Big is what will not fit in RAM in an ordinary use case. If it would
easily fit in memory use a memory file.
Right, but in my case the data won't often fit in memory.
Not sure what you're expecting from the software, I expect developers to
try
Note, HDF5 is normally used for permanent storage not ephemeral. Another
way would be to write a special terminal VFD driver that does exactly
what you want.
I do if fact have one. Is it possible to implement discard feature with
VFD hooks?
Best wishes,
Andrey Paramonov
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
> Big is what will not fit in RAM in an ordinary use case. If it would
> easily fit in memory use a memory file.
Right, but in my case the data won't often fit in memory.
If you have a rather simple space problem and the data
can be organized in some simple fashion without HDF,
then why not think about just memory mapped files
or simply rely on system's paging capabilities.
Big is what will not fit in RAM in an ordinary use case. If it would
easily fit in memory use a memory file.
Right, but in my case the data won't often fit in memory.
If you have a rather simple space problem and the data
can be organized in some simple fashion without HDF,
then why not think about just memory mapped files
or simply rely on system's paging capabilities.
(Probably you really want to do it using HDF...)
Yes, I do. I really like HDF5 query abilities, transparent compression. But most importantly, I can H5Ocopy data to permanent storage!
Best wishes,
Andrey Paramonov
···
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
