hello Elena, thank you for your reply. I know it is LZF compression. what I don’t know is the function or whatever to read it (and uncompressed it). I use HDF5.Pinvoke and I program in VB.NET
I forgot to add that to use the filter function you should have LZF compression library installed on your system.
The easiest way to bypass the plugin problem (as a quick hack) is to register the filter function with your application. For example, see section 5 in this document. Compile and link the filter function and the LZF library with your application. I am not sure from where to get the LZF library, but may be tar file from the THG GitHub plugins repo will be useful.
I quickly read waht you send, I will try next week thank you
I have zlib.dll and szip.dll
and I read the dataset like that
H5D.read(dsID, typeID, H5S.ALL, H5S.ALL, H5P.DEFAULT, pinnedArray.AddrOfPinnedObject())
what is not clear for me is how and when the data are uncompressed ?
Information about the applied filter(s) is stored in the header of a dataset, therefore HDF5 library knows which filter(s) to apply on write and read operations. HDF5 will search for the corresponding filter function and use it to encode/decode the data.
There are three mechanism in HDF5 to invoke filters: via plugins (the filter is loaded at run time), built-in with HDF5 (e.g., gzip, szip, n-bit, scale+offset), and built-in with HDF5 app (I pointed you to this one as the simplest to implement and debug). In all those cases decoding is applied automatically, so your current read call should work without modifications.
OK I see, ty. I currently use HDF5.Pinvoke, I just have downloaded hdf5-1.14.0-Std-win10_64-vs16.
For HDF5.Pinvoke I have
HDF.Pinvoke.dll (the one I import in my project), along with I have hdf5.dll, hdf5_hl.dll, zlid and szip.dll
When I read the array compressed I have no errors and the size seems also correct, but the array contain only zeros.
I installed hdf5-1.14.0-Std-win10_64-vs16 and I see in plugin directory libh5lzf.dll but I cannot include as a reference, should I do something with this ? Now I am quite lost with all the dll versions…
ty for your patience and support
Agree… It is confusing…But if everything is installed correctly, it should work without any work on your part 
I am on macOS system and downloaded hdf5-1.14.0-Std-macos11_64-clang.tar.gz to check binary packaging. It looks like that libh5lzf.so (equivalent to Windows libh5lzf.dll) contains the filter function and the LZF library. If your HDF5_PLUGIN_PATH environment variable points to the plugin directory, it should work.
I would try a few things to troubleshoot the problem:
-
Make sure you are using HDF5.Pinvoke based on HDF5 1.14.0
-
Try
h5dumpfrom this installation to dump the content of the file and see if it prints data. Use--enable-error-stackoption for h5dump to catch any errors. If the tool cannot see theplugindirectory then it will print an error message. If it prints zeros without displaying an error, I would suspect that the dataset was not written at all. When you use-poption, check theSIZEvalue for the LZF compressed dataset’s output. It should not be 0 if data was written. -
Try a simple C example (see section 3.1 in this document, and use 32000 for the filter ID to set LZF filter in the H5Pset_filter function. You should be able to see a filter parameter used for your file from the
h5dump -pHoutput. Also, section 3.2 shows the error you should be getting on read if the filter is not available.
Where I can get the file in questions? I will try it on my system in the next few days.
Elena
Hello Elena
1.
I have installed HDF.Pinvoke.NETStandard and HDF.Pinvoke.1.10 from the Nuget in VS
I copy HDF5.dll in my project from hdf5-1.14.0-Std-win10_64-vs16
but nothing changed
- I am sure the data are written in the file, I checked with binary editor and draw the images (data are images) with imageJ (of course it can nt read them properly because of compression, but it doesn’t contain zeros for sure). here is the file FileSender
the data are stored in “1/EBSD/Data/Unprocessed Patterns” tag
Elena mentioned it, but you did not indicate y/n -> If your HDF5_PLUGIN_PATH environment variable points to the plugin directory, it should work.
Also remember that windows needs to have any dlls in the “PATH” environment variable as well. (dll is the runtime component of libraries on windows)
I am wondering if this discussion is relevant to the problem. If files are OK, I would suspect that some dlls are not found.
Yes probably, but as I don’t get any error, I don’t know what to do. I have hdf5.dll, hdf5_hl.dll, zlid and szip.dll with HDF5.Pinvoke.1.10.dll (I had HDF5.Pinvoke.dll before, it was the same problem)
by the way,I cannot see this compressed data in HDFviewer
I will look all that asap, thanks
@benoitbeausir1 I have extended PureHDF to add support for LZF compressed data (https://www.nuget.org/packages/PureHDF.Filters.Lzf/1.0.0-alpha.25). If you are using a newer version of .NET you can run the two commands below and copy pase the code to your “Program.cs” file (and insert the correct path to your h5oina file):
// dotnet new console
// dotnet add package PureHDF.Filters.LZF --prerelease
using PureHDF;
using PureHDF.Filters;
H5Filter.Register(
identifier: (H5FilterID)32000,
name: "lzf",
filterFunction: H5Lzf.FilterFunction);
using var file = H5File.OpenRead(<path to h5oina file>);
var dataset = file.Dataset("/1/EBSD/Data/Unprocessed Patterns");
var data = dataset.Read<short>();
Console.WriteLine("Successfully read data.");
I was able to read the data from the file you have provided earlier. But as I do not know which format the image data has, I could not verify the output.
It should also work with .NET Framework 4.8 but lower versions will likely not work.
Edit: I forgot you are using VB.NET, but I think you can easily translate C# to VB.NET.
1 Like