# What do you want to see in "HDF5 2.0"? **URL:** \ **Category:** HDF5 Library\ **Created:** [July 21, 2022, 3:14pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003 "2022-07-21T15:14:39Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![epourmal1](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/82dd89/32.png) [@epourmal1](https://forum.hdfgroup.org/u/epourmal1)\ **Post date:** [September 1, 2022, 9:30pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/21 "2022-09-01T21:30:36Z")
Actually, we are almost there after new indexing for chunked datasets were introduced in 1.10.0. Current APIs and programming model still require to use H5Pset\_chunk but this call could be omitted if there is only one chunk (i.e, contiguos storage). Then compression can also be used on “contiguous” dataset. I couldn’t convince Quincey to introduce this change in 1.10.0 and the rest is history. Elena ---
**Author:** ![dave.allured](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/d/8e8cbc/32.png) [@dave.allured](https://forum.hdfgroup.org/u/dave.allured)\ **Post date:** [September 27, 2022, 5:21pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/22 "2022-09-27T17:21:13Z")
Requests: - Set filters with ID code strings, not numbers. - Set filter parameters with string keyword arguments. - Official registry for filter ID code strings. The current registry is a good start, if made official for existing ID code strings, not just the numbers. ---
**Author:** ![dave.allured](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/d/8e8cbc/32.png) [@dave.allured](https://forum.hdfgroup.org/u/dave.allured)\ **Post date:** [September 27, 2022, 6:07pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/23 "2022-09-27T18:07:07Z")
The use of contiguous storage is well entrenched in the HDF5 universe. It has certain advantages such as plain simplicity, and optimal subset access on local storage. I would prefer that support for contiguous is sustained. If compression is desired, just go to chunked storage, as intended by design. ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [September 27, 2022, 7:04pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/24 "2022-09-27T19:04:25Z")
Why should contiguous storage not just be a simple case of chunked storage? Contiguous storage is basically just chunked storage with a single chunk and no filters, no? ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [September 27, 2022, 8:32pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/25 "2022-09-27T20:32:46Z")
> [@dave.allured](#): > > ID code strings Could you give an example of an ID code string? ---
**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\ **Post date:** [September 27, 2022, 8:46pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/26 "2022-09-27T20:46:48Z")
> [@dave.allured](#): > > - Set filters with ID code strings, not numbers. I think I understand and agree with the spirit of this request. Its much easier to remember “gzip” or maybe “lzma2” as the identifier for a filter than “032105”. That said, can’t this already be achieved by adding a layer on top of the existing interface that keeps a mapping between strings and numbers? I don’t think the table would ever get so large that a linear search of it would have a negative performance impact. And, I honestly don’t think this needs to wait for an HDF5-2.0 or for THG to implement it to make it happen. It may already be implemented somewhere in the world 😉 ---
**Author:** ![dave.allured](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/d/8e8cbc/32.png) [@dave.allured](https://forum.hdfgroup.org/u/dave.allured)\ **Post date:** [September 27, 2022, 9:38pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/27 "2022-09-27T21:38:02Z")
“Filter ID code strings”. That is a mouthful, sorry, but I was trying to be complete in more than one way. I am referring to the “Name” column in [the HDF5 registry](https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins), in addition to simple names for the built-in filters. Here are a few examples. gzip, n-bit, scale-offset, shuffle, BZIP2, LPC-Rice ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [September 27, 2022, 9:43pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/28 "2022-09-27T21:43:20Z")
You mean like h5py? [https://docs.h5py.org/en/stable/high/dataset.html#filter-pipeline](https://docs.h5py.org/en/stable/high/dataset.html#filter-pipeline) ```auto dset = f.create_dataset("zipped", (100, 100), compression="gzip") ``` ---
**Author:** ![dave.allured](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/d/8e8cbc/32.png) [@dave.allured](https://forum.hdfgroup.org/u/dave.allured)\ **Post date:** [September 27, 2022, 9:53pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/29 "2022-09-27T21:53:29Z")
Yes, like that, but build the name registry into the HDF5 core library, so that standard filter names are available and guaranteed, outside of python. ---
**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\ **Post date:** [October 1, 2022, 4:56am UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/30 "2022-10-01T04:56:25Z")
Another thing to think about for HDF5 2.0…if you examine a lot of the functionality needed to manage metadata in an HDF5 file, you will find it is quite similar to what file systems have to do to manage their “media”. This kind of layering of the same abstractions, one upon the other, for purposes of providing ever more abstract storage objects is very similar to the [IP protocol stack](https://www.technologyuk.net/computing/computer-networks/internet/tcp-ip-stack.shtml). On the way hand, this layering permits implementation of various abstractions. On the other hand, it feels duplicative, wasteful and unnecessarily complex. Are there ways that an HDF5 2.0 could avoid this and instead utilize some of the pieces of lower-level media abstractions directly instead of reimplementing their own to produce an HDF5 “container” inside of a file system container (inside of a spinning disk container), etc. ---
**Author:** ![elvis.stansvik](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/46a35a/32.png) [@elvis.stansvik](https://forum.hdfgroup.org/u/elvis.stansvik)\ **Post date:** [October 1, 2022, 5:29pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/31 "2022-10-01T17:29:25Z")
> thread-parallel reads (and maybe writes) for same and different datasets in same file +1 This is the only thing I wish for 🙏 In general, thread-efficiency across the library, not just thread-safety through the mega-lock. ---
**Author:** ![vittynext](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/v/3bc359/32.png) [@vittynext](https://forum.hdfgroup.org/u/vittynext)\ **Post date:** [October 3, 2022, 10:24am UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/32 "2022-10-03T10:24:51Z")
Why should contiguous storage not just be a simple case of chunked storage? ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [October 9, 2022, 3:38am UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/33 "2022-10-09T03:38:43Z")
Thread-parallel reads is not too hard to pull off now or soon. 1. Locate the data ([`H5Dget_offset`](https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html#title21) or [`H5Dchunk_iter`](https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html#title6)). 2. Memory map the file (you may need to turn off locking). 3. Read the memory mapped file with multiple threads. 4. Do parallel decompression if needed What I really need the library to do is to efficiently tell me _where_ to find or put the data rather than actually doing it. ---
**Author:** ![philip.lee](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/958977/32.png) [@philip.lee](https://forum.hdfgroup.org/u/philip.lee)\ **Post date:** [October 15, 2022, 2:38pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/34 "2022-10-15T14:38:46Z")
UTF-16 for all strings. ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [October 18, 2022, 1:02pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/35 "2022-10-18T13:02:54Z")
I’m curious, why UTF-16 rather than UTF-8? ---
**Author:** ![philip.lee](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/958977/32.png) [@philip.lee](https://forum.hdfgroup.org/u/philip.lee)\ **Post date:** [October 18, 2022, 1:12pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/36 "2022-10-18T13:12:27Z")
.NET uses some form of UTF-16 internally, not sure about other platforms, so it would make translation to-from easier. Also UTF-8 requires 1-4 bytes per character which means that when creating fixed length strings we ideally should allocate 4x max string length to be safe. UTF-16 should hopefully mean only 2x although I’m aware it’s not that simple. ---
**Author:** ![ajelenak](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/3d9bf3/32.png) [@ajelenak](https://forum.hdfgroup.org/u/ajelenak)\ **Post date:** [October 21, 2022, 12:16pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/37 "2022-10-21T12:16:19Z")
I am not an HDF5 library developer but I think support for UTF-16 can likely be done even before “HDF5 2.0”. Strings in HDF5 are stored as bytes of a specific encoding. Currently only two encodings are recognized when defining a string datatype: ASCII and UTF-8. UTF-16 would just be another constant to add. It’s up to the client application to decode these bytes according to the datatype encoding information. ---
**Author:** ![philip.lee](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/958977/32.png) [@philip.lee](https://forum.hdfgroup.org/u/philip.lee)\ **Post date:** [October 21, 2022, 1:36pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/38 "2022-10-21T13:36:20Z")
Yes agreed, in my understanding the ‘encoding’ is just a hint to the client how to decode the stored bytes, it doesn’t affect anything in the library. I’ve gone for assuming everything it UTF-8 (and setting all appropriate encoding flags) since that’s a superset of ASCII. ---
**Author:** ![aragilar](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/aragilar/32/1744_2.png) [@aragilar](https://forum.hdfgroup.org/u/aragilar)\ **Post date:** [October 22, 2022, 9:42am UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/39 "2022-10-22T09:42:12Z")
The problem with UTF-16 is the not-uncommon issue that what is said to be UTF-16 is invalid UTF-16 (see [https://utf8everywhere.org/](https://utf8everywhere.org/) and [https://simonsapin.github.io/wtf-8/](https://simonsapin.github.io/wtf-8/)). UTF-8 is less commonly invalidly labelled (and can more easily be recognised as such through libraries like chardet). ---
**Author:** ![heiko.bauke](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/eb9ed0/32.png) [@heiko.bauke](https://forum.hdfgroup.org/u/heiko.bauke)\ **Post date:** [December 21, 2022, 1:14pm UTC](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/40 "2022-12-21T13:14:08Z")
.NET and Qt represent strings internally in UTF-16 because UTF-16 used to be the unicode representation of Windows NT and later versions. Though, UTF-8 is much more common today and even Windows is slowly moving to using UTF-8 as the default encoding, see [https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page](https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page) Thus, I am not sure if UTF-16 support is worth it. [Previous page](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003.md?page=1) [Next page](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003.md?page=3)