I’m working on covering more parts of the HDF5 specification in the pure python reader https://github.com/jjhelmus/pyfive (which we plan to use for recovering data from corrupt files).
Files that store groups while tracking the order (indexed groups) use a b-tree v2 with fractal heap. The specs are unclear about what a “heap ID” is (or at least I don’t get it):
Version 2 B-tree, Type 5 Record Layout - Link Name for Indexed Group
ID | This is a 7-byte sequence of bytes and is the heap ID for the link record in the group’s fractal heap.
Then when looking at the fractal heap
Fields: Fractal Heap Direct Block
Object Data | This section of the direct block stores the actual data for objects in the heap. The size of this section is determined by the direct block’s size minus the size of the other fields stored in the direct block (for example, the Signature, Version, and others including the Checksum if it is present).
How is the “heap ID” from the b-tree record related to the direct block data of the fractal heap? How do I know which direct block it is and what the offset+size is of the data in the block’s data?
As a more general question about fractal heaps: how is the “object data” from a direct block structured? Can data be extracted from this binary blob without knowing anything else?
2 Likes
The definition of the fractal heap ID can be found in the file format document after the definition of the fractal heap indirect block. As an overview, the ID can refer to a “tiny”, “huge”, or “managed” object. If it’s tiny, the ID contains the actual data and the heap itself does not need to be read from. If it’s huge, the ID contains the address on disk of the data or a b-tree key that can be used to find this address. If it’s managed, then it contains the offset and length within the virtual fractal heap address space (i.e. inside a direct block, possibly indexed by one or more indirect blocks). You can compute which direct and indirect blocks contains the data, and the offset within the direct block mathematically using the various parameters and algorithms described at the start of the fractal heap section - it describes an array of blocks of increasing size within a linear address space.
I hope that also answers your question about the data structure in general. It should be possible to extract data from a fractal heap using only a heap ID (and the sizeof offsets and sizeof lengths). Without a heap ID you could still walk through the heap and reconstruct a linear blob of bytes for managed data that information could potentially be extracted from, and individual huge objects could be pulled from the huge object b-tree if huge objects are indirectly accessed. Tiny objects and directly accessed huge objects would be lost though. The fractal heap stuff is a bit dense so let me know if you have further questions.
1 Like
So if I understand (correct me if I’m wrong):
- The “heap ID” from the records in a B-tree v2 are actually the tiny, huge or managed heap ID’s from III.G. Disk Format: Level 1G - Fractal Heap. The mixing of words “object ID” and “heap ID” got me confused. I thought the fractal heap was a store for object ID’s. So I was totally on the wrong track.
- If you concatenate all direct blocks from a fractal heap (including their header) then the offset in the heap ID of a manage object is the offset in this concatenate sequence (virtual linear address space).
- The data referenced by a heap ID from Type 5 and Type 6 B-tree v2 records is actually a link message (IV.A.2.g. The Link Message). So “objects” are actually “messages”.
None of these things were clear to me after reading the specs. Maybe it was just me or maybe this can help you improve the specs.
Thanks for your fast reply @nfortne2!
- Correct
- Correct. I’m not certain if all direct blocks are always allocated in order or not (if not there may be unallocated blocks you would have to fill in with zeroes), if it’s important I can look at the code and figure it out.
- Correct in the case of the fractal heap used to store link messages. Fractal heaps are used for other things which is why the fractal heap file format spec references uses the term “objects”. A “heap” in HDF5 is just a bucket for storing opaque packets of bytes, having no knowledge of their contents.
There are certainly places where the wording could be clearer. We’ll take your feedback into account when we get a chance to work on the spec. Thanks!
That shouldn’t affect things as you can assume the unallocated direct blocks take up as much space as they would if they were alocated. If you’re actually building up the serialized heap in memory you can just allocate enough space to include them. You shouldn’t even need to clear these sections because heap IDs should never point to unallocated blocks.
An unallocated direct block, if it is possible, would be represented by HADDR_UNDEF in the parent indirect block’s “Child Direct Block Address” field.
Ok that should answer all my questions. Thanks again @nfortne2!
Will data ever span direct blocks? I’m wondering if I would need to stitch together the virtual linear address space, or if I just need to be able to index an offset within it (which is easier)?
