# Storing an unknown number of arrays

**URL:** <https://forum.hdfgroup.org/t/storing-an-unknown-number-of-arrays/3393>\
**Category:** Hdf-forum archives\
**Created:** [January 14, 2015, 2:01pm UTC](https://forum.hdfgroup.org/t/storing-an-unknown-number-of-arrays/3393 "2015-01-14T14:01:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Robert\_van\_Loenhout](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/d6d6ee/32.png) [@Robert\_van\_Loenhout](https://forum.hdfgroup.org/u/Robert_van_Loenhout)\
**Post date:** [January 14, 2015, 2:01pm UTC](https://forum.hdfgroup.org/t/storing-an-unknown-number-of-arrays/3393/1 "2015-01-14T14:01:14Z")

</div>

Hi,

What would be the best way of storing in hdf5 a large number of arrays (millions) that all have the same size, however when writing the arrays, you do not yet know how many there are.  
Could you use a multi-dimensional array and write blocks to it? Is there any way to read back how many blocks have been written to it?

Regards,  
Robert

---

<div class="post-metadata">

**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\
**Post date:** [January 18, 2015, 11:35pm UTC](https://forum.hdfgroup.org/t/storing-an-unknown-number-of-arrays/3393/2 "2015-01-18T23:35:51Z")

</div>

Robert,

> **···**
>
> On Jan 14, 2015, at 8:01 AM, Robert van Loenhout \<[vanLoenhout@riscure.com](mailto:vanLoenhout@riscure.com)\<[mailto:vanLoenhout@riscure.com](mailto:vanLoenhout@riscure.com)\>\> wrote:
> 
> Hi,
> 
> What would be the best way of storing in hdf5 a large number of arrays (millions) that all have the same size, however when writing the arrays, you do not yet know how many there are.
> 
> Data arrays are stored in HDF5 as HDF5 datasets. There is no restriction on the number of datasets in HDF5 file. Are you familiar with the HDF5 datasets? If not, please take a look at the HDF5 Tutorial [http://www.hdfgroup.org/HDF5/Tutor/](http://www.hdfgroup.org/HDF5/Tutor/) and the misc. examples [http://www.hdfgroup.org/HDF5/examples/api18-c.html](http://www.hdfgroup.org/HDF5/examples/api18-c.html)
> 
> Could you use a multi-dimensional array and write blocks to it?
> 
> What do you mean by a “block”? A sub-array? One can write or read a subset of an array. The HDF5 subsets may be quite complex. The simplest subsets are an array element (a point) or a sub-array (a hyperslab).  
> Is there any way to read back how many blocks have been written to it?  
> There are functions to find out the current size of the dataset and amount of space used by its data in the file.
> 
> Elena
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
> Elena Pourmal The HDF Group [http://hdfgroup.org](http://hdfgroup.org)  
> 1800 So. Oak St., Suite 203, Champaign IL 61820  
> 217.531.6112  
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Regards,  
> Robert
> 
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> Hdf-forum is for HDF software users discussion.  
> [Hdf-forum@lists.hdfgroup.org](mailto:Hdf-forum@lists.hdfgroup.org)\<[mailto:Hdf-forum@lists.hdfgroup.org](mailto:Hdf-forum@lists.hdfgroup.org)\>  
> [http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org)  
> Twitter: [https://twitter.com/hdf5](https://twitter.com/hdf5)