# SWMR - extend dataset by 1000 records and write records one by one

**URL:** <https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363>\
**Category:** HDF5 Library\
**Created:** [February 5, 2019, 11:02am UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363 "2019-02-05T11:02:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![maoz.guttman](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/4491bb/32.png) [@maoz.guttman](https://forum.hdfgroup.org/u/maoz.guttman)\
**Post date:** [February 5, 2019, 11:02am UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363/1 "2019-02-05T11:02:00Z")

</div>

Hi,

My question is on SWMR. I have an one dimension array with unlimited size. In the writer, I get the records one by one (not in a block of records).  
My implementation is based on swmr\_addrem\_writer.c from hdf5 tests.  
For each record, I call H5Dset\_extent, H5Dget\_space, H5Sselect\_hyperslab, and H5Dwrite.  
I’m trying to improve runtime performance by calling H5Dset\_extent and H5Dget\_space less. I extended the dataset by 1000 Instead of by 1, and then called H5Sselect\_hyperslab, and H5Dwrite 1000 times for each record. I got ~50% runtime improvement. For example in pseudo code:

```
hsize_t size[] = {1000};
H5Dset_extent(dataset, size);
hid = hidFileSpace = H5Dget_space(dataset);
for (int i = 0; i < 1000; i++)
{
  H5Sselect_hyperslab ... // move to last record
  H5Dwrite ... // write one record
}

```

The output \*.h5 is fine. Just have “zeroed” records at the end of the dataset for the unwritten records (1000 - H5Dwrite calls in the last chunk).

Is it safe to do it? Will the SWMR reader see “zeroed” records during the writing (in the middle/end)?  
Is it safe from SWMR point of view to reduce the dataset size to the exact size with H5Dset\_extent just before closing the dataset?

Thanks,  
Maoz

---

<div class="post-metadata">

**Author:** ![steven](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/steven/32/117_2.png) [@steven](https://forum.hdfgroup.org/u/steven)\
**Post date:** [February 5, 2019, 11:13am UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363/2 "2019-02-05T11:13:51Z")

</div>

May I ask what is the throughout you are getting with this approach?

sizeof(object) \* nelements / 1 second and type of computer / hdd

---

<div class="post-metadata">

**Author:** ![maoz.guttman](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/4491bb/32.png) [@maoz.guttman](https://forum.hdfgroup.org/u/maoz.guttman)\
**Post date:** [February 5, 2019, 12:51pm UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363/3 "2019-02-05T12:51:55Z")

</div>

I’m trying to improve writer run-time by calling hdf5 APIs less. I got an improvement, but don’t know if it will \* work when SWMR reader is running in parallel.

```
Common for both cases:
Computer: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Hdd: NFS - I know SWMR is not supported on it. I'm currently developing a writer as a POC.
OS: Sles11
I/O from strace:
* write: 1055
* lseek: 48
* other: 10
H5Pset_chunk is set to 1000 and H5Pset_chunk_cache is set to allocate all 1000 records in the cache.

Without improvement (extend by 1):
--------------------------------------------------------------
| dataset | sizeof(object) | number of elements | Total size |
| --------|----------------|--------------------|------------|
| 1 | 32 | 163,940 | 5,246,080 |
| 2 | 24 | 180,324 | 4,327,776 |
| 3 | 32 | 163,930 | 5,245,760 |
| 4 | 32 | 163,930 | 5,245,760 |
| 5 | 24 | 163,930 | 3,934,320 |
| 6 | 32 | 163,930 | 5,245,760 |
|---------|----------------|--------------------|------------|
| Total | | | 29,245,456 |
--------------------------------------------------------------
file size (bytes) : 29,306,340
user time (seconds) : 12.58
system time (seconds) : 0.21
total time (seconds) : 12.79
write rate (bytes/second): 2,286,587

With improvement (extend by 1000):
--------------------------------------------------------------
| dataset | sizeof(object) | number of elements | Total size |
| --------|----------------|--------------------|------------|
| 1 | 32 | 164,000 | 5,248,000 |
| 2 | 24 | 181,000 | 4,344,000 |
| 3 | 32 | 164,000 | 5,248,000 |
| 4 | 32 | 164,000 | 5,248,000 |
| 5 | 24 | 164,000 | 3,936,000 |
| 6 | 32 | 164,000 | 5,248,000 |
|---------|----------------|--------------------|------------|
| Total | | | 29,272,000 |
--------------------------------------------------------------
file size (bytes) : 29,306,340
user time (seconds) : 7.28
system time (seconds) : 0.18
total time (seconds) : 7.46
write rate (bytes/second): 3,923,860

```

Write rate is up from 2,286,587 to 3,923,860.

Same test case when dumping a binary file with C++ ofstream gives ~4.5X writing rate comparing to hdf5.  
Total run-time that I used in the writing rate formula includes the “overhead” that generates the objects from a simulator.

Thanks,  
Maoz

---

<div class="post-metadata">

**Author:** ![paramon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/8491ac/32.png) [@paramon](https://forum.hdfgroup.org/u/paramon)\
**Post date:** [February 5, 2019, 1:02pm UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363/4 "2019-02-05T13:02:10Z")

</div>

Hi Maoz!

Extending by one record is no way currently, but might it be that the bug is fixed someday  
(or at least, enters JIRA)?

> [@\[BUG\] Compression doesn't compress data on SWMR](https://forum.hdfgroup.org/t/bug-compression-doesnt-compress-data-on-swmr/5280):
>
> Hello! To reproduce: Build and run attached [test980.c](https://hdf-discourse-1.s3.amazonaws.com/original/2X/d/d0e3c52264470ae2f2e33e99b68092cb1c75417c.c) (1.4 KB) Observe the size of test.h5 is 5,705 bytes (correct). Uncomment H5Fstart\_swmr\_write(file); rebuild and run. Now the file size exploded to 1,913,763 bytes! Originally reported in in [Compression doesn't compress data on SWMR · Issue #980 · h5py/h5py · GitHub](https://github.com/h5py/h5py/issues/980) (see more complete example there for motivation behind H5Fflushing repeatedly). Best wishes, Andrey Paramonov

Best wishes,  
Andrey Paramonov

---

<div class="post-metadata">

**Author:** ![steven](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/steven/32/117_2.png) [@steven](https://forum.hdfgroup.org/u/steven)\
**Post date:** [February 5, 2019, 9:49pm UTC](https://forum.hdfgroup.org/t/swmr-extend-dataset-by-1000-records-and-write-records-one-by-one/5363/5 "2019-02-05T21:49:21Z")

</div>

The reason I was asking for some metrics is to ballpark your effort. In the current version of H5CPP the record/element wise read/write rate is about on par with raw C IO: on a Lenovo x230 laptop **350MB/sec** for datasets size greater than system memory. And **1.5 - 3.5 GB/sec** for small burts IO requests.

The comparison was against [armadillo](http://arma.sourceforge.net/) **save/load:** arma\_binary, raw\_binary.  
Therefore if it is for speed improvement please consider evaluating H5CPP h5::append operator performance and or the techniques used there.  
Although h5::append filter chain is not plugged in yet, the operator is functional with good IO properties and can take arbitrary T :== POD types | fundamental types and stl::vector and major linear algebra objects upto rank 3.

here is an example to append an **std::vector** from examples/packet\_table

```cpp
...	
h5::fd_t fd = h5::create("example.h5",H5F_ACC_TRUNC);
// dataset descriptor is automatically converted to packet table descriptor
h5::pt_t pt = h5::create<double>(fd, "stream of vectors",
	h5::max_dims{H5S_UNLIMITED,5}, h5::chunk{1,5} );

	arma::vec V(5); for(int i=0; i<5; i++) V(i) = i;
// the actual append operation
	for( int i = 0; i < 7; i++)
	    h5::append( pt, V);

```