# Fletcher32 checksum error with szip compression and 64bit data **URL:** \ **Category:** HDF5 Library\ **Created:** [December 13, 2017, 4:55pm UTC](https://forum.hdfgroup.org/t/fletcher32-checksum-error-with-szip-compression-and-64bit-data/4141 "2017-12-13T16:55:59Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![paul\_mueller](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/db5fbb/32.png) [@paul\_mueller](https://forum.hdfgroup.org/u/paul_mueller)\ **Post date:** [December 13, 2017, 4:55pm UTC](https://forum.hdfgroup.org/t/fletcher32-checksum-error-with-szip-compression-and-64bit-data/4141/1 "2017-12-13T16:55:59Z")
Dear All, I am using h5py to store 2D and 3D data sets. When combining the fletcher32 checksum with szip compression and 64 bit data (float, int numpy arrays), I get a checksum error when I try to read the data. I created an issue with h5py, but it seems like this could be a bug in HDF5: > > > \* Operating System: Ubuntu 16.04.3 LTS > \* Python versions: 2.7.12 and 3.5.2 > \*… Where Python was acquired: system Python (apt-get) > \* h5py version: 2.6.0 > \* HDF5 version: 1.8.16 > \* The full traceback/stack trace shown (Python 3): > \`\`\` > Traceback (most recent call last): > File "error.py", line 12, in \ > print(h5\["image\_A"\]\[0\]) > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > File "/usr/lib/python3/dist-packages/h5py/\_hl/dataset.py", line 482, in \_\_getitem\_\_ > self.id.read(mspace, fspace, arr, mtype, dxpl=self.\_dxpl) > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/h5d.c:3123) > File "h5py/\_proxy.pyx", line 130, in h5py.\_proxy.dset\_rw (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1769) > File "h5py/\_proxy.pyx", line 84, in h5py.\_proxy.H5PY\_H5Dread (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1411) > OSError: Can't read data (Data error detected by fletcher32 checksum) > > \`\`\` > error.py: > \`\`\`python > import h5py > import numpy as np > > with h5py.File("test.h5", "w") as h5: > h5.create\_dataset("image\_A", > data=np.zeros(10000, dtype=np.float64), > fletcher32=True, > compression="szip", > ) > > with h5py.File("test.h5", "r") as h5: > print(h5\["image\_A"\]\[0\]) > \`\`\` > > The error does not occur, if either: > \- \`\`compression\`\` is set to something other than "szip" > \- \`\`dtype\`\` is set to something other than 64 bit > > This can be very hard to reproduce (try to increase the number of zeros saved). The checksum error seems to be a result of szip compression with 64 bit numbers. Here is the example code: [0xF034E8E3.asc](https://hdf-discourse-1.s3.amazonaws.com/original/1X/98ac2d88dc31d5bb527a1d9c20937a6e4ca0f1e9.asc) (3.04 KB) > **···** > > -------------- > import h5py > import numpy as np > > with h5py.File("test.h5", "w") as h5: >     h5.create\_dataset("image\_A", >                       data=np.zeros(10000, dtype=np.float64), >                       fletcher32=True, >                       compression="szip", >                       ) > > with h5py.File("test.h5", "r") as h5: >     print(h5["image\_A"][0]) > -------------- > > and here is the error message: > > -------------- > Traceback (most recent call last): >   File "error.py", line 12, in \ >     print(h5["image\_A"][0]) >   File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) >   File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) >   File "/usr/lib/python3/dist-packages/h5py/\_hl/dataset.py", line 482, > in \_\_getitem\_\_ >     self.id.read(mspace, fspace, arr, mtype, dxpl=self.\_dxpl) >   File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) >   File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) >   File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/h5d.c:3123) >   File "h5py/\_proxy.pyx", line 130, in h5py.\_proxy.dset\_rw > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1769) >   File "h5py/\_proxy.pyx", line 84, in h5py.\_proxy.H5PY\_H5Dread > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1411) > OSError: Can't read data (Data error detected by fletcher32 checksum) > -------------- > > and this is my setup: > > \* Operating System: Ubuntu 16.04.3 LTS > \* Python versions: 2.7.12 and 3.5.2 > \* Where Python was acquired: system Python (apt-get) > \* h5py version: 2.7.1 > \* HDF5 version: 1.8.18 > \* The full traceback/stack trace shown (Python 3): > > I was not able to find anything in previous posts in the mailing list > archive. > > I would really like to use szip compression, because the resulting file > sizes are considerably smaller. > > Thanks for your help. > > Regards, > Paul ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 7, 2018, 11:20pm UTC](https://forum.hdfgroup.org/t/fletcher32-checksum-error-with-szip-compression-and-64bit-data/4141/2 "2018-01-07T23:20:29Z")
Hi Paul, I believe you need to specify szip first and then fletcher32. I just tried a C example (modified [https://support.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1\_10/C/H5D/h5ex\_d\_szip.c\](https://support.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_10/C/H5D/h5ex_d_szip.c%5C)) with HDF5 1.10.1 and it works when szip is the first filter in a filter pipeline. I am getting an error when filters are reversed. HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:   #000: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Dio.c line 171 in H5Dread(): can't read data     major: Dataset     minor: Read failed   #001: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Dio.c line 544 in H5D\_\_read(): can't read data     major: Dataset     minor: Read failed   #002: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Dchunk.c line 2050 in H5D\_\_chunk\_read(): unable to read raw data chunk     major: Low-level I/O     minor: Read failed   #003: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Dchunk.c line 3405 in H5D\_\_chunk\_lock(): data pipeline read failed     major: Data filters     minor: Filter operation failed   #004: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Z.c line 1374 in H5Z\_pipeline(): filter returned failure during read     major: Data filters     minor: Read failed   #005: /home/hdftest/snapshots-hdf5\_1\_10\_1/current/src/H5Zszip.c line 324 in H5Z\_filter\_szip(): szip\_filter: decompression failed     major: Resource unavailable     minor: No space available for allocation And it makes sense, because your buffer has 4 bytes with checksum in fog and szip is confused. I am attaching the C example that failed. If you comment out the first call to H5Pset\_fletcher32 and uncomment the second one, the program will work. Hope it will be an easy fix for your Python script. Thank you! Elena [szip\_example.c](https://hdf-discourse-1.s3.amazonaws.com/original/1X/c2304125b4aa20ee473667e7f927963781c5232c.c) (5.4 KB) > **···** > > On Dec 13, 2017, at 10:55 AM, Paul Müller \\> wrote: > > Dear All, > > I am using h5py to store 2D and 3D data sets. When combining the > fletcher32 checksum with szip compression and 64 bit data (float, int > numpy arrays), I get a checksum error when I try to read the data. > > I created an issue with h5py, but it seems like this could be a bug in HDF5: > > > > > > > \* Operating System: Ubuntu 16.04.3 LTS > > \* Python versions: 2.7.12 and 3.5.2 > > \*… Where Python was acquired: system Python (apt-get) > > \* h5py version: 2.6.0 > > \* HDF5 version: 1.8.16 > > \* The full traceback/stack trace shown (Python 3): > > \`\`\` > > Traceback (most recent call last): > > File "error.py", line 12, in \ > > print(h5\["image\_A"\]\[0\]) > > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > > File "/usr/lib/python3/dist-packages/h5py/\_hl/dataset.py", line 482, in \_\_getitem\_\_ > > self.id.read(mspace, fspace, arr, mtype, dxpl=self.\_dxpl) > > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > > File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/h5d.c:3123) > > File "h5py/\_proxy.pyx", line 130, in h5py.\_proxy.dset\_rw (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1769) > > File "h5py/\_proxy.pyx", line 84, in h5py.\_proxy.H5PY\_H5Dread (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1411) > > OSError: Can't read data (Data error detected by fletcher32 checksum) > > > > \`\`\` > > error.py: > > \`\`\`python > > import h5py > > import numpy as np > > > > with h5py.File("test.h5", "w") as h5: > > h5.create\_dataset("image\_A", > > data=np.zeros(10000, dtype=np.float64), > > fletcher32=True, > > compression="szip", > > ) > > > > with h5py.File("test.h5", "r") as h5: > > print(h5\["image\_A"\]\[0\]) > > \`\`\` > > > > The error does not occur, if either: > > \- \`\`compression\`\` is set to something other than "szip" > > \- \`\`dtype\`\` is set to something other than 64 bit > > > > This can be very hard to reproduce (try to increase the number of zeros saved). The checksum error seems to be a result of szip compression with 64 bit numbers. > > Here is the example code: > > -------------- > import h5py > import numpy as np > > with h5py.File("test.h5", "w") as h5: >    h5.create\_dataset("image\_A", >                      data=np.zeros(10000, dtype=np.float64), >                      fletcher32=True, >                      compression="szip", >                      ) > > with h5py.File("test.h5", "r") as h5: >    print(h5["image\_A"][0]) > -------------- > > and here is the error message: > > -------------- > Traceback (most recent call last): > File "error.py", line 12, in \ >    print(h5["image\_A"][0]) > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > File "/usr/lib/python3/dist-packages/h5py/\_hl/dataset.py", line 482, > in \_\_getitem\_\_ >    self.id.read(mspace, fspace, arr, mtype, dxpl=self.\_dxpl) > File "h5py/\_objects.pyx", line 54, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2577) > File "h5py/\_objects.pyx", line 55, in h5py.\_objects.with\_phil.wrapper > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_objects.c:2536) > File "h5py/h5d.pyx", line 181, in h5py.h5d.DatasetID.read > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/h5d.c:3123) > File "h5py/\_proxy.pyx", line 130, in h5py.\_proxy.dset\_rw > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1769) > File "h5py/\_proxy.pyx", line 84, in h5py.\_proxy.H5PY\_H5Dread > (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/\_proxy.c:1411) > OSError: Can't read data (Data error detected by fletcher32 checksum) > -------------- > > and this is my setup: > > \* Operating System: Ubuntu 16.04.3 LTS > \* Python versions: 2.7.12 and 3.5.2 > \* Where Python was acquired: system Python (apt-get) > \* h5py version: 2.7.1 > \* HDF5 version: 1.8.18 > \* The full traceback/stack trace shown (Python 3): > > I was not able to find anything in previous posts in the mailing list > archive. > > I would really like to use szip compression, because the resulting file > sizes are considerably smaller. > > Thanks for your help. > > Regards, > Paul > \<0xF034E8E3.asc\>\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Hdf-forum is for HDF software users discussion. > Hdf-forum@lists.hdfgroup.org > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > Twitter: [https://twitter.com/hdf5](https://twitter.com/hdf5) ---
**Author:** ![paul\_mueller](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/db5fbb/32.png) [@paul\_mueller](https://forum.hdfgroup.org/u/paul_mueller)\ **Post date:** [April 11, 2018, 9:51pm UTC](https://forum.hdfgroup.org/t/fletcher32-checksum-error-with-szip-compression-and-64bit-data/4141/3 "2018-04-11T21:51:22Z")
Dear Elena, thanks again for your thorough answer. I created a PR which just got merged: [https://github.com/h5py/h5py/pull/989](https://github.com/h5py/h5py/pull/989) It looks like this bug will not be present in h5py \>= 2.7.2. Cheers, Paul