User-defined filter parameter parsing broken in h5repack?

Hdf-forum archives
1

Consider the attached noop_plugin.c. It contains a dynamically loaded
filter plugin, with filter number 42, which does nothing except print out
the received `cd_values` array in the filter function.

Build the test plugin (on *NIX) with `make`.

Then test the following:

estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
input.hdf5 output.hdf5
1
estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
input.hdf5 output.hdf5
2 2
estan@newton:~/noop_plugin$

Notice how in the second invokation (with UD=42,2,1,2), the cd_values
contains { 2, 2 }, despite { 1, 2 } being passed on the command line.

Has anyone seen this strangeness before, or can reproduce?

Thanks in advance,
Elvis Stansvik

noop_plugin.tar.gz (929 Bytes)

2

Consider the attached noop_plugin.c. It contains a dynamically loaded
filter plugin, with filter number 42, which does nothing except print out
the received `cd_values` array in the filter function.

Build the test plugin (on *NIX) with `make`.

Then test the following:

estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
input.hdf5 output.hdf5
1
estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
input.hdf5 output.hdf5
2 2
estan@newton:~/noop_plugin$

Notice how in the second invokation (with UD=42,2,1,2), the cd_values
contains { 2, 2 }, despite { 1, 2 } being passed on the command line.

I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.

Elvis

···

2016-08-22 10:16 GMT+02:00 Elvis Stansvik <elvis.stansvik@orexplore.com>:

Has anyone seen this strangeness before, or can reproduce?

Thanks in advance,
Elvis Stansvik

3

I can reproduce on OS X, same version of HDF5. Compiling the library with --enable-debug and getting the trace (HDF5_DEBUG=“trace”) shows the following call:

H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0, cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;

So I bet it's a bug in h5repack.

Andrea

···

On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <elvis.stansvik@orexplore.com> wrote:

2016-08-22 10:16 GMT+02:00 Elvis Stansvik <elvis.stansvik@orexplore.com>:
Consider the attached noop_plugin.c. It contains a dynamically loaded filter plugin, with filter number 42, which does nothing except print out the received `cd_values` array in the filter function.

Build the test plugin (on *NIX) with `make`.

Then test the following:

estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1 input.hdf5 output.hdf5
1
estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2 input.hdf5 output.hdf5
2 2
estan@newton:~/noop_plugin$

Notice how in the second invokation (with UD=42,2,1,2), the cd_values contains { 2, 2 }, despite { 1, 2 } being passed on the command line.

I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.

Elvis

Has anyone seen this strangeness before, or can reproduce?

Thanks in advance,
Elvis Stansvik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Andrea Bedini
@andreabedini, http://www.andreabedini.com

See the impact of my research at https://impactstory.org/AndreaBedini
use https://keybase.io/andreabedini to send me encrypted messages
Key fingerprint = 17D5 FB49 FA18 A068 CF53 C5C2 9503 64C1 B2D5 9591

4

I can reproduce on OS X, same version of HDF5. Compiling the library with
--enable-debug and getting the trace (HDF5_DEBUG=“trace”) shows the
following call:

H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0,
cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;

So I bet it's a bug in h5repack.

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The
parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I
haven't fully understood it yet. But I suspect it might be ignoring
anything past the first user parameter, so anything I'm seeing past that is
just uninialized memory.. not sure though.

Elvis

···

2016-08-22 14:21 GMT+02:00 Andrea Bedini <andrea@andreabedini.com>:

Andrea

> On 22 Aug 2016, at 4:18 PM, Elvis Stansvik <elvis.stansvik@orexplore.com> > wrote:
>
> 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <elvis.stansvik@orexplore.com>
:
> Consider the attached noop_plugin.c. It contains a dynamically loaded
filter plugin, with filter number 42, which does nothing except print out
the received `cd_values` array in the filter function.
>
> Build the test plugin (on *NIX) with `make`.
>
> Then test the following:
>
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
input.hdf5 output.hdf5
> 1
> estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
input.hdf5 output.hdf5
> 2 2
> estan@newton:~/noop_plugin$
>
> Notice how in the second invokation (with UD=42,2,1,2), the cd_values
contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
>
> I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
>
> Elvis
>
>
> Has anyone seen this strangeness before, or can reproduce?
>
> Thanks in advance,
> Elvis Stansvik
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

--
Andrea Bedini
@andreabedini, http://www.andreabedini.com

See the impact of my research at https://impactstory.org/AndreaBedini
use https://keybase.io/andreabedini to send me encrypted messages
Key fingerprint = 17D5 FB49 FA18 A068 CF53 C5C2 9503 64C1 B2D5 9591

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Hi Bálint,

Thank you for the patch! I entered bug HDFFV-9974 for this issue.

-Barbara

···

========================================================
Barbara Jones, The HDF Group Helpdesk, help@hdfgroup.org
Support Services: https://www.hdfgroup.org/services/

On Tue, 23 Aug 2016, Bálint Balázs wrote:

Dear HDF Group,

I recently discovered an issue with h5repack: the parameter parsing for
user defined filters doesn't work properly, and because of this it's not
possible to use these filters with h5repack. This issue was also
recently discussed on the HDF5 forum:
http://hdf-forum.184993.n3.nabble.com/User-defined-filter-parameter-parsing-broken-in-h5repack-td4029137.html

I made some modifications to fix the issue in the 1.8.17 version of
h5repack_parse.c and successfully compiled it for Windows. Please find
the file in the attachment. If you find it appropriate, could you
include it in a newer version of HDF5?

Kind regards,
Balint Balazs

--

6

Hi Bálint,

Thank you for the patch! I entered bug HDFFV-9974 for this issue.

Thanks Barbara,

Is there a way to follow bugs? (Some open bug tracker?) Or is that some
internal bug number?

Elvis

···

2016-08-23 15:45 GMT+02:00 <help@hdfgroup.org>:

-Barbara

========================================================
Barbara Jones, The HDF Group Helpdesk, help@hdfgroup.org
Support Services: https://www.hdfgroup.org/services/

On Tue, 23 Aug 2016, Bálint Balázs wrote:

Dear HDF Group,

I recently discovered an issue with h5repack: the parameter parsing for
user defined filters doesn't work properly, and because of this it's not
possible to use these filters with h5repack. This issue was also
recently discussed on the HDF5 forum:
http://hdf-forum.184993.n3.nabble.com/User-defined-filter-
parameter-parsing-broken-in-h5repack-td4029137.html

I made some modifications to fix the issue in the 1.8.17 version of
h5repack_parse.c and successfully compiled it for Windows. Please find
the file in the attachment. If you find it appropriate, could you
include it in a newer version of HDF5?

Kind regards,
Balint Balazs

--
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

7

Hi Elvis,

Our bug tracking system is not open to the public (but we are planning to change this).
Please feel free to contact the helpdesk (help@hdfgroup.org<mailto:help@hdfgroup.org>) if you ever want to check on the status of an issue.

-Barbara

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elvis Stansvik
Sent: Thursday, August 25, 2016 2:28 AM
To: HDF Users Discussion List
Cc: Bálint Balázs
Subject: Re: [Hdf-forum] User-defined filter parameter parsing broken in h5repack

2016-08-23 15:45 GMT+02:00 <help@hdfgroup.org<mailto:help@hdfgroup.org>>:
Hi Bálint,

Thank you for the patch! I entered bug HDFFV-9974 for this issue.

Thanks Barbara,
Is there a way to follow bugs? (Some open bug tracker?) Or is that some internal bug number?
Elvis

-Barbara

========================================================
Barbara Jones, The HDF Group Helpdesk, help@hdfgroup.org<mailto:help@hdfgroup.org>
Support Services: https://www.hdfgroup.org/services/

On Tue, 23 Aug 2016, Bálint Balázs wrote:
Dear HDF Group,

I recently discovered an issue with h5repack: the parameter parsing for
user defined filters doesn't work properly, and because of this it's not
possible to use these filters with h5repack. This issue was also
recently discussed on the HDF5 forum:
http://hdf-forum.184993.n3.nabble.com/User-defined-filter-parameter-parsing-broken-in-h5repack-td4029137.html

I made some modifications to fix the issue in the 1.8.17 version of
h5repack_parse.c and successfully compiled it for Windows. Please find
the file in the attachment. If you find it appropriate, could you
include it in a newer version of HDF5?

Kind regards,
Balint Balazs

--
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

8

Alright, thanks. Looking forward to when it's open.

Elvis

···

2016-08-25 14:51 GMT+02:00 Barbara Jones <bljones@hdfgroup.org>:

Hi Elvis,

Our bug tracking system is not open to the public (but we are planning to
change this).

Please feel free to contact the helpdesk (help@hdfgroup.org) if you ever
want to check on the status of an issue.

-Barbara

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Elvis Stansvik
*Sent:* Thursday, August 25, 2016 2:28 AM
*To:* HDF Users Discussion List
*Cc:* Bálint Balázs
*Subject:* Re: [Hdf-forum] User-defined filter parameter parsing broken
in h5repack

2016-08-23 15:45 GMT+02:00 <help@hdfgroup.org>:

Hi Bálint,

Thank you for the patch! I entered bug HDFFV-9974 for this issue.

Thanks Barbara,

Is there a way to follow bugs? (Some open bug tracker?) Or is that some
internal bug number?

Elvis

-Barbara

========================================================
Barbara Jones, The HDF Group Helpdesk, help@hdfgroup.org
Support Services: https://www.hdfgroup.org/services/

On Tue, 23 Aug 2016, Bálint Balázs wrote:

Dear HDF Group,

I recently discovered an issue with h5repack: the parameter parsing for
user defined filters doesn't work properly, and because of this it's not
possible to use these filters with h5repack. This issue was also
recently discussed on the HDF5 forum:
http://hdf-forum.184993.n3.nabble.com/User-defined-
filter-parameter-parsing-broken-in-h5repack-td4029137.html

I made some modifications to fix the issue in the 1.8.17 version of
h5repack_parse.c and successfully compiled it for Windows. Please find
the file in the attachment. If you find it appropriate, could you
include it in a newer version of HDF5?

Kind regards,
Balint Balazs

--
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

9

Hi Elvis,

I hope your rebuild will work.

I actually haven't reported this yet (will do now), but I think regarding
the nature of this change, it's very likely they will include it in a
future version.

Balint

···

2016-08-22 15:48 GMT+02:00 <hdf-forum-request@lists.hdfgroup.org>:

Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems
it's not so easy to just recompile the h5repack tool. But I might try a
full build of HDF5 at some point. Your corrected h5repack_parse.c will most
probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it
will be fixed in a future version?

Elvis

10

Hi Elvis,

I hope your rebuild will work.

I actually haven't reported this yet (will do now), but I think regarding
the nature of this change, it's very likely they will include it in a
future version.

Alright, thanks a lot.

Elvis

···

2016-08-23 13:16 GMT+02:00 Bálint Balázs <bluqesh@gmail.com>:

Balint

2016-08-22 15:48 GMT+02:00 <hdf-forum-request@lists.hdfgroup.org>:

Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems
it's not so easy to just recompile the h5repack tool. But I might try a
full build of HDF5 at some point. Your corrected h5repack_parse.c will
most
probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it
will be fixed in a future version?

Elvis

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

11

Hi Elvis,

I also had this problem a few weeks ago on Windows 7, both version 1.8.15
and 1.8.17. I managed to fix it in version 1.8.17, it was indeed in
h5repack_parse.c, as you suspected. Please find attached the corrected
file, hopefully if will work with 1.8.16 as well.

Balint

h5repack_parse.zip (4.38 KB)

···

2016-08-22 14:43 GMT+02:00 <hdf-forum-request@lists.hdfgroup.org>:

Send Hdf-forum mailing list submissions to
        hdf-forum@lists.hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_
lists.hdfgroup.org

or, via email, send a message with subject or body 'help' to
        hdf-forum-request@lists.hdfgroup.org

You can reach the person managing the list at
        hdf-forum-owner@lists.hdfgroup.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."

Today's Topics:

   1. Re: HDF5 running on ARM V 7 (Gerd Heber)
   2. Re: User-defined filter parameter parsing broken in h5repack?
      (Elvis Stansvik)

----------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
hdfgroup.org/attachments/20160822/d3e1f46f/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 22 Aug 2016 14:42:21 +0200
From: Elvis Stansvik <elvis.stansvik@orexplore.com>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] User-defined filter parameter parsing broken
        in h5repack?
Message-ID:
        <CAE-7VJk1AF7P088CpFU+_eCzkJ1fs3Q7+mUREeNDa6dCGd4+6g@
mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

2016-08-22 14:21 GMT+02:00 Andrea Bedini <andrea@andreabedini.com>:

> I can reproduce on OS X, same version of HDF5. Compiling the library with
> --enable-debug and getting the trace (HDF5_DEBUG=?trace?) shows the
> following call:
>
> H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0,
> cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;
>
> So I bet it's a bug in h5repack.
>

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The
parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I
haven't fully understood it yet. But I suspect it might be ignoring
anything past the first user parameter, so anything I'm seeing past that is
just uninialized memory.. not sure though.

Elvis

> Andrea
>
>
> > On 22 Aug 2016, at 4:18 PM, Elvis Stansvik < > elvis.stansvik@orexplore.com> > > wrote:
> >
> > 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <
elvis.stansvik@orexplore.com>
> :
> > Consider the attached noop_plugin.c. It contains a dynamically loaded
> filter plugin, with filter number 42, which does nothing except print out
> the received `cd_values` array in the filter function.
> >
> > Build the test plugin (on *NIX) with `make`.
> >
> > Then test the following:
> >
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
> input.hdf5 output.hdf5
> > 1
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,2,1,2
> input.hdf5 output.hdf5
> > 2 2
> > estan@newton:~/noop_plugin$
> >
> > Notice how in the second invokation (with UD=42,2,1,2), the cd_values
> contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
> >
> > I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
> >
> > Elvis
> >
> >
> > Has anyone seen this strangeness before, or can reproduce?
> >
> > Thanks in advance,
> > Elvis Stansvik
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@lists.hdfgroup.org
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_
lists.hdfgroup.org
> > Twitter: https://twitter.com/hdf5
>
> --
> Andrea Bedini
> @andreabedini, http://www.andreabedini.com
>
> See the impact of my research at https://impactstory.org/AndreaBedini
> use https://keybase.io/andreabedini to send me encrypted messages
> Key fingerprint = 17D5 FB49 FA18 A068 CF53 C5C2 9503 64C1 B2D5 9591
>
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.
hdfgroup.org/attachments/20160822/812a0c74/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

------------------------------

End of Hdf-forum Digest, Vol 86, Issue 20
*****************************************

12

Hi Elvis,

I also had this problem a few weeks ago on Windows 7, both version 1.8.15
and 1.8.17. I managed to fix it in version 1.8.17, it was indeed in
h5repack_parse.c, as you suspected. Please find attached the corrected
file, hopefully if will work with 1.8.16 as well.

Thanks a lot for sharing Balint.

I haven't built HDF5 myself, but using the Ubuntu package, and it seems
it's not so easy to just recompile the h5repack tool. But I might try a
full build of HDF5 at some point. Your corrected h5repack_parse.c will most
probably work fine.

Did you report the issue to the HDF5 maintainers, and do you know if it
will be fixed in a future version?

Elvis

···

2016-08-22 15:09 GMT+02:00 Bálint Balázs <bluqesh@gmail.com>:

Balint

2016-08-22 14:43 GMT+02:00 <hdf-forum-request@lists.hdfgroup.org>:

Send Hdf-forum mailing list submissions to
        hdf-forum@lists.hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.
hdfgroup.org

or, via email, send a message with subject or body 'help' to
        hdf-forum-request@lists.hdfgroup.org

You can reach the person managing the list at
        hdf-forum-owner@lists.hdfgroup.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."

Today's Topics:

   1. Re: HDF5 running on ARM V 7 (Gerd Heber)
   2. Re: User-defined filter parameter parsing broken in h5repack?
      (Elvis Stansvik)

----------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgrou
p.org/attachments/20160822/d3e1f46f/attachment-0001.html>

------------------------------

Message: 2
Date: Mon, 22 Aug 2016 14:42:21 +0200
From: Elvis Stansvik <elvis.stansvik@orexplore.com>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] User-defined filter parameter parsing broken
        in h5repack?
Message-ID:
        <CAE-7VJk1AF7P088CpFU+_eCzkJ1fs3Q7+mUREeNDa6dCGd4+6g@mail.
gmail.com>
Content-Type: text/plain; charset="utf-8"

2016-08-22 14:21 GMT+02:00 Andrea Bedini <andrea@andreabedini.com>:

> I can reproduce on OS X, same version of HDF5. Compiling the library
with
> --enable-debug and getting the trace (HDF5_DEBUG=?trace?) shows the

> following call:
>
> H5Pset_filter(plist=167772178 (genprop list), filter=42, flags=0,
> cd_nelmts=2, cd_values=0x7fff50f2b10c {2, 2}) = SUCCEED;
>
> So I bet it's a bug in h5repack.
>

Thanks for testing Andrea. Yes I'm pretty sure it's a parsing bug. The
parsing code in tools/h5repack/h5repack_parse.c looks pretty gnarly and I
haven't fully understood it yet. But I suspect it might be ignoring
anything past the first user parameter, so anything I'm seeing past that
is
just uninialized memory.. not sure though.

Elvis

> Andrea
>
>
> > On 22 Aug 2016, at 4:18 PM, Elvis Stansvik < >> elvis.stansvik@orexplore.com> >> > wrote:
> >
> > 2016-08-22 10:16 GMT+02:00 Elvis Stansvik <
elvis.stansvik@orexplore.com>
> :
> > Consider the attached noop_plugin.c. It contains a dynamically loaded
> filter plugin, with filter number 42, which does nothing except print
out
> the received `cd_values` array in the filter function.
> >
> > Build the test plugin (on *NIX) with `make`.
> >
> > Then test the following:
> >
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f UD=42,1,1
> input.hdf5 output.hdf5
> > 1
> > estan@newton:~/noop_plugin$ HDF5_PLUGIN_PATH=. h5repack -f
UD=42,2,1,2
> input.hdf5 output.hdf5
> > 2 2
> > estan@newton:~/noop_plugin$
> >
> > Notice how in the second invokation (with UD=42,2,1,2), the cd_values
> contains { 2, 2 }, despite { 1, 2 } being passed on the command line.
> >
> > I forgot to say: This was with HDF5 1.8.16 on Ubuntu 16.04.
> >
> > Elvis
> >
> >
> > Has anyone seen this strangeness before, or can reproduce?
> >
> > Thanks in advance,
> > Elvis Stansvik
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@lists.hdfgroup.org
> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.
hdfgroup.org
> > Twitter: https://twitter.com/hdf5
>
> --
> Andrea Bedini
> @andreabedini, http://www.andreabedini.com
>
> See the impact of my research at https://impactstory.org/AndreaBedini
> use https://keybase.io/andreabedini to send me encrypted messages
> Key fingerprint = 17D5 FB49 FA18 A068 CF53 C5C2 9503 64C1 B2D5 9591
>
>
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgrou
p.org/attachments/20160822/812a0c74/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

------------------------------

End of Hdf-forum Digest, Vol 86, Issue 20
*****************************************

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

13

Sorry to bring up this old issue again, but the problem is present in the 1.10.0 release (the one in Ubuntu 18.04 repositories). The test case I posted still produces those strange values.

Was the fix never applied to 1.10? Only 1.8?

This prevents h5repack from being used correctly with e.g. the Blosc compression filter (which has 7 parameters) :frowning:

Would have been good if the fix was applied also in 1.10, since we use the Ubuntu packaged version it’s too late now, unless we get Ubuntu to incorporate the patch in their package.

14

I’ve now also tested with the develop branch from Git, and the problem remains there as well.

15

I have double-checked and the source code in h5repack_parse.c in develop, 1.10 and 1.8 are exactly the same (except for a single whitespace character in 1.8). Our dev process is to change the develop code base first, test and review the changes before merging to 1.10 and finally 1.8 (if necessary).
This particular fix was completed on 2016/10/19 for 1.10.1 release.

So we will need to look for another explanation.

Allen

16

I’ve just run into this problem with h5repack ignoring some of the parameters:

h5repack -L -f UD=32001,7,0,0,0,0,1,1,1 input.h5 output.h5

The first 4 parameters are set to zero because the hdf5 blosc plugin reserves and inserts these (blosc version, blosc filter version format, datatype size, chunk size: https://github.com/Blosc/hdf5-blosc/blob/master/src/blosc_filter.c#L97)

So I only explicitly set the remaining 3 parameters: compression level, shuffle, compression algorithm.

However, the output seems to only include the 4 reserved parameters:

h5dump -pH out3.h5 
HDF5 "out3.h5" {
GROUP "/" {
   DATASET "data" {
      DATATYPE  H5T_STD_U16LE
      DATASPACE  SIMPLE { ( 100, 1536, 2048 ) / ( H5S_UNLIMITED, 1536, 2048 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 1, 1536, 2048 )
         SIZE 8244247 (76.313:1 COMPRESSION)
      }
      FILTERS {
         USER_DEFINED_FILTER {
            FILTER_ID 32001
            COMMENT blosc
            PARAMS { 2 2 2 6291456 }      <==== Missing the last three parameters (1,1,1) here
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE  0
      }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}
}

I have the same behaviour in h5repack from hdf5 1.10.2 and 1.10.3.

17

I have checked our blosc filter tests and they correctly display the parameters. The test actually steps through the different compression types used by blosc. The blosc version is 1.11.1.

Allen

18

Do these tests exercise h5repack?

I think the real problem is really with h5repack and its parsing of parameters for user-defined filters. I think it ignores parameters after the first one and the blosc filter just happens to “work” because it has a default set of parameters (but with unexpected results, like not using the desired compression algorithm).

I took a python snippet from Elvis and modified it to load an (uncompressed) dataset from another file - and then write it out in compressed chunks, using the compression parameters 0, 0, 0, 0, 1, 1, 1.

import h5py
import sys

input_file = sys.argv[1]

with h5py.File(input_file, 'r') as infile:
    input_data = infile['data']
    with h5py.File('output.hdf5', 'w') as outfile:
        outfile.create_dataset(
            'data',
            shape=input_data.shape,
            data=input_data,
            shuffle=False,
            compression=32001,                               #<==== blosc filter
            compression_opts=(0, 0, 0, 0, 1, 1, 1),    #<=== my input parameters. The first 4 are set by filter
            maxshape=input_data.maxshape,
            chunks=input_data.chunks
        )

In this case the output file has the expected compression parameters when read through h5dump:

h5dump -pH output.hdf5 
HDF5 "output.hdf5" {
GROUP "/" {
   DATASET "data" {
      DATATYPE  H5T_STD_U16LE
      DATASPACE  SIMPLE { ( 100, 1536, 2048 ) / ( H5S_UNLIMITED, 1536, 2048 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 1, 1536, 2048 )
         SIZE 8839621 (71.173:1 COMPRESSION)
      }
      FILTERS {
         USER_DEFINED_FILTER {
            FILTER_ID 32001
            COMMENT blosc
            PARAMS { 2 2 2 6291456 1 1 1 }      <======= all the parameters
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_ALLOC
         VALUE  H5D_FILL_VALUE_DEFAULT
      }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}
}

Where the same dataset compressed with h5repack does not include all of these parameters (see trace on my previous comment).

19

Yes, the tests use h5repack.
Here are the commands and h5dump output:
h5repack -v -f UD=32001,0,7,0,0,0,0,5,1,0 h5repack_layout.h5 h5repack_convert_lz.h5
GROUP “/” {
DATASET “dset1” {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 40, 20 ) / ( 40, 20 ) }
STORAGE_LAYOUT {
CHUNKED ( 40, 20 )
SIZE 367 (8.719:1 COMPRESSION)
}
FILTERS {
USER_DEFINED_FILTER {
FILTER_ID 32001
COMMENT HDF5 blosc filter; see http://www.hdfgroup.org/services/contributions.html
PARAMS { 2 2 4 3200 5 1 0 }
}
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}

h5repack  -v -f UD=32001,0,7,0,0,0,0,5,1,1 h5repack_layout.h5 h5repack_convert_lz4.h5
GROUP "/" {
DATASET "dset1" {
  DATATYPE  H5T_STD_I32LE
  DATASPACE  SIMPLE { ( 40, 20 ) / ( 40, 20 ) }
  STORAGE_LAYOUT {
     CHUNKED ( 40, 20 )
     SIZE 326 (9.816:1 COMPRESSION)
  }
  FILTERS {
     USER_DEFINED_FILTER {
        FILTER_ID 32001
        COMMENT HDF5 blosc filter; see http://www.hdfgroup.org/services/contributions.html
        PARAMS { 2 2 4 3200 5 1 1 }
     }
  }
  FILLVALUE {
     FILL_TIME H5D_FILL_TIME_IFSET
     VALUE  H5D_FILL_VALUE_DEFAULT
  }
  ALLOCATION_TIME {
     H5D_ALLOC_TIME_INCR
  }

}

h5repack  -v -f UD=32001,0,7,0,0,0,0,5,1,2 h5repack_layout.h5 h5repack_convert_lz4hc.h5
GROUP "/" {
DATASET "dset1" {
  DATATYPE  H5T_STD_I32LE
  DATASPACE  SIMPLE { ( 40, 20 ) / ( 40, 20 ) }
  STORAGE_LAYOUT {
     CHUNKED ( 40, 20 )
     SIZE 324 (9.877:1 COMPRESSION)
  }
  FILTERS {
     USER_DEFINED_FILTER {
        FILTER_ID 32001
        COMMENT HDF5 blosc filter; see http://www.hdfgroup.org/services/contributions.html
        PARAMS { 2 2 4 3200 5 1 2 }
     }
  }
  FILLVALUE {
     FILL_TIME H5D_FILL_TIME_IFSET
     VALUE  H5D_FILL_VALUE_DEFAULT
  }
  ALLOCATION_TIME {
     H5D_ALLOC_TIME_INCR
  }

}

h5repack  -v -f UD=32001,0,7,0,0,0,0,5,1,4 h5repack_layout.h5 h5repack_convert_zlib.h5
GROUP "/" {
DATASET "dset1" {
  DATATYPE  H5T_STD_I32LE
  DATASPACE  SIMPLE { ( 40, 20 ) / ( 40, 20 ) }
  STORAGE_LAYOUT {
     CHUNKED ( 40, 20 )
     SIZE 337 (9.496:1 COMPRESSION)
  }
  FILTERS {
     USER_DEFINED_FILTER {
        FILTER_ID 32001
        COMMENT HDF5 blosc filter; see http://www.hdfgroup.org/services/contributions.html
        PARAMS { 2 2 4 3200 5 1 4 }
     }
  }
  FILLVALUE {
     FILL_TIME H5D_FILL_TIME_IFSET
     VALUE  H5D_FILL_VALUE_DEFAULT
  }
  ALLOCATION_TIME {
     H5D_ALLOC_TIME_INCR
  }

}

20

I finally located the possible issue. User-defined filters need to know about the mandatory flag. So h5repack added a new parameter right after the filter number. From the release.txt:

- h5repack

  h5repack always set the User Defined filter flag to H5Z_FLAG_MANDATORY.

  Added another parameter to the 'UD=' option to set the flag by default
  to '0' or H5Z_FLAG_MANDATORY, the other choice is '1' or H5Z_FLAG_OPTIONAL.

  (ADB - 2017/08/31, HDFFV-10269)