Transposing a 2d matrix via hyperslab

A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

Thanks,
Isaac

Hyperslabs won’t do this, but you could use a point selection instead.

    Quincey

···

On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:

A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab. It seems that the grammer provided by start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?

Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!

···

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Isaac,

  without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html

To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

             Werner

···

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov > <mailto:koziol@lbl.gov>> wrote:

    > On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com > <mailto:isaac.gerg@gergltd.com>> wrote:
    >
    > A coworker and I are mulling over if its possible to take a
    simple 3x3 matrix (stored row major) in a dataspace and write it
    out transposed to disk using a hyperslab. It seems that the
    grammer provided by start, stride, blocks, and count won't allow
    for this. We hope we are wrong! It is possible to do this?

            Hyperslabs won�t do this, but you could use a point
    selection instead.

                    Quincey

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy
does their indexing by strides, the stride for each dimension. hdf5 does
their "striding" across the data in the sense of more of a selection than a
stride. I would be inclined to say that even a negative stride wouldnt
allow this. The problem is the stride is a simply stride and not one based
on dimension index so the stride can't wrap so you end up with out of
bounds issues with the datasets. Does this seem right?

···

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/
rotation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Isaac,

it's just thought that it could be done like this:

Copy operation: start = 0, stride = 1

    target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

    target_index = max_index + source_index * -1

So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

            Werner

···

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each dimension. hdf5 does their "striding" across the data in the sense of more of a selection than a stride. I would be inclined to say that even a negative stride wouldnt allow this. The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:

    Hi Isaac,

     without proof, but I could imagine it's possible to implement a
    shear operation via hyperslabs. If so, then hyperslabs could be
    used to implement rotations:

    e.g.
    https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
    <https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>

    To do a transpose, one would need a reflection in addition to
    rotation (e.g. http://techieme.in/matrix-rotation/
    <http://techieme.in/matrix-rotation/> ) , and I guess doing this
    operation of a reflection is not possible via hyperslabs, so doing
    a transpose is out of reach. Probably it would require to allow
    something like a negative stride value...

                Werner

    On 25.04.2017 22:24, Isaac Gerg wrote:

    Hi Quincy, I just was reading about point selections and was
    wondering if that's the way we would go. Thanks for confirming!

    On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov >> <mailto:koziol@lbl.gov>> wrote:

        > On Apr 25, 2017, at 1:02 PM, Isaac Gerg >> <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> wrote:
        >
        > A coworker and I are mulling over if its possible to take a
        simple 3x3 matrix (stored row major) in a dataspace and write
        it out transposed to disk using a hyperslab. It seems that
        the grammer provided by start, stride, blocks, and count
        won't allow for this. We hope we are wrong! It is possible
        to do this?

                Hyperslabs won�t do this, but you could use a point
        selection instead.

                        Quincey

        _______________________________________________
        Hdf-forum is for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter:https://twitter.com/hdf5

    -- ___________________________________________________________________________
    Dr. Werner Benger Visualization Research
    Center for Computation & Technology at Louisiana State University (CCT/LSU)
    2019 Digital Media Center, Baton Rouge, Louisiana 70803
    Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?

···

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation: start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.

           Werner

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.

···

On 26.04.2017 00:13, Isaac Gerg wrote:

I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:

    Hi Isaac,

    it's just thought that it could be done like this:

    Copy operation: start = 0, stride = 1

       target_index = 0 + source_index * 1

    Reflection: start = max_index, stride = -1

       target_index = max_index + source_index * -1

    So a negative stride in one dimensions would go "backwards", thus
    do a reflection in this direction. In practice the stride
    parameter is probably an unsigned integer and thus running out of
    bounds instead of going backwards, as you say.

               Werner

    On 25.04.2017 23:48, Isaac Gerg wrote:

    Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each
    dimension. hdf5 does their "striding" across the data in the
    sense of more of a selection than a stride. I would be inclined
    to say that even a negative stride wouldnt allow this. The
    problem is the stride is a simply stride and not one based on
    dimension index so the stride can't wrap so you end up with out
    of bounds issues with the datasets. Does this seem right?

    On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger >> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:

        Hi Isaac,

         without proof, but I could imagine it's possible to
        implement a shear operation via hyperslabs. If so, then
        hyperslabs could be used to implement rotations:

        e.g.
        https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
        <https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>

        To do a transpose, one would need a reflection in addition to
        rotation (e.g. http://techieme.in/matrix-rotation/
        <http://techieme.in/matrix-rotation/> ) , and I guess doing
        this operation of a reflection is not possible via
        hyperslabs, so doing a transpose is out of reach. Probably it
        would require to allow something like a negative stride value...

                    Werner

        On 25.04.2017 22:24, Isaac Gerg wrote:

        Hi Quincy, I just was reading about point selections and was
        wondering if that's the way we would go. Thanks for confirming!

        On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol >>> <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:

            > On Apr 25, 2017, at 1:02 PM, Isaac Gerg >>> <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> >>> wrote:
            >
            > A coworker and I are mulling over if its possible to
            take a simple 3x3 matrix (stored row major) in a
            dataspace and write it out transposed to disk using a
            hyperslab. It seems that the grammer provided by
            start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?

                    Hyperslabs won�t do this, but you could use a
            point selection instead.

                            Quincey

            _______________________________________________
            Hdf-forum is for HDF software users discussion.
            Hdf-forum@lists.hdfgroup.org
            <mailto:Hdf-forum@lists.hdfgroup.org>
            http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
            <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
            Twitter: https://twitter.com/hdf5

        _______________________________________________
        Hdf-forum is for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter:https://twitter.com/hdf5

        -- ___________________________________________________________________________
        Dr. Werner Benger Visualization Research
        Center for Computation & Technology at Louisiana State University (CCT/LSU)
        2019 Digital Media Center, Baton Rouge, Louisiana 70803
        Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

        _______________________________________________ Hdf-forum is
        for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter:https://twitter.com/hdf5

    -- ___________________________________________________________________________
    Dr. Werner Benger Visualization Research
    Center for Computation & Technology at Louisiana State University (CCT/LSU)
    2019 Digital Media Center, Baton Rouge, Louisiana 70803
    Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

I think of hyperslab as a sort of fancy click+drag selection tool of a
tensor (fancy in the sense that you can do strided blocks of selection).
The original problem stemmed from a desire to do matlab shiftdim-like or
numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because
of processing efficiency, but wish to store the data X,Y,Z for
visualization efficiency. Its really not a big deal to simply do a memcopy
but why waste the memory if I don't need to; this gives us a chance to
really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the source
for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis
under the good). The solution is rather elegant for them. Every array in
np stores a stride tuple which are the strides over dimension. So to do a
transpose they simply permute this tuple. Perhaps having a similar
interface for HDF is something to be considered? On the other hand, I
could see providing another interface to mangle data as "giving yourself
more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in digging
around how others (i.e numpy) handle it given their use cases.

Isaac

···

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:

Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.

On 26.04.2017 00:13, Isaac Gerg wrote:

I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation: start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.

           Werner

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:

Hi Isaac,

without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Isaac,

  just out of curiosity, which operation do you consider where something like Z,X,Y order is so much more efficient to justify such a data transformation at I/O?

         Werner

···

On 26.04.2017 13:58, Isaac Gerg wrote:

I think of hyperslab as a sort of fancy click+drag selection tool of a tensor (fancy in the sense that you can do strided blocks of selection). The original problem stemmed from a desire to do matlab shiftdim-like or numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because of processing efficiency, but wish to store the data X,Y,Z for visualization efficiency. Its really not a big deal to simply do a memcopy but why waste the memory if I don't need to; this gives us a chance to really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the source for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis under the good). The solution is rather elegant for them. Every array in np stores a stride tuple which are the strides over dimension. So to do a transpose they simply permute this tuple. Perhaps having a similar interface for HDF is something to be considered? On the other hand, I could see providing another interface to mangle data as "giving yourself more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in digging around how others (i.e numpy) handle it given their use cases.

Isaac

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:

    Probably correct, Quincey might know better how far the hyperslab
    implementation could be modified to allow a negative stride. Maybe
    it wouldn't be much of a change, just there was no need so far to
    do it. If that is the case, then at least it would mean that the
    "hyperslab grammar" would indeed allow operations like transpose
    (and rotation) of a dataset, just the implementation does not
    support it. That aside, it's probably not efficient as it would
    require multiple hyperslab operations, so a point selection would
    be faster, but for big data and out-of-core data management,
    sometimes seemingly inefficient operations turn out to be better.

    On 26.04.2017 00:13, Isaac Gerg wrote:

    I am not sure if you are allowed a negative stride but even if
    you are, I believe the transpose still won't work because the
    stride doesnt stride dimension like numpy but strides across the
    memory space and wrap is not allowed. Does this line of thinking
    make sense?

    On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger >> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:

        Hi Isaac,

        it's just thought that it could be done like this:

        Copy operation: start = 0, stride = 1

           target_index = 0 + source_index * 1

        Reflection: start = max_index, stride = -1

           target_index = max_index + source_index * -1

        So a negative stride in one dimensions would go "backwards",
        thus do a reflection in this direction. In practice the
        stride parameter is probably an unsigned integer and thus
        running out of bounds instead of going backwards, as you say.

                   Werner

        On 25.04.2017 23:48, Isaac Gerg wrote:

        Hi Werner, thanks for the reply. Let's wrestle with this a
        bit. So numpy does their indexing by strides, the stride
        for each dimension. hdf5 does their "striding" across the
        data in the sense of more of a selection than a stride. I
        would be inclined to say that even a negative stride wouldnt
        allow this. The problem is the stride is a simply stride
        and not one based on dimension index so the stride can't
        wrap so you end up with out of bounds issues with the
        datasets. Does this seem right?

        On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger >>> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:

            Hi Isaac,

             without proof, but I could imagine it's possible to
            implement a shear operation via hyperslabs. If so, then
            hyperslabs could be used to implement rotations:

            e.g.
            https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
            <https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>

            To do a transpose, one would need a reflection in
            addition to rotation (e.g.
            http://techieme.in/matrix-rotation/
            <http://techieme.in/matrix-rotation/> ) , and I guess
            doing this operation of a reflection is not possible via
            hyperslabs, so doing a transpose is out of reach.
            Probably it would require to allow something like a
            negative stride value...

                        Werner

            On 25.04.2017 22:24, Isaac Gerg wrote:

            Hi Quincy, I just was reading about point selections
            and was wondering if that's the way we would go. Thanks
            for confirming!

            On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol >>>> <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:

                > On Apr 25, 2017, at 1:02 PM, Isaac Gerg >>>> <isaac.gerg@gergltd.com >>>> <mailto:isaac.gerg@gergltd.com>> wrote:
                >
                > A coworker and I are mulling over if its possible
                to take a simple 3x3 matrix (stored row major) in a
                dataspace and write it out transposed to disk using
                a hyperslab. It seems that the grammer provided
                by start, stride, blocks, and count won't allow for
                this. We hope we are wrong! It is possible to do this?

                Hyperslabs won�t do this, but you could use a point
                selection instead.

                Quincey

                _______________________________________________
                Hdf-forum is for HDF software users discussion.
                Hdf-forum@lists.hdfgroup.org
                <mailto:Hdf-forum@lists.hdfgroup.org>
                http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
                <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
                Twitter: https://twitter.com/hdf5

            _______________________________________________
            Hdf-forum is for HDF software users discussion.
            Hdf-forum@lists.hdfgroup.org
            <mailto:Hdf-forum@lists.hdfgroup.org>
            http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
            <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
            Twitter:https://twitter.com/hdf5

            -- ___________________________________________________________________________
            Dr. Werner Benger Visualization Research
            Center for Computation & Technology at Louisiana State University (CCT/LSU)
            2019 Digital Media Center, Baton Rouge, Louisiana 70803
            Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

            _______________________________________________
            Hdf-forum is for HDF software users discussion.
            Hdf-forum@lists.hdfgroup.org
            <mailto:Hdf-forum@lists.hdfgroup.org>
            http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
            <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
            Twitter: https://twitter.com/hdf5

        _______________________________________________
        Hdf-forum is for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter:https://twitter.com/hdf5

        -- ___________________________________________________________________________
        Dr. Werner Benger Visualization Research
        Center for Computation & Technology at Louisiana State University (CCT/LSU)
        2019 Digital Media Center, Baton Rouge, Louisiana 70803
        Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

        _______________________________________________ Hdf-forum is
        for HDF software users discussion.
        Hdf-forum@lists.hdfgroup.org
        <mailto:Hdf-forum@lists.hdfgroup.org>
        http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
        <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
        Twitter: https://twitter.com/hdf5

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter:https://twitter.com/hdf5

    -- ___________________________________________________________________________
    Dr. Werner Benger Visualization Research
    Center for Computation & Technology at Louisiana State University (CCT/LSU)
    2019 Digital Media Center, Baton Rouge, Louisiana 70803
    Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>

    _______________________________________________ Hdf-forum is for
    HDF software users discussion. Hdf-forum@lists.hdfgroup.org
    <mailto:Hdf-forum@lists.hdfgroup.org>
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
    Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Werner,

Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.

  I like the idea quite a lot. :slight_smile: However, it might be quite difficult to implement. :frowning: The hyperslabs are “compiled” into [complex] internal data structures as they are built up, in order to make the I/O operations as fast as possible. It would be possible to defer that compilation until when the actual I/O operation occurs, but that would be another layer of data structures, etc. I believe it would be worthwhile, particularly so that transposes are possible, but it would definitely be a fair bit of work…

  Quincey

···

On Apr 26, 2017, at 12:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:

On 26.04.2017 00:13, Isaac Gerg wrote:

I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,

it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
   target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
   target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.

           Werner

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each dimension. hdf5 does their "striding" across the data in the sense of more of a selection than a stride. I would be inclined to say that even a negative stride wouldnt allow this. The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,

without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:
e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html <https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>
To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab. It seems that the grammer provided by start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <tel:%28225%29%20578-4809> Fax.: +1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--

___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <tel:%28225%29%20578-4809> Fax.: +1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--

___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Werner,

I am doing remote sensing into the seafloor with refraction being modeled.
Our beamformer creates imagery via layers and its a useful way to deal with
the refraction. But when you go to display the data, its useful to look at
the side projection (where the vertical axis is depth and the horizontal is
along track or range).

Isaac

···

On Wed, Apr 26, 2017 at 3:14 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

just out of curiosity, which operation do you consider where something
like Z,X,Y order is so much more efficient to justify such a data
transformation at I/O?

        Werner

On 26.04.2017 13:58, Isaac Gerg wrote:

I think of hyperslab as a sort of fancy click+drag selection tool of a
tensor (fancy in the sense that you can do strided blocks of selection).
The original problem stemmed from a desire to do matlab shiftdim-like or
numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because
of processing efficiency, but wish to store the data X,Y,Z for
visualization efficiency. Its really not a big deal to simply do a memcopy
but why waste the memory if I don't need to; this gives us a chance to
really do a deep dive on hyperslab to see if it can do the job.

In trying to get this to work with hyperslabbing, this led me to the
source for np.rollaxis which led me to np.tranpose (how numpy implements
rollaxis under the good). The solution is rather elegant for them. Every
array in np stores a stride tuple which are the strides over dimension. So
to do a transpose they simply permute this tuple. Perhaps having a similar
interface for HDF is something to be considered? On the other hand, I
could see providing another interface to mangle data as "giving yourself
more rope to trip over" so to speak.

In any case, I figured I'd share where the problem came from and in
digging around how others (i.e numpy) handle it given their use cases.

Isaac

On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:

Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.

On 26.04.2017 00:13, Isaac Gerg wrote:

I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation: start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.

           Werner

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >>> wrote:

Hi Isaac,

without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.h
dfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

*I* have to chime in and say that I'm happy with the way HDF works now.
Please don't take our single usecase as a need to spend a lot of work on
this. I expected a simple yes or no to my question! Im totally fine with
HDF not being the end all of every usecase.

···

On Thu, Apr 27, 2017 at 7:13 PM, Quincey Koziol <koziol@lbl.gov> wrote:

Hi Werner,

On Apr 26, 2017, at 12:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:

Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.

I like the idea quite a lot. :slight_smile: However, it might be quite difficult to
implement. :frowning: The hyperslabs are “compiled” into [complex] internal data
structures as they are built up, in order to make the I/O operations as
fast as possible. It would be possible to defer that compilation until
when the actual I/O operation occurs, but that would be another layer of
data structures, etc. I believe it would be worthwhile, particularly so
that transposes are possible, but it would definitely be a fair bit of work…

Quincey

On 26.04.2017 00:13, Isaac Gerg wrote:

I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?

On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:

Hi Isaac,

it's just thought that it could be done like this:

Copy operation: start = 0, stride = 1

   target_index = 0 + source_index * 1

Reflection: start = max_index, stride = -1

   target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.

           Werner

On 25.04.2017 23:48, Isaac Gerg wrote:

Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?

On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:

Hi Isaac,

without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:

e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...

            Werner

On 25.04.2017 22:24, Isaac Gerg wrote:

Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!

On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:

> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?

        Hyperslabs won’t do this, but you could use a point selection
instead.

                Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>

_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5