A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?
Thanks,
Isaac
Hyperslabs won’t do this, but you could use a point selection instead.
Quincey
On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:
A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab. It seems that the grammer provided by start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?
Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!
On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:
e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...
Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go. Thanks for confirming!
On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov > <mailto:koziol@lbl.gov>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com > <mailto:isaac.gerg@gergltd.com>> wrote:
>
> A coworker and I are mulling over if its possible to take a
simple 3x3 matrix (stored row major) in a dataspace and write it
out transposed to disk using a hyperslab. It seems that the
grammer provided by start, stride, blocks, and count won't allow
for this. We hope we are wrong! It is possible to do this?Hyperslabs won�t do this, but you could use a point
selection instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy
does their indexing by strides, the stride for each dimension. hdf5 does
their "striding" across the data in the sense of more of a selection than a
stride. I would be inclined to say that even a negative stride wouldnt
allow this. The problem is the stride is a simply stride and not one based
on dimension index so the stride can't wrap so you end up with out of
bounds issues with the datasets. Does this seem right?
On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/
rotation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.
Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each dimension. hdf5 does their "striding" across the data in the sense of more of a selection than a stride. I would be inclined to say that even a negative stride wouldnt allow this. The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?
On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a
shear operation via hyperslabs. If so, then hyperslabs could be
used to implement rotations:e.g.
https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
<https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>To do a transpose, one would need a reflection in addition to
rotation (e.g. http://techieme.in/matrix-rotation/
<http://techieme.in/matrix-rotation/> ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing
a transpose is out of reach. Probably it would require to allow
something like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was
wondering if that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov >> <mailto:koziol@lbl.gov>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg >> <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> wrote:
>
> A coworker and I are mulling over if its possible to take a
simple 3x3 matrix (stored row major) in a dataspace and write
it out transposed to disk using a hyperslab. It seems that
the grammer provided by start, stride, blocks, and count
won't allow for this. We hope we are wrong! It is possible
to do this?Hyperslabs won�t do this, but you could use a point
selection instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________ Hdf-forum is for
HDF software users discussion. Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?
On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if
that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3
matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.
On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?
On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus
do a reflection in this direction. In practice the stride
parameter is probably an unsigned integer and thus running out of
bounds instead of going backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each
dimension. hdf5 does their "striding" across the data in the
sense of more of a selection than a stride. I would be inclined
to say that even a negative stride wouldnt allow this. The
problem is the stride is a simply stride and not one based on
dimension index so the stride can't wrap so you end up with out
of bounds issues with the datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger >> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to
implement a shear operation via hyperslabs. If so, then
hyperslabs could be used to implement rotations:e.g.
https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
<https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>To do a transpose, one would need a reflection in addition to
rotation (e.g. http://techieme.in/matrix-rotation/
<http://techieme.in/matrix-rotation/> ) , and I guess doing
this operation of a reflection is not possible via
hyperslabs, so doing a transpose is out of reach. Probably it
would require to allow something like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was
wondering if that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol >>> <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg >>> <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> >>> wrote:
>
> A coworker and I are mulling over if its possible to
take a simple 3x3 matrix (stored row major) in a
dataspace and write it out transposed to disk using a
hyperslab. It seems that the grammer provided by
start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?Hyperslabs won�t do this, but you could use a
point selection instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________ Hdf-forum is
for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________ Hdf-forum is for
HDF software users discussion. Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I think of hyperslab as a sort of fancy click+drag selection tool of a
tensor (fancy in the sense that you can do strided blocks of selection).
The original problem stemmed from a desire to do matlab shiftdim-like or
numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because
of processing efficiency, but wish to store the data X,Y,Z for
visualization efficiency. Its really not a big deal to simply do a memcopy
but why waste the memory if I don't need to; this gives us a chance to
really do a deep dive on hyperslab to see if it can do the job.
In trying to get this to work with hyperslabbing, this led me to the source
for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis
under the good). The solution is rather elegant for them. Every array in
np stores a stride tuple which are the strides over dimension. So to do a
transpose they simply permute this tuple. Perhaps having a similar
interface for HDF is something to be considered? On the other hand, I
could see providing another interface to mangle data as "giving yourself
more rope to trip over" so to speak.
In any case, I figured I'd share where the problem came from and in digging
around how others (i.e numpy) handle it given their use cases.
Isaac
On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:
Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Isaac,
just out of curiosity, which operation do you consider where something like Z,X,Y order is so much more efficient to justify such a data transformation at I/O?
Werner
On 26.04.2017 13:58, Isaac Gerg wrote:
I think of hyperslab as a sort of fancy click+drag selection tool of a tensor (fancy in the sense that you can do strided blocks of selection). The original problem stemmed from a desire to do matlab shiftdim-like or numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because of processing efficiency, but wish to store the data X,Y,Z for visualization efficiency. Its really not a big deal to simply do a memcopy but why waste the memory if I don't need to; this gives us a chance to really do a deep dive on hyperslab to see if it can do the job.
In trying to get this to work with hyperslabbing, this led me to the source for np.rollaxis which led me to np.tranpose (how numpy implements rollaxis under the good). The solution is rather elegant for them. Every array in np stores a stride tuple which are the strides over dimension. So to do a transpose they simply permute this tuple. Perhaps having a similar interface for HDF is something to be considered? On the other hand, I could see providing another interface to mangle data as "giving yourself more rope to trip over" so to speak.
In any case, I figured I'd share where the problem came from and in digging around how others (i.e numpy) handle it given their use cases.
Isaac
On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu > <mailto:werner@cct.lsu.edu>> wrote:
Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe
it wouldn't be much of a change, just there was no need so far to
do it. If that is the case, then at least it would mean that the
"hyperslab grammar" would indeed allow operations like transpose
(and rotation) of a dataset, just the implementation does not
support it. That aside, it's probably not efficient as it would
require multiple hyperslab operations, so a point selection would
be faster, but for big data and out-of-core data management,
sometimes seemingly inefficient operations turn out to be better.On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if
you are, I believe the transpose still won't work because the
stride doesnt stride dimension like numpy but strides across the
memory space and wrap is not allowed. Does this line of thinking
make sense?On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger >> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards",
thus do a reflection in this direction. In practice the
stride parameter is probably an unsigned integer and thus
running out of bounds instead of going backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a
bit. So numpy does their indexing by strides, the stride
for each dimension. hdf5 does their "striding" across the
data in the sense of more of a selection than a stride. I
would be inclined to say that even a negative stride wouldnt
allow this. The problem is the stride is a simply stride
and not one based on dimension index so the stride can't
wrap so you end up with out of bounds issues with the
datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger >>> <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to
implement a shear operation via hyperslabs. If so, then
hyperslabs could be used to implement rotations:e.g.
https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
<https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>To do a transpose, one would need a reflection in
addition to rotation (e.g.
http://techieme.in/matrix-rotation/
<http://techieme.in/matrix-rotation/> ) , and I guess
doing this operation of a reflection is not possible via
hyperslabs, so doing a transpose is out of reach.
Probably it would require to allow something like a
negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections
and was wondering if that's the way we would go. Thanks
for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol >>>> <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg >>>> <isaac.gerg@gergltd.com >>>> <mailto:isaac.gerg@gergltd.com>> wrote:
>
> A coworker and I are mulling over if its possible
to take a simple 3x3 matrix (stored row major) in a
dataspace and write it out transposed to disk using
a hyperslab. It seems that the grammer provided
by start, stride, blocks, and count won't allow for
this. We hope we are wrong! It is possible to do this?Hyperslabs won�t do this, but you could use a point
selection instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________ Hdf-forum is
for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5-- ___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>_______________________________________________ Hdf-forum is for
HDF software users discussion. Hdf-forum@lists.hdfgroup.org
<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
Hi Werner,
Probably correct, Quincey might know better how far the hyperslab implementation could be modified to allow a negative stride. Maybe it wouldn't be much of a change, just there was no need so far to do it. If that is the case, then at least it would mean that the "hyperslab grammar" would indeed allow operations like transpose (and rotation) of a dataset, just the implementation does not support it. That aside, it's probably not efficient as it would require multiple hyperslab operations, so a point selection would be faster, but for big data and out-of-core data management, sometimes seemingly inefficient operations turn out to be better.
I like the idea quite a lot. 
However, it might be quite difficult to implement. 
The hyperslabs are “compiled” into [complex] internal data structures as they are built up, in order to make the I/O operations as fast as possible. It would be possible to defer that compilation until when the actual I/O operation occurs, but that would be another layer of data structures, etc. I believe it would be worthwhile, particularly so that transposes are possible, but it would definitely be a fair bit of work…
Quincey
On Apr 26, 2017, at 12:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:
On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I believe the transpose still won't work because the stride doesnt stride dimension like numpy but strides across the memory space and wrap is not allowed. Does this line of thinking make sense?
On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a reflection in this direction. In practice the stride parameter is probably an unsigned integer and thus running out of bounds instead of going backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So numpy does their indexing by strides, the stride for each dimension. hdf5 does their "striding" across the data in the sense of more of a selection than a stride. I would be inclined to say that even a negative stride wouldnt allow this. The problem is the stride is a simply stride and not one based on dimension index so the stride can't wrap so you end up with out of bounds issues with the datasets. Does this seem right?
On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu <mailto:werner@cct.lsu.edu>> wrote:
Hi Isaac,without proof, but I could imagine it's possible to implement a shear operation via hyperslabs. If so, then hyperslabs could be used to implement rotations:
e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html <https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html>
To do a transpose, one would need a reflection in addition to rotation (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this operation of a reflection is not possible via hyperslabs, so doing a transpose is out of reach. Probably it would require to allow something like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering if that's the way we would go. Thanks for confirming!
On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov <mailto:koziol@lbl.gov>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com <mailto:isaac.gerg@gergltd.com>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple 3x3 matrix (stored row major) in a dataspace and write it out transposed to disk using a hyperslab. It seems that the grammer provided by start, stride, blocks, and count won't allow for this. We hope we are wrong! It is possible to do this?Hyperslabs won’t do this, but you could use a point selection instead.
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5–
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <tel:%28225%29%20578-4809> Fax.: +1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <tel:%28225%29%20578-4809> Fax.: +1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is for HDF software users discussion. Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Werner,
I am doing remote sensing into the seafloor with refraction being modeled.
Our beamformer creates imagery via layers and its a useful way to deal with
the refraction. But when you go to display the data, its useful to look at
the side projection (where the vertical axis is depth and the horizontal is
along track or range).
Isaac
On Wed, Apr 26, 2017 at 3:14 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
just out of curiosity, which operation do you consider where something
like Z,X,Y order is so much more efficient to justify such a data
transformation at I/O?Werner
On 26.04.2017 13:58, Isaac Gerg wrote:
I think of hyperslab as a sort of fancy click+drag selection tool of a
tensor (fancy in the sense that you can do strided blocks of selection).
The original problem stemmed from a desire to do matlab shiftdim-like or
numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because
of processing efficiency, but wish to store the data X,Y,Z for
visualization efficiency. Its really not a big deal to simply do a memcopy
but why waste the memory if I don't need to; this gives us a chance to
really do a deep dive on hyperslab to see if it can do the job.In trying to get this to work with hyperslabbing, this led me to the
source for np.rollaxis which led me to np.tranpose (how numpy implements
rollaxis under the good). The solution is rather elegant for them. Every
array in np stores a stride tuple which are the strides over dimension. So
to do a transpose they simply permute this tuple. Perhaps having a similar
interface for HDF is something to be considered? On the other hand, I
could see providing another interface to mangle data as "giving yourself
more rope to trip over" so to speak.In any case, I figured I'd share where the problem came from and in
digging around how others (i.e numpy) handle it given their use cases.Isaac
On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:
Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >>> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.h
dfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
*I* have to chime in and say that I'm happy with the way HDF works now.
Please don't take our single usecase as a need to spend a lot of work on
this. I expected a simple yes or no to my question! Im totally fine with
HDF not being the end all of every usecase.
On Thu, Apr 27, 2017 at 7:13 PM, Quincey Koziol <koziol@lbl.gov> wrote:
Hi Werner,
On Apr 26, 2017, at 12:34 AM, Werner Benger <werner@cct.lsu.edu> wrote:
Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe it
wouldn't be much of a change, just there was no need so far to do it. If
that is the case, then at least it would mean that the "hyperslab grammar"
would indeed allow operations like transpose (and rotation) of a dataset,
just the implementation does not support it. That aside, it's probably not
efficient as it would require multiple hyperslab operations, so a point
selection would be faster, but for big data and out-of-core data
management, sometimes seemingly inefficient operations turn out to be
better.I like the idea quite a lot.
implement.
structures as they are built up, in order to make the I/O operations as
fast as possible. It would be possible to defer that compilation until
when the actual I/O operation occurs, but that would be another layer of
data structures, etc. I believe it would be worthwhile, particularly so
that transposes are possible, but it would definitely be a fair bit of work…Quincey
On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if you are, I
believe the transpose still won't work because the stride doesnt stride
dimension like numpy but strides across the memory space and wrap is not
allowed. Does this line of thinking make sense?On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards", thus do a
reflection in this direction. In practice the stride parameter is probably
an unsigned integer and thus running out of bounds instead of going
backwards, as you say.Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a bit. So
numpy does their indexing by strides, the stride for each dimension. hdf5
does their "striding" across the data in the sense of more of a selection
than a stride. I would be inclined to say that even a negative stride
wouldnt allow this. The problem is the stride is a simply stride and not
one based on dimension index so the stride can't wrap so you end up with
out of bounds issues with the datasets. Does this seem right?On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <werner@cct.lsu.edu> >> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to implement a shear
operation via hyperslabs. If so, then hyperslabs could be used to implement
rotations:e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r
otation_by_shearing.html
To do a transpose, one would need a reflection in addition to rotation
(e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this
operation of a reflection is not possible via hyperslabs, so doing a
transpose is out of reach. Probably it would require to allow something
like a negative stride value...Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections and was wondering
if that's the way we would go. Thanks for confirming!On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <koziol@lbl.gov> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg <isaac.gerg@gergltd.com> >>>> wrote:
>
> A coworker and I are mulling over if its possible to take a simple
3x3 matrix (stored row major) in a dataspace and write it out transposed to
disk using a hyperslab. It seems that the grammer provided by start,
stride, blocks, and count won't allow for this. We hope we are wrong! It
is possible to do this?Hyperslabs won’t do this, but you could use a point selection
instead.Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 225 578-5362 <%28225%29%20578-5362>_______________________________________________ Hdf-forum is for HDF
software users discussion. Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 578-5362 <(225)%20578-5362>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
