compressed rectilinear grids

Hi, Quincy et al.
I originally thought that perhaps I could do this with region references,
but it looks like I cannot, from what I've read.

I have a Cartesian, rectilinear grid of--tens, hundreds or more--millions of
points on which I am solving the fully compressible Navier Stokes equations.
The flow variables need to be specified at each point, so for each flow
variable we have a large three dimensional data set. Since the grid
transformation to map from computational space (a Cartesian grid with unity
spacing between elements) to physical space is so simple, rather than
storing the grid as large three dimensional data sets (one 3D data set for
x, y, and z each) over millions of points, we can simply store the grid as
1D array data sets (again one for x, y, and z). We can do this since each
physical coordinate is only a function of one computational coordinate with
some one dimensional stretching function applied. (There are no off-diagonal
terms in the grid Jacobian matrix.) Each coordinate, x,y and z can be of
different length/number of elements corresponding to the number of grid
points in that direction or along that edge of our computational box. This
means hat we can store all the info we need for our grid as O(N^(1/3)) many
floating point values instead of O(N) many floating point values where N is
very large.

The simulation code uses meta data or a flag to tell it that we are using a
Cartesian rectilinear grid (rather than a rectilinear structured grid) so it
can read in the grid data (x,y,z coordinates) as a set of three one
dimensional arrays.

The problem with this, however, is that Tecplot (and perhaps other vis
packages) require that each variable we load have the same dimensionality
and shape. So, in order that I may load my grid file and then load the flow
variables I would need the grid data sets to appear to be 3D arrays rather
than one dimensional arrays. I was originally thinking that maybe I could
have some sort of pointers or region references where I can set up what
appears to be a three dimensional array data set but it is actually some
sort of pointer to the appropriate elements of a one dimensional data set.
(A many-to-one type of pointer setup.)

Correct me if I am wrong, but as of now it appears that there is no way for
me to accomplish this, short of writing a custom data filter. I am thinking
that perhaps, for ease of use, we might just have to store the grid data
sets as full 3D arrays despite the redundant and wasteful nature of doing
this.

I hope I have been clearer than before. If not, and you have access to
matlab, look at the help for the 'meshgrid' function. I am basically trying
to figure out a way to make it look like I am storing the output arrays X,
Y, Z, but actually store the (smaller) input arrays, x, y, z. As I said
above, I suspect that there is no simple way to do this and I might just
have to deal with the extra expense of storing the redundant 3D arrays.
Please correct me if I am wrong about this.

Thanks,
Izaak Beekman

···

On Sun, Oct 9, 2011 at 12:00 PM, <hdf-forum-request@hdfgroup.org> wrote:

> Hi Izaak,
>
> On 10/5/2011 4:09 PM, Zaak Beekman wrote:
>>
>> 2) Is there a way to create a compressed rectilinear (Cartesian) grid? I
would like to be able to just specify 3 1-D arrays for each edge of my 3D
cartesian grid, and then create something that looks like a 3D dataset but
where x(:,j/=1,k/=1) points to x(:,1,1). (The reason for this is I would
like to be able to read our grid file directly into Tecplot 360 without
wasting tons of disk space. The simulation can use the 1-D arrays without
any issues.)
> I do not fully understand the issue. Others may be able to help?

       Hmm, yes, I don't understand the notation being used here... Izaak,
can you expand a bit more for us Tecplot laymen? :slight_smile:

       Quincey

===================================
(301)244-9367
Princeton University Doctoral Candidate
Mechanical and Aerospace Engineering
ibeekman@princeton.edu

UMD-CP Visiting Graduate Student
Aerospace Engineering
ibeekman@umiacs.umd.edu
ibeekman@umd.edu

>> 2) Is there a way to create a compressed rectilinear (Cartesian) grid?
>> I would like to be able to just specify 3 1-D arrays for each edge of my 3D
>> cartesian grid, and then create something that looks like a 3D dataset but
>> where x(:,j/=1,k/=1) points to x(:,1,1).

Since the grid
transformation to map from computational space (a Cartesian grid with unity
spacing between elements) to physical space is so simple, rather than
storing the grid as large three dimensional data sets (one 3D data set for
x, y, and z each) over millions of points, we can simply store the grid as
1D array data sets (again one for x, y, and z).

To the best of my knowledge, HDF has no native understanding of a grid
(structured or otherwise). Assembling semantic understanding about
the contents of your HDF files is up to your simulation and
visualization software. Quincey, please correct me if I am wrong.

I would be very, very surprised if Tecplot cannot product a 3D
rectilinear grid from the outer product of three 1D arrays. It's
probably just hidden in one of their myriad unusable interface
dialogs. You might try asking in a Tecplot or CFD forum.

Using Xdmf's "3DRectMesh" with the "VXVYVZ" option should work for you
(http://www.xdmf.org/index.php/XDMF_Model_and_Format). Both Paraview
and VisIt will accept Xdmf. No idea if Tecplot does. Paraview is not
amazingly robust in the presence of errors in your Xdmf description so
have patience if you choose to try. I have successfully visualized a
nontrivial 3D rectilinear CFD field in this manner using Paraview
(though I don't have my sample Xdmf handy).

Best of luck,
Rhys

Hi Izaak,

···

On Oct 9, 2011, at 6:42 PM, Zaak Beekman wrote:

On Sun, Oct 9, 2011 at 12:00 PM, <hdf-forum-request@hdfgroup.org> wrote:
> Hi Izaak,
>
> On 10/5/2011 4:09 PM, Zaak Beekman wrote:
>>
>> 2) Is there a way to create a compressed rectilinear (Cartesian) grid? I would like to be able to just specify 3 1-D arrays for each edge of my 3D cartesian grid, and then create something that looks like a 3D dataset but where x(:,j/=1,k/=1) points to x(:,1,1). (The reason for this is I would like to be able to read our grid file directly into Tecplot 360 without wasting tons of disk space. The simulation can use the 1-D arrays without any issues.)
> I do not fully understand the issue. Others may be able to help?

       Hmm, yes, I don't understand the notation being used here... Izaak, can you expand a bit more for us Tecplot laymen? :slight_smile:

       Quincey

Hi, Quincy et al.
I originally thought that perhaps I could do this with region references, but it looks like I cannot, from what I've read.

I have a Cartesian, rectilinear grid of--tens, hundreds or more--millions of points on which I am solving the fully compressible Navier Stokes equations. The flow variables need to be specified at each point, so for each flow variable we have a large three dimensional data set. Since the grid transformation to map from computational space (a Cartesian grid with unity spacing between elements) to physical space is so simple, rather than storing the grid as large three dimensional data sets (one 3D data set for x, y, and z each) over millions of points, we can simply store the grid as 1D array data sets (again one for x, y, and z). We can do this since each physical coordinate is only a function of one computational coordinate with some one dimensional stretching function applied. (There are no off-diagonal terms in the grid Jacobian matrix.) Each coordinate, x,y and z can be of different length/number of elements corresponding to the number of grid points in that direction or along that edge of our computational box. This means hat we can store all the info we need for our grid as O(N^(1/3)) many floating point values instead of O(N) many floating point values where N is very large.

The simulation code uses meta data or a flag to tell it that we are using a Cartesian rectilinear grid (rather than a rectilinear structured grid) so it can read in the grid data (x,y,z coordinates) as a set of three one dimensional arrays.

The problem with this, however, is that Tecplot (and perhaps other vis packages) require that each variable we load have the same dimensionality and shape. So, in order that I may load my grid file and then load the flow variables I would need the grid data sets to appear to be 3D arrays rather than one dimensional arrays. I was originally thinking that maybe I could have some sort of pointers or region references where I can set up what appears to be a three dimensional array data set but it is actually some sort of pointer to the appropriate elements of a one dimensional data set. (A many-to-one type of pointer setup.)

Correct me if I am wrong, but as of now it appears that there is no way for me to accomplish this, short of writing a custom data filter. I am thinking that perhaps, for ease of use, we might just have to store the grid data sets as full 3D arrays despite the redundant and wasteful nature of doing this.

I hope I have been clearer than before. If not, and you have access to matlab, look at the help for the 'meshgrid' function. I am basically trying to figure out a way to make it look like I am storing the output arrays X, Y, Z, but actually store the (smaller) input arrays, x, y, z. As I said above, I suspect that there is no simple way to do this and I might just have to deal with the extra expense of storing the redundant 3D arrays. Please correct me if I am wrong about this.

  Hmm, it's a bit crude, but you may be able to achieve something close to what you want, if you use chunked dataset storage in HDF5, along with a compression filter. Chunks with no data stored in them will not be created, and sparsely filled chunked should compress very well.

  Otherwise, a meshing library on top of HDF5, like MOAB, might be a viable option.

    Quincey