appending data

Hello,

I am wondering if there is a way to append previously written datasets into a larger dataset without having to read everything into memory and then write it out. For instance, two files that contain a cloud of data for a a set of temperature probes on the east and west.

I would like to read in the east temperatures and append them to the west temperatures (without reading in the west temperatures).

Cheers,

Allen

Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu

Hi Allen,

···

On Oct 20, 2009, at 2:20 PM, Allen Sanderson wrote:

Hello,

I am wondering if there is a way to append previously written datasets into a larger dataset without having to read everything into memory and then write it out. For instance, two files that contain a cloud of data for a a set of temperature probes on the east and west.

I would like to read in the east temperatures and append them to the west temperatures (without reading in the west temperatures).

  This isn't currently possible, but if the datasets were chunked (and otherwise compatible) it would be theoretically possible to just "transfer" the ownership of the chunks from one dataset's chunk index to another.

  Quincey

> I am wondering if there is a way to append previously written
> datasets into a larger dataset without having to read everything
> into memory and then write it out. For instance, two files that
> contain a cloud of data for a a set of temperature probes on the
> east and west.
>
> I would like to read in the east temperatures and append them to the
> west temperatures (without reading in the west temperatures).

If the 'west' dataset is expandable, are you not able to write the 'east'
values to positions beyond the 'west' dataset, and automatically expand the
'west' dataset? The number of rows of the 'west' data set should be known,
and therefore it shouldn't be much of a problem to append with out having to
read the existing values.

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Hi all

I am kind of disoriented, not sure what Ray wants to do.... Ray, consider
that you have a table. If you want to add rows to the table, then you wanna
make that table's leading dimension expandable. If you wanna add something
like a new colmun, and maybe you do maybe you don't, the only way to do that
and that I know of that has reasonable performance, is to make a dataset of
references that reference the other two datasets. Make that table expandable
and the overhead to add another column is minimal. I hope this makes sense
:-/

good luck

-- dimitris

···

2009/10/20 Ray Burkholder <ray@oneunified.net>

> > I am wondering if there is a way to append previously written
> > datasets into a larger dataset without having to read everything
> > into memory and then write it out. For instance, two files that
> > contain a cloud of data for a a set of temperature probes on the
> > east and west.
> >
> > I would like to read in the east temperatures and append them to the
> > west temperatures (without reading in the west temperatures).
>

If the 'west' dataset is expandable, are you not able to write the 'east'
values to positions beyond the 'west' dataset, and automatically expand the
'west' dataset? The number of rows of the 'west' data set should be known,
and therefore it shouldn't be much of a problem to append with out having
to
read the existing values.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org