···
-----Original Message-----
From: Dimitris Servis [mailto:servisster@gmail.com]
Sent: Thursday, January 31, 2008 3:21 PM
To: Ramakrishnan Iyer
Cc: hdf-forum@hdfgroup.org
Subject: Re: Writing a stream of data/Opaque Data type
Hi Ramakrishnan,
no offense, I did not say you don't understand the difference, I said
you confuse them probably. I see it the other way around:
serialization - The object stored has necessary info so that it can be
created back while deserializing.
persistence - A process of saving an object into a storage medium.
There, I think that fixes it 
My point is that you first have to
decide what you want to do: sure, you can save an opaque data type into
an HDF5 file, but is that what you really want to do? My rule of thumb
goes like that:
- If you want different instances of the same application to access a
volatile state of one of them, you are closer to serialization. Note,
the serialized object tree can be stored in any resource: flatfile,
database, HDF5, network etcetera.
- If you want (a) To share data between clients and platforms (hw, sw,
oses, vms) at any point in space and time (no need to version, share
business etcetera) (b) To have random access to this data, you should
look for persistence.
Now whether that's HDF5, MySQL, SQLite and so on will depend on the way
your clients are designed to work with the data and on the data itself.
Many of us are more than willing to design a toolkit that persists a
collection of objects and pointed-to arrays you described, because this
effort pays off under one or more of the above mentioned circumstances.
Mix and match strategies exist in cases where you might need to store
both "native" HDF5 datasets and opaque data for a plethora of reasons.
Still, the two might coexist, for me it's difficult to say if one can
replace the other under the same requirements. That's why I am saying
you confuse the two, they're not interchangeable. Or maybe your
requirements are not clear.
HTH
-- dimitris
2008/1/31, Ramakrishnan Iyer <ramakrishnan.iyer@altair.com>:
Dear Dimitris ,
I do understand the difference between serialization and persistence .
serialization - A process of saving an object into a storage medium
Persistence - The object stored has necessary info so that it can be
created back while deserializing .
My question relates to serialization only .
I want to know whether it is advisable to have 3 types of datasets which
consists opaque data types as one of its data components . How will the
performance be affected while writing and reading the datasets ? Will it
be slower when I compare it with datasets which do not have an opaque
datatype .
Also does compression apply to opaque datatypes ( I know it is not
effective on variable length datatypes )
Given below is your reply for Opening Of Datasets wherein you have
mentioned that writing opaque datatypes is faster .
-----Original Message-----
From: Dimitris Servis [mailto:servisster@gmail.com]
Sent: Wednesday, January 30, 2008 7:56 PM
Cc: hdf-forum@hdfgroup.org
Subject: Re: Writing a stream of data/Opaque Data type
Errr...
I think Ramakrishnan confuses serialization with persistence...
-- dimitris
2008/1/30, McCloskey, David L. <McClosDL@westinghouse.com>:
If you use option 1, are you saying you'd create 3 compound data types?
Data, Data1, and Data2? If so, you could make 3 datasets, one for each
of these. You can make pointer type objects within HDF5 to point from
an entry in Data to which entry in the Data1 and Data2 datasets
correspond to this entry in Data. Then when you read an entry from
Data, you can create new entries to Data1 and Data2, populate from those
datasets, and then point to the newly created entries for what you have
in memory as *d1 and *d2. Note that these compound data types will only
be efficient if they are of fixed size.
For option 2, if you write opaque data, it won't be readable
cross-platform if the other platforms have different byte ordering or
word size. Also, you won't be able to store the data very efficiently
in HDF5 and your performance will suffer.
Dave McCloskey
From: Ramakrishnan Iyer [mailto:ramakrishnan.iyer@altair.com]
Sent: Tuesday, January 29, 2008 10:01 PM
To: hdf-forum@hdfgroup.org
Subject: FW: Writing a stream of data/Opaque Data type
FYI
-----Original Message-----
From: Ramakrishnan Iyer
Sent: Monday, January 28, 2008 2:48 PM
To: 'hdf-forum-help@hdfgroup.org'
Subject: Writing a stream of data/Opaque Data type
Dear HelpDesk Team,
The query is with hdf51.6.5
I want to write a structure Data whicb is defined as follows
struct Data{
Data1 *d1;
unsigned int id;
Data2 *d2;
}
The Data1 and Data2 are complicated Data structures .
Now if I am to write Data into HDF5 i can adopt the following approaches
1st approach - Create a corresponding data structure for the Data struct
in hdf5 and write the instances . BUt for this I will have to create
corresponding data structures for Data1 and Data2 . Since these are
quite complicated it will be a very difficult task to create
apporpriate structures for them . It will require a lot of effort . Also
for pointers i will have to use references .
2nd approach - Write the entire data structure as a blob using Opaque
Datatype. Since the Data strucure contains pointers it won't be safe to
assume that the addresses which I have written will be correct when i
read back the data . The right approach will be writing the actual
values pointed to by Data1 and Data2 pointers , as a stream of data . To
further simplify I can write the dataset with a datatype which contains
an unsigned int and a stream of bytes ( for Data1 and Data2)
My Queries:
What is your opinion on these 2 approaches ? Which one ie better/faster
?
Is there any other way by which i can write a stream of data in hdf5
other than the paque datatype ?
Is there a possibility of writing stream of data in hdf5 without knowing
the size of data to be written until the time of actual writing ?
Regards
Ramakrishnan
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
