Hallo all,
my hdf5 data is generated by a solver compiled in 64bit. My question is if it is possible/safe to read this data with a software using the 32bit libraries?
Thx,
Jan
Hi Jan,
In general, yes, I believe it should be.
I can think of some special cases you might need to consider if they
apply to you...
a) file size is on 64-bit system grows to be greater than 2^(32-1) (2
Gigs) and you don't have large file support on your 32-bit system. You
won't be able to read your file on the 32 bit system. You can 'fix' this
by using the 'family' Virtual File Driver I think.
b) you write 64-bit integer dataset AND the values stored in it are
indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
have an appropriate integer type (e.g. long int or long long or
something) to read it back into. Note, if you write 'int' on your 64 bit
system and the values are small enough to fit into 32 bit, I am not sure
if HDF5 will handle reading it into an 'int' on your 32 bit system or
not. I think it will but I am not sure. You could explicitly attempt to
read it into long int or long long or int64 if your system supports
those types AND they are indeed 64 bit.
These situations are highly UNlikely. So, I'd expect you would have no
problems. But, I think its important to mention them for completeness. I
can't think if any other cases where there could be problems.
Mark
···
On Sun, 2010-05-23 at 08:50, Jan Linxweiler wrote:
Hallo all,
my hdf5 data is generated by a solver compiled in 64bit. My question is if it is possible/safe to read this data with a software using the 32bit libraries?
Thx,
Jan
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
Hi all,
Hi Jan,
In general, yes, I believe it should be.
I can think of some special cases you might need to consider if they
apply to you...a) file size is on 64-bit system grows to be greater than 2^(32-1) (2
Gigs) and you don't have large file support on your 32-bit system. You
won't be able to read your file on the 32 bit system. You can 'fix' this
by using the 'family' Virtual File Driver I think.
Yes, that's true.
b) you write 64-bit integer dataset AND the values stored in it are
indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
have an appropriate integer type (e.g. long int or long long or
something) to read it back into. Note, if you write 'int' on your 64 bit
system and the values are small enough to fit into 32 bit, I am not sure
if HDF5 will handle reading it into an 'int' on your 32 bit system or
not. I think it will but I am not sure. You could explicitly attempt to
read it into long int or long long or int64 if your system supports
those types AND they are indeed 64 bit.
Yes, the conversion will work, but your values may be truncated.
These situations are highly UNlikely. So, I'd expect you would have no
problems. But, I think its important to mention them for completeness. I
can't think if any other cases where there could be problems.
I agree. This is a primary use case for HDF5 and should work without any difficulty.
Quincey
···
On May 23, 2010, at 11:27 PM, Mark Miller wrote:
Mark
On Sun, 2010-05-23 at 08:50, Jan Linxweiler wrote:
Hallo all,
my hdf5 data is generated by a solver compiled in 64bit. My question is if it is possible/safe to read this data with a software using the 32bit libraries?
Thx,
Jan
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
So, on this point, I just wanted to make sure I understand. Will HDF5
silently and without error convert a dataset consisting of 64 bit ints
to 32 bit ints if all values are within range of the destination type?
If just one value is out of range, will it fail the WHOLE read or just
truncate that one value and if latter, what kind of error, if any, is
triggered?
Mark
···
On Mon, 2010-05-24 at 09:40, Quincey Koziol wrote:
Note, if you write 'int' on your 64 bit
> system and the values are small enough to fit into 32 bit, I am not sure
> if HDF5 will handle reading it into an 'int' on your 32 bit system or
> not. I think it will but I am not sure. You could explicitly attempt to
> read it into long int or long long or int64 if your system supports
> those types AND they are indeed 64 bit.Yes, the conversion will work, but your values may be truncated.
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
If 64-bit values are small enough in their numerical range to fit into
32bit range, it would be better to store them as 32bit "int32_t" and not
int's anyway, even on a 64bit machine...
It's overkill to store all and everything in 64bit even if not required.
For instance, graphics hardware doesn't know about 64bit int's on indices,
even on 64bit machines handling otherwise huge data beyond 2GB.
Werner
···
On Mon, 24 May 2010 11:40:46 -0500, Quincey Koziol <koziol@hdfgroup.org> wrote:
On May 23, 2010, at 11:27 PM, Mark Miller wrote:
b) you write 64-bit integer dataset AND the values stored in it are
indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
have an appropriate integer type (e.g. long int or long long or
something) to read it back into. Note, if you write 'int' on your 64 bit
system and the values are small enough to fit into 32 bit, I am not sure
if HDF5 will handle reading it into an 'int' on your 32 bit system or
not. I think it will but I am not sure. You could explicitly attempt to
read it into long int or long long or int64 if your system supports
those types AND they are indeed 64 bit.Yes, the conversion will work, but your values may be truncated.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
Hi Mark,
Note, if you write 'int' on your 64 bit
system and the values are small enough to fit into 32 bit, I am not sure
if HDF5 will handle reading it into an 'int' on your 32 bit system or
not. I think it will but I am not sure. You could explicitly attempt to
read it into long int or long long or int64 if your system supports
those types AND they are indeed 64 bit.Yes, the conversion will work, but your values may be truncated.
So, on this point, I just wanted to make sure I understand. Will HDF5
silently and without error convert a dataset consisting of 64 bit ints
to 32 bit ints if all values are within range of the destination type?
Yes.
If just one value is out of range, will it fail the WHOLE read or just
truncate that one value and if latter, what kind of error, if any, is
triggered?
It will normally just truncate that one value and won't issue an error. If an application desires to see all the exceptions that occur during datatype conversion when performing I/O on a dataset, it can use the H5Pset_type_conv_cb() routine (http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetTypeConvCb) to register a callback that can intervene or record them.
Quincey
···
On May 24, 2010, at 11:46 AM, Mark Miller wrote:
On Mon, 2010-05-24 at 09:40, Quincey Koziol wrote:
Hi Werner,
Hmm. Well, it certainly is true that 'int' on a 64 bit machine can wind
up being twice as many bytes as 'int' on a 32 bit machine.
But, unless I am really, really cramped for memory, I would just as soon
write my code to use 'int' everywhere and let the compiler and I/O
library handle everything for me automagically.
I don't want to have to worry about sizeof(int) unless I really need to.
And, most often, I don't. Unless I am moving data between 64-bit and
32-bit systems a lot. And then, I am still most interested in portable
code and data.
But, I do see your point that there is a 2x memory savings to be had by
choosing the type carefully.
Mark
···
On Mon, 2010-05-24 at 09:57, Werner Benger wrote:
On Mon, 24 May 2010 11:40:46 -0500, Quincey Koziol <koziol@hdfgroup.org> wrote:
> On May 23, 2010, at 11:27 PM, Mark Miller wrote:
>>
>> b) you write 64-bit integer dataset AND the values stored in it are
>> indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
>> have an appropriate integer type (e.g. long int or long long or
>> something) to read it back into. Note, if you write 'int' on your 64 bit
>> system and the values are small enough to fit into 32 bit, I am not sure
>> if HDF5 will handle reading it into an 'int' on your 32 bit system or
>> not. I think it will but I am not sure. You could explicitly attempt to
>> read it into long int or long long or int64 if your system supports
>> those types AND they are indeed 64 bit.
>
> Yes, the conversion will work, but your values may be truncated.
>
If 64-bit values are small enough in their numerical range to fit into
32bit range, it would be better to store them as 32bit "int32_t" and not
int's anyway, even on a 64bit machine...It's overkill to store all and everything in 64bit even if not required.
For instance, graphics hardware doesn't know about 64bit int's on indices,
even on 64bit machines handling otherwise huge data beyond 2GB.Werner
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
Hi Mark,
it's not just the memory savings aspect. I was recently stumbling across
the issue to get meshes on a graphics card, and of course you'll try to be
as general enough to store it all just using "int". However, as GPU's don't
support 64bit integer, it required to convert all the 64bit int's to 32bit
prior to shuffling them to the graphics card. On a 32bit machine that of
course was not required, making the rendering on the 32bit machine faster
than on 64bit. This should just not be the case... so unless there are
data which require more than 4 billion vertices (which can't be directly
rendered any more anyway), doing consistently 32bit everywhere is just
more efficient (in this situation).
Werner
···
On Mon, 24 May 2010 12:10:02 -0500, Mark Miller <miller86@llnl.gov> wrote:
Hi Werner,
Hmm. Well, it certainly is true that 'int' on a 64 bit machine can wind
up being twice as many bytes as 'int' on a 32 bit machine.But, unless I am really, really cramped for memory, I would just as soon
write my code to use 'int' everywhere and let the compiler and I/O
library handle everything for me automagically.I don't want to have to worry about sizeof(int) unless I really need to.
And, most often, I don't. Unless I am moving data between 64-bit and
32-bit systems a lot. And then, I am still most interested in portable
code and data.But, I do see your point that there is a 2x memory savings to be had by
choosing the type carefully.Mark
On Mon, 2010-05-24 at 09:57, Werner Benger wrote:
On Mon, 24 May 2010 11:40:46 -0500, Quincey Koziol <koziol@hdfgroup.org> wrote:
> On May 23, 2010, at 11:27 PM, Mark Miller wrote:
>>
>> b) you write 64-bit integer dataset AND the values stored in it are
>> indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
>> have an appropriate integer type (e.g. long int or long long or
>> something) to read it back into. Note, if you write 'int' on your 64 bit
>> system and the values are small enough to fit into 32 bit, I am not sure
>> if HDF5 will handle reading it into an 'int' on your 32 bit system or
>> not. I think it will but I am not sure. You could explicitly attempt to
>> read it into long int or long long or int64 if your system supports
>> those types AND they are indeed 64 bit.
>
> Yes, the conversion will work, but your values may be truncated.
>
If 64-bit values are small enough in their numerical range to fit into
32bit range, it would be better to store them as 32bit "int32_t" and not
int's anyway, even on a 64bit machine...It's overkill to store all and everything in 64bit even if not required.
For instance, graphics hardware doesn't know about 64bit int's on indices,
even on 64bit machines handling otherwise huge data beyond 2GB.Werner
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
Hi Mark,
thank you for your answer.
I made a simple test and did not observe any problems so far. Until now I'm not dealing with large integers and doubles.
Thx,
Jan
···
Today's Topics:
1. Re: 64bit and 32bit interoperability (Mark Miller)
----------------------------------------------------------------------
Message: 1
Date: Sun, 23 May 2010 21:27:33 -0700
From: Mark Miller <miller86@llnl.gov>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] 64bit and 32bit interoperability
Message-ID: <1274675252.18786.8.camel@114 miller86-1.llnl.gov
miller86-1
Content-Type: text/plainHi Jan,
In general, yes, I believe it should be.
I can think of some special cases you might need to consider if they
apply to you...a) file size is on 64-bit system grows to be greater than 2^(32-1) (2
Gigs) and you don't have large file support on your 32-bit system. You
won't be able to read your file on the 32 bit system. You can 'fix' this
by using the 'family' Virtual File Driver I think.b) you write 64-bit integer dataset AND the values stored in it are
indeed large enough to REQUIRE 64 bits and your 32 bit system does NOT
have an appropriate integer type (e.g. long int or long long or
something) to read it back into. Note, if you write 'int' on your 64 bit
system and the values are small enough to fit into 32 bit, I am not sure
if HDF5 will handle reading it into an 'int' on your 32 bit system or
not. I think it will but I am not sure. You could explicitly attempt to
read it into long int or long long or int64 if your system supports
those types AND they are indeed 64 bit.These situations are highly UNlikely. So, I'd expect you would have no
problems. But, I think its important to mention them for completeness. I
can't think if any other cases where there could be problems.Mark
On Sun, 2010-05-23 at 08:50, Jan Linxweiler wrote:
Hallo all,
my hdf5 data is generated by a solver compiled in 64bit. My question is if it is possible/safe to read this data with a software using the 32bit libraries?
Thx,
Jan
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851------------------------------
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.orgEnd of Hdf-forum Digest, Vol 11, Issue 24
*****************************************