I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
HDF5 does not have transactions, concurrency or replication features. It is not a database. It is designed for storing and sharing large array-like datasets in a hierarchically structured file (see http://www.hdfgroup.org/HDF5/whatishdf5.html for some more details and documentation).
Quincey
···
On Aug 18, 2010, at 12:28 AM, elisa sibarani wrote:
Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
what I mean is not really the same feature as SQL Server has, but all features regarding reading from a file. I've read your reply to hdf forum as follows:
True concurrent access to a file when it is being written to is not
currently supported. We are working to add a single-writer/multiple-reader
access mode for the next release (1.10.0), but it's not available yet. The
current way to have pseudo-concurrent access to a file is found here: http://www.hdfgroup.org/hdf5-quest.html#grdwt
Quincey
therefore, this H5Fflush is able to handle concurrency for reading a file?
From: Quincey Koziol <koziol@hdfgroup.org>
Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Wednesday, August 18, 2010, 9:47 AM
Hi Elisa,
On Aug 18, 2010, at 12:28 AM, elisa sibarani wrote:
Hi all,
I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
HDF5 does not have transactions, concurrency or replication features\. It is not a database\. It is designed for storing and sharing large array\-like datasets in a hierarchically structured file \(see http://www.hdfgroup.org/HDF5/whatishdf5.html for some more details and documentation\)\.
Quincey
Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
what I mean is not really the same feature as SQL Server has, but all features regarding reading from a file. I've read your reply to hdf forum as follows:
True concurrent access to a file when it is being written to is not
currently supported. We are working to add a single-writer/multiple-reader
access mode for the next release (1.10.0), but it's not available yet. The
current way to have pseudo-concurrent access to a file is found here: http://www.hdfgroup.org/hdf5-quest.html#grdwt
Quincey
therefore, this H5Fflush is able to handle concurrency for reading a file?
Yes, when the SWMR feature is finished, it will support a particular form of concurrent access to HDF5 files. Along those "database" lines, we are finishing up support for journaling metadata updates to HDF5 files, which resembles transactions on modifications to HDF5 files, at least in the sense that all the changes from an HDF5 API routine will either be atomically reflected in the file, or not. It is also possible to implement a "mirroring" virtual file driver for replicating modifications to HDF5 files, although we don't have one currently implemented.
Quincey
···
On Aug 18, 2010, at 9:57 AM, elisa sibarani wrote:
From: Quincey Koziol <koziol@hdfgroup.org>
Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Wednesday, August 18, 2010, 9:47 AM
Hi Elisa,
On Aug 18, 2010, at 12:28 AM, elisa sibarani wrote:
> Hi all,
>
> I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
HDF5 does not have transactions, concurrency or replication features. It is not a database. It is designed for storing and sharing large array-like datasets in a hierarchically structured file (see http://www.hdfgroup.org/HDF5/whatishdf5.html for some more details and documentation).
Quincey
> Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
>
> Thanks all.
>
> Regards,
> Elisa
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Thank you for your answer. It is indeed help so much, but is it possible that the next release (1.10.0) is already finish and when will be the distribution of the new library?
Where can I find the proposal documentation or any kind of document which explain about the journaling metadata and the mirroring virtual file driver for replication?
From: Quincey Koziol <koziol@hdfgroup.org>
Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Thursday, August 19, 2010, 8:04 AM
Hi Elisa,
On Aug 18, 2010, at 9:57 AM, elisa sibarani wrote:
what I mean is not really the same feature as SQL Server has, but all features regarding reading from a file. I've read your reply to hdf forum as follows:
True concurrent access to a file when it is being written to is not
currently supported. We are working to add a single-writer/multiple-reader
access mode for the next release (1.10.0), but it's not available yet. The
current way to have pseudo-concurrent access to a file is found here: http://www.hdfgroup.org/hdf5-quest.html#grdwt
Quincey
therefore, this H5Fflush is able to handle concurrency for reading a file?
Yes, when the SWMR feature is finished, it will support a particular form of concurrent access to HDF5 files\. Along those "database" lines, we are finishing up support for journaling metadata updates to HDF5 files, which resembles transactions on modifications to HDF5 files, at least in the sense that all the changes from an HDF5 API routine will either be atomically reflected in the file, or not\. It is also possible to implement a "mirroring" virtual file driver for replicating modifications to HDF5 files, although we don't have one currently implemented\.
Quincey
From: Quincey Koziol <koziol@hdfgroup.org>
Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Wednesday, August 18, 2010, 9:47 AM
Hi Elisa,
On Aug 18, 2010, at 12:28 AM, elisa sibarani wrote:
> Hi all,
>
> I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
HDF5 does not have transactions, concurrency or replication features\. It is not a database\. It is designed for storing and sharing large array\-like datasets in a hierarchically structured file \(see http://www.hdfgroup.org/HDF5/whatishdf5.html for some more details and documentation\)\.
Quincey
> Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
>
> Thanks all.
>
> Regards,
> Elisa
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Thank you for your answer. It is indeed help so much, but is it possible that the next release (1.10.0) is already finish and when will be the distribution of the new library?
The 1.10.0 is not finished yet and may not be ready for another 6+ months.
Where can I find the proposal documentation or any kind of document which explain about the journaling metadata and the mirroring virtual file driver for replication?
Here's the URL for the metadata journaling RFC:
The mirroring VFD was just an idea I've tossed around, it's not implemented (or on our current plan to implement) and doesn't have an RFC describing it. I don't think it would be too difficult to implement however, if you'd like to give it a try.
Quincey
···
On Aug 26, 2010, at 4:02 PM, elisa sibarani wrote:
From: Quincey Koziol <koziol@hdfgroup.org>
Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Thursday, August 19, 2010, 8:04 AM
Hi Elisa,
On Aug 18, 2010, at 9:57 AM, elisa sibarani wrote:
> what I mean is not really the same feature as SQL Server has, but all features regarding reading from a file. I've read your reply to hdf forum as follows:
>
> True concurrent access to a file when it is being written to is not
> currently supported. We are working to add a single-writer/multiple-reader
> access mode for the next release (1.10.0), but it's not available yet. The
> current way to have pseudo-concurrent access to a file is found here:
> http://www.hdfgroup.org/hdf5-quest.html#grdwt
>
> Quincey
>
> therefore, this H5Fflush is able to handle concurrency for reading a file?
Yes, when the SWMR feature is finished, it will support a particular form of concurrent access to HDF5 files. Along those "database" lines, we are finishing up support for journaling metadata updates to HDF5 files, which resembles transactions on modifications to HDF5 files, at least in the sense that all the changes from an HDF5 API routine will either be atomically reflected in the file, or not. It is also possible to implement a "mirroring" virtual file driver for replicating modifications to HDF5 files, although we don't have one currently implemented.
Quincey
> thanks
>
>
>
> --- On Wed, 8/18/10, Quincey Koziol <koziol@hdfgroup.org> wrote:
>
> From: Quincey Koziol <koziol@hdfgroup.org>
> Subject: Re: [Hdf-forum] Handle concurrency, transaction (or any features provided) in HDF5 compare to SQL Server
> To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
> Date: Wednesday, August 18, 2010, 9:47 AM
>
> Hi Elisa,
>
> On Aug 18, 2010, at 12:28 AM, elisa sibarani wrote:
>
> > Hi all,
> >
> > I want to ask is there anyone of you can shares information about HDF5 features available until now? If I want to recommend my company to use HDF5, now I have to make sure it'll work at least as reliable as SQL Server. Therefore, I want to make a comparison about services/features available in SQL Server such as transaction (locking process), concurrency, replication (for back up) and so on. How I can get such information about HDF5?
>
> HDF5 does not have transactions, concurrency or replication features. It is not a database. It is designed for storing and sharing large array-like datasets in a hierarchically structured file (see http://www.hdfgroup.org/HDF5/whatishdf5.html for some more details and documentation).
>
> Quincey
>
> > Or maybe if anyone has experience (at least once) to move from SQL Server to HDF5, maybe you can share about what list of things I should put in mind for this changes?
> >
> > Thanks all.
> >
> > Regards,
> > Elisa
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@hdfgroup.org
> > http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
The compilation process failed with the following message :
"""
Intel(R) Visual Fortran Intel(R) 64 Compiler Professional for
applications running on Intel(R) 64, Version 11.1 Build 20100203
Package ID: w_cprof_p_11.1.060
Copyright (C) 1985-2010 Intel Corporation. All rights reserved.
30 DAY EVALUATION LICENSE
ifort: NOTE: The evaluation period for this product ends on 29-sep-2010 UTC.
: catastrophic error: Variable H5P_FLAGS too large for NTCOFF. Bigger
than 2GB. Use heap instead
in file amelethdf.obj, line 0, column 0
compilation aborted for XXX.f90 (code 3)
"""
I don't understand what happens.
Is it a known issue ?
