ACID transactions?

HDF5 Library
1

Hello HDF5 folks!

I know there is SQL interface for HDF5:


and Kita
https://www.hdfgroup.org/solutions/hdf-kita
which is presented as a high-performance enterprisey database.

But to my knowledge, none of those does support Atomicity, Consistency,
Isolation of ACID; nor plain HDF5 library does. Are there any
attempts/plans/intentions to implement proper ACID transactions over
HDF5? Let’s consider the following example:

  1. Create group /pub/bottles/99
  2. Create dataset /pub/bottles/99/chromatogram
  3. Put data into /pub/bottles/99/chromatogram
  4. Update attribute “total” of /pub
    all in ACID way.

Best wishes,
Andrey Paramonov

2

@paramon, as always thanks for the thought-provoking questions.

We discussed this internally – including HDF colleague Gerd Heber who focuses on application and data services that sit stop HDF5 – and we suggest:

  1. If by ACID you’re just generally / loosely speaking about how to scale access to multiple people – i.e. “database” for HDF5 – then there’s a number of options, e.g. HDF5 on Hadoop (HBase/Drill + HDFS + HDF5 + HDFS VFD).

  2. However, if you’re talking about strict ACID compliance, then we’re not working on anything right now, but would strongly encourage you to lead the creation and discussion of an RFC, e.g. explore different notions of transactions and ACID-ity for HDF5, use cases, etc.

Thanks,

– Dave

3

Hi David,
Nice to hear from you :slight_smile:

  1. However, if you’re talking about strict ACID compliance

This!

then we’re
not working on anything right now, but would strongly encourage you
to lead the creation and discussion of an RFC, e.g. explore
different notions of transactions and ACID-ity for HDF5, use cases, etc.

Thanks for the proposal. Unfortunately it wouldn’t work this way: I’ve
been promised JIRA write access for ~2 years and BitBucket pull-request
access for ~1 year but nothing actually happened. It’s just too slow a
development model to practically consider creating/contributing
non-trivial features.

Best wishes,
Andrey Paramonov

4

Andrey,

I was trying to reach you for some time to provide you with the credential for Bitbucket without much success.

Could you please reach me at epourmal@hdfgroup.org and we will go from there?

Thanks,

Elena