I am writing to find out if there is an open source C/C++ library which allows one to supply a collection of particle (position + additional channels like velocity, density, temperatures, vorticity etc.) and builds/layout them in an Octree fashion and persisting them to disk based on HDF5, which has corresponding methods to retrieve those particles at non-empty leaf node, one node at a time.
I am looking for a memory efficient way to retrieve small chunks (at known bounds, hence octree node) of a larger data sets which has 10^8 particles per time step on commodity PC machines with 8-16Gig of RAM of which I targeting a memory use budget of 1-2Gig as there are other geometry in memory.
Cheers
···
--
Nicholas Yue
Graphics - Arnold, Alembic, RenderMan, OpenGL, HDF5
Custom Dev - C++ porting, OSX, Linux, Windows http://au.linkedin.com/in/nicholasyue
I'm working on similar problems of storing big data including especially but not only large particle sets (point clouds). The layout we use to store particles with additional fields such as velocity, temperature etc. is the Fiberbundle HDF5 (F5) model, which is an open source library, described at http://www.fiberbundle.net/ .
It's just a layout "recommendation" on top of HDF5, it does nothing by itself in terms of data transformation or storing points as octrees, it merely suggests a layout for complex data types. An octree can be imposed on it, as well as a multi-resolution hierarchy of particles based on clustering. On our side, we're using kD-trees, not octrees, but actually the idea would be to store points via space-filling curves which are like an "image" of a kd-tree. This part however is work in progress and not yet implemented. This could be done via multiple index sets that provide alternative point iteration schemes. Accessing subsets based on data queries could be connected with the fastbit library, which provides bit-optimized indexing of HDF5 datasets; it may be part of a future HDF5 release, I am just experimenting with it to get first experiences.
However, these indexing abilities are yet under development on our side, so it depends on your time scale and urgency how far you could use it. For the mere data layout, the F5 library is available right now. It can still be coupled with another indexing library as well.
Cheers,
Werner
···
On 06.01.2015 05:57, Nicholas Yue wrote:
Hi,
I am writing to find out if there is an open source C/C++ library which allows one to supply a collection of particle (position + additional channels like velocity, density, temperatures, vorticity etc.) and builds/layout them in an Octree fashion and persisting them to disk based on HDF5, which has corresponding methods to retrieve those particles at non-empty leaf node, one node at a time.
I am looking for a memory efficient way to retrieve small chunks (at known bounds, hence octree node) of a larger data sets which has 10^8 particles per time step on commodity PC machines with 8-16Gig of RAM of which I targeting a memory use budget of 1-2Gig as there are other geometry in memory.
Cheers
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
