IEX2H5: Fast IEX TOPS to HDF5 Conversion - Call the Doctor 8/5/25
In this episode, The HDF Group’s Gerd Heber will interview Steven Varga (@steven.varga), a long-time HDF5 supporter and author of H5CPP, about his newly released IEX2H5 conversion utility. IEX2H5 is a high-performance C++ utility that converts IEX (Investors EXchange) Transport Protocol (IEX-TP) packet captures, specifically IEX TOPS (Top of Book) market data, into structured HDF5 datasets. This tool is designed to facilitate the efficient storage, analysis, and integration of large-scale financial data for applications such as backtesting, algorithmic trading, and scientific/financial time-series analytics. It supports both Regular Time Series (RTS) for aggregated snapshots and Irregular Time Series (IRTS) for high-fidelity tick streams.
To join, just jump on the zoom:
Launch Meeting - Zoom
August 5, 2025,12:20 p.m. central time US/Canada
Note: There will be no Call the Doctor next week, on August 12.
1 Like
During this HDF Clinic episode, The HDF Group’s Gerd Heber and Steven Varga of Varga Consulting met to discuss the challenges of processing and storing tick data, a critical component for financial analytics and algorithmic trading. Steven has just released IEX2H5, a tool designed to convert raw network traffic data in PCAP format into analytics-ready HDF5, optimizing access time and storage through compression. Using standard Gzip, compression ratios of 3:1 to 4:1 can be easily achieved. However, much higher compression ratios are possible with more bespoke compression methods.
HDF5 is perhaps the most popular format for storing large amounts of financial data due to its ability to load data into various systems, its compatibility with virtually all programming languages, its direct control over tick streams, and its longevity and archival qualities. Steven’s tool, IEX2H5, is used for backtesting and building frameworks with IEX Exchange data to extract and analyze trade counts and identify top-performing stocks from this vast dataset with high-speed processing. It represents a significant step towards data liberation and reproducibility in financial market research. Give your research a boost by using IEX2H5 and spread the word by starring the GitHub repository at https://github.com/vargaconsulting/iex2h5!
For additional information on processing financial data (or any other large datasets!) with HDF5, reach out to The HDF Group.
2 Likes
Thank you Gerd and the HDF Group for the kind invitation and for hosting the Call the Doctor session! It was a pleasure to present IEX2H5 and share how HDF5 enables efficient storage and analytics on high-frequency market data.
For those interested, full documentation, benchmarks, and usage examples are available here:
IEX2H5 - VargaLABS
Feel free to reach out with any questions or suggestions — always happy to collaborate and push the limits of HDF5 for financial data.
— Steven Varga
1 Like
