I’m wondering if I’m the only person seeing this (in which case it’s something to do with my system), or if anyone else has encountered this:
When running some code across many different files (they follow the same format, but shape of datasets will vary), I’m getting strange deadlocks sometimes, whereby the process gets stuck waiting for a futex (based on looking at proc/using strace). Each file runs in it’s own process (the code is designed to take a single file and print out or plot some information from said file), and there appears to be no pattern as to whether a file will trigger this or not. Sometimes the deadlock never occurs (when running the same code on the same files, with the same version of dependencies), but other times it occurs quite rapidly. I’m not using multiprocessing, nor spawning my own threads directly (though it’s entirely possible some library is). I’ve seen this on HDF5 1.10.8 and 1.12.2 (from Debian), with h5py 3.9 (I need to check earlier versions and see if this is a regression).
I’m currently running the code under ltrace to see if that gives me a hint as to where the deadlock occurs, but currently it’s not deadlocked yet.