Callbacks, coroutines, and API calls from callbacks

The documentation warns about the use of callbacks, subsequent API calls, and co-routines.

Before considering co-routines, are there any restrictions on what APIs if any can be called safely from a callback? My guess is that if I am iterating attributes via a callback, I probably should not be modifying the list of attributes I am iterating over.

Now for co-routines:

Attention
C++ Developers using HDF5 C-API functions beware:
Several functions in this C-API take function pointers or callbacks as arguments. Examples include H5Pset_elink_cb(), H5Pset_type_conv_cb(), H5Tconvert(), and H5Ewalk2(). Application code must ensure that those callback functions return normally such to allow the HDF5 to manage its resources and maintain a consistent state. For instance, those functions must not use the C setjmp / longjmp mechanism to leave those callback functions. Within the context of C++, any exceptions thrown within the callback function must be caught, such as with a catch(…) statement. Any exception state can be placed within the provided user data function call arguments, and may be thrown again once the calling function has returned. Exceptions raised and not handled inside the callback are not supported as it might leave the HDF5 library in an inconsistent state. Similarly, using C++20 coroutines cannot be used as callbacks, since they do not support plain return statements. If a callback function yields execution to another C++20 coroutine calling HDF5 functions as well, this may lead to undefined behavior.

Co-routines also exist in Go, Julia, and even Python, so the concern here extends beyond C++ now.

We should catch all errors thrown in a callback, but I’m not clear what other special situations would apply to co-routines that would not apply to all callbacks.

One application of using co-routines is to turn callback based iteration into yield based iteration. Here the callback would place information into a first-in-first-out queue or channel and then yield to the receiver of the channel. The receiver would then do some processing before returning control flow to the original callback. The advantage to this is that iteration can adapted to the native interation protocol or interface of the language.

My request here is if we can be more specific about the concern here. I suspect that if a callback or related co-routines do not make nested calls into the HDF5 library then all should be fine.

The two main sources of concern are the library and file states (= the sources of all evil…). Calling APIs that don’t change the file state in a callback should be fine. Calling APIs that change the file state may have knock-on effects, which are not apparent from the API.

W.r.t. coroutines, things might be more complicated. How do languages store/represent the execution state of coroutines? Different languages may implement that differently, and there might be even different implementations for the same language for different compilers. How do we know that successive execution states of coroutines are consistent with the file and library states?

Yes, we will try to be more precise here.

G.

1 Like