Favoring Doxygen for documentation over independent wiki documentations

I imagine the net result will be the Doxygen-generated superset of (including the) versions of H5Literate, H5Literate1, and H5Literate2. To get there we’ll have to annotate the H5L* source code. Since HDF5 1.8.22, 1.10.7, 1.12.1, etc. will be upon us sooner or later, we’ll need to think about what has to go into which branches. Unfortunately, but realistically, the story is non-trivial. We have deprecations, renamings, etc. going on. (Strictly speaking, we would need to know how the library API-compatibility is configured to create accurate release documentation, but let’s ignore that.) I think any artifacts/outlines you can produce that help us understand what it really takes start to finish to make this happen would be helpful. If Doxygen creates a lot of other stuff automagically, that’d be a bonus, but let’s drill a bit into the nitty-gritty details. OK? G.

Ok, I’ll give it a run. As I would do it, I’d document H5Literate2(), since that is the most modern function and thus the one that is recommended to be used. From there, there can be a reference to the documented H5Literate1() function, which is annotated with a warning that it should no longer be used as its outdated, and a comment that a H5Literate() define exists which maps to either of those functions depending on the library’s compilation defines, such as those defined by the respective config.h or as provided by the build system (which could override the settings in the config.h) . This documentation would be part of the respective release branches anyway, so each release gets its own address once placed on the web. If that address is f
ixed, then a documentation of 1.12.1 can also link to the documentation of 1.10.7 , but I am not sure if that is useful.

I’m not sure how to create an account for the HDF5 wiki such to edit a page there (“The supplied credentials could not be used for account creation.”), but here’s a possible doxygen version for the H5L group of functions, where I got H5Literate2() annotated for use with doxygen; it references the H5Literate() macro as well:

https://www.fiberbundle.net/hdf5-1.12.0/group___h5_l.html

The comment in H5L.c in doxygen-compatible form then looks like the following, which is a rather small change to the current status from the HDF distribution:

/**@ingroup H5L
 * Iterates over links in a group, with user callback routine,
 *              according to the order within an index.
 *
 *              Same pattern of behavior as H5Giterate.
 *
 * @return      Success:    The return value of the first operator that
 *                          returns non-zero, or zero if all members were
 *                          processed with no operator returning non-zero.
 *              Failure:    Negative if something goes wrong within the
 *                          library, or the negative value returned by one
 *                          of the operators.
 *
 * @param group_id The group ID to iterate over
 * @param idx_type The index type
 * @param order    Specifies the order how to iterate
 * @param idx_p    Pointer to an iteration index such to allow continuing
 *                 a previous iteration
 * @param op       Function pointer for a callback operation to be invoked
 *                 at each iteration
 * @param op_data  User-defined data structure that will be passed on to
 *                 the callback function
 *
 *
 * @see See also H5Literate_by_name(), H5Lvisit2(), H5Lvisit_by_name()
 *
 * @note This function is also available through the H5Literate() macro.
 *
 *
 *-------------------------------------------------------------------------
 */

btw @steven Doxygen uses Bison for source code parsing by default, but it also has the option to enable a clang-based parser.

https://www.hdfgroup.org/register/

https://hdf5.wiki/index.php/RM:Requirements/Doxygen

Seems my older account worked to login, but I cannot edit that page:

You do not have permission to edit this page, for the following reason:
The action you have requested is limited to users in the group: writer.

Does it allow to upload files? I’d upload the doxygen config file and the modified H5L.c then if that’s of interest.

Yes (upload) https://hdf5.wiki/index.php/Special:Upload

No:

Permitted file types: png, gif, jpg, jpeg, webp.

Can’t use this for text files such as source code or the config file.

OK, great start. Let’s maybe dig a little into the details. If you compare https://www.fiberbundle.net/hdf5-1.12.0/group___h5_l.html and https://hdf5.wiki/index.php/RM:H5Literate2 you notice that there’s quite a bit more context on the latter in the sense that I can see all the enumerations, structures, and callback types on the same page. Yes, I can click the corresponding links on the former, but that’s a bit of a cognitive challenge, because, by the time I get to the destination, I can’t remember the context from where I came. Notice also, that these are NOT copies (which would just be an invitation for spreading errors), but transclusions, i.e., there’s only one copy, and if that changes, it changes everywhere. Does Doxygen have a notion of transclusion or (user-defined) templates? The HDF5 documentation is just full of references and stereotypes like that, and you don’t wanna repeat tables such as https://hdf5.wiki/index.php/Template:file_access_flags. G.

Yes, I noticed in the wiki version you had the documentation of the structures used by the function directly under the function. Actually, I found that rather annoying as I missed the option to click on the function parameters to get the information about the parameter types right there, as I am so much used to that way. It is a widespread way of documentation also for the QT library, for instance:

https://doc.qt.io/qt-5/qwidget.html#grab

Using the QT library’s documentation tool, qtdoc, would be an option as well. It has well grown over the years to handle large codes, but I am not sure whether it is as flexible as doxygen. In the past, qtdoc was not freely available, that is why doxygen was created.

Back to the function documentation at hand: There are a couple ways to group functions and data structures together. So far I have not explored all those options, as it’s just to explore in which direction the HDF5 documentation wants to go. In general, doxygen is rather object-oriented focusing around the data structures, and then allows for functions to be related to this data structure. So this design is somewhat the other way round than the current wiki page for H5Literate, which starts with the function and then lists the data structures related to this function. It may be possible to tweak the doxygen docu more in that direction, but it’s a somewhat unusual way as from I am used to from other libraries.

If you say “if it changes it changes everywhere” - does that mean the structure documentation on that page is replicated to multiple places to other functions as well? So if that page is printed, then the printed versions contain redundantly the same information multiple times as it’s same on multiple functions? The doxygen / QTdoc approach avoids redundancy as there is one page for one structure that uniquely defines that data structures and the functions using that data structure just refer to that one. I find that much easier than redundancy, but I could try to look how much such replicated data-inline documentation is doable via doxygen, if that is a requirement. Doxygen also allows for macros and conditionals, so there may be ways.

The table like the file access flags are like the enum documentions in QT, e.g.

https://doc.qt.io/qt-5/qt.html#GestureType-enum

Doxygen does the same:

https://www.fiberbundle.net/hdf5-1.12.0/_h5public_8h.html#a6a6ddd1504d1ed61939d46d91d9441b9

Of course, it can be annotated more.

Here a version of the documentation with the data structure placed into the same H5L group, so its both on the same group page as the function, and in a style with a site navigation panel on the left:

https://www.fiberbundle.net/hdf5-1.12.0-v2/group___h5_l.html

Others should chime in, but I was just following (HDF) precedent. I think that as long as we are not switching to a different page to get the related type info, that would work for me.

What I mean is this. If you look at the source of RM:H5Literate2 you won’t see, for example, the definition of H5_index_t. That’s transcluded in the source as {{H5_index_t}}, which is a reference to Template:H5_index_t. The source (as text) exists only there. If there’s a change or correction to that, I only have to do that in once place and it “spreads” from there, because there are no static pages (There’s caching, but it’s all rendered per request if any of the dependencies changes). H5_index_t appears all over the place and it would be costly to maintain occurrences-by-copy. Transclusion solves that problem. It also works in reverse, i.e., if I would like to know which pages transclude a given template I just go to Special:WhatLinksHere and type in the template name, and that returns all pages using that template.

I’m not sure I understand what you mean by that. If a page contains multiple transclusions of the same content, it will be rendered that many times. (And printed if the user presses the print button.) We don’t have any pages like that, but that’s what would happen.

G.

I like the second attempt much better. What’s involved annotation-wise in the code and the doxygen file? I can create a Git repo and you can check’em in if you like. Let me know! G.

Well, with hand-written documentation of course you can do more, but it’s lots of work. With doxygen you do nothing and you already get a full-featured comprehensive documentation, with HTML pages only one out of 12 (currently) options. For instance, it can also create a help file that is integrated for immediate lookup in eclipse or QT Creator. You don’t get what with a wiki. It can also produce a linear manual via LaTeX to create a PDF file from it, for those who prefer papers. And starting from this do-nothing approach, things can be customized and beautified to a certain extent, even including hand-written HTML code if needed. But it won’t necessarily be exactly the same as a hand-coded wiki page, unless you’d go even that far to modify the doxygen source or contact its developer to extent it (he’s ok, I had contact with him a couple of years ago and was quite open to include a suggestion).

What seems closest to your transclusion is doxygen’s \include{doc} special command:
https://www.doxygen.nl/manual/commands.html#cmdinclude
It allows to insert some documentation from one file in another place, and of course you can use that repeatedly and multiple places. That should be the same as the wiki’s transclusion; I wouldn’t do it though since it is additional work - like editing a wiki - and I personally prefer the documentation of types on another page anyway, like QT does it as well (as one of the documentation pages where I hang around lots).

For the git repo, sure, we can do that. I basically used the source from HDF5 1.12.0, edited two files there and created a doxygen configuration file. The layout with the side menu is the HTML output style option whether to generate a navigation panel or not (which is, like the two C++ websites, a matter of taste). There are more options, you’d need to explore which ones you like, and if users build their own documentation pages, they can use it up to their choice. The more effort is spent on the docu, the better it will become of course.

Would you want me to copy the entire hdf5 1.12 source in this demo repo or just those files that I modified?

Just the files that you modified plus the Doxygen config file will do. I can make you a contributor or you can make a pull request against https://github.com/HDFGroup/doxygen-rm. Thanks, G.

ok done. I also added a trivial makefile to ease running it under windows when it’s not in the path.

Excellent. Thank you very much. I’ll try it and report back. I see there is hope for Fortran as well. :slightly_smiling_face:

1 Like

Following @werner’s example, I’ve mocked up the same examples as in MediaWiki. See H5F and H5L. I worked off the HDF5 development branch and the source file changes can be seen in H5F.c and H5L.c. The Doxygen transclusion and templating are rather primitive and brittle but might be good enough for the task at hand. G.

2 Likes

Looks good! For classes and data structures, Doxygen produces a nice alphabetical listing. Unfortunately, not for functions. Such could probably be created semi-automatically as an extra doxygen page via a script.

We’ve just begun the conversion in earnest. Progress can be tracked here. The underlying Bitbucket branch is this one. Comments welcome! G.

3 Likes