Doxygen Squirrel filter

[Wormnest]

Since doxygen can't handle Squirrel scripts very well without preprocessing I decided to have a go at a filter to improve that situation.

Requirements
The filter uses python 2.7. It may work with other versions but no guarantees.
It was tested on Windows 7. Other OS'es may work but again no guarantees.

Known problems

• Functions need to be defined inside the class or they won't be shown. It's not easy to fix this: we would have to parse the source twice. First to find all the functions not defined inside the class and then a second time to add them inside the class. Example: MailAI TownManager.nut
• Inline code in the file outside of any function can confuse doxygen too sometimes. Example: AILib.List main.nut the code at the bottom.
• Multi line string constants not supported: (starting with @" ).

Usage instructions, readme and download
https://bitbucket.org/jacobb/squirrel2doxygen/overview

Bug reports and comments welcome. Since I use python only occasionally I wouldn't be suprised if there are parts that could be improved.

What we now need is for the AI and GS libraries to be made compatible with doxygen documentation so we can have a nice online documentation for them

[frosch]

Nice idea! It would make sense to run this on the #Coop Devzone compile farm, since it hosts mosts libraries.

I've added a ticket: https://dev.openttdcoop.org/issues/7700

[planetmaker]

Oh, that's a nice addition indeed

EDIT: I've now installed that script as well as doxygen on the DevZone. AI and game script authors can now already create a build target which builds the doxygen documentation. We'll discuss how to automate that for the existing repositories

Further, I created also a local clone of your repository: https://dev.openttdcoop.org/projects/squirrel2doxygen I would very welcome you to register with DevZone (and providing an ssh key) so that I can give you write access to that repository; maybe you can maintain that tool there? It automatically updates with each push the corresponding toolchain on the DevZone.

[Wormnest]

Further, I created also a local clone of your repository: https://dev.openttdcoop.org/projects/squirrel2doxygen I would very welcome you to register with DevZone (and providing an ssh key) so that I can give you write access to that repository; maybe you can maintain that tool there? It automatically updates with each push the corresponding toolchain on the DevZone.

You should already have my ssh key. My Ai WormAI is available there too.

I'll try to maintain it but I sometimes loose interest for a while. I do intend to see this weekend if I can manage to add support for functions belonging to a class being defined outside of the class.

[planetmaker]

Further, I created also a local clone of your repository: https://dev.openttdcoop.org/projects/squirrel2doxygen I would very welcome you to register with DevZone (and providing an ssh key) so that I can give you write access to that repository; maybe you can maintain that tool there? It automatically updates with each push the corresponding toolchain on the DevZone.

You should already have my ssh key. My Ai WormAI is available there too.

I'll try to maintain it but I sometimes loose interest for a while. I do intend to see this weekend if I can manage to add support for functions belonging to a class being defined outside of the class.

Ai, right. Well, then you do have commit access Things are up and running on the DevZone - it just needs configuring projects to build and publish their documentation. Any particular wishlist on your part from the list of AI/GS and libraries from http://dev.openttdcoop.org/projects ?

[Wormnest]

Ai, right. Well, then you do have commit access Things are up and running on the DevZone - it just needs configuring projects to build and publish their documentation. Any particular wishlist on your part from the list of AI/GS and libraries from http://dev.openttdcoop.org/projects ?

Nice, thanks. Is there any info yet on how to configure it or is that still in the works? Or is that something that has to be done serverside.

The libraries I use myself are SuperLib and AILib.List.
Basic support in SuperLib should be as easy as changing the comment start from /* to /**.

AILib.List needs the functions added inside the class or my parser adapted as mentioned before.

[planetmaker]

Ai, right. Well, then you do have commit access Things are up and running on the DevZone - it just needs configuring projects to build and publish their documentation. Any particular wishlist on your part from the list of AI/GS and libraries from http://dev.openttdcoop.org/projects ?

Nice, thanks. Is there any info yet on how to configure it or is that still in the works? Or is that something that has to be done serverside.

The libraries I use myself are SuperLib and AILib.List.
Basic support in SuperLib should be as easy as changing the comment start from /* to /**.

AILib.List needs the functions added inside the class or my parser adapted as mentioned before.

While all the rest works somewhat automatically, setting up a project to build unfortunately is something which does not yet work automated. So I'll see to configure build jobs for those libraries first You should talk with Zuu about Superlib Maybe he accepts patches.

[Zuu]

I'm indeed open to accept a patch. Though, it is a bit more work for you than just flipping a few /* to /**.

Some classes like eg. datastore.nut have their functions divided into sections which I don't know if that is supported by Doxygen. But the actual function comments can be exposed using /** instead of /*. Some comments are however made using // as well and sometimes there is one comment for many similar methods. Or a paragraph below a section heading describing the methods belonging to that section.

Another thing is that ideally a Doxygen doc should not expose internal names such as _SuperLib_DataStore but rather SuperLib.DataStore.

There are also some "strange" things in main.nut that control that story.nut is only imported and exposed in the GS version of SuperLib. It is basically a keyword that the AI -> GS translation script look for.

[planetmaker]

I've created jobs for all GS and AI libraries which I found hosted on the DevZone as well as for WormAI. The generated documentation for these projects can be found on the bundles server in the respecitve dirs: http://bundles.openttdcoop.org/ I fear most, if not all projects, will need some comment adoption in order for the documentation to become usable.

If you feel like, you are invited to do so not only for your project(s) (including to improve the squirrel2doxygen filter) but also for the old AI "standard libraries" (A*, Pathfinder Road, Pathfinder Rail, Binary Heap, Fibonacci Heap, Priority Queue, AILib Tile, AILib String, AILib, AILib Direction, AILib common, AILib List). For other libraries I suggest to contact their maintainers and / or post patches as issue in their project tracker.

There are some libraries where the repos are not structured suitably for the default build rules; they would need special care with commits to them with a specifically crafted makefile and Doxygen file (MinchinWeb AI meta library, Script communication Library, GS Toy Library)

Additionally I'll happily create build tasks for game scripts and AIs as needed on request (but libraries seemed more important to me right now).

[Wormnest]

Some classes like eg. datastore.nut have their functions divided into sections which I don't know if that is supported by Doxygen. But the actual function comments can be exposed using /** instead of /*. Some comments are however made using // as well and sometimes there is one comment for many similar methods. Or a paragraph below a section heading describing the methods belonging to that section.

I knew it would be more work than just using /** that's why I mentioned basic support
You can group member functions together and have a name/description for that group. Groups can however not be nested.

Comments that get attached to multiple functions should be possible as long as the following settings is added for doxygen:

Code: Select all

DISTRIBUTE_GROUP_DOC   = YES

Attached as an example a fixed version of datastore.nut and example output of the datastore class.

datastore-doxygen-output.png

(73.66 KiB) Downloaded 5 times

About replacing of _SuperLib_XXX and the GS versus AI. I will have to think about that to find a preferable generic solution.

[Wormnest]

I've created jobs for all GS and AI libraries which I found hosted on the DevZone as well as for WormAI. The generated documentation for these projects can be found on the bundles server in the respecitve dirs: http://bundles.openttdcoop.org/ I fear most, if not all projects, will need some comment adoption in order for the documentation to become usable.

Is the squirrel filter getting called? A few hours ago I added some doxygen documentation for WormAI.
Looking at the generated documentation it stops right after the imports.
This is how it looked to me before I wrote the filter suggesting that it's possibly not running correctly yet.

If you feel like, you are invited to do so not only for your project(s) (including to improve the squirrel2doxygen filter) but also for the old AI "standard libraries" (A*, Pathfinder Road, Pathfinder Rail, Binary Heap, Fibonacci Heap, Priority Queue, AILib Tile, AILib String, AILib, AILib Direction, AILib common, AILib List).

I'll try to do that. First need to see about a few improvements to the filter and I think there's another bug because it has trouble parsing cEngineLib. Or at least doxygen has trouble parsing our output there.

[planetmaker]

I've created jobs for all GS and AI libraries which I found hosted on the DevZone as well as for WormAI. The generated documentation for these projects can be found on the bundles server in the respecitve dirs: http://bundles.openttdcoop.org/ I fear most, if not all projects, will need some comment adoption in order for the documentation to become usable.

Is the squirrel filter getting called? A few hours ago I added some doxygen documentation for WormAI.
Looking at the generated documentation it stops right after the imports.
This is how it looked to me before I wrote the filter suggesting that it's possibly not running correctly yet.

If you feel like, you are invited to do so not only for your project(s) (including to improve the squirrel2doxygen filter) but also for the old AI "standard libraries" (A*, Pathfinder Road, Pathfinder Rail, Binary Heap, Fibonacci Heap, Priority Queue, AILib Tile, AILib String, AILib, AILib Direction, AILib common, AILib List).

I'll try to do that. First need to see about a few improvements to the filter and I think there's another bug because it has trouble parsing cEngineLib. Or at least doxygen has trouble parsing our output there.

Ho ho, you're right. There was (still is?) a lapse on my part... I didn't properly put the FILTER_PATTERNS as I didn't specify the path to the filter script properly... but it's still odd...
You see the actually used Doxygen file in your working dir: https://jenkins.openttdcoop.org/job/ai-worm/ (and similarily for other projects)
The file is found in the DevZone repo and can be adjusted: http://dev.openttdcoop.org/projects/hom ... le.devzone

[Wormnest]

Ho ho, you're right. There was (still is?) a lapse on my part... I didn't properly put the FILTER_PATTERNS as I didn't specify the path to the filter script properly... but it's still odd...
You see the actually used Doxygen file in your working dir: https://jenkins.openttdcoop.org/job/ai-worm/ (and similarily for other projects)
The file is found in the DevZone repo and can be adjusted: http://dev.openttdcoop.org/projects/hom ... le.devzone

In the console output from Jenkins I see 2 errors for WormAI and 1 from AILib.List.
The common error is

Code: Select all

10:36:46 /usr/bin/env: python
: No such file or directory

Following that I see no console output at all of my script so I don't think it's used.

The other error is doxygen complaining about a missing end quote on line 106 which is probably the line where the version number is added. The problem here is most likely that I use a variable instead of a number and maybe doxygen wants a number there. It does however show the variable name instead of the number in the documentation header.

Is it possible to have the following non default values added to the default Doxygen. Especially the first since it seems that a lot of scripts assume that @brief is not necessary. The second one makes it possible to document multiple functions with one comment.

Code: Select all

JAVADOC_AUTOBRIEF      = YES
DISTRIBUTE_GROUP_DOC   = YES

[Wormnest]

I have updated the script. It now adds function declaration for any class member functions that are declared outside the class back inside the class.
It now also handles enum declarations that don't have a ";" after the closing brace "}".

I'm not too sure what to do about class names with a dot in it. It confuses doxygen somewhat but usually not too bad.
I think this is not allowed in c++. To get it working we would have to replace the dot with something else.
However since the effects I have seen so far are not too bad I'm probably going to leave it like this for now.

[planetmaker]

Ho ho, you're right. There was (still is?) a lapse on my part... I didn't properly put the FILTER_PATTERNS as I didn't specify the path to the filter script properly... but it's still odd...
You see the actually used Doxygen file in your working dir: https://jenkins.openttdcoop.org/job/ai-worm/ (and similarily for other projects)
The file is found in the DevZone repo and can be adjusted: http://dev.openttdcoop.org/projects/hom ... le.devzone

In the console output from Jenkins I see 2 errors for WormAI and 1 from AILib.List.
The common error is

Code: Select all

10:36:46 /usr/bin/env: python
: No such file or directory

Following that I see no console output at all of my script so I don't think it's used.

It seems that the problem in this case were the type of line endings. I converted the file to unix-style line endings and it seems to work now.

The other error is doxygen complaining about a missing end quote on line 106 which is probably the line where the version number is added. The problem here is most likely that I use a variable instead of a number and maybe doxygen wants a number there. It does however show the variable name instead of the number in the documentation header.

Yes, that's a case where my automatic extraction of the version from the script fails, though I expected it to give different results than I see in the the Doxygen file... I'll try to optimize that later today. The code I use is in the build script itself: https://hg.openttdcoop.org/misc/files/t ... ild.sh#L88

Is it possible to have the following non default values added to the default Doxygen. Especially the first since it seems that a lot of scripts assume that @brief is not necessary. The second one makes it possible to document multiple functions with one comment.

Code: Select all

JAVADOC_AUTOBRIEF      = YES
DISTRIBUTE_GROUP_DOC   = YES

Sure. Added.

In essence the build log now looks much more promising, I think: https://jenkins.openttdcoop.org/job/ai-worm/12/console
http://bundles.openttdcoop.org/ai-worm/ ... ormAI.html

[Zuu]

If possible, it would be good if library.nut is not parsed. In the link below, member functions from library.nut are included in the library class. Those methods are only accessible to OpenTTD, not AI/GS authors.

http://bundles.openttdcoop.org/superlib ... erLib.html

A bit misleading it seems that attributes of a class is described as "public attributes". Technically Squirrel doesn't support private attributes, but some may by convention be private. It would perhaps be better to label it as just 'attributes' or is there a way to tell Doxygen which attributes that are supposed to be private?

[Wormnest]

In essence the build log now looks much more promising, I think: https://jenkins.openttdcoop.org/job/ai-worm/12/console
http://bundles.openttdcoop.org/ai-worm/ ... ormAI.html

Ah yes, the output looks mostly the way I see it here too with a few minor differences possibly because I use a different version of doxygen.
All the doxygen warnings about argument not found are to be expected since Squirrel has untyped parameters and doxygen expects c++ syntax.

Edit: Maybe it would be a good idea to regenerate the documentation for the other libraries to have an idea which ones need the most work or to see possible problems with the script.

If possible, it would be good if library.nut is not parsed. In the link below, member functions from library.nut are included in the library class. Those methods are only accessible to OpenTTD, not AI/GS authors.

http://bundles.openttdcoop.org/superlib ... erLib.html

I agree including library.nut and even info.nut is generally not that useful. It is possible to exclude files from being parsed by adding them to Doxyfile like this

Code: Select all

EXCLUDE                = info.nut \
                         library.nut 

I wouldn't mind if that would be added to the default Doxyfile.

A bit misleading it seems that attributes of a class is described as "public attributes". Technically Squirrel doesn't support private attributes, but some may by convention be private. It would perhaps be better to label it as just 'attributes' or is there a way to tell Doxygen which attributes that are supposed to be private?

Doxygen expects c++ as input so what the filter does is to try to make Squirrel code look more like c++.
Making everything inside a class public was on purpose. I add the public: keyword as the first word inside a class definition otherwise doxygen would think everything is private. Private symbols are by default not shown in doxygen documentation so that would not be very useful.

I have seen two versions of marking private functions and variables in Squirrel code:
Starting the name with an underscore and a comment like /* private: */ versus /* public: */.

Taking both types into consideration and adding the correct c++ keywords for them is on my todo list but I think in most cases it should already be obvious that those are to be considered private.

[Zuu]

You don't need to overdo it by covering every possible way to express it, but it may be useful to document what ways to tell Doxygen or your script which attributes that are private and which are public. And other useful hints perhaps as a wiki page in the AI section: https://wiki.openttd.org/AI:Main_Page

[planetmaker]

In essence the build log now looks much more promising, I think: https://jenkins.openttdcoop.org/job/ai-worm/12/console
http://bundles.openttdcoop.org/ai-worm/ ... ormAI.html

Ah yes, the output looks mostly the way I see it here too with a few minor differences possibly because I use a different version of doxygen.
All the doxygen warnings about argument not found are to be expected since Squirrel has untyped parameters and doxygen expects c++ syntax.

Edit: Maybe it would be a good idea to regenerate the documentation for the other libraries to have an idea which ones need the most work or to see possible problems with the script.

Sure. But as that's a manual task I first wanted some feedback whether it now gives the expected results
EDIT: I triggered a re-build for all libraries.

If possible, it would be good if library.nut is not parsed. In the link below, member functions from library.nut are included in the library class. Those methods are only accessible to OpenTTD, not AI/GS authors.

http://bundles.openttdcoop.org/superlib ... erLib.html

I agree including library.nut and even info.nut is generally not that useful. It is possible to exclude files from being parsed by adding them to Doxyfile like this

Code: Select all

EXCLUDE                = info.nut \
                         library.nut 

I wouldn't mind if that would be added to the default Doxyfile.

The default can be changed to whatever suits you (plural, AI and GS developers) best.
But I take it, that both, info.nut and library.nut are only to be excluded when we talk about a library, yes? Or do those two nut files never contain 'useful' information?

[Alberth]

I think excluding files is only useful when you want to present an API. In that case you don't want the clutter of the meta-information.
If you see it as a program, then imho all source code is relevant, even if probably don't look much in info.nut