Pure Client-side Dojo API Docs 32


It was a project last year, where yet another inline documentation syntax was “created” and suggested to be used. That triggered our brains and they started spinning caused by the unhappiness of reinventing the wheel, especially for something that is still too much of a step-child for frontend engineers and doesn’t get the necessary dedication anyway, the JavaScript inline docs. So let’s give them some love :-).

Fast forwarding … what came out of it is the API docs viewer which, we here at uxebu, built over the christmas holidays. It is completely client-based, for creating the documentation and retreiving all the information, no server-side code is needed anymore, it’s all done in dojo.

If you want to see a detailed introductional video, sit back, relax and enjoy the following screencast:



Inline doc styles

There are a couple of JavaScript inline documentation tools and styles out there, but no real standard has evolved yet. Maybe because of JavaScript’s dynamic nature it’s hard to fix it to just one style. Though this just makes the task more interesting, actually.

Let’s take a step back and take a look at just three of the various different inline doc styles.

The jQuery way, pretty much textual.

// Execute a callback for every element in the matched set.
// (You can seed the arguments with an array of args, but this is
// only used internally.)
each: function( callback, args ) {

The YUI way, as you will see it’s very JavaDoc like, the method is documented above the actual definition and uses the @ sign for identifiers.

/**
 * Returns a string without any leading or trailing whitespace.  If
 * the input is not a string, the input will be returned untouched.
 * @method trim
 * @since 2.3.0
 * @param s {string} the string to trim
 * @return {string} the trimmed string
 */

trim: function(s){

The dojo way, with proprietary identifiers, inside the function code!

dojo.trim = function(str){
    //  summary:
    //      Trims whitespace from both sides of the string
    //  str: String
    //      String to be trimmed
    //  returns: String
    //      Returns the trimmed string
    //  description:
    //      This version of trim() was selected for inclusion into the base due ...

We see that there are different levels of depth and explicitness in the inline comments. But they all have in common, that important and interesting information are contained inside the source code, that should be exposed to the users of those APIs.

The __doc__ approach

In python, a certain type of comments is available as the docstring property doc of the function it is in. This was actually the initial idea, to provide the docs in this kind of way:

dojo.trim = function(str){}
dojo.trim.<strong>doc</strong> = {
    summary:"Trims whitespace from both sides of the string",
    returns:{type:"String", summary:"Returns the trimmed string"}
}

It’s is pretty obvious, that this is is pretty ugly source code in the first place and secondly a comment is not a comment anymore but becomes source code. Prone to errors, and not really leading to higher productivity, we realized that this approach was actually just a learning stage.
Still it is pretty handy to use the command line in FireBug and have the docs at hand via autocompletion e.g. just typing dojo.trim.doc__ and the docs are there (this feature is currently also implemented, but the __doc property must be generated by a function call afterwards and it is not in the main focus for the current state of the project).

The less disruptive approach

Since the doc approach has the ugly side effect that library authors would have to change all their source code it was dismissed pretty quickly. New ways had to be explored. Driven by the flexibility of JavaScript and it’s dynamic nature we found a nice mix of reflection and minimal file parsing that allows us to extract the information from the existing raw (unpacked and not minified!) source code, without the need to modify it upfront.

First comes the reflection. Using the simple for-in loop over the prototype and the object itself allows us to extract all the methods and properties. Applying a little bit of logic using toString() and alikes and we know if the implementation of this object overrides, inherites or creates this method, we find out the default value of properties and a lot more. In the special case of dojo we can also easily find out the parent class(es). The reflection is actually worth a separate article, since it is quite powerful and allows deep insights into a class’s inner values.

Second step is simple file parsing. Since we know that object “foo” has the method “bar”, we know that in the source code we can find the definition by looking for the following patterns:

foo.bar = function(){}
// OR
foo = {
    bar: function(){}
}
// OR
foo = new function(){
    this.bar =  function(){}
}

There are a couple of special but rare other cases, but the patterns are very simple to detect as you can see. So our FileParser doesn’t really need to understand the JavaScript syntax, it is just looking for exact patterns and is therefore quite fast (we can surely still improve on the performance!).
Using regular expressions we also find out the parameters and the actual docstring itself, which then is parsed by the DocStringParser.

Let’s summarize. We have an object’s/class’s (if you want to call it a class) methods, properties, their according docstring and the docstrings of the class itself. We know the inheritance structure, can determine the type (public, private, etc.) from the name (for now we assume it is private when it starts with an “_”) or soon we will use the docstring identifier “tags” (which was just shortly introduced into dojo as you can see it used here). And we have example code, which is contained in a lot of docstrings. So this is already quite an amount of useful data. Let’s make them useable.

Plugging it all together

The pieces are there, they just need to be plugged together and hopefully a useful API docs viewer will arise. For this special dojo use case, we also had to build a mapper which enables the mapping of a namespace to the correct file(s). E.g. dojo.query() is actually a function inside the dojo object, which would naturally mean it is inside dojo.js, but in this case this method is in a separate file query.js which is “the knowledge” of the mapper.

Building the namespace tree was another challenge, loading all the files in the browser and inspecting the entire namespace via JavaScript would just be an aweful waste of resources. So currently there is a small python script which generates the JSON files which contain the contents of the tree to the left. But the high flexibility of using dynamic extending of objects (like e.g. NodeList-fx does) mixins and other things make it hard to rely purely on the generated contents of the tree, so they are partly hand-crafted too.

dojo UI exampleThe UI is purely dojo, showing off some of the neat features, like the slick ExpandoPane. In my eyes it looks very nice, with a lot of attention to the detail. The UI was actually the part that needed the least time to build, thanks to Nikolai’s deep knowledge it was a piece of cake. There are of course a couple of glitches like the wrapping of the tabs, but hopefully this is just another kick in the right direction to speed up those kind of fixes. Because what is there is already much more than just the base of a GUI toolkit with some widgets, it is a full-fledged-ready-to-go-batteries-included-kick-ass set of UI components. (Ok, enough wallowing, yes I am biased :-).) So look for yourself and let us know what you think.

The future

As you know it too, working on those kind of projects always spins off a lot of new ideas. So we want to share some with you and maybe even get some help on some of them. First: we will put the source code out on google code (or alike) and it will be open for contributions, of course. We just need to do some renaming to ensure future-proof enhancements won’t just clobber this thing and make it unmaintainable.

Another idea is to configure the API docs viewer for custom namespaces and as long as the inline comments are dojo style it should be just a snap of a finger to get your own project’s API docs out. Well, actually the snap of a finger is the creation of the above mentioned JSON file, which contains the tree structure. But one day we will also have an auto-detector which builds the tree initially out of traversing through a given existing namespace (given that it is entirely loaded).

Idea No. 734 :-) was to provide the API viewer for other libraries too. The architecture is made this way, that the reflection and file parsing are separated and could be extended to work with another inline doc style syntax (we made first experiments with it). In order to do this right we think about having a common interface where to put and get the apidoc data from. A json-schema for API doc data is required. I have started on one, but this task really is huge when done right.

Of course the doc property will be provided. Currently when working with libraries or function that I don’t know I mostly use toString() on it and learn from the source code. I could also very well imagine myself using doc if it was there. So keep looking for that, I think it can change the way one uses the FireBug console and learns about libraries.

Since we know all the objects, methods on the one hand and the according docs on the other hand we can tell how good the doc coverage is. This heads into the direction of continuous integration and imho that is really where JavaScript still has a long way to go.

And there are a couple more ideas, like running the inline doctests in the UI, integrating the API viewer with Bespin and allowing to throw the api viewer onto any page using a bookmarklet, which would nicely let you browse the JavaScript API of any site, no matter if you want to learn, fix something or whatever. To mention just some of the ideas …

Now if you want to get your hands dirty and play with the code, see how it can integrate with your own code or just have offline dojo api documentation, drop us an email ( labs (a t) uxebu.com ) and we will make sure a copy of the code lands in your mailbox. Also, if you find bugs, have ideas, want to contribute, contact us and we will exchange ideas to make this tool the best available javascript api viewer out there.


About Wolfram Kriesing

Wolfram Kriesing has more than fourteen years professional experience in IT. The early involvement in web technologies provides him with deep knowledge and experience for designing and implementing stable and scalable architectures.

  • http://alex.dojotoolkit.org Alex Russell

    Amazing work!

    A couple of questions:

    * will the source be available? Say as a Google Code project?
    * In the tree view, some things show up as classes (e.g., dojo.NodeList) but they’re not listed as classes when I open the “dojo” object in the main list…am I doing something wrong?
    * how fresh are these docs. Can there be a version of this system that works against a continuously updated SVN checkout of Dojo? That would be a huge boon to the folks writing inline docs.
    * how does this system work with custom namespaces? It looks like it would be simple to make it work, but I’m curious about the ease of integration
    * can you use Neil’s new JSON/XML output from the PHP parser to help speed up initial parsing/load-time for some modules? Is there some way for those systems to collaborate?

    Again, this is just phenomenal work, and it’s beautiful to look at. Nice work!

    Regards

  • http://www.uxebu.com Wolfram Kriesing

    @Alex, thanks for the nice words.
    * Yes we will put it on google code, there is even the project created already http://code.google.com/p/dools/ , we just wanted to focus on getting it out before changing the infrastructure again :-) we had been to slow already anyways hehe.
    * Actually in the dojo NS there are only the methods and properties listed, not the classes, because that could be endless but its worth a thought … thx
    * the docs are generated right from the underlying source at the time you are clicking, so modifying the sourcecode on the server and reloading the class/method/whatever has the updated docs in them, so it can be against any version of svn, yes
    * any namespace can be added. only two limitations currently 1) the inline docs have to be dojo style 2) the JSON file for the tree (on the left) has to be created either by hand or by a script like the one we used http://dojodocs.uxebu.com/uxebu/apps/apidoc/tools/make-tree.py
    * i think speeding up the stuff would be done by generating the output and putting it into a JSON file, much like Neils approach I guess. Which is actually what the approach of creating an according json-schema is for

    thx again, looking forward to more input

    Wolfram

  • http://www.uxebu.com Wolfram Kriesing
  • http://unscriptable.com/ unscriptable

    Excellent work guys!

  • http://lazutkin.com/ Eugene Lazutkin

    Very nice and useful project!

  • http://almaer.com/blog Dion Almaer

    Great stuff. If we can help with the Bespin integration in anyway let us know.

    I am refactoring the Bespin codebase to enable the ability to reuse the editor in a much easier way, that should help.

  • http://www.cb1inc.com Chris Barber

    Congrats! This looks so sweet!

  • http://www.uxebu.com Nikolai Onken

    @Dion

    that’s great, we have a _very_ prototypal integration running already as a proof of concept and would love to talk to you guys about how to push this forward.

  • http://qooxdoo.org Fabian Jakobs

    Great work guys. A standalone API viewer is really a plus. Its important to push the standards of client side JavaScript development. The UI reminds me of our qooxdoo API viewer . Did you take a look at it?

  • http://www.uxebu.com Tobias Klipstein

    Hey Fabian, the qooxdoo API viewer definitely was one of the JavaScript API viewers we’ve looked at very often :-) Thanks for your preliminary work

  • Pingback: Ajaxian » JavaScript API Viewer: Client-side parsing with Dojo

  • Les

    Very nice work!

    I have two suggestions:
    1.) Overflowing tabs in the detail pane should slide. It would look better if they didn’t display in multiple rows.
    2.) Force visibility of the tree node for the selected tab. For example, if I click a tab ‘dijit.MenuBarItem’, the corresponding tree node should be selected and become visible.

  • http://www.uxebu.com Nikolai Onken

    Hi Les,

    thanks for your comments! There is a patch available for floating tabs, we need to review this asap and get it into codebase.
    Also good idea with the tree, we will file a ticket and check if we can add that feature.

  • Pradeep Maddineni

    The work you have done is fabulous!!! I have a question, in your screencast, you have asked to email you guys to get the source code for this tool, I am so excited to check out the source code, would you mind sending it to me? Any help would be highly appreciated.

    Thanks,
    Pradeep

  • http://weblogs.asp.net/bleroy Bertrand Le Roy

    This is great, but I’d like to point out two omissions. First, the OpenAjax format for documentation meta-data, which is getting mature and quite usable and is already supported in a few JS IDE: http://www.openajax.org/member/wiki/OpenAjax_Metadata_Specification, and the .NET documentation format, which has been supported in Visual Studio for a while: http://weblogs.asp.net/bleroy/archive/2007/04/23/the-format-for-javascript-doc-comments.aspx
    Cheers!

  • http://www.uxebu.com Nikolai Onken

    @Pradeep, thats great, could you send an email to labs _at_ uxebu.com so we can send the code to you?

    @Bertrand, you are absolutely right, we are looking into providing XML or Json which validates against the OpenAjax Metadata spec. The great thing about this is that all the doc strings are stored in objects – turning those into a standardized format should be fairly simple.

  • Pingback: JavaScript API Viewer: Client-side parsing with Dojo | Guilda Blog

  • http://www.at9t.com at9t

    wow~It’s a little bit Python like

  • http://www.uxebu.com Wolfram Kriesing

    I have committed the first stuff to the SVN http://code.google.com/p/dools/source/list
    We have decided to put it all under the namespace “dools” which shall mean “dojo tools” and we hope “dools” doesnt mean anything bad :-).
    Why “dojo tools”? Simply because there is much more to come.

    Wolfram

  • http://www.uxebu.com Wolfram Kriesing

    @at9t Python like in which sense?

    btw the api tree widget (the one seen on the left hand side of the api docs viewer) has landed in the dools svn last night
    step by step we are getting there to have al commponents under dools :-)

  • http://javascriptmvc.com Justin

    Awesome work. I’d like to get JMVC’s JS doc parser working with this.

  • http://happycloudmoments.blogspot.com Joyce

    This is great!

  • matsuri

    great job!

    best part is the real-time doc building and the awesome interface.. of corz everything with dojo :]
    looking forward to use it for my own modules

    see u soon for a beer in munich

  • ofr

    Amazing work!

    Just an idea from my side:
    That thing as an AIR application could be amazing! It need not be AIR, just something that allows local file access. Then you could keep the docs cached in some database, only requiring an update if the .js file has a more recent last change date.
    Following this approach, you could even auto-generate the tree from time to time or on user request.

    And of course, I’d love to see downloadable archives… But please take the time to get it stable!

  • http://www.uxebu.com Nikolai Onken

    Hey ofr,

    thanks for your comments. An AIR version would be great and we will work on this once we have moved everything into google code.
    Keep checking the blog for updates on this.

  • Michael Johnson

    This looks great, so far.

    I only have one complaint. It doesn’t work in Opera. When I try to load a item, I get a Cannot convert undefined or null to Object error. Unfortunately, I don’t have time to track down the reason right now, and I’m not sure when I will. But it would be awesome if it could be made to work.

  • http://shawnzhu.blogspot.com/ Shawnzhu

    Is dojo 1.3 required? what about dojo 1.2?

  • http://www.uxebu.com Nikolai Onken

    Shawnzu,

    the tool works with any version of Dojo, though the tree which lets you browse includes modules which are 1.3 specific.
    So if you would want to run the tool against Dojo < 1.3 all you’d need to do is to make sure that new modules are not in the navigation (just change the json)

  • http://www.uxebu.com Nikolai Onken

    @Michael Johnson, we will look at that, it should work in Opera as well :)

  • http://[email protected] Asceffige
  • http://www.baolimotuo.cn 暴力摩托下载

    btw the api tree widget (the one seen on the left hand side of the api docs viewer) has landed in the dools svn last night

  • http://www.sayrich.com Rich

    Thanks for your code , i add some it in my codes, useful.