The perfect excuse for acceptance testing

It may surprise some, that open source, as revolutionary phenomenon as it has been, is actually a very conservative way to develop software. But that also makes it such a great fit for stable organisations like established universities. Successfully participating in open source projects require long term (and often personal) commitment, but should also result in pleasant surprises.

One of such surprising result from our open source collaboration has been the ability to generate documentation screenshots as a side-effect from acceptance testing. Or to put it other way, we are able to script our documentation screenshot as inline acceptance tests for the end-user features being documented. We are even able to do "documentation driven development": Write acceptance criteria into documentation skeleton and see the documentation complete with screenshot as the project develops.

We are able to script our documentation screenshot as inline acceptance tests for the end-user features being documented. We are even able to do "documentation driven development".

For example, once the required build tools and configuration boiler plate is in place, writing our end-user documentation with scripted (and therefore always up-to-date) screenshots may look like this:

Submitting a new application
============================

For submitting a new application, simply click the button and fill the form
as shown in the pictures below:

..  figure:: submit-application-01.png

    Open the form to submit a new application by pressing the button.

..  figure:: submit-application-02.png

    Fill in the required fields and press *Submit* to complete.

..  code:: robotframework

    *** Test Cases ***

    Show how to submit a new application
        Go to  ${APPLICATION_URL}

        Page should contain element
         ...  css=input[value="New application"]
        Capture and crop page screenshot
        ...  submit-application-01.png
        ...  css=#content

        Click button  New application

        Page should contain  Submit new application
        Page should contain element
        ...  css=input[value="Submit"]

        Input text  id=form-widgets-name  Jane Doe
        Input text  id=form-widgets-email  jane.doe@example.com
        Capture and crop page screenshot
        ...  submit-application-02.png
        ...  css=#content

        Click button  Submit
        Page should contain  New application submitted.

This didn't become possible just by overnight, and it would not have been possible without ideas, contributions and testing from the community. It all started almost by an accident: a crucial part between our then favourite Python testing framework and Robot Framework based cross-browser acceptance testing with Selenium was missing. we needed that part to enable one of our interns to tests their project, we choosed to implement it, many parts clicked together, and a few years later we had this new development model available in our toolbox.
The specific technology for writing documentation with acceptance testing based screenshots is a real mashup by itself:

• The final documentation is build with Sphinx, which is a very popular software documentation tool written in Python.
• The extensibility of Sphinx is based on a plain text formatting syntax called ReStructuredText and its mature compiler implementation Docutils, which also also written in Python.
• Together with a Google Summer of Code student, whom I mentored, we implemented a Sphinx-plugin to support inline plain text Robot Framework test suites within Sphinx-documentation.
• In Robot Framework test suites, we can use its Selenium-keywords to test the web application in question and capture screenshots to be included in the documentation.
• We also implemented a library of convenience keywords for annotating and cropping screenshots by bounding box of given HTML elements.
• For Plone, with a lot of contributions from its friendly developer community, a very sophisticated Robot Framework integration was developed, to enable setting up and tearing down complete Plone server with app specific test fixtures directly from Robot Framework test suites with a few keywords.
• Finally, with help from the Robot Framework core developers, the new ReStructuredText support for Robot Framework was implemented, which made it possibly to also run the written documentation with scripted screenshots as a real test suite with the Robot Framework's primary test runner (pybot).

Once you can script both the application configuration and screenshots, fun things become possible. For example, here's an old short scripted Plone clip presenting all the languages supported by Plone 4 out of the box. Only minimal editing was required to speedup the clip and add the ending logo:

Any cons? Yes. It's more than challenging to integrate this approach into workflows of real technical writers, who don't usually have a developer background. In practice, automated acceptence tests must be written by developers, and also ReStructuredText is still quite technical syntax to write documentation. Therefore, even for us this toolchain still remain quite underused.