Case Study: Creating a Course Registration System in Short Notice (Jedi Style!)

Here is a nice story, that almost got lost in translation (literally - its published in Finnish last summer).

It was a warm, late afternoon on May 2016, when we at plone-support received a message, asking for help. Perhaps we weren't like Obi-Wan Kenobi on that, but close enough: A customer needed to have a new course registration system and migration from the old one so soon as August 2016. Whoa, three monts, summer coming on and other projects on the road too. Should we just skip it or take the million credits to get even with Jabba?

Well, here's a spoiler alert: https://www.fdpe.fi/

 

1. The case of The Finnish Doctoral Programme in Economics

A Joint Doctoral Programme of the Economics Disciplines of 10 Participating Universities in Finland. The Finnish Doctoral Programme in Economics (FDPE) promotes high-quality doctoral education in economics in Finland by organising systematic and diverse doctoral courses and seminars.

 Here's is a rough translation of the initial email we received in May 2016.

We will need a platform, where we can manage nation wide economics doctoral programmes cooperation. To be more clear: we need a web-based system, that allows students from participating universities to apply for doctoral studies courses from another universities. Ideally the principles of the system would be: 
- Each university inserts a batch of courses as part of the cooperation
- The courses became a course pool, that will be communicated to the studens via the system
- The students from each univeristy may register for the coursers on the course pool
- After a course a teacher may communicate the grade of each student to the student's home university via the system
- Once per year the registrations are gathered and counted, how many participants there are from university x to university y
- Based on those calculations it is possible to settle the compensation between universityes. 

The point is to build a platform to gather registrations and communicate students grades and registration information. The platform should be as simple and robust as possible. 

Our schedule is uncomfortably tight in the sense that the system (or at least te application feature) should be online at latest in August.
****************

Ok, so we got the message. And why they chose to ask us, is because of another Plone-site that shows events and gathers applications there: http://inforte.jyu.fi

We discussed about the case, and thought it was quite a challenge to present so late at spring, (with all the vacations and other projects we had in store for the summer). But we also thought, that we had a quite toolset for this kind of system:

• Plone would take care of the website, permissions, workflows, security and robustness. 
• Courses would be dexterity content types - easy to create and to modify if needed
• The registration form could be easily created using Plone Form Gen add on - of which we had lot's of experience 
• The information about participants could be delivered in the system using plone permission management and intranet features
• This was a good case to try out new Plone 5 and its out of the box responsive layout

So we were set to talk with the customer and see what else they might have in mind.

After a positive meeting, we also had this information:

• Migration was needed, but only for visible content
• The visual desing of the site would be secondary feature
• If only the application forms and course information were online already in August, the rest of system could be finished by September
• We could use HAKA indentity federation service to allow students and staff from different universities to login with their own userid and password. We could have created internal useris's for the admins and could have forced students to insert more information at the application form, but this was way better solution. We had used HAKA before with Plone in many occasions (Like Moniviestin video publishing platform)
• The couldn't use national Opintopolku.fi -service (at least not yet) for doctoral studies. 
• The project had fixed deadline and the customer was willing to prioritize and to focus on the most important features first

So we decided to set out for this venture!

2. Plan B. 

Actually, we had no plan B, so we came up with plan A, to divide the development into clear phases, visible to the customer:

1. At first we would set up a preview-site: the customer could then start using Plone and to learn how to use it. 
2. A light theme customization
3. Site name, domain, certificate and HAKA-federation bureaucracy was set to motion
4. Migration to the old public content (scrape public content)
5. Shibboleth-integration (HAKA) development for Plone 5
6. An event like Course content type would be created
7. Using a Plone Form Gen -solution we had created before on another case to automatically populate basic user information about the person filling the form.  
8. Using a previous solution to generate reports on every separate application form. 
9. A Plone folder, where a teacher stores the saved Excel-files about course grades
10. Other features, like managing permissions,  communication, instructions, planning the application form and theming.

So, we had plans.

Some features, like good security, content management, intranet features,  permission and group management, responsive web design and workflows are built in to Plone CMS. We used the latest Plone 5 version on this case.

The actual work

We started by creating a preview-site in june 2016. This way it was easy for the customer to follow progress. So we added an empty Plone 5 site with minor theming and gave admin permissions to the customer so they could get to know how Plone works. We also did the migration of old public content at this point.

When the migrated content was checked and deemed okay, and the bureaucracy with the domain name done, we published the new site in june 2016 - only without the form functionalities. However, the customer could already do some content editing and checking that everything was okay.

Next up was the HAKA federation. Using HAKA federation, every higher education student in Finland could log in to the site using his/her own user id and password to make logging in easy. At the same time our customer could easily gather the name, email and organisation of every enrolled student. In addition they could add new site administrators from different organizations. Plone allows very fine-grained permission management.

Last, but not least, we developed the registration form and a new content type called Course, with some basic data (start date, end date, information). When a student goes to a course, there is a registration link which opens the form with all necessary fields already populated. The registration form was done using Plone FormGen add on. We had used that add on for over a thousand forms at JYU, so we knew it was stable and customizable.

The registrations were collected to a .csv-table, easy to open in Excel or Google Sheets and easy to use from every organization participating FDPE programme.

In the end we created short instructions for the site admins on how to manage content. The instructions were just Plone pages with some images and links.

Example one: Course Page

Example 2: Registration Form

Summary

The customer was very satisfied with the development and results, and so were we.

This kind of development in this short time period wouldn't have been possible if we didn't have the suitable technology available and knowhow on how to properly utilize it.

Additionally the customer was really cooperative - they focused on getting things done, they did their part on the content and other bureaucracy, and most importantly they knew what to prioritize.

We had limited time and limited resources - and usually it is a good thing if you can actually prioritize and altogether just cut out some features if needed. First things first, and then some.

By using free Open Source components and no need to pay any licence fees, the only thing we (and the customer) needed to focus on, was the actual development.

Now, after almost a year of usage there has been near to none feature requests or incidents using the system, and definitely nothing major. But there have been hundreds of successful registrations and everything seems to be working fine. 

If you want to know more, please contact plone-support@jyu.fi

It is party time!

The perfect excuse for acceptance testing

It may surprise some, that open source, as revolutionary phenomenon as it has been, is actually a very conservative way to develop software. But that also makes it such a great fit for stable organisations like established universities. Successfully participating in open source projects require long term (and often personal) commitment, but should also result in pleasant surprises.

One of such surprising result from our open source collaboration has been the ability to generate documentation screenshots as a side-effect from acceptance testing. Or to put it other way, we are able to script our documentation screenshot as inline acceptance tests for the end-user features being documented. We are even able to do "documentation driven development": Write acceptance criteria into documentation skeleton and see the documentation complete with screenshot as the project develops.

We are able to script our documentation screenshot as inline acceptance tests for the end-user features being documented. We are even able to do "documentation driven development".

For example, once the required build tools and configuration boiler plate is in place, writing our end-user documentation with scripted (and therefore always up-to-date) screenshots may look like this:

Submitting a new application
============================

For submitting a new application, simply click the button and fill the form
as shown in the pictures below:

..  figure:: submit-application-01.png

    Open the form to submit a new application by pressing the button.

..  figure:: submit-application-02.png

    Fill in the required fields and press *Submit* to complete.

..  code:: robotframework

    *** Test Cases ***

    Show how to submit a new application
        Go to  ${APPLICATION_URL}

        Page should contain element
         ...  css=input[value="New application"]
        Capture and crop page screenshot
        ...  submit-application-01.png
        ...  css=#content

        Click button  New application

        Page should contain  Submit new application
        Page should contain element
        ...  css=input[value="Submit"]

        Input text  id=form-widgets-name  Jane Doe
        Input text  id=form-widgets-email  jane.doe@example.com
        Capture and crop page screenshot
        ...  submit-application-02.png
        ...  css=#content

        Click button  Submit
        Page should contain  New application submitted.

This didn't become possible just by overnight, and it would not have been possible without ideas, contributions and testing from the community. It all started almost by an accident: a crucial part between our then favourite Python testing framework and Robot Framework based cross-browser acceptance testing with Selenium was missing. we needed that part to enable one of our interns to tests their project, we choosed to implement it, many parts clicked together, and a few years later we had this new development model available in our toolbox.
The specific technology for writing documentation with acceptance testing based screenshots is a real mashup by itself:

• The final documentation is build with Sphinx, which is a very popular software documentation tool written in Python.
• The extensibility of Sphinx is based on a plain text formatting syntax called ReStructuredText and its mature compiler implementation Docutils, which also also written in Python.
• Together with a Google Summer of Code student, whom I mentored, we implemented a Sphinx-plugin to support inline plain text Robot Framework test suites within Sphinx-documentation.
• In Robot Framework test suites, we can use its Selenium-keywords to test the web application in question and capture screenshots to be included in the documentation.
• We also implemented a library of convenience keywords for annotating and cropping screenshots by bounding box of given HTML elements.
• For Plone, with a lot of contributions from its friendly developer community, a very sophisticated Robot Framework integration was developed, to enable setting up and tearing down complete Plone server with app specific test fixtures directly from Robot Framework test suites with a few keywords.
• Finally, with help from the Robot Framework core developers, the new ReStructuredText support for Robot Framework was implemented, which made it possibly to also run the written documentation with scripted screenshots as a real test suite with the Robot Framework's primary test runner (pybot).

Once you can script both the application configuration and screenshots, fun things become possible. For example, here's an old short scripted Plone clip presenting all the languages supported by Plone 4 out of the box. Only minimal editing was required to speedup the clip and add the ending logo:

Any cons? Yes. It's more than challenging to integrate this approach into workflows of real technical writers, who don't usually have a developer background. In practice, automated acceptence tests must be written by developers, and also ReStructuredText is still quite technical syntax to write documentation. Therefore, even for us this toolchain still remain quite underused.