RoboCon 2018 and Robot Framework Jupyter support

It's already over a week since I got back home from the first Robot Framework conference ever – RoboCon 2018. It was a pleasure to be there, and I really feel privileged that I was accepted there as a speaker.

My RoboCon 2018

RoboCon 2018 was a single day conference about Robot Framework test automation ecosystem, held in the heart of Helsinki, Finland, on 18th of January 2018. The conference venue was quite if not completely full, so there must have been around 250 participants. The event was in English and had pretty good international participation. Yet, most of the participants came from Finland, where Robot Framework has become de-facto standard for test automation.

Pekka Klärck presenting the history, present and future of Robot Framework

RoboCon 2018 had only a single track, so that had to be packed to include something for everyone in its diverse audience. In addition, there was plenty of time and a separate space for networking with the other participants and conference sponsors. There was also organized social program before and after the conference, but unfortunately, I was unable to attend those at this time.

In my opinion the program was well balanced: The conference started with introductory talks, continued with variety of differnet case studies (Kone, Plone and Texas Instruments), and ended with more technical talks about specific Robot Framework addons (SeleniumLibrary, the most awesome new REST library and pabot). And in the middle of everything, there was my personal favorite: Ed Manlove's talk about building successful open source communities. My presentation was called Robot Framework in Plone CMS Project: a case study, story and some technical details, how Robot Framework got successfully adopted in distributed open source community behind Plone.

The most important part of this conference, of course, was getting a lot of Robot Framework users and developers to meet in the same place at the same time. After all, RoboCon 2018 was the first Robot Framework conference ever. My personal absolute highlight during the conference was meeting a former Plonista, Ed Manlove. He was the one who first introduced me to Selenium testing in San Francisco Plone Conference in 2011, and whom I had not seen after that. Until now. I really hope takes less than seven yeras to see him again...

Ed Manlove presenting his talk: The Importance of Open Source Communities

Jupyter kernel for Robot Framework

After the conference on Thursday came the single day conference sprint on Friday. And if the conference was a success, the sprint was even more so: the sprint venue, three large office rooms, was packed full of sprinters, many of them participating their first open source sprint (and got a good introduction to open source development from Ed).

The sprint facilities were provided by Eficode

Because I had to leave early in Friday, I had planned a very specific sprint goal for myself: a MVP Robot Framework kernel for Jupyter notebook.

Jupyter notebook (previously known as IPython notebook) is an open-source web application for creating and sharing documents that contain live code, equations, visualizations and narrative text. The architecture behind Jupyter notebook separates the notebook application from its language specific ”kernels” that are responsible for executing the code in notebooks. Syntax highlighting in notebooks, on the other hand, is provided by CodeMirror project for the interactive frontend, and Pygments for server side generated highlighting.

I'm happy to say that I made it. And the more I use it, the more confident I get on that Jupyter makes a great platform for learning also Robot Framework. And not only for learning by yourself, but also for sharing your notes with others.

Check my example notebooks to judge the kernel by yourself.

Please, note that while these examples are static renderings at http://nbviewer.jupyter.org/, they can be opened and interacted live in any running Jupyter notebook with the new kernel and required Python packages. See the repository for more details.

The main Jupyter Robot Framework support features shown in those examples are:

• Support for defining and executing Robot Framework test suite cumulatively step by step in successive notebook cells. The main limitation is that each cell should start with a test suite section header (settings, variables, keywords or test suites) even when the same header was already defined in some cell before.
• HTML log and report files are linked below the executed cells containing the tests. Both files are actually bundled with the notebook in a way that sharing the notebook also shares the log and report files.
• Images generated during test execution are shown below the executed cells generating the images. Similarly to HTML logs and reports also images are bundled with the notebook for sharing.
• Support for %%python module LibaryName ”cell magic” to allow defining custom Robot Framework keyword libraries in Python in fly. Once thell cell with a Python library class definition is executed, it can be imported in a successive Robot Framework code cell.
• Syntax highlighting. But, unfortunately, until the CodeMirror plugin derived from brackets-robotframework-project is accepted into upstream, it must be manually patched into CodeMirror version shipped with Jupyter notebook-distribution. (I have not yet submitted a pull for it.)
• If the last keyword of the last test case in the executed cell returns JSON string, it is rendered as cell execution output. I added this quite specific feature to make it more fun to learn RESTinstance library with Jupyter (Output keyword of RESTinstance library returns JSON).

Obviously, while the current versions is already fully functional on Python 3, there's still a lot of work (QA, packaging and Python 2 support) left to polish the code for release. I'm looking forward to finish it during the spring.

Happy hacking! And hopefully see you in RoboCon next year – or whenever it is organized and I'll manage to participate it for the next time! :)

Plone Mephisto Sprint 2016 Report

Plone Mosaic 2.0 RC has now been released for Plone 5. Similarly to previous releases, you can deploy a demo site of it at Heroku with a single click. Please, join the bug hunt and help with reporting and fixing issues which matter you the most. Please note that Mosaic 2.0 is a major release because it no longer supports for Plone 4.3 (yet, most of the bug fixes could be cherry-picked to 1.0 branch by those who care). The final release will be made once possible migration problems from 1.0 have been solved.

Yes, there was a yet another Plone community sprint. Plone Mephisto Sprint 2016 happened at Leipzig, Germany, from 5th to 9th September, and was focused on improving TTW (through-the-web) experience of customizing Plone sites for better flexibility and hackability. The sprint was organized by Maik Derstappen and it was sponsored by Derico, e-ventis and Plone Foundation. Once the sprint was approved as a strategic sprint by the foundation, enhancing Plone Mosaic became the main target of the sprint.

The participants included Maik Derstappen (derico), Peter Holzer (agitator Weblösungen - BDA), Thomas Massmann (it-spirit), Asko Soukka (University of Jyväskylä), Thomas Lotze, Andreas Jung, Stefania Trabucchi, Jens Klein (Klein & Partner KG - BDA), Christoph Scheid (Uni Marburg), Kristin Kuche (Uni Marburg), Stephan Klinger (der Freitag), Nathan Van Gheem (Windcard Corp), Veit Schiele, Gil Forcada Codinachs (der Freitag), Ramon Navarro Bosch, Michael Töpfl (e-ventis.de) and Christoph Töpfl (e-ventis.de). During the sprint there was also remote participation by Dylan Jay and Rodrigo Ferreira de Souza (Simples Consultoria).

Let's start with some Github Activity statistics collected by Nathan Vangheem from the sprint. During the sprint, at least the following packages were touched with these great numbers:

• plone.app.mosaic (23 PR, 154 commits, 2 outstanding PRs)
• plone.app.blocks (4 PR, 18 commits)
• plone.app.standardtiles (9 PR, 18 commits, 2 outstanding PRs)
• plone.app.tiles (4 PR, 9 commits)
• plone.tiles (2 PR, 16 commits)

Behind those numbers, there was a lot of bug fixes, new issues reported, some refactoring and simplification, few major changes to the foundations of Plone Mosaic, and the long awaited initiative for more complete configuration documentation. Thanks to all the bug fixes, Mosaic Editor and TinyMCE in it will break much less often. Also layout related permission work now as expected. To name the other major fixes and changes to Plone Mosaic, we:

• deprecated dedicated image and attachment tiles in favor of using image and file content types and linking them with rich text tiles as without Mosaic.
• removed complex universal pluggable grid system (tm) implementation in favor of simply using such CSS grid class names that it's possibly to build any current grid system for those class names. This finally makes Mosaic edit and view mode show the same grid by default on Plone 5.
• removed Mosaic Editor from content add forms and introduced a simple add form with just title and description for those content types, which have Mosaic enabled and its layout view defined as their default view. This removes all experienced confusion of add forms some times behaving differently from edit forms and not being able to save images directly inside the new containerish content.
• unified HTML tile implementations and removed dedicated example tiles for headers, lists, etc, in favor of a single rich text tile (which would still allow similar dedicated or templated tiles).
• introduced outline mode for the editor (pressing Alt modifier key while in editor) to keep the default editor experience simple, but still allow more technical look into the layout and make it easier to split the layout into separate rows.
• fixed a major issue where versioning of content with blobs in its tiles' configuration created empty blobs into filesystem. This has been a major issue for collective.cover users and is now fixed with new plone.app.tiles releases (1.1.0 for collective.cover).

In addition

• we fixed a few site layouts related compatibility issues with Plone 5 and added a support for enabling site layouts support with a single line in buildout. Site layouts are not yet enabled by default.
• we implemented a new, but transparent, tile configuration and data storage to mostly avoid using annotation objects with shared content layouts (and be more friendly for ZODB connection cache).
• Michael and Christoph (Töpfl) worked on Rob Gietema's ReactJS based Mosaic Editor experiment, and Ramon started a new Angular 2 based Mosaic Editor experiment. Hopefully at least the other one will succeed to give us a fresh flexbox based layout editor during the next year.
• Maik lead work on example content layout to be shipped with the final Mosaic 2.0 release.
• Andreas started work on allowing custom views for existing content tile.
• Dylan worked on user interface and TinyMCE support for allowing tiles inside rich text tiles.
• Rodrigo worked on refactoring code of RSS and calendar portlets to be easily re-usable with respective tiles.

We also discussed about the path to get Mosaic into Plone core. The current plan is to get Mosaic dependencies into Plone core first (plone.tiles, plone.app.tiles, plone.app.blocks, plone.app.standardtiles), but only PLIP the user interface package (plone.app.mosaic) when it works well enough with the other add-ons shipped with Plone (e.g. multilingual). Unfortunately, no PLIPs were written yet.

A few photos from the sprint are available at Google Photos. In addition to full day sprinting, we did get a city tour around Leipzig and enjoyed Maik's barbequing at the sprint garden. Maik did a great job in organizing the sprint, and I really hope, we made it worth the effort. And as always, the work around Plone Mosaic continues also after the sprint. It's still huge effort left, but while it's still not ready for Plone core, it can already be customized to give real return on investment, as seen in Castle CMS. Finally, Big thanks for everyone participating the sprint. It was a pleasure to work with you, and I hope to see you all again!

Plone Barcelona Sprint 2016 Report

For the last week, I was lucky enough to be allowed to participate Plone community sprint at Barcelona. The print was about polishing the new RESTful API for Plone, and experimenting with new front end and backend ideas, to prepare Plone for the next decade (as visioned in its roadmap). And once again, the community proved the power of its deeply rooted sprinting culture (adopted from the Zope community in the early 2000).

Just think about this: You need to get some new features for your sophisticated software framework, but you don't have resources to do it on your own. So, you set up a community sprint: reserve the dates and the venue, choose the topics for the sprint, advertise it or invite the people you want, and get a dozen of experienced developers to enthusiastically work on your topics for more for a full week, mostly at their own cost. It's a crazy bargain. More than too good to be true. Yet, that's just what seems to happen in the Plone community, over and over again.

To summarize, the sprint had three tracks: At first there was the completion of plone.restapi – a high quality and fully documented RESTful hypermedia API for all of the currently supported Plone versions. And after this productive sprint, the first official release for that should be out at any time now.

Then there was the research and prototyping of a completely new REST API based user interface for Plone 5 and 6: An extensible Angular 2 based app, which does all its interaction with Plone backend through the new RESTful API, and would universally support both server side and browser side rendering for fast response time, SEO and accessibility. Also these goals were reached, all the major blockers were resolved, and the chosen technologies were proven to be working together. To pick of my favorite sideproduct from that track: Albert Casado, the designer of Plone 5 default theme in LESS, appeared to migrate the theme to SASS.

Finally, there was our small backend moonshot team: Ramon and Aleix from Iskra / Intranetum (Catalonia), Eric from AMP Sport (U.S.), Nathan from Wildcard (U.S.) and yours truly from University of Jyväskylä (Finland). Our goal was to start with an alternative lightweight REST backend for the new experimental frontend, re-using the best parts of the current Plone stack when possible. Eventually, to meet our goals within the given time constraints, we agreed on the following stack: aiohttp based HTTP server, the Plone Dexterity content-type framework (without any HTML views or forms) built around Zope Toolkit, and ZODB as our database, all on Python 3.5 or greater. Yet, Pyramid remains as a possible alternative for ZTK later.

I was responsible for preparing the backend track in advance, and got us started with a a simple aiohttp based HTTP backend with experimental ZODB connection supporting multiple concurrent transaction (when handled with care). Most of my actual sprint time went into upgrading Plone Dexterity content-type framework (and its tests) to support Python 3.5. That also resulted in backwards compatible fixes and pull requests for Python 3.5 support for all its dependencies in plone.* namespace.

Ramon took the lead in integrating ZTK into the new backend, implemented a content-negotiation and content-language aware traversal, and kept us motivated by rising the sprint goal once features started clicking together. Aleix implemented an example docker-compose -setup for everything being developd at the sprint, and open-sourced their in-house OAuth-server as plone.oauth. Nathan worked originally in the frontend-team, but joined us for the last third of the sprint for pytest-based test setup and asyncio-integrated Elasticsearch integration. Eric replaced Zope2-remains in our Dexterity fork with ZTK equivalents, and researched all the available options in integrating content serialization of plone.restapi into our independent backend, eventually leading into a new package called plone.jsonserializer.

The status of our backend experiment after the sprint? Surprisingly good. We got far enough, that it's almost easier to point the missing and incomplete pieces that still remain on our to do:

• We ported all Plone Dexterity content-type framework dependencies to Python 3.5. We only had to fork the main plone.dexterity-package, which still has some details in its ZTK integration to do and tests to be fixed. Also special fields (namely files, richtext and maybe relations) are still to be done.
• Deserialization from JSON to Dexterity was left incomplete, because we were not able to fully re-use the existing plone.restapi-code (it depends on z3c.form-deserializers, which we cannot depend on).
• We got a basic aiohttp-based Python 3.5 asyncio server running with ZODB and asynchronous traverse, permissions, REST-service mapping and JSON-serialization of Dexterity content. Integration with the new plone.oauth and zope.security was also almost done, and Ramon promised to continue to work on that to get the server ready for their in-house projects.
• Workflows and their integration are to be done. We planned to try repoze.worklfow at first, and if that's not a fit, then look again into porting DCWorkflow or other 3rd party libraries.
• Optimization for asyncio still needs more work, once the basic CRUD-features are in place.

So, that was a lot of checkbox ticked in a single sprint, really something to be proud of. And if not enough, an overlapping Plone sprint at Berlin got Python 3.5 upgrades of our stack even further, my favorite result being a helper tool for migrating Python 2 version ZODB databases to Python 3. These two sprints really transformed the nearing end-of-life of Python 2 from a threat into a possibility for our communitt, and confirmed that Plone has a viable roadmap well beyond 2020.

Personally, I just cannot wait for a suitable project with Dexterity based content-types on a modern asyncio based http server, or the next change meet our wonderful Catalan friends! :)