Midsummer Sprint Final Report

Plone Midsummer Sprint was held at Jyväskylä, Finland, on 3rd–7th of July 2017. The main goal for the sprint was to polish content editors' user experience on the current Plone CMS release. While a lot was accomplished for that original goal, the final sprint topics adapted to follow the interests of its volunteering participants.

The sprint was sponsored by the University of Jyväskylä. In addition, the sprint was accepted to receive strategic funding from Plone Foundation to cover the possible travel and accommodation costs of Albert Casado, the designer of Plone 5 user interface.

This is the final report of Midsummer Sprint.

Participants

There was 18 participants at Midsummer Sprint in total. The participants included, of course, our organisers, our local development team for Plone based and related services, a GSOC student working on a Plone Foundation mentored topic, a professional UX designer and many well known names from Plone community.

Our sprint organisers:

• Asko Soukka, University of Jyväskylä
• Rikupekka Oksanen, University of Jyväskylä

Our local development team:

• Jussi Rajala, University of Jyväskylä
• Jussi Talaskivi, University of Jyväskylä
• Matti Pohjanvirta, University of Jyväskylä
• Valtteri Karppinen, University of Jyväskylä

Google Summer of Code student:

• Mikko Hursti, University of Jyväskylä

Plone community participants:

• Albert Casado, Hewlett Packard
• Alexander Loechel, Ludwig-Maximilians-Universität München
• Eric Steele
• Maik Derstappen, Derico
• Paul Roeland, Clean Clothes Campaign
• Peter Holzer, agitator Weblösungen
• Philip Bauer, Starzel.de
• Stephan Klinger, der Freitag
• Sven Strack
• Timo Stollenwerk, kitconcept GmbH
• Victor Fernández de Alba, kitconcept GmbH

Thank you for all the participants! Many of you came to Jyväskylä with members of your family, and it was our pleasure to host every one of you!

Results

According to the feedback that we got from our participants, the sprint was considered a very successful one. Maybe the sprint was unable to fix as many content editing related issues as we were hoping for, but it definitely managed to get the best out of its participants:

At first, thanks to this sprint, all members of our local Plone development team at IT Services of University of Jyväskylä got to learn how Plone is being developed. They got their contributor agreements submitted to be able to push their patches to upstream, they learned how those patches were made, tested and reviewed in practice, and they got to know many experienced Plone developers in person. Of course, they also got to feel the spirit of an international open source development sprint, from the community with probably the longest continuing tradition on it.

The same was true for Mikko Hursti, a local student attending the sprint while working on a Google Summer of Code project for Plone Foundation. This year only four students from Finland were accepted for Google Summer of Code program. Mikko's project is about implementing a drag and drop user interface for customising content listings on Plone.

During the sprint, our local development team mainly focused on confirming and fixing the issues reported for PloneFormGen on Plone 5. PloneFormGen is the most famous form builder for Plone, used by most of the content editors at our university. Also, its unrivalled flexibility really showcases many of the unique features of Plone CMS. Before the end of the sprint, Jussi Rajala and Valtteri Karppinen submitted at least four related pull requests for it and got very familiar with the large code base in question. Not bad at all for their first sprint.

• https://github.com/plone/Products.Archetypes/pull/93
• https://github.com/plone/Products.Archetypes/pull/94
• https://github.com/smcmahon/Products.PloneFormGen/pull/199
• https://github.com/smcmahon/Products.PloneFormGen/pull/200

Security is one of the most advertised features of Plone CMS. What makes it particularly great feature is that content editors don't really need to know or think about it. It just works. Maik Derstappen made spectacular work at the sprint by cleaning up and fixing the code base for a major security feature of Plone called safe HTML transform. It's a configurable HTML filter, which prevents content editors from accidentally (or with malicious objectives) introducing security issues in their edits. Maik's work guarantees that the feature really works as advertised also on Plone 5. With the new synchronised configuration, it also works better than ever with TinyMCE, the WYSIWYG HTML editor used on Plone.

• https://github.com/plone/Products.CMFPlone/pull/2098
• https://github.com/plone/Products.PortalTransforms/pull/20
• https://github.com/plone/plone.outputfilters/pull/25

The latest major Plone release, Plone 5, introduced a completely redesigned content management toolbar for all logged-in users. But not all logged-in Plone users are content editors, and when they are not, the new toolbar could a distraction. At the sprint, Peter Holzer continued on earlier work by Jens Klein and introduced Show Toolbar -permission to control when the full toolbar is shown, and an alternative member tools -dropdown menu to show a minimal set of required actions (including logout) for logged-in users when the real toolbar is not shown. The default behaviour, of course, remains unchanged.

• https://github.com/plone/plonetheme.barceloneta/pull/131
• https://github.com/plone/Products.CMFPlone/pull/2096
• https://github.com/plone/Products.CMFPlone/pull/2100
• https://github.com/plone/plone.app.upgrade/pull/127
• https://github.com/plone/plone.app.layout/pull/128

Plone 5 was also the first Plone CMS release to ship with a complete multilingual content support. With the bundled Multilingual Support -add-on enabled, Plone supports easy-to-use one-to-one translation of each content page for any language. In addition to fixing some reported issues in it, Peter Holzer and Asko Soukka redesigned its key user interface element, Translate-menu on the toolbar, to be much simpler and more consistent with the other menus in Plone's editing toolbar. With the redesigned Translate-menu soon in place, we really expect to see a good drop in translation related support requests that we currently receive from our content editors.

• https://github.com/plone/plone.app.drafts/pull/6
• https://github.com/plone/plone.app.multilingual/pull/283
• https://github.com/plone/plone.app.multilingual/pull/284
• https://github.com/plone/plone.app.multilingual/pull/285

Plone provides, and has always provided, readable and meaningful web addresses for any content stored on it. Even better, it's really easy to content around on Plone. But what happens to those old web addresses when content gets moved? Correct. Of course Plone keeps track of moved content and maintains a redirection table for them. Unfortunately, until now, the redirection table has been invisible for Plone web masters. Luckily, Stephan Klinger championed to implement the PLIP for adding the missing redirection management UI for Plone, based on the existing (but lacking) RedirectionTool add-on. While Stephan made the most of the work, Asko helped him by implementing paging and filtering for the new redirection management control panel.

• https://github.com/plone/Products.CMFPlone/commits/plip-1486-redirection

Our farthest participant was Eric Steel, the Plone release manager, who came to the sprint all the way from the United States. Eric started his sprint by releasing the latest beta of Plone 5.1 for developers, but then, together with Philip Bauer, completed the long-awaited refactoring of Plone login forms. The current login forms used in Plone 5 are relying on a form framework, which is no longer used anywhere else in Plone. Upgrading Plone login forms to use the current form frameworks in Plone has been one of many prerequisites for upgrading Plone to Python 3. It's just awesome that this has finally been done, and we cannot wait for being able to customise Plone login forms in a more sustainable way now. In addition, Philip did also fix a few known issues in the in-place content migration from Plone 4 to Plone 5.

• https://github.com/plone/Products.CMFPlone/issues/2092
• https://github.com/plone/plone.login/commits/master
• https://github.com/plone/plone.app.contenttypes/pull/413

A lots and lots of work was also done to enhance the official Plone documentation at the sprint. To really get an idea, how Plone documentation applies traditional CI practices to ensure high quality documentation, we really recommend watching the presentation that Sven Strack gave us at the sprint. Sven Strack, Paul Roeland and Alexander Loechel worked at least for better tools for building and testing the documentation, better theme for the upcoming Plone 5.1 documentation, better readability of the docs on mobile devices, refactoring robot generated screenshots (helped also by Asko), documentation on developing better documentation for contributing the docs itself and probably much more. In addition, Paul also fixed a few reported accessibility issues on Plone 5 and tested Asko's approach for building Plone themes with Webpack. Alexander used some of his sprint time to mentor his GSOC student Pavithira KC on implementing new add-ons listing for plone.org.

• https://github.com/plone/sphinxtheme.plone/commits/master
• https://github.com/plone/documentation/commits/5.1
• https://github.com/plone/Products.CMFPlone/commits/robodoc
• https://github.com/plone/Products.CMFPlone/issues/1619
• https://github.com/plone/Products.CMFPlone/pull/2089
• https://github.com/plone/plone.app.layout/pull/126
• https://github.com/plone/plonetheme.barceloneta/pull/130

Other interesting developments at the sprint included Maik implementing sub-templates into bobtemplates.plone, Alexander adding tox based test setup into bobtemplates.plone and Jussi Talaskivi enhancing plone.docker. Once Maik's enhancements for Plone add-on templates at bobtemplates.plone are available, they will allow to make significantly cleaner initial templates and provide sub-template based scaffolding throughout the add-on lifetime. Alexander's examples for tox based test setups pave the way for testing add-ons easily on both Python 2 and Python 3. Jussi clarified documentation on using Plone Docker images (also as a cluster), investigated into slimming Plone Docker images and then researched the remaining issues in installing Plone with pip, the official Python package installer.

• https://github.com/plone/bobtemplates.plone/pull/206
• https://github.com/plone/plone.docker/issues/69
• https://github.com/plone/plone.docker/issues/70

Finally, there was the dream team of Albert Casado, Victor Fernández de Alba and Timo Stollenwerk working on a new CMS user interface and user experience concept called Pastanaga UI. If something makes this project special, it's the team. Timo is expert on test driven development, and experienced Plone entrepreneur, who really knows the challenges on the current CMS market. Victor is expert on Plone as enterprise CMS. He has authored a book on using Plone for intranets, and has managed many hundreds of Plone sites during his prior position at Universitat Politècnica de Cataluny. Albert, of course, is best known for his design of the current Plone 5 default theme called Barceloneta. And the best part: they all have successfully worked together before.

At the sprint, Albert completed many of his earlier designs for Pastanaga UI components and composed a visual style guide to help with the reference implementation and later adaptations. Thanks for Alberts prior experience on Plone, Pastanaga UI simplifies many Plone key features to be easier to use than ever, especially on mobile. All this his available on request until the reference implementation has been completed. Victor and Timo supplemented Albert's work by bootstrapping a ReactJS based reference implementation of the design. Their goal is to complete Plone REST API compatible minimal viable product with the new UI soon, and attract resources for a more complete implementation later. Since the first implementations will be fully JavaScript based, it not yet sure, if Pastanaga UI would be adaptable for the current Plone UI.

• https://github.com/plone/pastanaga

Many of our sprint days started with a presentation on some of the sprinted or otherwise current topics, including a presentation on Pastanaga UI by Timo and Albert. Recordings of those presentations are now publicly available on our Plone based video publishing platform Moniviestin.

Program

While the code is important, Plone sprints are also about the people, who participate them. Plone community has a long tradition for dining or just hanging around together after those very intensive work days, and Midsummer Sprint did not make an exception. We also tried our best to take account that many of the participants travelled with their families.

Most of us dined together every day from Sunday evening to Friday evening. While many of the participants were still arriving on Sunday, we dined at Restaurant Shalimar inside the train station, to be able to welcome travelling participants and their families as they arrived.

In the Monday evening we dined at Restaurant Sohwi, which has always been popular place among the students and staff of our university due to its close proximity to our main campus areas.

On Tuesday, we wrapped up the day a bit earlier than usual and took a guided tour at Alvar Aalto museum (tour being sponsored by the university) to learn and get inspired by Aalto's design. In the evening, we walked a short forest trail at Haukanniemi together, and experienced a beautiful and warm sunset while enjoying a traditional Finnish camp fire with Finnish sausages (and surprisingly without mosquitoes).

In the Wednesday evening we dined together at viking restaurant Harald, where it was possible to enjoy many Finnish delicacies including reindeer, gravlax, Scandinavian rye and malt bread, lingonberries and cheese bread with cloudberries. We were lucky enough to catch the summer discount of the restaurant to make that evening a bit more affordable.

On Thursday, we organised a traditional Finnish sauna evening with street food at Opinkivi Sauna, a private sauna apartment owned by The Student Union of the University of Jyväskylä. Two separate dressing rooms and saunas at Opinkivi Sauna ensured that everyone was able to enjoy Finnish sauna in comfort.

And finally, in the Friday evening, we ended the sprint with the remaining participants by dining together at restaurant Trattoria Aukio, near the harbour of Jyväskylä, and then, said the goodbyes.

Thank you all, and see you soon!

Case Study: Creating a Course Registration System in Short Notice (Jedi Style!)

Here is a nice story, that almost got lost in translation (literally - its published in Finnish last summer).

It was a warm, late afternoon on May 2016, when we at plone-support received a message, asking for help. Perhaps we weren't like Obi-Wan Kenobi on that, but close enough: A customer needed to have a new course registration system and migration from the old one so soon as August 2016. Whoa, three monts, summer coming on and other projects on the road too. Should we just skip it or take the million credits to get even with Jabba?

Well, here's a spoiler alert: https://www.fdpe.fi/

 

1. The case of The Finnish Doctoral Programme in Economics

A Joint Doctoral Programme of the Economics Disciplines of 10 Participating Universities in Finland. The Finnish Doctoral Programme in Economics (FDPE) promotes high-quality doctoral education in economics in Finland by organising systematic and diverse doctoral courses and seminars.

 Here's is a rough translation of the initial email we received in May 2016.

We will need a platform, where we can manage nation wide economics doctoral programmes cooperation. To be more clear: we need a web-based system, that allows students from participating universities to apply for doctoral studies courses from another universities. Ideally the principles of the system would be: 
- Each university inserts a batch of courses as part of the cooperation
- The courses became a course pool, that will be communicated to the studens via the system
- The students from each univeristy may register for the coursers on the course pool
- After a course a teacher may communicate the grade of each student to the student's home university via the system
- Once per year the registrations are gathered and counted, how many participants there are from university x to university y
- Based on those calculations it is possible to settle the compensation between universityes. 

The point is to build a platform to gather registrations and communicate students grades and registration information. The platform should be as simple and robust as possible. 

Our schedule is uncomfortably tight in the sense that the system (or at least te application feature) should be online at latest in August.
****************

Ok, so we got the message. And why they chose to ask us, is because of another Plone-site that shows events and gathers applications there: http://inforte.jyu.fi

We discussed about the case, and thought it was quite a challenge to present so late at spring, (with all the vacations and other projects we had in store for the summer). But we also thought, that we had a quite toolset for this kind of system:

• Plone would take care of the website, permissions, workflows, security and robustness. 
• Courses would be dexterity content types - easy to create and to modify if needed
• The registration form could be easily created using Plone Form Gen add on - of which we had lot's of experience 
• The information about participants could be delivered in the system using plone permission management and intranet features
• This was a good case to try out new Plone 5 and its out of the box responsive layout

So we were set to talk with the customer and see what else they might have in mind.

After a positive meeting, we also had this information:

• Migration was needed, but only for visible content
• The visual desing of the site would be secondary feature
• If only the application forms and course information were online already in August, the rest of system could be finished by September
• We could use HAKA indentity federation service to allow students and staff from different universities to login with their own userid and password. We could have created internal useris's for the admins and could have forced students to insert more information at the application form, but this was way better solution. We had used HAKA before with Plone in many occasions (Like Moniviestin video publishing platform)
• The couldn't use national Opintopolku.fi -service (at least not yet) for doctoral studies. 
• The project had fixed deadline and the customer was willing to prioritize and to focus on the most important features first

So we decided to set out for this venture!

2. Plan B. 

Actually, we had no plan B, so we came up with plan A, to divide the development into clear phases, visible to the customer:

1. At first we would set up a preview-site: the customer could then start using Plone and to learn how to use it. 
2. A light theme customization
3. Site name, domain, certificate and HAKA-federation bureaucracy was set to motion
4. Migration to the old public content (scrape public content)
5. Shibboleth-integration (HAKA) development for Plone 5
6. An event like Course content type would be created
7. Using a Plone Form Gen -solution we had created before on another case to automatically populate basic user information about the person filling the form.  
8. Using a previous solution to generate reports on every separate application form. 
9. A Plone folder, where a teacher stores the saved Excel-files about course grades
10. Other features, like managing permissions,  communication, instructions, planning the application form and theming.

So, we had plans.

Some features, like good security, content management, intranet features,  permission and group management, responsive web design and workflows are built in to Plone CMS. We used the latest Plone 5 version on this case.

The actual work

We started by creating a preview-site in june 2016. This way it was easy for the customer to follow progress. So we added an empty Plone 5 site with minor theming and gave admin permissions to the customer so they could get to know how Plone works. We also did the migration of old public content at this point.

When the migrated content was checked and deemed okay, and the bureaucracy with the domain name done, we published the new site in june 2016 - only without the form functionalities. However, the customer could already do some content editing and checking that everything was okay.

Next up was the HAKA federation. Using HAKA federation, every higher education student in Finland could log in to the site using his/her own user id and password to make logging in easy. At the same time our customer could easily gather the name, email and organisation of every enrolled student. In addition they could add new site administrators from different organizations. Plone allows very fine-grained permission management.

Last, but not least, we developed the registration form and a new content type called Course, with some basic data (start date, end date, information). When a student goes to a course, there is a registration link which opens the form with all necessary fields already populated. The registration form was done using Plone FormGen add on. We had used that add on for over a thousand forms at JYU, so we knew it was stable and customizable.

The registrations were collected to a .csv-table, easy to open in Excel or Google Sheets and easy to use from every organization participating FDPE programme.

In the end we created short instructions for the site admins on how to manage content. The instructions were just Plone pages with some images and links.

Example one: Course Page

Example 2: Registration Form

Summary

The customer was very satisfied with the development and results, and so were we.

This kind of development in this short time period wouldn't have been possible if we didn't have the suitable technology available and knowhow on how to properly utilize it.

Additionally the customer was really cooperative - they focused on getting things done, they did their part on the content and other bureaucracy, and most importantly they knew what to prioritize.

We had limited time and limited resources - and usually it is a good thing if you can actually prioritize and altogether just cut out some features if needed. First things first, and then some.

By using free Open Source components and no need to pay any licence fees, the only thing we (and the customer) needed to focus on, was the actual development.

Now, after almost a year of usage there has been near to none feature requests or incidents using the system, and definitely nothing major. But there have been hundreds of successful registrations and everything seems to be working fine. 

If you want to know more, please contact plone-support@jyu.fi

It is party time!

University of Jyväskylä Tech Blog - Technology, Development and Culture

Did you know that University of Jyväskylä IT Services does a lot of in-house software development?

In this blog we will tell you about what technology choices we have made and why, how we develop software, how we utilize Open Source software and how we keep up inspiring culture in our teams.

In this article I will tell shortly about who we are and what services we have. In later posts we will dig into the interesting how.

Screenshot of our build monitor

University of Jyväskylä - facts and figures

The University of Jyväskylä is a nationally and internationally significant research university and an expert on education that focuses on human and natural sciences.

• Seven faculties
• 15 000 students
• 2600 staff members
• Budget of about 211 M€/year

University of Jyväskylä resides in the middle of Finland.

Compared to many other higher education organizations in Finland we have relatively large group of people dedicated to IT - 90 people, of which

• about 30 do IT services (network, servers, email, AD etc.)
• about 40 work at IT support and software support
• about 20 work with maintaining software and developing new software, of which about 15 are developers

Our web-based, inhouse services, some examples

• Study data system Korppi (used by all students and teachers)
• Plone CMS for our websites (80 sites, over 250 000 pages of content, 2 000 000 visitors/month), e.g. https://www.jyu.fi
• Moniviestin video publishing platform - since 2003, thousands of videos
• Koppa and Avoin Koppa LMS - integrated to Korppi and plagiarism detection
• Payment systems - over 1 M€ gross/year and about 25 000 payment transactions made/year
• TUTKA research portal
• iRODS research data store
• Multiple customised web-services
• Jybar - simple yet powerful toolbar connecting our different web based services
• ROTI - study information system 
• Integrations to nation wide systems
• Migrations from legacy software to new ones
• and much more!

What's to come?

We will tell concrete examples of about

• Technology choices we have made 
• Systems we have developed and maintained with relatively little amount of people
• Automated software testing, Continuous deployment 
• Docker, Ansible, Gitlab
• Java, Python, C#, JavaScript, HTML, CSS etc
• Open Source
• Being Agile 
• How to cope with public sector bureaucracy
• How to do integrations between different systems
• Successful (and perhaps not so successful) customer use cases
• And more

Stay tuned!