Lizmap is almost 10 years old. 10 years of new features, documentation, and numerous version release. Many software libraries updates, many contributors (31 if github is correct) and contributions later we've released version 3.4 at the end of 2020.
Whilst in the release cycle, we realized that our release process was lacking something. We were unsure of the quality of this release, and we were missing some tools to assess whether the release would contain regressions.
In order to avoid these issues in future releases and updates, we decided to implement a number of processes and changes in both how we release and build lizmap.
First : we have decided to not accept code patches without tests. Be these tests unit tests, integration tests or just some lines of text explaining how to test for the bug or regression being fixed by the patch. We are now maintaining a list of tests in our code repository.
Secondly we changed our release process to be able to test. We now have a dedicated time slot between the time when our code is ready and the time our code is released to the world. This will let us test internally and will let us gather feedback from users testing our release candidates.
Our third point is community centered. It's now quite easy to set up a lizmap stack or two using our Docker images. Using these instructions you'll have a docker running lizmap in no time on your computer. Add a project or two to it, test and see if you see anything that doesn't work like your production environment and let us know by opening a ticket. Our aim here is to have more user test our release candidates (we do these for major releases) so we can quickly fix regressions before we officially release.
Fourth we are setting up internal processes to make our code easier to test (docker is a good example of this) and easier to read. Hence, are growing use of linters. We also added a code analyzer for our backend: PHPstan.
Finally, we spent a good amount of time updating our demos (see demo.lizmap.com), making sure they work. This makes our demos a reference for most of our testing.
We had realized in June 2020 some of theses issues around quality. That's why we had an intern refactor lizmap. That refactor had two goals. One was to make it easier to add tests to its code and the second one was to add more tests. All these changes have made it to the 3.4 branch. They aren't in the 3.3 branch because it would have meant too much work. This means 3.3 is a bit less tested than 3.4 (hint upgrading from 3.3 is very easy).
Now that most of our backend is covered we are focusing our efforts on the front-end. This means we will update some of the libraries we use, we will refactor our code and will add as much test as possible. This is going to be the work of an intern. We have chosen Cypress to build end2end tests.
All of these automated tests (backend, front-end, end2end) are being run from our githubrepo through github actions.
All these changes have been made possible by the growth of 3liz during 2020.