Test Status

Introduction

ERP5 is used for many critical applications. Our customers relies on it to run their every day business, and they can not accept having regularly bugs coming again and again. Every software issue could have important consequences, especially if you think about banking, aerospace applications and many others. Therefore Nexedi considers the quality as one of its top priorities and decided to use automated testing since the early age of ERP5.

Unit / Functional / Deployment and Scalability Tests

While the code of ERP5 is evolving, more and more tests are added. When issues are reported, we write the test and fix the problem. Like this we make sure that the same issue is not going to happen again. Also, when new features are done, tests are added to make sure we will provide a well working experience. We have both unit tests and functional tests. Unit tests are used to test isolated software component, while functional tests are used to verify scenarios closer to user experience.

Development policy

We tried for years to keep the quantity of failures as low as possible. While the size of ERP5 was growing up, it started to be more and more time consuming to track regressions. In particular, it's quite common that a change in some part of the code generates a regression in another place that seems unrelated without detailed analysis. And then it happened that we had several errors, and it was difficult to see new ones.

Therefore, after our switch to git, we introduced the rule "contributions must not introduce ANY regression". So we started by fixing test failures, we checked all random issues, and we started to reject any code introducing failures. The idea is simply to check all unit tests before contributing changes to the central place. This can easily be done thanks to our Testing Infrastructure.

Thanks to this rule, about any revision of ERP5 could be considered as stable. This is a really great success. With a quantity of test growing, we have nearly 0 errors/failures all the time.

Testing Infrastructure

We are using what we call "ERP5 Test Nodes". They are processes installed on many different machines thanks to SlapOS cloud tools. We define several lists of tests to run, and ERP5 test nodes will look regularly to several code repositories to detect new commits, then they launch software constructions and finally tests are automatically run. Theses days we are using about 180 CPU cores to run tests for our various projects and for different versions.

Description of tests

Please find below a short description of all tests currently running:

• ERP5.UnitTest-Master (former ERP5-MASTER)
  An unit tests which covers all ERP5 code base.
• Wendelin.UnitTest-Master ( former WENDELIN-MASTER-DEV)
  An unit tests which covers Wendelin (Big Data Platform) code base
• NEO.UnitTest-Master (former NEO-MASTER)
  An unit tests which covers the NEO (distributed, redundant and transactional storage DB)
• JIO.UnitTest-Master (former JIO-MASTER)
  An unit tests (javascript) for our  JIO framework running on a web browser
• JIO.UnitTest-Master.NodeJs (former JIO-MASTER-NODE)
  An unit tests (javascript) for our  JIO framework running on NodeJS
• RenderJs.UnitTest-Master (former RENDERJS-MASTER)
  An unit tests (javascript) for our RenderJS framework
• Cloudooo.UnitTest-Master (former CLOUDOOO-MASTER)
  An  unit tests which tests our conversion daemon called cloudoo
• ERP5.StandaloneDeploymentScriptTest-Debian.Stretch (former SLAPOS-DEPLOY-erp5-standalone-stretch)
  A deployment tests which assures that we can install ERP5 into a Debian Stretch OS
• SlapOS-Master.StandaloneDeploymentScriptTest-Debian.Stretch (former SLAPOS-DEPLOY-slapos-master-standalone-stretch)
  Tests slapos' master installation on Debian 9 (stretch)
• ERP5.ScalabilityTest-Master (former ERP5-SCALABILITY)
  Tests how scalable an ERP5 instance is by simulating heavy intensive operations against a set of cluster configurations
• ERP5.PerformanceTest-Master (former PERF-ERP5-MASTER)
  Tests pure ERP5 performance like creating and storing an object which should happen without any regressions for a already predefined time set
• OfficeJS.Appstore.UnitTest-Master (former P-OJS.Appstore-master)

  An unit tests which covers the OfficeJS Appstore code base
• SlapOS.SlapRunner.ResilienceTest-Master (former SLAPOS-RESILIENCE-WR-MASTER)
  Tests that the data produced by a simple Software Release inside a resilient webrunner is correctly backup-ed and restored in a backup webrunner, in order ensure a disaster recovery.
• SlapOS.SlapRunner.ResilienceTest-Master.ERP5 (former SLAPOS-RESILIENCE-WR-ERP5-MASTER)
  Tests that the data produced by ERP5 Software Release (from master branch) inside a resilient webrunner is correctly backup-ed and restored in a backup webrunner, in order ensure a disaster recovery.
• SlapOS.SoftwareReleases.IntegrationTest-Master (former SLAPOS-SR-TEST-MASTER)
  On every change on master branch, it compile several Software Releases, like ERP5, Wendelin, WebRunner, KVM .... (see the full list) on the testnodes (currently we have testnodes running on Debian 9 and Debian 10).
• SlapOS.Eggs.UnitTest-Master,  SlapOS.Eggs.UnitTest-Master.Python2 and SlapOS.Eggs.UnitTest-Master.Python3 (former SLAPOS-EGG-TEST and SLAPOS-EGG-TEST-PY2 and SLAPOS-EGG-TEST-PY3)
  Tests all eggs which are been developed under SlapOS Project, and it may include recipes (slapos.cookbook, slapos.recipe.build...) and also tools (slapos.toolbox). All tests here should run just using "python setup.py test". Tests for different Python versions.
• SlapOS.KVM.ResilienceTest-Master (former SLAPOS-RESILIENCE-KVM-MASTER)

  Tests that the disk used by a KVM is correctly backuped and can be reused by a second KVM, in order to ensure a disaster recovery.
• SlapOS.SlapRunner.ResilienceTest-Master.Gitlab (former SLAPOS-RESILIENCE-WR-GITLAB-MASTER)

  Tests if Gitlab Software Release (from master branch) inside a webrunner resilient works (produce a clone and replace main instance with the clone).
• SlapOS.SoftwareReleases.IntegrationTest-1.0 (former SLAPOS-SR-TEST-1.0)
  On every change on 1.0 branch  (a.k.a Release Candidate Branch), it compile several Software Releases, like ERP5, Wendelin, WebRunner, KVM .... (see the full list) on the testnodes (currently we have testnodes running on Debian 9 and Debian 10).