top of page

Parallelization of Philips Testing Framework

Context: Philips has nightly bulids for the PIC iX project where tests are run against the existing codebase to test whether or not any checked in changes affected the product in ways that negatively impact its functionality. With the addition of new product functionality day by day, over time, the execution of the nightly build began to approach 24 hours. If the nightly build were to hit 24 hours, it would no longer be effective in verifying the product code changes made since nightly builds would overlap.

​

Question: How can we dramatically reduce the execution time of our product tests and prevent the nightly build from taking 24 hours to complete?

​

Solution: Upgrade the previous sequential testing to parallel testing through a system of reservation and releasing clients for usage.

Philips Testing Infrastructure

IOCSPDB: Database storing testing agents (i.e remote agent, which connects the database to all testing clients) and logs in case a testing client crashes mid-test execution

​

IOCSP1, IOCSP2, IOCSPPHY: Philips monitors with all necessary binary files and tools installed to run product tests. For nightly build pipeline use as well as personal testing.

Reservation

Previously, the database would be connected to multiple testing clients at once, however these clients would not be running tests from the same pipeline at once. For example, if the nightly build was executing, the pipeline would take IOCSPPHY to run tests on it. IOCSP1 and IOCSP2 would either be free or in use by another pipeline (validation, acceptance). The issue with this approach is that integration and validation would commonly finish before nighttime, and personal testing occuring during the night is quite rare. This would leave resources available for testing just sitting with nothing to do.

 

How Reservation System Works In Practice

Through the reservation system, the database is able to 'lease' a specific client. This way, if a pipeline or personal tester tries to use a client already in use, there will be no conflict as the client will not be available for them to test on. 

The remote agent is able to keep track of leases through a simple hashmap. When a client has finished running tests, the client sends a finished signal to the remote agent which then releases the client from the document. The agent also stores a queue of reservation requests, so if a pipeline is also trying to access a client in use, they will wait for the earliest finishing client to become free.

In order to prevent test collision, meaning if IOCSP1 and IOCSP2 are both running product tests on the web, test categories were made to separate collision. For example if one test sets a shared bed "Bed_01" to have an alarm ring with background color green, and another test sets the alarm to ring with red, then one or both of the two tests may fail due to the eventual color not reflecting what their test reflects. By separating tests into batches, tests can be run in parallel but also with order, so all alarm color tests are put in Test Category 1, and all heart rate tests are put in Test Category 2. 


 

Result: Since many tests share categories, the test execution time does not decrease by 66%, however build start and end times do show significant cuts in execution before vs. after the added parallelization. 

As seen in our pipeline results, Network Tests - 1 duration for PIC iX 4.6 acceptance was cut by about 40% compared to PIC iX 4.5, resulting in complete pipeline execution in 5 less hours

bottom of page