INDUSTRY

Financial Banking

PROJECT

Compulsory Compliance - A collaboration of two to six week sprint cycles with over 500 onshore and offshore developers and over 100 testers delivering fully automated regression testing, quality gate into UAT and improved test assets.

SYNOPSIS

Testing Performance / Fimatix - Performance Testing Case Study

THE BACKGROUND

Prior to Testing Performance’s involvement, the automation project had already seen nine months of effort.

During this period the incumbent had only managed to automate less than 50 scripts, which equated to less than 10% of required coverage. The incumbent team were 10 sprints behind the development, and automated testing was deemed to be failing.

THE REQUIREMENTS

Conduct a full regression test against each release:

•  Test up to two releases per day in three environments - six releases per day, six test executions per day.

•  Imminent, next and development release, e.g. 7.0, 7.1, 7.2.

•  Identify bugs and feed back to development after each regression test execution.

•  Provide a quality gate into UAT.

•  Continually update regression test pack to match new development.

•  Continually improve the test assets.

Test automation was seen as only way of achieving requirements​.

THE CHALLENGES

This was a considerably pressured and highly political environment – the incumbent had failed to deliver an adequate solution in order for the bank to avoid receiving a £1bn fine.

Rapid Application Development – two-to-six-week sprint cycles, test up to two releases per day in three environments.

Deliver Regression Test Packs with 100% Test Automation that could be continually updated to match new functionality.

THE SOLUTION

After initially evaluating the automation project, our Lead Automation Consultant made tool recommendations and process adjustments.

A combination of commercial and open-source tools was implemented to create, execute, monitor and report on tests.

•  Dashboard - Bespoke – A bespoke application that reports in real time on a test execution. Reports on multiple tests concurrently and provides a summary view.

•  Code builds – Maven – Builds testing releases. Integrate new and updated scripts. Supports multiple code versions.

•  Development IDE – IntelliJ – Used to develop test scripts and update existing scripts. Used to execute unit / standalone tests.​​

•  Framework – Serenity (open-source) – A framework that integrates scripts which implements BDD (Business Driven Design). Provides good reporting.

•  Test Tool – Selenium Web Driver – (open-source) - Actual test scripts that interface with application​. Scripts executed via Serenity.​

•  Scheduling tool – Jenkins – (open-source) - Schedules the testing tasks.

The major process adjustment was to add a triage step which assessed the following:

•  Are environments all up and at the correct version?

•  Did all the hardware start-up as expected?

•  Was the test data available?

•  Was the correct code version deployed?

•  Did the application build complete without error?

Our consultant worked with developers / project / automated testers to determine what new test scripts / assets were required.  Functionality was identified to be included in the next test run and the framework was updated to include new or modified tests. This then instigated automation test builds, which instigated test execution producing results for review.  

Results could be seen in real time via a dashboard or post execution via reports. These results to prepare summary reports for daily stand-up calls.

Testing Performance provided a team of six automation specialists to execute the automation test plan and were able to up or down scale the team size as required.

THE OUTCOME

After Testing Performance took over Automation Testing for the project we were able to:

•  Vastly improve process, focus and proficiency.

•  Automate over 1,000 scripts which were divided into 17 jobs over a full year of automation.

•  Bring automation within one sprint of development which is an ideal agile position.

•  Change the perception of automated testing - it is now deemed to be best practice and a Quality Gate.  

•  Move automation testing from managed Regional deployments to Global deployments.  

•  Bring about a 50% reduction in manual Functional Tester FTEs and 50% reduction in Automation FTEs

•  Enable the bank to complete the project within deadline thereby avoiding a £1bn fine.

THE EXAMPLE DEMONSTRATES

•  How fully automated regression testing can be achieved in an agile environment.

•  Our consultancy skills to step into a failing project and turn it around to be highly successful.

•  Automation testing is repeatable, reusable and although requires an initial investment it produces great results in time and cost savings.

•  Our ability to lead and build an automation test team with highly qualified specialists that can be scaled to requirement.