INDUSTRY

Financial Banking

PROJECT

Compulsory Compliance

SYNOPSIS

A collaboration of two to six week sprint cycles with over 500 onshore and offshore developers and over 100 testers delivering fully automated regression testing, quality gate into UAT and improved test assets.

Background

Prior to Testing Performance’s involvement the automation project had already had nine months of effort.  During this period the incumbent had only managed to automate less than 50 scripts, which equated to less than 10% of required coverage.  The incumbent team were 10 sprints behind the development.  Automated testing was deemed to be failing.

Requirement

Conduct a full regression test against each release

  • Test up to two releases per day in three environments - six releases per day, six test executions per day.
  • Imminent, next and development release, e.g. 7.0, 7.1, 7.2
  • Identify bugs and feed back to development after each regression test execution.
  • Provide a quality gate into UAT.
  • Continually update regression test pack to match new development.
  • Continually improve the test assets.

Challenges

This was a considerably pressured and highly political environment – the incumbent had failed to deliver an adequate solution in order for the bank to avoid receiving a £1bn fine.

Rapid Application Development – two to six week sprint cycles, test up to two releases per day in three environments.

Deliver Regression Test Packs with 100% Test Automation that could be continually updated to match new functionality.

Solution

After initially evaluating the automation project, our Lead Automation Consultant made tool recommendations and process adjustments.

A combination of commercial and open-source tools were implemented that created, executed, monitored and reported on tests.  The major process adjustment was to add a Triage step which assessed the following:

  • Were environments all up and at the correct version?
  • Did all the hardware start-up as expected?
  • Was the test data available?
  • Was the correct code version deployed?
  • Did the application build complete without error?

Our consultant worked with developers/project/automated testers to determine what new test scripts/assets were required.  Functionality was identified to be included in the next test run and the framework was updated to include new or modified tests.  This then instigated automation test builds, which instigated test execution producing results for review.  Results could be seen in real time via a dashboard or post execution via reports.  He used the results to prepare summary reports for daily stand up calls.

Testing Performance provided a team of six automation specialists to execute the automation test plan and were able to up or down scale the team size as required.

Outcome

After Testing Performance took over Automation Testing for the project we were able to:

  • Vastly improve process, focus and proficiency.
  • Automate over 1,000 scripts which were divided into 17 jobs over a full year of automation.
  • Bring automation within one sprint of development which is an ideal agile position.
  • Change the perception of automated testing - it is now deemed to be best practice and a Quality Gate.  
  • Move automation testing from managed Regional deployments to Global deployments.  
  • Bring about a 50% reduction in manual Functional Tester FTEs and 50% reduction in Automation FTEs
  • Enable the bank to complete the project within deadline thereby avoiding a £1bn fine.

This example demonstrates

How fully automated regression testing can be achieved in an agile environment

Our consultancy skills to step into a failing project and turn it around to be highly successful.

Automation testing is repeatable, reusable and although requires an initial investment it produces great results in time and cost savings.

Our ability to lead and build an automation test team with highly qualified specialists that can be scaled to requirement.