Load Testing-Artifical Bottlenecks

As discussed in the Load Test Environments article, load test environments tend to be smaller versions of the production environment. These differences can include;

• Less memory
• Fewer and smaller physical CPU’s
• Fewer and less efficient disk arrays
• Single or fewer instances of servers, e.g. 2 database servers in performance test, but 3 database servers in production.

There can be other differences as well. System software & hardware versions and specification can vary. Data can be older and less comprehensive than in production. There also may be different authentication, firewall or load balancing configurations in place.

The performance test itself is full of compromise. For instance;

• Normally only a subset of functions are automated. The path through the function may vary slightly, but generally the same steps are repeated each iteration.
• Data can be mass produced in a uniform manner, affecting the way data is stored and accessed on the database. Some database tables can contain too little data, others too much.
• The behaviour of users can be unrealistic, for instance, if a requisition is raised, it is not generally fulfilled in reality until some days down the track. In a performance test, it may be 5 minutes later.
• Workload being processed varies in the normal course of events, during a performance test it can remain uniform.

An artificial bottleneck is essentially a performance problem that is a direct result of a difference between the production environment or workload and the performance test environment or workload.

When a performance bottleneck is found, the performance tester must investigate the symptoms in an attempt to try and pinpoint the cause of the issue. Care must be taken to distinguish between a genuine performance bottleneck and an artificial bottleneck brought about purely because of differences in performance test compared to production.

Examples of artificial bottlenecks include;

• Database locking. Performance testing results in a subset of functionality being automated. The entire workload is then spread across this subset of automation resulting in some functions been driven at a higher frequency during performance test than would be seen in production. This can result in database locking which would not occur in production. The solution? The problem function should be driven no higher than the maximum peak workload expected in production.

• Poor response times and excessive database logical I/O. If a large amount of data has been recently added to the database via a data creation exercise, database tables can become disorganised, resulting in inefficient use of indexes. Additionally, if performance testing started with the database tables nearly empty of data, the optimiser can incorrectly decide that index use is not required. The solution? Run a database reorganisation and refresh optimiser statistics at regular intervals is large amounts of data is being added to the database.

• Poor response times and excessive database physical I/O. The assumption here is that the database bufferpool is smaller in performance test than it is in production due to a shortage of memory in the performance test environment. Unfortunately, this is a case of poor performance of some users impacting the performance of all users. The solution? Once the problem function (or functions) is identified, attempt to decrease the workload for those functions to minimise physical I/O. If this is not possible, it may be time for a memory upgrade of the server. Memory upgrades are usually relatively straightforward in terms cost and time.