Load testing - 10 pointers to improving performance
People often equate load testing with performance testing. Load
testing is seen as a way of answering the question "How fast
does the system respond?" This view then tends to mean that load
testing is seen as an end of project activity. Only at the end
of development will we have the final implementation for
performance testing and so we can confirm only then that it
performs quickly enough in the real world and smoothly
transition into live service.
Wrong approach! This is extremely risky and misses out on the
many benefits of starting load testing early and applying it
throughout the project. With this approach does the system sail
through load testing and transition smoothly into service?
Occasionally yes. But more frequently the system starts to fail
as load starts to be applied, even with small increases in
volume.. For the first time there are concurrent demands on the
system and arbitration over resources is required. Paths through
code that have never been executed are triggered, situations
arise that nobody really thought through. Transactions fail.
Systems crash. After these problems are fixed and more load is
applied in a test, we then encounter problems like resource
exhaustion, buffer overflows, timeouts and inconsistent
behaviour. The real work needed to turn a functional
pre-production system into a robust solution has only just begun.
Examples abound of products that failed when load testing
started and, after lots of effort, stress and expenditure, have
been shelved. Worse still are the ones that missed load testing
altogether and failed dramatically during live operation. An
internet portal developer recently stopped development of a new
service, one that had completed functional development, when
load testing revealed fundamental structural problems and
inefficient coding which led to a poorly performing and unstable
system.
So what should you do to avoid these risks? We all know it is
better to find faults early when they cost far less to fix yet
load testing is still left until as late as possible. The types
of faults it finds frequently need architectural changes and
major rewrites which are by then are hugely expensive to
implement. The answer is that you should start early. Different
forms of load testing should be repeatedly applied throughout
the project to identify problems early and to check that the
system is not going off track.
This is a natural extension of the practice of test led
development. Test led development, where automated tests are
written first and code must pass these tests as it is developed,
offers major benefits. However, in its current form, the focus
of this testing is on functionality. As it evolves the
functional status of the software is always known and hence
manageable, functional faults are nipped in the bud avoiding
high cost fixes, the functional risk is greatly reduced. Not so
other risks. If a project performs early and continuous load
testing it gets much wider and comprehensive risk reduction. To
make this effective:
1. Study the system and perform a risk analysis to help to order
the threats to the system, this will help you to prioritise load
testing activities.
2.Collect data to allow comparison of the efficiency of
different builds. This permits monitoring of the long term
trend, "Is the system using more and more processor time to do
the same work?" This data can be used to predict resource
requirements at different levels of demand and so support
scalability predictions.
3. Execute tests that aim to assess the behaviour of the system
and to trigger faults under load. Use workloads that simulate
expected patterns of demand to observe the aggregate behaviour
of the system. Use specially targeted extreme workloads to probe
the vulnerabilities of the system.
4. Include the full spectrum of load tests into the test suite.
This means performance testing with typical and busy period work
loads; stress testing to check both atypical demand spikes and
resource exhaustion impacts; endurance testing that uses both
operational period and cumulative operation tests; reliability
testing that runs lots of transactions and then checks whether
occasional transactions fail; concurrency testing of two users
working on the same account at the same time.
5. Design measurement activities as scientists would design an
experiment, design them to provide data that can be analysed.
Sample the system under different steady state workloads to
provide multiple data sets to support interpolation. Chose the
workloads to permit estimation of the resource costs for each
transaction type.
6. Target the middleware first with generic activities and
evolve the suite as functionality is developed. Start early and
then test each incremental release of the system, firstly with
the previous suite and then with a modified suite that addresses
new functionality.
7. Invest the time and resources to work at a representative
scale. Maybe the test bed can't be full scale but it should not
be two orders of magnitude smaller than the intended system. Be
smart and innovative to use resources effectively to provide an
appropriate scale test bed. The costs that will be incurred if
this is not done will far exceed the cost of providing the test
bed.
8. Don't delay; test an increment as soon as possible. Don't
skip one or you'll end up skipping them all. Compare the
measurements and behaviour with the previous one, is it better
or worse?
9. Provide a background load for functional testing. Features
that work offload may fail when the system has other things to
think about.
10. Consider occasional events such as server failures and
reconfiguration of the system. Do these need to be tested under
load?
In conclusion, you need to incorporate load testing throughout
the development process. Leaving load testing until the final
run in to live service is a recipe for disaster. If this became
common practice then a lot more applications and systems that
work would be delivered on time and to budget.