Testing Strategy

Managers of teams testing and lead tester often develop than necessary working documents and artifacts, documents a higher level, describing the general approaches to system testing and development of the software testing services process in the project. On one of these documents, artifacts, and will be discussed below.

Many of us have been developing a testing strategy, particularly often, such artifacts are interested in the customers of major projects, the development period which exceeds one year. Let’s try to clarify the concept of testing strategy and to answer some questions dismantling the few examples in practice.

Terminology

STRATEGY - the art of leadership, a general plan of this work, based on the current reality at the present stage of development.

There is another option, with a military slant:

The science of warfare, the art of warfare. The overall plan of warfare, military operations.

I cite two definitions, although the first to accurately describe what actually we are dealing with. And although our everyday lives can be seen as waging war for the quality of the product, as we shall see anything "military" in the testing strategy is just not.

Definition

Testing Strategy - a plan of work on testing the system or module-specific functionality and dependencies with other components of systems and platforms. The strategy defines the types of tests that need to perform for a given functional system includes a description of the approaches in terms of testing purposes, and may specify or describe the requirements necessary to conduct the test tools and infrastructure.

It turns a little scary? Let’s try to split into more detailed parts using, for example, the breakdown of the issues that the strategy meets the test.

The strategy responds to the questions:

·         How, how testing will answer that this functionality works?

·         What should be done and what use of the tools for achieving the goals of testing?

·         When a specific functionality will be tested and therefore when to expect the results?

As an additional problem to be solved in the process of understanding the strategy of testing, we can consider the problem of minimizing the cost of testing. Below is an example we analyze a specific case, but now just confine ourselves to stating the fact.

We see that the testing strategy, as an artifact, fits well into the test plan. Templates, test plans offered by different methodologies, often just one of the sections include a description of a testing strategy or include a description of the strategy in points of the plan is responsible for testing of specific parts of the functional. For example, the pattern Rational Unified Process defines testing strategy as the biggest section of the test plan.

That the ideologically more correct: to provide a strategy for the entire project, or to develop specific approaches for each area of ??testing depends on the project. In modern distributed systems with several types of jobs and system agents that also act as users of the system, allocate logical strategy for each set of functional test: the module or subsystem. For simpler applications you can offer testing of all applications on a single strategy.

The customer often wants to control the testing process and see the understanding of the problem of testing for implementing the project. For him, testing strategy is less detailed document, the vision (vision) of how the system will be tested during development.

Practice

In order to move from theory to practical application, try to develop a testing strategy for the two different software solutions: desktop applications, working in single user mode and client-server system of average complexity.

Parsed following examples use a simplified model of the application in the context of interaction between application components and systems, as well as the internal logic of the application.

As a simple desktop application, take, for example, a new scientific calculator.

Scientific calculator

The whole set of functions that applications can be divided into two logical groups: the logic of engineering operations and interaction with the operating system (allocating and freeing resources, the use of system components interact with the system clipboard, etc.).

Of the requirements for the application to provide support for the 5 operating systems in 4 main languages ??of localization and implementation, for example, 50 engineering functions, each of which is clearly covered by one test. Tests, of course, include checking test cases, checking the boundary values, etc. In addition, the application can perform, for example, 5 functions in the interaction with the system (running an application, exit the application, save results to a file, work with clipboard, etc.).

Full coverage of the requirements specifies a set of 5 * 4 * (50 +5) = 1100 test runs. So how to perform all operations available applications for all supported operating systems and language localization.

For example, as you know, picked up not by accident. Let’s try to illustrate the testing strategy in action.

We return to the breakdown of the application functionality into logical groups. There is a set of engineering functionality is system interaction.

It is obvious that the work and the validity of the calculations, implemented within an application and do not use external components are not dependent on the localization of the system. Moreover, if the application is successfully running under a supported operating system and basic functionality for working with engineering functions work, then having a successful run all the engineering tests with the analysis of test results, you can talk about the performance of engineering functions under all supported operating systems and localization.

Thus, to guarantee the availability of engineering operations, the calculator can be run of 50 tests under the same environment.

Interaction with the system, starting, stopping the application, work with the buffer, writing the results to a file in this example, delivers a 5 tests. We have 5 versions of the operating system and 4 localization, which gives 20 combinations and 20 * 5 = 100 test runs.

Thus, for this example, the full scope of functionality is determined by 50 +100 = 150 test runs, including 50 tests of engineering functions are performed under the same configuration and 5 tests of system functions under the 1920 test environment.

Testing strategy in action

In this case, we figured out that it was necessary to test, defined the requirements for tests, conducted a simple analysis of test conditions and dependencies. In essence, defined the strategy: "how" and "what" test. Develop tests can be in parallel to develop a strategy that in any case, the design and development of test strategy is not a substitute.

Full coverage of the requirements specifies 1100 test runs. Parsing functionality and definition of the required set of testing gives the output from the strategy of testing 150 test runs.

For this particular example, the development of a testing strategy provides a direct benefit in the cost of testing at 1100/150 = 7, (3) times. As you can see, it makes sense. Try to consider steps for developing a testing strategy for more complex applications - distributed client-server system.

Distributed system

Approach in developing a testing strategy for distributed systems in much the same developing a testing strategy for a desktop calculator. For example, similar to the previous example, we need to identify the main areas that can be tested separately. To test the complex systems, also useful to distinguish not only, so to speak, the operational steps for testing (that is, what, how and where to be tested), but also to analyze the tactical steps for testing with the development of the system over time.

We return to a strategy of testing complex distributed systems. Carry out an indicative breakdown of functionality to the test area in order to understand how to plan testing and minimize costs.

Suppose that the system has a "thick client, application server and database server, which can operate on different physical platforms. The basic logic input / output validation of values ??and the construction of printed forms focus on the client, application server provides the presentation level and the necessary service logic (automatic data archiving, alerting users about abnormal situations, etc.), and the database provides the addition of direct storage certain part of the data sold, for example, in the form of packages of functions. Nothing particularly difficult to be selected and let the description does not scare some over action - we just outlined the problem for testing.

In order to not overly complicate the problem, restrict the range of test environments by fixed configurations of the database server (often in the industrial exploitation of a dedicated server or a cluster solution for databases) and application server. The client application provides the under with 2 operating systems and are rigidly fixed configuration of the operating system components (for example, on all user machines are regularly installed all the updates and service packs + a set of software on client machines is strictly controlled by the internal IT policy).

Volume problems

Try to describe the problem in terms of test scenarios, then there will be used to assess the complexity of the problem or part of the number of tests required for testing the functional.

Client: 50 tests to work with the data (input forms, data calculation, based on data stored in the dictionaries, search data, editing dictionaries, etc.), 10 tests to work with the printed forms (forming periods of sampling, the choice of types of records, print or export to a predefined list formats, etc.). Let the testing of the system is required, as in the previous case, 5 more tests. By analogy with the previous example we can get the following picture: having 2 environment, we obtain (5 +10) * 2 = 30 test runs to verify the functionality associated with the operating system (including functional printing and exporting in which to create new files in the filesystem) . We assume that the 50 test runs of implementing validation logic of the data can be performed under the same environment. Outcome - 80 test runs to test the client system.

Uniting under the example in testing the functionality of the application server and database. Let the application server implements the 20 teams for data processing and user sessions (excluding the operation of system connection pooling, compression functions transferred over the network traffic, etc.). The database server implements the 1910 system for data archiving operations, construction, usage statistics report, and several more operations. The general sense is that we have a finite set of test operations, and as the configuration defined in advance, we can talk about the final set of tests that must be done that would test the server functionality of the system. Outcome - 30 tests on the server side. Note that in this example, we do not touch the load carrying component testing: we are talking only about the functional testing services.

 Depending on the project artifacts

More interesting from the standpoint of developing a testing strategy in this example, will factor in the development of the whole system: the creation of new modules in the client application or entry into a system of new server agents.

What should I consider developing a testing strategy for complex distributed or client-server systems. One of the main factors influencing the strategy of testing (after separation of the test areas and an understanding of test problems in each area), is an analysis of the schedule the appearance of new functionality in the system, and often plan to develop designs and specifications for the modules and system components (from This depends on the task of design tests). A clear project plan, divided by tasks of planning, design, design and implementation of testing gives the manager the foundation needed to solve the problem, which raises the question of "when", ie issues related to test planning. Although the temporary assessment and evaluation of complexity of the problem logically extend beyond the definition of the strategy and are planning to direct testing, it is understood the application / system in the context of its development over the project is a strategic component of the test plan.

Thus, developing a testing strategy for small applications, and serious enough systems similar to the stage of the selection of test areas and understand the dependencies of the application functionality from external modules and components. Vary the strategy depending on the type of application under test can in part understand the development of the system over time, since large systems are often developed in several stages and are characterized by large in comparison with desktop applications, a set of functionality that requires Retesting in each new version. Understanding of what exactly should be tested as how specific functionality will be tested, that is the result of satisfying the objectives of testing is often the result of the development strategy of testing.

Conclusions

That in general gives a strategy of testing? Analysis of test tasks on components, the selection of test areas and, ultimately, a more complete understanding of the test tasks in a specific project. As we have seen in testing engineering calculator, understanding the problem allows us to separate the functionality of the application under test or system areas that can be tested independently, thereby reducing (and sometimes quite significant!) The costs of testing.

Let us return to the definition of strategy: "a general plan of work based on the current reality at this stage of development" - is applicable to the testing strategy is understanding "what" "where" (in what environment) and "when" will be tested. The answer to the question "how" we can give an analysis of system requirements and design tests. Agree that having a plan before the development of functional and can be confident enough to schedule tasks for testing, prepare the necessary data and test environment. At any given time, the head, relying on a strategy, he knows where he is and where to go next. Planning a test, the head of department or lead tester does not begin to understand the system and directly engaged in planning, resource allocation and deadlines for specific tasks.

Visit: Software testing companies