At my new customer there are a number of various types of testing, BT, BIT, FIT, EST, FT, ST etc. etc. You can imaging how hard it can be to get a grip on what all those acronyms actually mean. And more than that, even if you can decipher the acronyms, what to they really mean? Is Basic Integration Test “basic” with respect to what is integrated or does it imply that there is only some basic tests run? System Test is testing which system?
As we are introducing Agile techniques here, many people are asking me what is the difference between “acceptance test” and xT? (Where x can be almost any combination of letters…)
In my mind there are only two levels of tests: unit tests and functional tests.
We all know what a Unit Test is, a test verifying that the implementation of a small software unit, usually a class, matches the expected behaviour. And it should run fast. This means that unit tests should decouple the unit from almost every other unit. However we may define a unit to be some small set of classes that interacts, but a unit could never include a database or the network. (As we all know, in the “real world” we need to make compromises all the time…)
A Functional Test is a test that verifies some function of the product from a user/customer perspective. It should demonstrate some Customer Benefit. (This is probably not the only definition of “Functional Test” that exists, but I reused it instead of trying to invent another name for tests…)
But the funny thing is that Functional Tests are Acceptance Tests. At least until we are confident that the functionality is actually implemented. Then they becomes a Regression Tests. And running such tests for functionalities that require other applications makes them Integration Tests? And if we run the same tests in an environment simulating a real environment, then they become System Tests!
So there are at least two dimensions when we are talking about Tests. One is on granularity, units or the whole as percieved by some user. The other is in which environment and for which purpose are we running the tests. I find it helpful to talk about Tests when I talk about test cases, either or unit or functional level, and Testing when I talk about the environment and purpose running some tests.
So in Implementation Testing we usually run all Unit Tests and all Automated Functional Tests, typically using Continuous Integration in a development environment. The purpose is to catch errors in logic, setup and data.
System Testing is about running Functional Tests in an environment as close to a real environment as possible. Usually your application is not the only one which is exercised at the same time.
Performance Testing is about running many Functional Test at the same time to find performance bottle necks and errors caused by limited resources etc.
Its all in the name.