Some recent publications

In the last weeks, the Testing team has got some successes disseminating its work:

1. Pérez Lamancha B., Polo M., Caivano D., Piattini M. and Visaggio G. (2012). Automated generation of Test Oracles using a model-driven approach. Information and Software Technology. Accepted and in press. Available at: http://dx.doi.org/10.1016/j.infsof.2012.08.009
2. Reales P., Polo M. and Offutt J. (2012). Mutation at the multi-class and system levels. Science of Computer Programming. Accepted and in Press. Available at: http://dx.doi.org/10.1016/j.scico.2012.02.005
3. Reales P., Polo M. and Fernández JL. (2012). Validating 2nd-order mutation at system level. IEEE Transactions on Software Engineering. Accepted and in Press. Available at: http://doi.ieeecomputersociety.org/10.1109/TSE.2012.39
4. Reales P. and Polo M. (2012). Parallel mutation testing. Software Testing, Verification and Reliability. Accepted and in press. Available at: http://onlinelibrary.wiley.com/doi/10.1002/stvr.1471/abstract

Moreover, we have published a text book (in Spanish) where we present the main concepts of combinatorial and mutation testing: Técnicas combinatorias y de mutación para testing de sistemas software, ed. Ra-Ma, Madrid, Spain, 2012: http://www.ra-ma.es/libros/TECNICAS-COMBINATORIAS-Y-DE-MUTACION-PARA-TESTING-DE-SISTEMAS-SOFTWARE/75098/978-84-9964-146-1

expoQA 2012

Last 6 of June, we were at the International Conference on Software Testing & Quality (expoQA 2012), which was organized by nexoQA and was held in the Hotel Auditorioum, in Madrid.
Federico had a tutorial about performance testing, whilst Macario gave a talk entitled "Mutation: a testing technique ready to its transference to industry", which got the Technical Committee Award to the best technical paper.
Moreover, we did some demos of the Bacterio tool for mutation testing at the stand of Panel Sistemas Informáticos.

Semaphores: a new Java system for testing

Some time ago, we developed a small Java system for doing a first approach to analyze test case traceability, that we published in the SEKE 2009 conference (Int. Conf. on Software Engineering and Knowledge Engineering: download here the paper Some experiments on test case traceability). Our goal was to study how the coverage reached by a test suite on a high-level representation of a system is related to the coverage reached by those very same test cases in the system implementation.

The system simules a street with two semaphores. Pedestrians may press a button on any of the two semaphores for requesting red. There is a Manager that controls the light flow of two semaphores: when there are no pedestrians, the manager changes the light of both semaphores (a and b) sending them the change event every a fixed number of seconds (60, 63, 66, 83, and 86). However, a pedestrian may request the red light in any of the semaphores: if the semaphore where red is requested is in yellow or red, nothing happens; if it is in green and the semaphore is a, then the managers changes a to yellow either 20 seconds after the request or, if less than 20 seconds remain, in this time. If the red light is requested on b, then the request is passed to a:

The structure of the system is shown in the next figure: the window is represented by the JSemaphores class, which knows an instance of the Manager. User events are passed from JSemaphores to the Manager, who deals with the two instances of Sempahore it knows. There is also a Watch for controlling the color cycles where no pedestrians request red.

The behavior of the Manager class is given by the following state machine:

A text file representation of this state machine, valid to be processed by CTWeb for generating test cases, is the following:

% This is a small example of a state machine description file

Initial node

JC

% Transitions have: source state TAB symbol of the alphabet TAB target state

Transitions

JC setA A established

A established setB GG

GG requestRedOnA GG

GG requestRedOnB GG

GG setTime60 YG

YG setTime63 RY

RY setTime66 RR

RR setTime83 GR

GR requestRedOnA TGR

GR requestRedOnB TGR

TGR setTime86 GG

GR setTime86 GG

% Symbols can be mapped to method calls of the system using: symbol TAB method.

Symbol aliases

setA Manager manager=new Manager(this); Semaphore a=new Semaphore(manager); manager.setA(a);

setB Semaphore b=new Semaphore(manager); manager.setB(b);

requestRedOnA manager.requestRed(a);

requestRedOnB manager.requestRed(b);

setTime60 manager.setTime(60);

setTime63 manager.setTime(63);

setTime66 manager.setTime(66);

setTime83 manager.setTime(83);

setTime86 manager.setTime(86);

% States can also be used to the further creation of action oracles.

% The syntax is State TAB expression and the label is State aliases. For example:

State aliases

A established assertTrue(a.toString().equals("GREEN"));

GG assertTrue(manager.toString().equals("GREEN,GREEN,false"));

YG assertTrue(manager.toString().equals("YELLOW,GREEN,false"));

RY assertTrue(manager.toString().equals("RED,YELLOW,false"));

RR assertTrue(manager.toString().equals("RED,RED,false"));

GR assertTrue(manager.toString().equals("GREEN,RED,false"));

TGR assertTrue(manager.toString().equals("GREEN,RED,true"));

If you process the state machine with CTWeb with All pairs coverage, you get the following tests to path:

Path: [1] [setA, setB, requestRedOnA, requestRedOnA, requestRedOnB, requestRedOnA, setTime60, setTime63, setTime66, setTime83, requestRedOnA, setTime86, requestRedOnA]

Path: [2] [setA, setB, requestRedOnB, requestRedOnB, setTime60]
Path: [3] [setA, setB, setTime60]
Path: [4] [setA, setB, setTime60, setTime63, setTime66, setTime83, requestRedOnA, setTime86, requestRedOnB]
Path: [5] [setA, setB, setTime60, setTime63, setTime66, setTime83, requestRedOnA, setTime86, setTime60]
Path: [6] [setA, setB, setTime60, setTime63, setTime66, setTime83, setTime86, requestRedOnA]
Path: [7] [setA, setB, setTime60, setTime63, setTime66, setTime83, setTime86, requestRedOnB]
Path: [8] [setA, setB, setTime60, setTime63, setTime66, setTime83, setTime86, setTime60]
Path: [9] [setA, setB, setTime60, setTime63, setTime66, setTime83, requestRedOnB, setTime86]
Path: [10] [setA, setB, setTime60, setTime63, setTime66, setTime83, setTime86]

The code for the JUnit test case for the first path, generated by CTWeb is:

Manager manager=new Manager(this); Semaphore a=new Semaphore(manager); manager.setA(a);
assertTrue(a.toString().equals("GREEN"));
assertTrue(a.toString().equals("GREEN"));
Semaphore b=new Semaphore(manager); manager.setB(b);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.requestRed(a);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.requestRed(a);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.requestRed(b);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.requestRed(a);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.setTime(60);
assertTrue(manager.toString().equals("YELLOW,GREEN,false"));
manager.setTime(63);
assertTrue(manager.toString().equals("RED,YELLOW,false"));
manager.setTime(66);
assertTrue(manager.toString().equals("RED,RED,false"));
manager.setTime(83);
assertTrue(manager.toString().equals("GREEN,RED,false"));
manager.requestRed(a);
assertTrue(manager.toString().equals("GREEN,RED,true"));
manager.setTime(86);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));
manager.requestRed(a);
assertTrue(manager.toString().equals("GREEN,GREEN,false"));

Here, you can download the complete code of this system and a test suite covering All pairs generated by CTWeb from the test description of the state machine. Feel free to use this stuff.

The test suite may be executed by Bacterio, our mutation testing tool, which is free for students, professors and researchers. The next figure shows the results of executing the test suite (which proceeds from applying All pairs to the state machine) against the semaphores implementation: note that only 36.95% of mutants are killed.

MATE: Methodology for Automated model-driven TEsting

Some days ago, we have a published an article in the Testing Experience magazine explaining MATE.
Besides a typical uruguayan and argentine beverage, MATE is also complete methodology for automated model-driven testing that works both in traditional software development as in software product lines contexts.
MATE is completely based on OMG standards: UML, QVT, MOFM2T, UML Testing Profile... and has also capabilities for generating code.
The article can be downloaded for free from this link of the this link of the magazine web page.

The Java precision...

Hi all,

The other day i was testing a system that calcuates weights and i discobered a precision bug in Java.

Try to run this:

public void testPor3(){

double a = 3;

double b = a * 0.2;

System.out.println(b);

double af = 3;

double bf = a / 5;

System.out.println(bf);

}

The output that i got was:

0.6000000000000001

0.6

Then, i and Fede googled about this error and we found this (http://www.velocityreviews.com/forums/t139008-java-double-precision.html):

double val = 0;
for(int i=0;i<10;i++) {
val+=0.1;
System.out.println(val);
}

has the following (terrible) output:

0.1
0.2
0.30000000000000004
0.4
0.5
0.6
0.7
0.7999999999999999
0.8999999999999999
0.9999999999999999

What a mess!

What is mutation testing? (I)

[Do you want a spanish version of this page? Follow this link].

Suppose you are the head of a publishing house, and that you need to hire a style and spelling checker. If you have a set of candidates, a possible way for selecting the best one is to give them a text (such as that on the left side, which is the first paragraph of The Catcher in the Ray novel, by J.D. Salinger) with some errors, and then hiring that one who find more typos. 

If you realy want to hear about it, the first thing you'll probably want know is where I was born, an what my lousy child hood was like, and how my parents where occupied and all be fore they had me, and all that David Coperfield kind of crap, but I don't feel like going in it, if you want to know the true.


If we're not in a mistake, there are the following nine typos in it:

If you really want to hear about it, the first thing you'll probably want to know is where I was born, and what my lousy childhood was like, and how my parents were occupied and all before they had me, and all that David Copperfield kind of crap, but I don't feel like going into it, if you want to know the truth.

When mutation was proposed more than 30 years ago, the basic idea behind it was the same: to evaluate the quality of test suites as a function of the number of faults detected by the cases in the test suite, when are executed against the SUT. As more faults are detected, higher is the test suite quality.

As well as we have "injected" artificial typos in the original Salinger's text, mutation relies on the insertion of artificial faults in the code of the system under test (the SUT). These artificial faults are inserted by means of "mutation operators", which introduce some kind of syntactic change in the SUT.

In next chapter we'll talk about mutation operators :)

New version

A new version of Bacterio has been released:

http://www.alarcosqualitycenter.com/index.php/news?Itemid=93

News

[FEBRUARY 2012] New version BACTERIO Mutation Test System   Alarcos Quality Center has developed with Alarcos Research Group a new versión of the tool (BACTERIO Mutation Test System) with the follow features: Exploratory testing automation. "Mutant schema" implemented. Some types of mutation. Execution time storage. Parallel execution. Equivalent mutants detection assistance. You can download a demo version here.

Bacterio, free for students, professors and university researchers

Bacterio is a Java desktop-based tool for mutation testing. It is very easy to use and implements most of the techniques proposed in literature (bytecode mutation, mutant sampling, mutant schemata, parallel execution, n-order mutation, strong and weak mutation, etc.).
It is specially suitable for doing mutation at system and integration levels.
Besides strong and weak, it implements a novel comparison method called Flexible Weak Mutation.

• If you work or study in a university, feel free to write us and request a complete version.
• Otherwise, ask for a demo to Alarcos Quality Center.

Test case reduction: what is?

Both the execution of test cases against the system under test (the "SUT") and their maintenance have an associated cost: if the system changes, maybe the test cases must be changed, recompiled, reviewed and reexecuted. So, it is important to have test cases prioritized according, for example, to their ability to find faults on the SUT, or to any other "test requirement" (which, in general, will be some coverage criterion).

If we use the number of faults as test requirement, we'll be interested in executing firstly those test cases that find more faults in the SUT:

• In a new system, this prioritization can be made counting the number of artificial changes introduced in the SUT by a mutation tool.
• In an old system, the prioritization will depend on the number of faults found by the test cases in previous system releases.

Suppose a SUT for which 7 mutants have been generated, and a test suite composed by 6 test cases. The following matrix may show the number of faults found by each test case: tc1 finds the errors seeded on m1 and m2, tc2 the errors in m1, m2 and m3; tc6 does not find any fault.

	tc1	tc2	tc3	tc4	tc5	tc6
m1	X	X
m2	X	X			X
m3		X
m4			X
m5			X
m6			X	X
m7				X

tc1 and tc5 are redundant with respect to tc2, since the faults they find are a subset of the faults found by tc2: thus, tc2 is a better test case than the formers. To build a prioritized test suite, we would add test cases in this order: {tc2, tc3, tc4, tc1, tc5} (tc2 or tc3 could be added in a random order, since they find three faults). tc6 can be removed because it's an ineffective test case.

This kind of prioritizations are made by greedy algorithms. 

In some weeks we'll present a paper on this on the 15th International Conference on Fundamental Approaches to Software Engineering (FASE): Reduction of Test Suites Using Mutation.

Some other related papers are:

Reference	Brief summary
Harrold M, Gupta R and Soffa M (1993). A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology, 2(3), p. 270-285.	They give a greedy algorithm (usually referred to as HGS) for reducing the suite of test cases into another one, preserving the fulfillment of the test requirements reached by the original suite. The main steps of this algorithm are: Initially, all the test requirements are unmarked. Add to T’ those test cases that only exercise a test requirement. Mark the requirements covered by the selected test cases. Order the unmarked requirements according to the cardinality of the set of test cases exercising one requirement. If several requirements are tied (since the sets of test cases exercising them have the same cardinality), select the test case that would mark the highest number of unmarked requirements tied for this cardinality. If multiple such test cases are tied, break the tie in favor of the test case that would mark the highest number of requirements with testing sets of successively higher cardinalities; if the highest cardinality is reached and some test cases are still tied, arbitrarily select a test case from among those tied. Mark the requirements exercised by the selected test. Remove test cases that become redundant as they no longer cover any of the unmarked requirements. Repeat the above step until all testing requirements are marked.
Jeffrey D and Gupta N (2005). Test suite reduction with selective redundancy. International Conference on Software Maintenance.  Budapest (Hungary): IEEE Computer Society.	With Jeffrey, Gupta adds “selective redundancy” to the HGS algorithm. “Selective redundancy” makes it possible to select test cases that, for any given test requirement, provide the same coverage as another previously selected test case, but that adds the coverage of a new, different test requirement. Thus, maybe T’ reaches the All-branches criterion but not def-uses; therefore, a new test case t can be added to T’ if it increases the coverage of the def-uses requirement: now, T’ will not increase the All-branches criterion, but it will do with def-uses.
Tallam S and NG (2005). A concept analysis inspired greedy algorithm for test suite minimization. 6th ACM SIG-PLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering.	With Tallam, the test case selection method of HGS is based on Concept Analysis techniques. According to the authors, this version achieves same size or smaller size reduced test suites than prior heuristics as well as a similar time performance
Heimdahl M and George D (2004). Test-Suite Reduction for Model Based Tests: Effects on Test Quality and Implications for Testing. 19th IEEE International Conference on Automated Software Engineering.	Propose a greedy algorithm for reducing the test suite. Basically, they take a random test case, execute it and check the coverage reached. If this one is greater than the highest coverage, then they add it to the result. The algorithm is repeated five times to obtain five different reduced sets of test cases. Since chance is an essential component of this algorithm, the good quality of the results is not guaranteed.
McMaster S and Memon A (2005). Call Stack Coverage for Test Suite Reduction. 21st IEEE International Conference on Software Maintenance.  Budapest (Hungary).	McMaster and Memon present an also–greedy algorithm. The parameter taken into account to include test cases in the reduced suite is based on the “unique call stacks” that test cases produce on the program under test. As it is seen, the criterion for selecting test cases is not a “usual test requirement”, as it is common, but the number of unique call stacks.

Systems under test

Below appear some of the applications we have used as experimental stuff in some of our studies and publications. For all them, we provide their source code and test cases in JUnit format. All of them are implemented in Java.

• Chess: a RMI server of a chess game system.
• Cine: a small system to buy tickets for a cinema.
• citasMed: a system to get appointments for the doctor.
• ecal: a calculator, also available at sourceforge.net.
• monopoly: the domain logic of a system to play the famous Monopoly game.