Friday, January 23, 2015
Generating regression test cases
We miscounted—there are, in fact, 9,697 test cases so far.
My manager, S, has been responsible for generating the test case descriptions, stuff like:
Section 1
Originating device is XXXXXXXXXX XX XXXXX and is calling a terminator with our client on the device test cases.
- Terminating device can have the following features: XXXXXXX, XXXXXXXXXXXXX (2 options)
- Originating device can be a
sip:ortel:URI (2 options)- Terminating device can be a
sip:ortel:URI (2 options)- Originating can have XXXXXXX present or not (2 options)
- Terminator's XXXXXXX state is not applicable (1 option)
- Originating device can have two options for XXX (2 options)
- Terminator's XXX state is not applicable (1 option)
- Originating device can have XXXX values of XXXXXXXXXX, XXXXXXXXXXX, XXXXXXXXXXX (3 options)
- Terminator's XXXX state/capability is not applicable (1 option)
- XXXXXXXX XXXX XXXXXXXXXXX is either present or not for the Originating device (2 options)
- Terminator's XXXXXXXX XXXX XXXXXXXXXXX is not applicable (1 option)
- Originating Type of Number can either by CDMA or LTE (2 options)
- Terminating Type of Number can be: cdma+XXXX, cdma, lte (3 options)
- Terminating XXXXX can be: XXXXXX, XXXXXXX, XXXX (3 options)
- Originators XXXXX is not applicable (1 option)
- Total number of test cases in this section = 2 × 2 × 2 × 2 × 2 × 3 × 2 × 2 × 3 × 3 = 3456.
It's my job to generate the data required to run all these test cases, which means I need to run through the various variables and all their values, generating the data required to run the test and there's no better way of doing this than to brute force it. I could have written the code as:
-- variable names changed to protect the innocent.
for _,A in ipairs { false, true } do
for _,B in ipairs { 'tel' , 'sip' } do
for _,C in ipairs { 'tel' , 'sip' } do
for _,D in ipairs { false , true } do
for _,E in ipairs { false , true } do
for _, F in pairs { 'opt1' , 'opt2' , 'opt3' } do
...
test_case(A,B,C,E,E,F, ... )
end
end
end
end
end
end
end
and while that does the trick, it's verbose and it uses a convention
(using '_' to designate a result one doesn't care about, and in this case, I
don't care about the first return result from ipairs()) that
the other programmers here might not immediately pick up on (as far as I
know, I'm the only one using Lua at the Corporation).
No, I feel it's better to show my intent. I want the code to look like:
for A in bool() do
for B in list { 'tel' , 'sip' } do
for C in list { 'tel' , 'sip' } do
for D in bool() do
for E in bool() do
for F in list { 'opt1' , 'opt2' , 'opt3' } do
...
test_case(A,B,C,D,E,F, ... )
end
end
end
end
end
end
end
And that's exactly what I did. The generic for loop in Lua
is defined
as:
forvariableinfunction, state-variable, initial-value
where function is repeatedly invoked as
function(state-variable,initial-value) until the it returns
nil; when the result isn't nil, it's assigned to
the main loop variable. In the first example, ipairs() (a standard Lua
function):
function ipairs(list)
local function next_value(state,var)
var = var + 1
if var > #state then
return nil
else
return var,state[var]
end
end
return next_value,list,0
end
ipairs() returns a function that takes a state variable (in
this case, the variable list which should be an array) and an
initial value (0) and this is repeatedly called until that function returns
nil (here, when the index exceeds the number of items in the array).
For what I wanted, the function list() is easy:
function list(L)
local function next_item(state)
state.n = state.n + 1
return L[state.n]
end
return next_item,{ n = 0 }
end
The function next_item() only cares about the state, which
holds an index into the list (it's stored in a table because this is the
only way we can modify it) which we increment and return that item from the
passed in list L. We only return two values, the function and
the state. The missing third value will be nil, which we don't
care about. The one for cycling through booleans, bool()
looks a bit more complicated:
function bool()
return function(_,value)
if value == nil then
return false
elseif value == false then
return true
else
return nil
end
end
end
Here we just return a function; the other two values for is
expecting will then become nil. And since I don't care about
any state (the next value is readily apparent from the previous value we've
returned), that parameter in the returned function is named _
(the conventional name for ignored parameters). Since we initially gave
for a nil value, that's what we get on the first call, so we
return false. On the next call, value is
false so we return true. On the third call, we
call through, returning nil which ends the for
loop.
And those two functions make the test case generation code look much better. The intent is clearer, and you can easily match the code against the English description.
