Erlang EUnit - continuation 2 - Test Representation

Having passed through basics and some setup / teardown, it's time to discuss test representation, a very broad subject which invites for misconceptions.


Basics of Representation
As always, the foundation for the discussion is the EUnit manual pages, there it stated that a test can be defined in many different ways, and to keep the discussion very short, I shall supply some common examples of what is often seen/used.


Simple Test Objects

Each representation form will be shown with examples based on the mylist module defined in the first post.  So, each Simple Test Object (STO) can be defined by

Any nullary function (function with zero arguments)


What this implies is that your STO is encapsulated within a functional object and may be executed at will by EUnit. More or less you can put yourself in EUnit's shoes (cool) and the user gives you a "black box", which you simply execute. No need to care about other stuff. Just execute it.

nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end. 

nullarySTO2() -> fun nullarySTO1/0 

nullarySTO3() -> fun mylists_tests:nullarySTO1/0

A Module / Function Tuple


This is just a different way of reaching the nullary function by the module and function name. Here it is implicit that the functionname is a nullary function. Remember that all erlang functions are uniquely determined by M,F,Arity.

tupleSTO1() -> { mylist_tests, nullarySTO1 }

tupleSTO2() -> { mylist_tests, nullarySTO2 }

tupleSTO3() -> { mylist_tests, nullarySTO3 } 

A Linenumber / STO  Tuple (generated by _test(Expr) macro)



According do our best friend, (the user's guide) "LineNumber is a nonnegative integer ....  LineNumber should indicate the source line of the test". Wow, so it's the linenumber in the sourcecode for where the tests is. For some reason, this can be written by hand, but luckily the _test(Expr) macro does it for us.

tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end )

tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end )

tupleSTO6() -> ?_test( nullarySTO2() ) 

Test Sets and Test Generator Basics

One last thing on the basics of representation is that EUnit does not make any "type"-difference between a deep list of STO's and single STO, that is.

TestSet :=  [ TestSet ] |  STO 

So, a Test-Set can be a so called deep list of Test-Sets, to any level, or just a single STO. So, any of these are Test Sets

nullarySTO1() 

[ tupleSTO4() ] 

[ [ nullarySTO1() , nullarySTO2() ] , [ tupleSTO1(), tupleSTO2() ] ]

Also, a Test Generator is a function that ends with _test_() and returns a TestSet. End of story. Like this

my_generator1_test_() ->  nullarySTO1().

my_generator2_test_() -> [ nullarySTO1(), tupleSTO4() ]

What we have seen are all valid erlang EUnit STO statements. To prove it, I shall place it in a gran EUnit file, compile and run. Below follows the example STO mylists_tests.erl module, with compilation and running as usual.


Simple Test Object Example File

-module(mylist_tests).

-include_lib("eunit/include/eunit.hrl").

-compile(export_all).

nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end. 

nullarySTO2() -> fun nullarySTO1/0.

nullarySTO3() -> fun mylist_tests:nullarySTO1/0.

tupleSTO1() -> { mylist_tests, nullarySTO1 }.

tupleSTO2() -> { mylist_tests, nullarySTO2 }.

tupleSTO3() -> { mylist_tests, nullarySTO3 }.

tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).

tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).

tupleSTO6() -> ?_test( nullarySTO2() ).

tupleSTO7() -> { 16, nullarySTO3() }.

generator_test_() ->    

    [

     nullarySTO1(),

     nullarySTO2(),

     nullarySTO3(),

     tupleSTO1(),

     tupleSTO2(),

     tupleSTO3(),

     tupleSTO4(),

     tupleSTO5(),

     tupleSTO6(),

     tupleSTO7()

    ].

That is the end of the module, next is the compilation and running.

zen:EunitBasic3 zenon$ erlc -o ebin/ src/*.erl test/*.erl

zen:EunitBasic3 zenon$ erl -pa ebin/

Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)

1> eunit:test(mylist,[verbose]).

======================== EUnit ========================

module 'mylist'

  module 'mylist_tests'

    mylist_tests: nullarySTO1...ok

    mylist_tests: nullarySTO2...ok

    mylist_tests: nullarySTO1...ok

    mylist_tests: nullarySTO1...ok

    mylist_tests: nullarySTO2...ok

    mylist_tests: nullarySTO3...ok

    mylist_tests:13: tupleSTO4...ok

    mylist_tests:14: tupleSTO5...ok

    mylist_tests:15: tupleSTO6...ok

    mylist_tests:16: nullarySTO1...ok

    [done in 0.030 s]

  [done in 0.030 s]

=======================================================

  All 10 tests passed.

ok

2> 

Great! Now, as we know how to write, set up and clean STO's and TestSets. The next lesson is on Test Control!

Erlang EUnit - continuation 1 - Fixtures

This time we will learn how to create a specific environment for our specific tests. As usual, all this can be found in the "EUnit User's Guide".

Why would I want to create specific environments?

Consider the case that we wish to perform EUnit tests on on a stateful system (a system with state that is), it could be a server, an fsm, whatever holds and updates a state.

How do I create these environments?

Meet your friend the fixture! There are three kind of major fixtures, Setup / Node / Foreach, this blog entry will only cover the basics of Setup. A fixture is the EUnit name for test-environment. By using fixture statements, we can control how a state (fixture) is created and destroyed. In EUnit terms, this is called

setup/0
cleanup/1

Normally, our EUnit tests run in the same basic environment, as was the case for the mylist EUnit tests, as we shall se next, this is often not desirable when testing a server which contains state (a stateful process).

An example stateful server!

Before (a-hah!) I show some fixture examples, I will show the example code that will be tested using fixtures! It will become clear why the fixture is needed.

-module(numberserver).
-export([start/0,stop/0,op/2,get/0,init/0]).
start() ->
    Pid = spawn_link(?MODULE,init,[]),
    register(?MODULE,Pid),
    ok.
stop() -> ?MODULE ! stop, unregister(?MODULE).
op(Op,Num) -> ?MODULE ! {Op,Num}, ok.
get() ->
    ?MODULE ! {get_result,self()},
    receive
        X -> X
    end.
init() -> loop(basic).
loop(E) ->
    receive
        stop -> ok;
        {get_result,From} ->
            From ! E, loop(E);
        {Op,Num} -> loop(result(Op,E,Num))
    end.
result(_,basic,X) -> X;
result('+',X,Y) -> X + Y;
result('*',X,Y) -> X * Y;
result('-',X,Y) -> X - Y; 
result('/',X,Y) -> X / Y.

The code above gives us a small server which performs basic binary operations (operations that take two operands), based on which operator and number is sent with the op/2 function. Compilation , start , usage and stop(ing) is shown below. As usual, compilation and execution is performed from the root directory of my "project".

zen:EunitBasic2 zenon$ tree .
.
├ ── compile
├── ebin
├── src
│ └── numberserver.erl
└── test

(As can be seen, I have been a bad boy once again and drifted from the enlightened path of TDD by first writing the source and later writing the tests - all for your learning convenience)

zenon:EunitBasic2$ erlc -o ebin/ src/*.erl
zenon:EunitBasic2$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> numberserver:start().
ok
2> numberserver:op('+',10).
ok
3> numberserver:op('*',2).
ok
4> numberserver:op('-',5).
ok
5> numberserver:op('/',3).
ok
6> numberserver:get().
5.0
7> numberserver:stop().
true
8>
(note the 5.0, this is due to the division which produced a non-integer result).

As you could see, this process obviously maintain some kind of state (hint: the previous computation result), and the subsequent calls use the state as second operand for each binary operation. Thus, when testing the different computations, we wish a clean untouched server for each run. (To exemplify further: assume we are testing basic addition,in one test generating function and later wish to test multiplication, now, in between each test we would ideally want the server to be reset in some way). This is where the fixture comes in.

Basic fixturing (state creation and state teardown).

The most basic fixture example does setup only (=creation only). This would be achieved with the 3-tuple

{ setup, SETUP-FUNCTION , TEST/INSTANTIATOR }

SETUP-FUNCTION must be a function with arity 0 (zero) and shall perform all necessary operations for creating the encompassing environment.

TEST can be a an EUnit Control operator (more on this in next post), or a test primitive, an INSTANTIATOR is a function receiving the result from the SETUP-FUNCTION and generated instantiated test-sets. I shall use the basic _test(Expr) macro here as is customary (note: there is nothing special about the _test/1 macro, it just returns the the given Expr as a test-function.

first_additon_test_() ->
     { setup, 
       fun setup/0,
       fun cleanup/1,
       ?_test(
          begin
              numberserver:op('+',2),
              numberserver:op('*',3),
              ?assertEqual(6,numberserver:get())
          end)}.

This extremely simple test could as well have been written as

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").
first_additon_test_() ->
    numberserver:start(),
    numberserver:op('+',2),
    numberserver:op('*',3),
    ?assertEqual(6,numberserver:get()).

Let's test this, save the code in numberserver_tests.erl inside the test/ directory, compile and run.

zenon:EunitBasic2$ erlc -o ebin/ src/*.erl test/*.erl
zenon:EunitBasic2$ erl -pa ebin/
zenon:EunitBasic2$ erl

Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> eunit:test(numberserver,[verbose]).
======================== EUnit ========================
module 'numberserver'
module 'numberserver_tests'
numberserver_tests:6: first_additon_test_...ok
[done in 0.004 s]
[done in 0.004 s]
=======================================================
Test passed.
ok

Great, but what happens if we run it again?

2> eunit:test(numberserver,[verbose]).
======================== EUnit ========================
module 'numberserver'
module 'numberserver_tests'
*** context setup failed ***
::error:badarg
in function erlang:register/2
called as register(numberserver,<0.48.0>)
in call from numberserver:start/0


=======================================================
Failed: 0. Skipped: 0. Passed: 0.
One or more tests were cancelled.
error

Uh oh, obviously we can't register the same numberserver in the same environment again. Seems like we need to do some kind of cleanup (hint hint). The cleanup/1 function is automatically given one argument, the result from the setup.

setup/0 -> X :: any()
cleanup( X :: any() ) -> any()

So, if your setup returns a Pid, you could use that pid in the cleanup to perform stops / kills, whatnot. As is, setup/0 is excuted before any tests are performed, and cleanup/1 is executed after all tests have been performed regardless of test errors or crashes. This would use the 4 tuple setup

{ setup,
  SETUP-FUNCTION/0,
  CLEANUP-FUNCTION/1,
  TEST/INSTANTIATOR }

CLEANUP-FUNCTION is the arity-1 function that reverses the effect of SETUP, and TEST is as usual a test-primitive or an EUnit control term, the INSTANTIATOR receives the same argument as cleanup, namely the result from SETUP.

Setup - Cleanup Fixture Example

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").
first_additon_test_() ->
     { setup, 
       fun setup/0,
       fun cleanup/1,
       ?_test(
          begin
              numberserver:op('+',2),
              numberserver:op('*',3),
              ?assertEqual(6,numberserver:get())
          end)}.
first_multiply_test_() ->
     { setup, 
       fun setup/0,
       fun cleanup/1,
       ?_test( 
          begin
              numberserver:op('*',1),
              numberserver:op('*',3),
              numberserver:op('*',5),
              ?assertEqual(15,numberserver:get())
          end)}.

setup() ->
    numberserver:start().

cleanup(_Pid) ->
    numberserver:stop().

Let's run it!

zen:EunitBasic2 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EunitBasic2 zenon$ erl -pa ebin
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> eunit:test(numberserver).
All 2 tests passed.
ok

What is so special about this? Well, for starters, we added one more tests, this second test will now succeed but would have failed previously. Why? Well, because we have a teardown and setup of the numberserver.

This was the basic setup and cleanup, you can find more in the manual pages, next time we will look at test representations.

Erlang EUnit - introduction

Why do I want to have a battery of EUnit tests?

1. Show that each part of the module is working as expected (Unit testing)
2. Be able to see if/how new code breaks existing code (Regression testing)

As part of TDD (Test Driven Development) one should ideally write the tests first, and later write the code that makes the tests go through. This will automatically give you a set of Unit + Regression tests, so that each step in your development cycle is "fastened" in a secure place (a "green light").

How do I write my first EUnit test?

Now, assuming you have not followed TDD and have an untested list-processing module (module is for demonstration purposes only and serves no other reason)

-module(mylist).
-export([sum/1,product/1,odds/1]).

sum([X|R]) -> X + sum(R);
sum([]) -> 0.

product([X|R]) -> X * product(R);
product([]) -> 1.

odds(List) -> odds(List,1).
odds([X|R],N) when N rem 2 == 1 -> [X | odds(R,N+1)];
odds([_|R],N) -> odds(R,N+1);
odds([],_) -> [].

Thus this module could be tested with the separate test module called mylist_tests

(This is good practice since you do not wish to clutter the logic - your real module, with test cases, thus your tests also become portable and regression testing with different versions of code becomes easier).

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").

sum_test() ->
    ?assertEqual(0, mylist:sum([])),
    ?assertEqual(0, mylist:sum([0])),
    ?assertEqual(6, mylist:sum([1,2,3,4,-4])).

product_test() ->
    ?assertEqual(1, mylist:product([])),
    ?assertEqual(2, mylist:product([2])),
    ?assertEqual(36, mylist:product([2,3,2,3])).

odds_test() ->
    ?assertEqual([], mylist:odds([])),
    ?assertEqual([1], mylist:odds([1])),
    ?assertEqual([1,3,5], mylist:odds([1,2,3,4,5])).

That was the whole test module, now there are some interesting points with the test module.

1. The test module name ends with _tests, this is to allow the eunit test function to find the tests for your module by simply referencing the source module (mylist.erl)
2. The test module does not export any functions explicitly
3. The test module includes the eunit header file -include_lib("eunit/include/eunit.hrl"), this is a most important part since this will automatically export all the tests in the module and cause them to be executed once we start.
4. Test function names end with _test(), this is a requirement for eunit to identify tests.

How do I run my eunit tests?

Save the mylists.erl and mylists_tests.erl in the same directory (for this basic guide).

Compile them with erlc

zen:ErlangBlog zenon$ cd EUnitBasic/
zen:EUnitBasic zenon$ erlc *.erl
zen:EUnitBasic zenon$ ls *.beam
mylist.beam mylist_tests.beam

Next, start erl, and run the tests with the test() function, as follows

zen:EUnitBasic zenon$ erl
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.7.5 (abort with ^G)
1> eunit:test(mylist).
All 3 tests passed.
ok
2>

All right! Seems to work. However, we had the test module in the same directory as the source module, which is not good practice.

Testing like a gentleman and a good lady

Make a clear division between test code and source code (well, clearly all of it is source anyway), also we wish our beam files to be stored in a separate directory

zen:EUnitBasic zenon$ rm -iv *.beam
zen:EUnitBasic zenon$ mkdir src
zen:EUnitBasic zenon$ mkdir test
zen:EUnitBasic zenon$ mkdir ebin
zen:EUnitBasic zenon$ mv mylist.erl src/
zen:EUnitBasic zenon$ mv mylist_tests.erl test/

So, what did I do? I removed our previous mess (the beams), created the source directory, the test directory and the ebin directory. Each one of them should contain source modules, test modules and beam files.

Next off, lets compile properly and run the tests, this time standing at the top level

zen:EUnitBasic zenon$ tree .
.
├── ebin
├── src
│ └── mylist.erl
└── test
  └── mylist_tests.erl

3 directories, 2 files

 This proves that I am in the top directory of my little "project".

 Next let me compile and output all beams to the ebin/ also I have to tell erlc where to find the source module and the test module.

zen:EUnitBasic zenon$ erlc -o ebin/ src/*.erl test/*.erl

Next, let's check that all beam files where generated into the ebin/ directory as promised.

zen:EUnitBasic zenon$ tree .
.
├── ebin
│ ├── mylist.beam
│ └── mylist_tests.beam
├── src
│ └── mylist.erl
└── test
  └── mylist_tests.erl

Seems good, let's run the tests then. From the top level, but telling erl how to find the beams. Also, let's do it verbally.

zen:EUnitBasic zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.7.5 (abort with ^G)
1> eunit:test(mylist,[verbose]).

======================== EUnit ========================
module 'mylist'
module 'mylist_tests'
mylist_tests: sum_test...ok
mylist_tests: product_test...ok
mylist_tests: odds_test...ok
[done in 0.009 s]
[done in 0.009 s]
=======================================================
All 3 tests passed.

ok

Wonderful! This was what we wanted. Now we have the basics set, and next post on this topic will teach us how to do proper test environment setup and cleanup.

All the basis can be found in the EUnit users guide