GNU Autotest is GNU Autoconf’s unit testing tool, and is very generic, portable and simple. Autotest is therefore well suited to run tests using other testing tools, such as EUnit (Erlang/OTP’s unit testing tool), JUnit, etc. I just added to GNU Autoconf the AT_CHECK_EUNIT macro to run EUnit unit tests in Autotest testsuites. It is the first Autotest macro to integrate such an external testing tool, and will be included in the next version of GNU Autoconf (> 2.64). A complete, working example project using AT_CHECK_EUNIT can be downloaded in autotest-eunit-demo-1.0.tgz. This article shows how to use AT_CHECK_EUNIT by explaining the important parts of this example project.
How to write Autotest testsuites
In Autotest, each project has usually one testsuite to run all its tests. A testsuite is a set of test groups. The purpose of Autotest is to generate a single, portable shell script that performs all the tests in a testsuite. A testsuite is specified in a text file with file extension .at, typically named testsuite.at, and which contents are M4 macro calls. Autotest defines a set of M4 macros specific to implementing testsuites.
The main testsuite.at file must start with a call to AT_INIT, and optionally a call to macro AT_COPYRIGHT, and then typically includes other .at Autotest files that define the test groups. For instance, a testsuite.at file that includes autotest_eunit_demo.at is:
AT_INIT AT_COPYRIGHT([Copyright (c) 2009 Romain Lenglet]) m4_include([autotest_eunit_demo.at]) ...
An included .at file should contain a set of test groups. A call to AT_BANNER at the top of a .at file displays information about the set of test groups. Then, each test group consists in a number tests enclosed between AT_SETUP and AT_CLEANUP. AT_SETUP takes a descriptive text as an argument. Tests in Autotest consist in creating files using macro AT_DATA, and executing commands using AT_CHECK, mixed with Bourne Shell code.
For instance, a set of simple tests of command grep is:
AT_BANNER([Trivial tests of grep]) AT_SETUP([grep]) AT_DATA([testfile], [[line one line two ]]) AT_CHECK([grep -q one testfile]) AT_CLEANUP
AT_CHECK executes the given command line, and tests that its exit code is 0, and that the command outputs nothing on its standard outputs. AT_CHECK takes several optional arguments to specify which exit code is expected (0 is the default), whether the command’s output should be ignored, etc.
How to use the AT_CHECK_EUNIT macro
The new AT_CHECK_EUNIT macro is similar to AT_CHECK. However, instead of executing a command line, it executes a EUnit test, given a EUnit test specification. Its syntax is the following:
AT_CHECK_EUNIT(module, test-spec, [erlflags], [run-if-fail], [run-if-pass])
test-spec is the specification of the test to run. It must be a valid EUnit test specification. module is a unique Erlang module name. Under the hood, AT_CHECK_EUNIT generates a “wrapper” Erlang module named module, which calls the EUnit library to execute the test-spec, compiles that wrapper module and executes it. If the EUnit test passes, AT_CHECK_EUNIT passes, otherwise it fails. erlflags are optional command-line options to pass to the Erlang interpreter to execute the wrapper module. This can be used for instance to specify the paths to the compiled modules under test.
For instance, to test all the EUnit tests associated to a module autotest_eunit_demo, one can use EUnit specification {module, autotest_eunit_demo}. To perform that test in Autotest, it is sufficient to write this autotest_eunit_demo.at file:
AT_BANNER([Tests demonstrating the integration of GNU Autotest and EunitTrivial tests of grep])
AT_SETUP([autotest_eunit_demo])
AT_KEYWORDS([autotest_eunit_demo])
AT_CHECK_EUNIT([test],
[{module, autotest_eunit_demo}],
[-pa "${abs_top_builddir}/src"])
AT_CLEANUP
In this example, the project’s build system is assumed to compile the tested modules (autotest_eunit_demo.beam, etc.) into the src subdirectory of the project. Soerlflags is set to -pa "${abs_top_builddir}/src", which configures the Erlang VM to load compiled modules from that directory.
The call to AT_KEYWORDS macro, which is a standard Autotest macro, associates keywords to this test group. A testsuite can be setup to run only tests with specific keywords, instead of running all the tests in the testsuite.
How to generate testsuites
An Autotest testsuite should be generated using GNU Autoconf and GNU Automake. An Autotest testsuite is typically contained in a subdirectory tests of a project, which should contain all the .at files for the testsuite (testsuite.at, and all other included .at files), and a Makefile.am Automake file to generate and run the testsuite. This Makefile.am is mostly boilerplate given in the Autoconf manual. What varies between projects is the list of .at files to process (in bold):
# Always generate package.m4 into the source directory, not into the
# build directory, since it must be distributed, along with testsuite,
# configure, etc.
$(srcdir)/package.m4: $(top_srcdir)/configure.ac
:;{ \
echo '# Signature of the current package.' && \
echo 'm4_define([AT_PACKAGE_NAME], [$(PACKAGE_NAME)])' && \
echo 'm4_define([AT_PACKAGE_TARNAME], [$(PACKAGE_TARNAME)])' && \
echo 'm4_define([AT_PACKAGE_VERSION], [$(PACKAGE_VERSION)])' && \
echo 'm4_define([AT_PACKAGE_STRING], [$(PACKAGE_STRING)])' && \
echo 'm4_define([AT_PACKAGE_BUGREPORT], [$(PACKAGE_BUGREPORT)])'; \
echo 'm4_define([AT_PACKAGE_URL], [$(PACKAGE_URL)])'; \
} > $@-t
mv $@-t $@
EXTRA_DIST = testsuite.at autotest_eunit_demo.at package.m4 $(TESTSUITE)
TESTSUITE = $(srcdir)/testsuite
check-local: atconfig $(TESTSUITE)
$(SHELL) '$(TESTSUITE)' $(TESTSUITEFLAGS)
installcheck-local: atconfig $(TESTSUITE)
$(SHELL) '$(TESTSUITE)' $(TESTSUITEFLAGS)
clean-local:
test ! -f '$(TESTSUITE)' || \
$(SHELL) '$(TESTSUITE)' --clean
AUTOM4TE = autom4te
AUTOTEST = $(AUTOM4TE) --language=autotest
# Always generate testsuite into the source directory, not into the
# build directory, since it must be distributed, along with
# package.m4, configure, etc.
$(TESTSUITE): $(srcdir)/testsuite.at $(srcdir)/autotest_eunit_demo.at \
$(srcdir)/package.m4
$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
mv $@.tmp $@
DISTCLEANFILES = atconfig
The top directory must contain a Makefile.am that defines SUBDIRS to build at least the tests subdirectory (as well as the other subdirectories, etc.):
... SUBDIRS += tests ...
Finally, the configure.ac Autoconf configuration file must contain a call to macro AC_CONFIG_TESTDIR(dir) where dir is the subdirectory of the testsuite (in this case, tests). To enable the use of macro AT_CHECK_EUNIT in testsuites, configure.ac must also detect the Erlang interpreter and the Erlang compiler, using macros AC_ERLANG_PATH_ERL and AR_ERLANG_PATH_ERLC (or better, AC_ERLANG_NEED_ERL and AC_ERLANG_NEED_ERLC):
AC_PREREQ([2.64.6-683a0]) AC_INIT([Autotest Eunit integration demo], [1.0], [romain.lenglet@example.com], [autotest-eunit-demo]) AM_INIT_AUTOMAKE([1.10]) ... AC_ERLANG_NEED_ERL AC_ERLANG_NEED_ERLC ... AC_CONFIG_FILES([ Makefile tests/Makefile ...]) AC_CONFIG_TESTDIR([tests]) AC_OUTPUT
If the Erlang interpreter and compiler are not detected in configure.ac using those macros, all the test groups that use AT_CHECK_EUNIT will be skipped.
To obtain all the configuration and build files generated by Autoconf and Automake, and also the tests/testsuite script generated by Autotest to execute all the tests, run those commands:
autoreconf -vi ./configure (cd tests ; make testsuite)
Since the AC_CHECK_EUNIT macro requires a version of Autoconf newer than 2.64, and no such version has yet been released, you will have first to download and build Autoconf from the latest version in the Autoconf source repository, and make sure that the built versions of those tools are in the PATH when running the commands above.
The generated files (configure, Makefile.in, tests/testsuite, etc.) must be distributed to users. This is done automatically when generating an archive using the generated Makefiles and make dist:
./configure ; make dist
How to run Autotest testsuites
As an end-user, all that is needed to run the test suite is to configure and build the software, and then executing make check:
./configure ; make ; make check
For instance, to build and test my example project autotest-eunit-demo-1.0.tgz, you only need to execute:
tar xzf autotest-eunit-demo-1.0.tgz cd autotest-eunit-demo-1.0 ./configure ; make ; make check
End-users don’t need to install and use Autoconf, Automake, or Autotest. The generated distributed files don’t depend on those tools.
In the tests subdirectory, all that make check does is to execute the generated tests/testsuite shell script. testsuite can be executed directly, and accepts some options. For instance, running testsuite in verbose mode displays detailed information on the standard output, and also runs EUnit in verbose mode:
(cd tests ; ./testsuite --verbose)
One can also restrict the executed tests to only those associated to specific keywords (as specified using macro AT_KEYWORDS), for instance:
(cd tests ; ./testsuite -k autotest_eunit_demo)
Conclusion
Setting up an Autotest testsuite may seem tedious. However, it is quite easy once you have a working Autoconf / Automake configuration. Then, adding tests using AT_CHECK_EUNIT macros to run EUnit tests is trivial. This integration of Autotest and EUnit will mostly benefit developers and users of projects that use several languages, e.g. Erlang, Java, and C, who have to integrate tests written in different languages and using different testing systems. Autotest will be extended to integrate more and more testing systems, to be usable as a general driver of all a project’s tests.