<!--break-->

libgeoip-erlang installation

Install prerequisite software: Mercurial, and the GeoIP library. On Debian, those are packages mercurial, libgeoip1, and libgeoip-dev. Then, get the libgeoip-erlang sourcecode, and compile it:

hg clone http://bitbucket.org/mattsta/libgeoip-erlang/
cd libgeoip-erlang/
make

Then, make sure that the generated libgeoip-1.0.1 directory is in the code load path, e.g. by passing -pz .../libgeoip-1.0.1 to the erlinterpreter. I personally prefer to install everything into /usr/local, and to use GNU Stow (Debian package stow) to manage packages there:

sudo mkdir -p /usr/local/stow/libgeoip-1.0.1/lib/erlang/lib/
sudo cp -r libgeoip-1.0.1 /usr/local/stow/libgeoip-1.0.1/lib/erlang/lib/
sudo chown -R root:root /usr/local/stow/libgeoip-1.0.1/
sudo stow -d /usr/local/stow/ libgeoip-1.0.1

This has the effect of installing libgeoip-1.0.1 into /usr/local/lib/erlang/lib/ with the possibility to easily uninstall it like any Stow package, with one command: sudo stow -d /usr/local/stow/ -D libgeoip-1.0.1. After installing additional Erlang libraries into /usr/local/lib/erlang/lib/, those can be loaded simply by setting the ERL_LIBS=/usr/local/lib/erlang/lib environment variable, as shown below.

Get MaxMind's free GeoLite City database. On Debian, this can be done by running:

sudo sh /usr/share/doc/libgeoip1/examples/geolitecityupdate.sh

This command installs the database into /usr/share/GeoIP/GeoIPCity.dat.

Test that libgeoip works correctly:

$ ERL_LIBS=/usr/local/lib/erlang/lib erl
> application:start(libgeoip_app).
> libgeoip:set_db("/usr/share/GeoIP/GeoIPCity.dat").
> libgeoip:lookup(<<91,121,26,170>>).

This should give you the location of the www.berabera.info server in France.

Using log_geoip_stats to analyze web client locations

Logtilla's log_geoip_stats module uses the libgeoip library to count the number of parsed log entries per client country. Here is a sample usage to analyze a single Apache access.log file:

$ cd src
$ PATH=../c_src:$PATH ERL_LIBS=/usr/local/lib/erlang/lib erl
> application:start(libgeoip_app).
> libgeoip:set_db("/usr/share/GeoIP/GeoIPCity.dat").
> {ok, Pid} = gen_log_analyzer:start_link(log_geoip_stats, [], []).
> ok = gen_log_analyzer:parse(Pid, "/var/log/apache2/access.log").
> log_geoip_stats:get_stats(Pid, 10).

The get_stats/2 function orders the countries by number of hits, converts the numbers of hits into percentages, and returns the top N countries (here, N=10). For one of my access.log files, this prints out:

[{'US',40.94006639874192},
 {'JP',30.572543537771566},
 {'GB',4.403284990389656},
 {'FR',3.9664511619779836},
 {'BY',3.8266643368862483},
 {'TR',3.6286330013396237},
 {'CH',2.958821131108393},
 {'TH',0.9260877162327451},
 {'PR',0.9202632651872561},
 {'CA',0.9086143630962782}]

The vast majority of my visitors in that period came from the USA (40%) and Japan (30%).

Implementation overview

In module log_geoip_stats, most of the code is boilerplate to implement the gen_log_analyzer behaviour. The most interesting pieces are functions handle_log_entry/2 and get_stats/2:

% Analyze a parsed log entry:
handle_log_entry(LogEntry, State) ->
    case LogEntry#'LogEntry'.'remote-host' of
        {'ip-address', IPAddress} ->
            case libgeoip:lookup(list_to_binary(IPAddress)) of
                {geoip, Country, _, _, _, _, _, _, _} ->
                    % Address found:
                    State1 = update_country(list_to_atom(Country), State),
                    {ok, State1};
                [] ->
                    % Address Unknown to the GeoIP library:
                    State1 = update_country('unknown', State),
                    {ok, State1}
            end;
        _Else ->
            % If the client address is a hostname or an ip6-address,
            % count it as 'unknown':
            State1 = update_country('unknown', State),
            {ok, State1}
    end.

% Query the stats from the analysis process, and order and convert the data:
get_stats(Name, Length) ->
    Stats = dict:to_list(gen_log_analyzer:call(Name, get_stats)),
    Total = lists:foldl(fun({_, Count}, Total) -> Total + Count end,
                         0, Stats),
    Stats1 = lists:sort(fun({_, C1}, {_, C2}) -> C1 > C2 end, Stats),
    Stats2 = lists:sublist(Stats, Length),
    lists:map(fun({Country, Count}) -> {Country, Count*100/Total} end, Stats2).

One possible improvement would be to handle timeouts to calls to libgeoip:lookup/1. The implementation of that function implicitly imposes an arbitrary timeout of 200ms, which I have sometimes observed. My current implementation does not tolerate such timeouts.

<!--break-->

In Erlang, a port program (or C port) is a program that is executed as a subprocess of the Erlang interpreter, and communicates with Erlang via its standard input and output. In an Erlang program, a port program is started using standard function open_port/2, and killed using port_close/1. The Erlang program sends binary data to the process' standard input using port_command/2, and can receive data from the process' standard output as a standard Erlang message in the form {Port, {data, Data}}, where Data is a binary.

The main problem is to design and implement a protocol to structure the data exchanged between the Erlang program and the port program. One could choose for instance a XML-based protocol, or a simple text-based protocol with one command per line (simple, but requires parsing on the receiving side), etc. In the case of Logtilla, the purpose of the port program is to parse text files, and to return structured data to the Erlang program (Logtilla's gen_log_analyzer module), so I needed to define a protocol with reasonably complex, but well-structured data types. Since Erlang/OTP has an excellent ASN.1 compiler, and ASN.1's purpose is to define structured data types for protocols, ASN.1 is the formalism of choice.

An overview of ASN.1 and Logtilla's datatypes

Logtilla's datatypes corresponding to the Common Log Format and Apache's Combined Log Format are defined in ASN.1 module WebAccessLog, in file asn1/WebAccessLog.asn1. The main type it defines is LogEntry:

LogEntry ::= SEQUENCE {
  -- Entries in the common log format:
  remote-host         NetworkAddress,
  client-identity [0] UTF8String OPTIONAL,
  auth-user       [1] UTF8String OPTIONAL,
  time                GeneralizedTime,
  request             UTF8String,
  status              HTTPStatusCode,
  length              INTEGER (0..MAX) OPTIONAL,
  -- Additional entries in the combined log format:
  referrer        [2] UTF8String OPTIONAL,
  user-agent      [3] UTF8String OPTIONAL,
  ...
}

This definition is straightforward. A LogEntry is a SEQUENCE, which is similar to a struct in C or a record in Erlang. It is an ordered set of fields, of which each has a name, e.g. remote-host, and a type, e.g. UTF8String. Here the fields are a direct transcription of the log formats. ASN.1 defines many standard types; only the types in bold above are custom-defined. ASN.1 types can be constrained, e.g. the (0..MAX) constraint on field length limits the range of values that this INTEGER field can take. A field can also be OPTIONAL. The ellipse "..." at the end of the definition makes the type extensible: any parser of this type can ignore any new field added in future versions. This ensures that future versions of the definition, with fields added after the ellipse, are backward-compatible with older decoders.

When encoding a LogEntry structure, an ASN.1 encoder encodes each field in sequence, and attaches an unambiguous tag to each encoded field. Module WebAccessLog specifies that tags should be automatically inferred, with option IMPLICIT TAGS. In that case, the standard tag associated with the ASN.1 type is used for each field. However, in some cases tags have to be manually disambiguated, as in the case of the successive referrer and user-agent fields, which are both of type UTF8String and OPTIONAL: if only one string is transmitted, it would be impossible for a decoder to determine from the type's tag if it corresponds to one field or the other. Therefore, I had to specify application-specific tags [2] and [3] to disambiguate.

The NetworkAddress type definition is straightforward too:

NetworkAddress ::= CHOICE {
  hostname        UTF8String,
  ip-address  [0] OCTET STRING(SIZE (4)),
  ip6-address [1] OCTET STRING(SIZE (16)),
  ...
}

It is a CHOICE, corresponding roughly to a union in C: a network address is either a hostname, of an IPv4 address, or an IPv6 address, etc. Only the chosen field is encoded/decoded. Again, since the ip-address and ip6-address fields have the same type, they have to be disambiguated by specifying application-specific tags, for the decoder to determine which field has been chosen. An OCTET STRING is a simple sequence of bytes.

The HTTPStatusCode type is an enumeration of the HTTP status codes, as defined in standard IETF RFC 2616:

HTTPStatusCode ::= ENUMERATED {
  continue(100),
  switching-protocols(101),
  ok(200),
  created(201),
  accepted(202),
  non-authoritative-information(203),
  no-content(204),
  reset-content(205),
  partial-content(206),
  etc. etc. etc.
  ...
}

An ENUMERATION is similar to an enum in C.

Defining Logtilla's protocol using ASN.1

I am using the terminology from the RM-ODP (Reference Model of Open Distributed Processing, ITU-T X.903). The most common concept for modeling interactions between two systems (a client and a server) is the concept of operation. There are two kinds of operations:

• an interrogation, which consists of two interactions:
  • an invocation, initiated by a client to convey information to a server to request the server to perform a function, followed by
  • a termination, initiated by the server to convey information in response to the invocation.
• an announcement, which consists of only an invocation.

Examples of annoucements are Erlang messages, and CORBA "oneway" remote method calls, and common examples of interrogations are RPC calls, and CORBA remote method calls. In many cases, the possible operations that can be performed with a server are specified in an IDL (Interface Definition Language). In the OSI world, the IDL is ROSE (Remote Operations Service Elements, ITU-T X.880-X.882). I use ROSE's concepts to specify the protocol implemented in Logtilla. Unfortunately, the ASN.1 modules specified in ROSE cannot be compiled using the asn1c ASN.1-to-C compiler, which I use in Logtilla, so I wrote the protocol specification by hand in "pure" ASN.1.

Logtilla currently implements only one operation: ParseLogFile. It is an interrogation. I defined one ASN.1 type for each PDU, in file asn1/WebAccessLogParserOperations.asn1, used each for either an invocation or a termination. There are two possible terminations for a ParseLogFile operation: EndOfFile notifies the end of the parsing of a file, and CannotOpenFile notifies an error when opening the file.

-- Invocation:
ParseLogFile ::= SEQUENCE {
  invoke-id INTEGER,
  argument UTF8String -- the name of the file to parse
}
 
-- Erroneous termination of a ParseLogFile operation:
CannotOpenFile ::= SEQUENCE {
  invoke-id INTEGER
}
 
-- Normal termination:
EndOfFile ::= SEQUENCE {
  invoke-id INTEGER
}

Since there may be several operations performed simultaneously by a client, the client must be able to associate a received termination to a pending operation. Therefore, each invocation PDU contains an invoke ID field, which is an integer identifier that is unambiguous to the client. This invoke ID is sent back by the server in each termination PDU, also in an invoke ID field.

This protocol must be completed by additional operations to allow the Logtilla file parser (the server) to send back log entries (LogEntry data) to the Erlang client. I use the concept of linked reply. This concept is used a lot in systems management protocols, the most important being CMIP (Common Management Information Protocol, ITU-T X.711). The concept of linked reply is one of the few concepts that are very specific to systems management, cf. ODMA (Open Distributed Management Architecture, ITU-T X.703).

A linked reply is an operation that is initiated by the server, and identified as part of an operation that was initiated by the client, and that happens before the client operation terminates. In ROSE, this is implemented by adding a linked ID field to each invocation of a linked reply, which corresponds to the invoke ID of the client operation. In Logtilla, the log file parser returns LogEntry data in a sequence of ReturnLogEntry linked replies, which PDU very simply contains only a linked ID and a LogEntry:

-- Invocation:
ReturnLogEntry ::= SEQUENCE {
  linked-id INTEGER, -- an invoke-id passed in a ParseLogFile
  argument LogEntry
}

Note that ReturnLogEntry is an announcement, i.e. it doesn't require a termination, so there is no need to include an invoke ID in the ReturnLogEntry type.

The possible sequences of interactions / PDUs for a ParseLogFile operation are therefore:

• success:
  

  client -----ParseLogFile----> server
  client <---ReturnLogEntry---- server
  client <---ReturnLogEntry---- server
  ...
  client <---ReturnLogEntry---- server
  client <------EndOfFile------ server
  

• error:
  

  client ----ParseLogFile----> server
  client <---CannotOpenFile--- server
  

We still have to group the PDU types to identify all the PDUs that can be sent by the Erlang client (and received by the C server), and vice-versa. This is easily done by defining two CHOICE types:

-- PDUs sent by the client to the server:
ConsumerPDU ::= CHOICE {
  parse-log-file [1] ParseLogFile,
  ...
}
 
-- PDUs sent by the server to the client:
SupplierPDU ::= CHOICE {
  cannot-open-file [1] CannotOpenFile,
  return-log-entry [2] ReturnLogEntry,
  end-of-file [3] EndOfFile,
  ...
}

Implementing the protocol in Erlang

Since communication with port programs is based on streams of bytes, one must find a way to segment the data into PDUs. This is done in Logtilla by passing the {packet, 2} option to the open_port/2 function when starting the C port program. With this option each sent PDU is prepended with its size in 2 bytes, and the data sent by the port program is segmented by reading a PDU size in 2 bytes, reading a binary of exactly that size, and passing that binary to the Erlang program in a {Port, {data, Data}} message. This takes care of segmenting. To transmit PDUs over TCP connections, instead of between Erlang programs and port programs, one can use the TPKT packet format (IETF RFC 2126), which similarly prepends the PDU size to each PDU.

Logtilla's gen_log_analyzer only has to decode the encoded PDUs, to match the type of the PDU using normal pattern matching, and to call the behaviour module's appropriate callbacks:

handle_info({Port, {data, EncPDU}}, State) when is_port(Port) ->
  {ok, PDU} = 'WebAccessLogParserOperations':decode('SupplierPDU', EncPDU),
  case PDU of
    {'cannot-open-file', CannotOpenFile} ->
      #'CannotOpenFile'{'invoke-id'=InvokeId} = CannotOpenFile, ...,
      {noreply, NewState};
    {'return-log-entry', ReturnLogEntry} ->
      #'ReturnLogEntry'{'linked-id'=LinkedId, 'argument'=LogEntry} = ReturnLogEntry,
      ..., Mod:handle_log_entry(LogEntry, ModState), ...,
      {noreply, NewState};
    {'end-of-file', EndOfFile} ->
      #'EndOfFile'{'invoke-id'=InvokeId} = EndOfFile, ...,
      {noreply, NewState}
  end.

Sending PDUs to the port program is also easy:

handle_call({parse, FileName}, {From, Tag}, State) ->
  Port = ...,
  % Build and encode the PDU into a binary:
  InvokeId = ...,
  Invoke = #'ParseLogFile'{'invoke-id'=InvokeId, 'argument'=FileName},
  {ok, PDU} = 'WebAccessLogParserOperations':encode(
    'ConsumerPDU', {'parse-log-file', Invoke}),
  % Using the {packet, 2} option for the port, the PDU will be
  % prefixed by its size as a 16-bit integer:
  port_command(Port, PDU),
  ...,
  {noreply, NewState};

All the encoding/decoding is implemented in modules automatically generated by OTP's ASN.1 compiler, as well as the record definitions (CannotOpenFile, ReturnLogEntry, EndOfFile, LogEntry, etc.).

I will describe the C side of the protocol implementation in a separate article.

Conclusion

This article describes a step-by-step approach to define a lightweight protocol to allow interactions between heterogeneous systems (a C port program and an Erlang program, in the case of Logtilla). The encoded form of ASN.1 data is very compact, and ASN.1 is very expressive: it defines many standard datatypes, etc.

The linked reply pattern is very common in systems management, since when querying a large set of managed objects for their states we usually want to be able to start processing replies as soon as managed objects send them back, without having to wait for all objects to have replied. This is also useful in the case of Logtilla, where using linked replies allows the Erlang program to handle and analyze replied log entries as soon as they are parsed by the port program. I believe that the linked reply concept apply to many contexts where large volumes of data are exchanged, and should not be confined to systems management.

<!--break-->

Installation

To build it, pull the Git archive from the Logtilla Git repository, and then initialize the build system, configure, and build:

autoreconf -vi
./configure
make

This requires you to install Autoconf, Automake, the asn1c ASN.1-to-C compiler which is used by Logtilla (I have tested that both the released 0.9.21 version and the version in the asn1c SVN repository are usable for Logtilla), and of course any recent version of Erlang/OTP.

Overview of Logtilla

Logtilla consists essentially of a single behaviour module: gen_log_analyzer, which defines the following callbacks:

• init/1: Initialize the state.
  

  init(Args::any()) ->
      {'ok', State::any()}
      | 'ignore'
      | {'stop', Reason::any()}.
  

• handle_log_entry/2: Handle a parsed log entry. The LogEntry record type is defined in header file WebAccessLog.hrl.
  

  handle_log_entry(LogEntry::#'LogEntry'(), State::any()) ->
      {'ok', NewState::any()}
      | {'error', Reason::any(), NewState::any()}.
  

• handle_call/3: Handle an application-specific call. This callback is similar to the gen_server:handle_call/3 callback.
  

  handle_call(Msg::any(), {From::pid(), Tag::any()}, State::any()) ->
      {'reply', Reply::any(), NewState::any()}
      | {'reply', Reply::any(), NewState::any(), Timeout::timeout()}
      | {'noreply', NewState::any()}
      | {'noreply', NewState::any(), Timeout::timeout()}
      | {'stop', Reason::any(), Reply::any(), NewState::any()}.
  

• handle_cast/2: Handle an application_specific cast. This callback is similar to the gen_server:handle_cast/2 callback.
  

  handle_cast(Msg::any(), State::any()) ->
      {'noreply', NewState::any()}
      | {'noreply', NewState::any(), Timeout::timeout()}
      | {'stop', Reason::any(), NewState::any()}.
  

• terminate/2: Cleanup on termination. This callback is similar to the gen_server:terminate/2 callback.
  

  terminate(Reason::any(), State::any()) ->
      no_return().
  

• code_change/3: Update the state after a module upgrade. This callback is similar to the gen_server:code_change/3 callback.
  

  code_change({'down', OldVsn::any()} | OldVsn::any(), State::any(), Extra::any()) ->
      {'ok', NewState::any()}.
  

The most important callbacks to implement are init/1 and handle_log_entry/2.

Running example

Logtilla contains a basic example module, log/logtilla_test. It counts how many parsed log entries correspond to a query reply for which a length was returned, and how many don't have a length. This module has no practical purpose, but is useful to illustrate the behaviour callbacks. The module's most important parts are:

-module(logtilla_test).

% Implement Logtilla's gen_log_analyzer behaviour:
-behaviour(gen_log_analyzer).
% Include Logtilla's header for the definition of the LogEntry record:
-include("WebAccessLog.hrl"). 

% Define and initialize the state:
-record(state, {count_without_length=0, count_with_length=0}).
init([]) ->
  State = #state{},
  {ok, State}.

% Analyze the log entry and update the state:
handle_log_entry(LogEntry, State) ->
  case LogEntry#'LogEntry'.length of
    asn1_NOVALUE ->
      {ok, State#state{
        count_without_length=State#state.count_without_length+1}};
    _Length ->
      {ok, State#state{
        count_with_length=State#state.count_with_length+1}}
  end.

% Implement an application-specific call to return the stats:
handle_call(get_stats, _, State) ->
  {reply, {State#state.count_without_length, State#state.count_with_length},
   State}.

To execute this example to parse a file named /var/log/apache2/access.log:

$ cd src
$ PATH=../c_src:$PATH erl
> {ok, Pid} = gen_log_analyzer:start_link(logtilla_test, [], []).
> ok = gen_log_analyzer:parse(Pid, "/var/log/apache2/access.log").
> gen_log_analyzer:call(Pid, get_stats).

This prints out a tuple with the count of entries without a length and the count of entries with a length.

You must add the c_src directory to the PATH, as it is where the logtilla_parser program is generated, and this program is executed as a port program by gen_log_analyzer to parse the files, so this program must be found in the PATH.

I will soon write other blog posts on the internals of Logtilla (which is the most interesting), and on future works.

<!--break-->

Only the AC_RUN_IFELSE patch has been committed so far. The other patches are functional, but the Autoconf maintainers would prefer a better integration with the other Autoconf macros, for instance by making Autoconf's AC_CHECK_HEADER macro working for Erlang code when the test language is set to Erlang, which could be used like that:

AC_LANG([Erlang])
AT_CHECK_HEADER([eunit/include/eunit.hrl])

I agree that this is the right way to go, and it will benefit the other supported languages, but it will take me a lot more time to implement.

Ralf Wildenhus, one of GNU Autoconf's maintainers, has also found a bug in the AT_CHECK_EUNIT macro, which executes EUnit tests from a GNU Autotest testsuite: the macro was always failing when used with Erlang/OTP versions prior to R13. We finally fixed that bug, so that AT_CHECK_EUNIT will work with prior versions of Erlang/OTP, when it will be first available in the next release of Autoconf. Users don't have to worry about the details of that bug, but I just want to share below my frustration of using Erlang to implement command-line tools.

The problem with AT_CHECK_EUNIT was that the Erlang module generated internally by AT_CHECK_EUNIT for calling EUnit was calling function init:stop/1 (taking an exit code as an argument). But that function was introduced only in R13, and only init:stop/0 (without argument) was available in previous versions. There are only three high-level ways to stop the Erlang VM:

• by calling halt/1, which takes the exit code as an argument, and abruptly exits the VM with that exit code without flushing the standard output;
• by calling init:stop/0, which cleanly flushes the standard output, and always exits the VM with exit code 0;
• by calling init:stop/1, which both cleanly flushes the standard output, and exits the VM with the exit code given as an argument, but is available only from version R13.

In AT_CHECK_EUNIT, we need both to return custom exit codes, to integrate with Autotest, and to flush the standard output, to let Autotest capture the whole output produced by EUnit. So only init:stop/0 can be used. Internally, the Erlang test module was initially run within Autotest's standard AT_CHECK macro, basically like:

AT_CHECK([erl -s foobar start], [0])

AT_CHECK checks the exit code of the command in argument, and compares it to the expected exit code (here, 0). If it is the expected exit code, then the test succeeds. If the exit code is 77, the test is skipped, e.g. to indicate that a requirement to the test is not met. Otherwise, the test fails.

In the executed Erlang module, we determine that the exit code is 77 if EUnit is not available (i.e., if module eunit cannot be loaded); otherwise if EUnit was run successfully and the EUnit test passed it is 0; otherwise it is 1. That exit code was passed to init:stop/1 to exit the VM. In the fixed version of AT_CHECK_EUNIT, the module now writes out the exit code into a temporary file, and then calls init:stop/0 (and therefore the VM always exits with code 0 and normally never fails). The real exit code is then read from the file and checked in a subsequent test, like:

AT_CHECK([erl -s foobar start], [0])
AT_CHECK([test -f tempfile && (exit `cat tempfile`)])

As for now, Automake provides no help to generate those files. This is Autoconf's job. What Autoconf can do is help you avoid the manual duplication of information between several files, and help you synchronize the content of the .app and .rel files with the state of your system and your Erlang/OTP installation. The trick is to let Autoconf generate parts of the .app and .rel files for you.

To let Autoconf substitute parts of those files, first add their names into the AC_CONFIG_FILES macro call, in your configure.ac file:


AC_CONFIG_FILES([... sample.app sample.rel])


Then, name your files with extensions .app.in and .rel.in, e.g. sample.app.in and sample.rel.in, as the configure script will generate sample.app from sample.app.in and sample.rel from sample.rel.in. Your .app.in and .rel.in files should contain variables to substitute enclosed with @...@.

For instance, a sample.app.in file could be:


{application, @PACKAGE@,
[{description, "Sample Erlang server"},
{vsn, "@VERSION@"},
{modules, [sample_app, sample_sup, sample_server]},
{registered, [sample_sup, sample_server]},
{applications, [kernel, stdlib]},
{mod, {sample_app, []}}
]}.


The @PACKAGE@ and @VERSION@ parts are substituted with the information supplied in the AC_INIT macro call in configure.ac.

Likewise, a sample.rel.in file could be:


{release, {"@PACKAGE@", "@VERSION@"}, {erts, "5.6.1"},
[{kernel, "@ERLANG_LIB_VER_kernel@"},
{stdlib, "@ERLANG_LIB_VER_stdlib@"},
{@PACKAGE@, "@VERSION@"}]}.


The variables ERLANG_LIB_VER_kernel and ERLANG_LIB_VER_stdlib contain the version numbers of the installed libraries named kernel and stdlib. To define such variables automatically with the versions of the libraries actually installed in your system, you just have to call the standard AC_ERLANG_CHECK_LIB Autoconf macro in your configure.ac file, e.g.:


AC_ERLANG_CHECK_LIB([kernel])
AC_ERLANG_CHECK_LIB([stdlib])


Note also that the ERTS version is not yet automatically determined by the standard Autoconf macros. However, Ruslan also proposed a new macro, AC_ERLANG_SUBST_ERTS_VER, to do that:


# AC_ERLANG_SUBST_ERTS_VER
# -------------------------------------------------------------------
AC_DEFUN([AC_ERLANG_SUBST_ERTS_VER],
[AC_REQUIRE([AC_ERLANG_NEED_ERLC])[]dnl
AC_REQUIRE([AC_ERLANG_NEED_ERL])[]dnl
AC_CACHE_CHECK([for Erlang/OTP ERTS version],
[erlang_cv_erts_ver],
[AC_LANG_PUSH(Erlang)[]dnl
AC_RUN_IFELSE(
[AC_LANG_PROGRAM([], [dnl
Version = erlang:system_info(version),
file:write_file("conftest.out", Version),
halt(0)])],
[erlang_cv_erts_ver=`cat conftest.out`],
[AC_MSG_FAILURE([test Erlang program execution failed])])
AC_LANG_POP(Erlang)[]dnl
])
AC_SUBST([ERLANG_ERTS_VER], [$erlang_cv_erts_ver])
])# AC_ERLANG_SUBST_ERTS_VER


You only have to put the code of that macro for instance in a acinclude.m4 file, placed in the same directory as the configure.ac file, and to call it in configure.ac:


AC_ERLANG_SUBST_ERTS_VER


That way, the ERLANG_ERTS_VER variable gets automatically defined with the detected installed ERTS version, and can be substituted in your .rel.in file, e.g. in sample.rel.in:

{release, {"@PACKAGE@", "@VERSION@"}, {erts, "@ERLANG_ERTS_VER@"},
[{kernel, "@ERLANG_LIB_VER_kernel@"},
{stdlib, "@ERLANG_LIB_VER_stdlib@"},
{@PACKAGE@, "@VERSION@"}]}.


Note to myself: I should really really take a little time to submit the definition of Ruslan's AC_ERLANG_SUBST_ERTS_VER macro to the GNU Autotools team for inclusion into the standard Autoconf...

Dryverl supports <dev-c-local-variable/> elements to declare local variables in the generated C code. The first bug was that Dryverl allows such an element to be empty, which would mean that the generated C local variable declaration has no specific initial value. However, in that case, Dryverl generated invalid declarations, such as:

int some_var = ();

Dryverl 0.1.3 now correctly generates no initializer in such cases:

int some_var;

The second bug was in the generated code that encodes output data, i.e. data that is returned from the C side to the Erlang side of the driver. As several methods are available in the Erlang port driver API to communicate between C and Erlang, Dryverl-generated drivers try to automatically use the most efficient one. The most efficient of all is the port call method. In the case the C code is invoked by a port call from the Erlang side, the emulator allocates itself a buffer that must be used for returning the output encoded data. The code generated by Dryverl automatically uses that buffer whenever possible, to avoid unnecessary memory allocation. However, if the encoded data exceeds that buffer, the generated code automatically switches to another method, which is to encode the output data as a separately allocated Erlang binary term, and reference that term from the output data encoded in the port call buffer.

The bug was that when the port call output buffer is exceeded, the data already encoded into it was not copied into the newly allocated Erlang binary's buffer, i.e. the start of that binary was left as garbage. The code generated by Dryverl 0.1.3 now correctly handles that case by copying the port call output buffer data into the Erlang binary buffer, before using that Erlang binary to further encode output data.

That second bug is of a kind that is very hard to trigger, because of the very large number of possible combinations of methods that can be used to communicate between Erlang and C, and because the code generated by Dryverl, and the strategy chosen by that code, depend on several conditions: are binaries sent in input? in output? are asynchronous threads used? is the encoded output data too large to fit the port call output buffer?, etc.

Thanks a lot to Hunter Morris for spotting those two bugs and for submitting a patch.