log_geoip_stats, which gives the top N client countries, in terms of hits, from web access log files. This module uses the libgeoip-erlang library to get geolocations from clients' IP addresses.
]]>log_geoip_stats, which gives the top N client countries, in terms of hits, from web access log files. This module uses the libgeoip-erlang library to get geolocations from clients' IP addresses.
<!--break-->
Install prerequisite software: Mercurial, and the GeoIP library. On Debian, those are packages mercurial, libgeoip1, and libgeoip-dev. Then, get the libgeoip-erlang sourcecode, and compile it:
hg clone http://bitbucket.org/mattsta/libgeoip-erlang/ cd libgeoip-erlang/ make
Then, make sure that the generated libgeoip-1.0.1 directory is in the code load path, e.g. by passing -pz .../libgeoip-1.0.1 to the erlinterpreter. I personally prefer to install everything into /usr/local, and to use GNU Stow (Debian package stow) to manage packages there:
sudo mkdir -p /usr/local/stow/libgeoip-1.0.1/lib/erlang/lib/ sudo cp -r libgeoip-1.0.1 /usr/local/stow/libgeoip-1.0.1/lib/erlang/lib/ sudo chown -R root:root /usr/local/stow/libgeoip-1.0.1/ sudo stow -d /usr/local/stow/ libgeoip-1.0.1
This has the effect of installing libgeoip-1.0.1 into /usr/local/lib/erlang/lib/ with the possibility to easily uninstall it like any Stow package, with one command: sudo stow -d /usr/local/stow/ -D libgeoip-1.0.1. After installing additional Erlang libraries into /usr/local/lib/erlang/lib/, those can be loaded simply by setting the ERL_LIBS=/usr/local/lib/erlang/lib environment variable, as shown below.
Get MaxMind's free GeoLite City database. On Debian, this can be done by running:
sudo sh /usr/share/doc/libgeoip1/examples/geolitecityupdate.sh
This command installs the database into /usr/share/GeoIP/GeoIPCity.dat.
Test that libgeoip works correctly:
$ ERL_LIBS=/usr/local/lib/erlang/lib erl
> application:start(libgeoip_app).
> libgeoip:set_db("/usr/share/GeoIP/GeoIPCity.dat").
> libgeoip:lookup(<<91,121,26,170>>).
This should give you the location of the www.berabera.info server in France.
Logtilla's log_geoip_stats module uses the libgeoip library to count the number of parsed log entries per client country. Here is a sample usage to analyze a single Apache access.log file:
$ cd src
$ PATH=../c_src:$PATH ERL_LIBS=/usr/local/lib/erlang/lib erl
> application:start(libgeoip_app).
> libgeoip:set_db("/usr/share/GeoIP/GeoIPCity.dat").
> {ok, Pid} = gen_log_analyzer:start_link(log_geoip_stats, [], []).
> ok = gen_log_analyzer:parse(Pid, "/var/log/apache2/access.log").
> log_geoip_stats:get_stats(Pid, 10).
The get_stats/2 function orders the countries by number of hits, converts the numbers of hits into percentages, and returns the top N countries (here, N=10). For one of my access.log files, this prints out:
[{'US',40.94006639874192},
{'JP',30.572543537771566},
{'GB',4.403284990389656},
{'FR',3.9664511619779836},
{'BY',3.8266643368862483},
{'TR',3.6286330013396237},
{'CH',2.958821131108393},
{'TH',0.9260877162327451},
{'PR',0.9202632651872561},
{'CA',0.9086143630962782}]
The vast majority of my visitors in that period came from the USA (40%) and Japan (30%).
In module log_geoip_stats, most of the code is boilerplate to implement the gen_log_analyzer behaviour. The most interesting pieces are functions handle_log_entry/2 and get_stats/2:
% Analyze a parsed log entry:
handle_log_entry(LogEntry, State) ->
case LogEntry#'LogEntry'.'remote-host' of
{'ip-address', IPAddress} ->
case libgeoip:lookup(list_to_binary(IPAddress)) of
{geoip, Country, _, _, _, _, _, _, _} ->
% Address found:
State1 = update_country(list_to_atom(Country), State),
{ok, State1};
[] ->
% Address Unknown to the GeoIP library:
State1 = update_country('unknown', State),
{ok, State1}
end;
_Else ->
% If the client address is a hostname or an ip6-address,
% count it as 'unknown':
State1 = update_country('unknown', State),
{ok, State1}
end.
% Query the stats from the analysis process, and order and convert the data:
get_stats(Name, Length) ->
Stats = dict:to_list(gen_log_analyzer:call(Name, get_stats)),
Total = lists:foldl(fun({_, Count}, Total) -> Total + Count end,
0, Stats),
Stats1 = lists:sort(fun({_, C1}, {_, C2}) -> C1 > C2 end, Stats),
Stats2 = lists:sublist(Stats, Length),
lists:map(fun({Country, Count}) -> {Country, Count*100/Total} end, Stats2).
One possible improvement would be to handle timeouts to calls to libgeoip:lookup/1. The implementation of that function implicitly imposes an arbitrary timeout of 200ms, which I have sometimes observed. My current implementation does not tolerate such timeouts.
<!--break-->
To build it, pull the Git archive from the Logtilla Git repository, and then initialize the build system, configure, and build:
autoreconf -vi ./configure make
This requires you to install Autoconf, Automake, the asn1c ASN.1-to-C compiler which is used by Logtilla (I have tested that both the released 0.9.21 version and the version in the asn1c SVN repository are usable for Logtilla), and of course any recent version of Erlang/OTP.
Logtilla consists essentially of a single behaviour module: gen_log_analyzer, which defines the following callbacks:
init/1: Initialize the state.init(Args::any()) ->
{'ok', State::any()}
| 'ignore'
| {'stop', Reason::any()}.
handle_log_entry/2: Handle a parsed log entry. The LogEntry record type is defined in header file WebAccessLog.hrl.handle_log_entry(LogEntry::#'LogEntry'(), State::any()) -> {'ok', NewState::any()} | {'error', Reason::any(), NewState::any()}.
handle_call/3: Handle an application-specific call. This callback is similar to the gen_server:handle_call/3 callback.handle_call(Msg::any(), {From::pid(), Tag::any()}, State::any()) -> {'reply', Reply::any(), NewState::any()} | {'reply', Reply::any(), NewState::any(), Timeout::timeout()} | {'noreply', NewState::any()} | {'noreply', NewState::any(), Timeout::timeout()} | {'stop', Reason::any(), Reply::any(), NewState::any()}.
handle_cast/2: Handle an application_specific cast. This callback is similar to the gen_server:handle_cast/2 callback.handle_cast(Msg::any(), State::any()) -> {'noreply', NewState::any()} | {'noreply', NewState::any(), Timeout::timeout()} | {'stop', Reason::any(), NewState::any()}.
terminate/2: Cleanup on termination. This callback is similar to the gen_server:terminate/2 callback.terminate(Reason::any(), State::any()) -> no_return().
code_change/3: Update the state after a module upgrade. This callback is similar to the gen_server:code_change/3 callback.code_change({'down', OldVsn::any()} | OldVsn::any(), State::any(), Extra::any()) ->
{'ok', NewState::any()}.
The most important callbacks to implement are init/1 and handle_log_entry/2.
Logtilla contains a basic example module, log/logtilla_test. It counts how many parsed log entries correspond to a query reply for which a length was returned, and how many don't have a length. This module has no practical purpose, but is useful to illustrate the behaviour callbacks. The module's most important parts are:
-module(logtilla_test).
% Implement Logtilla's gen_log_analyzer behaviour:
-behaviour(gen_log_analyzer).
% Include Logtilla's header for the definition of the LogEntry record:
-include("WebAccessLog.hrl").
% Define and initialize the state:
-record(state, {count_without_length=0, count_with_length=0}).
init([]) ->
State = #state{},
{ok, State}.
% Analyze the log entry and update the state:
handle_log_entry(LogEntry, State) ->
case LogEntry#'LogEntry'.length of
asn1_NOVALUE ->
{ok, State#state{
count_without_length=State#state.count_without_length+1}};
_Length ->
{ok, State#state{
count_with_length=State#state.count_with_length+1}}
end.
% Implement an application-specific call to return the stats:
handle_call(get_stats, _, State) ->
{reply, {State#state.count_without_length, State#state.count_with_length},
State}.
To execute this example to parse a file named /var/log/apache2/access.log:
$ cd src
$ PATH=../c_src:$PATH erl
> {ok, Pid} = gen_log_analyzer:start_link(logtilla_test, [], []).
> ok = gen_log_analyzer:parse(Pid, "/var/log/apache2/access.log").
> gen_log_analyzer:call(Pid, get_stats).
This prints out a tuple with the count of entries without a length and the count of entries with a length.
You must add the c_src directory to the PATH, as it is where the logtilla_parser program is generated, and this program is executed as a port program by gen_log_analyzer to parse the files, so this program must be found in the PATH.
I will soon write other blog posts on the internals of Logtilla (which is the most interesting), and on future works.
]]>