Kerberized remote start of Erlang/OTP slave nodes

This post describes how to configure remote Kerberos-authenticated command execution to start remote slave Erlang nodes on Debian GNU/Linux.


Starting slave nodes requires to use a rsh-compatible command, which name is specified as an argument to the erl command. For instance, ssh is commonly used to start remote slave nodes:

erl -rsh ssh

When starting a slave node, using the functions of the standard slave module, that specified rsh-like command (for instance, ssh) is executed to start an Erlang VM Unix process on the remote host. That Unix process detaches itself from the rsh-like server process, to run in the background, and the rsh connection is terminated. Then, the slave node exclusively communicates using standard Erlang messages with the master node and the other nodes.

There are several ways to execute commands on a remote host, using Kerberos authentication:

  • remote shells commands: MIT kerberized telnet, MIT kerberized rlogin.
  • rsh-like commands: MIT kerberized rsh, kerberized OpenSSH, remctl.

In the following, I evaluate the several alternatives to rsh and ssh, that are based on Kerberos authentication. The advantage of Kerberos-based authentication vs. ssh key-pair-based authentication, in this case, is that it does not require to copy public keys on every node on which one wants to start slave nodes (like when using ssh): authentication is centralized. In addition, Kerberos is a single-sign-on (SSO) method for authentication, i.e. if your client system is well configured (e.g. using the libpam-krb5 package), there is no need for a user to enter any password in order to start slave nodes. Using ssh, it is necessary to use a ssh-agent, and to call ssh-add at least once for every client login session.

So far, I have been successful only in using remctl (see below).

MIT kerberized rsh

MIT kerberized rsh (Debian packages krb5-clients and krb5-rsh-server) is a drop-in replacement for the traditionnal rsh, adding strong authentication and encryption to it. However, it appears that it cannot be used to start remote slave Erlang nodes, because when the Erlang VM process detaches from the kshd server process after being started, the kshd process kills it and the rsh-like command terminates without any error message. I have found no way to remotely start a Unix command and put it in the background, using MIT kerberized rsh.

MIT kerberized rsh cannot therefore be used to start slave nodes.

Kerberized ssh

A ssh-krb5 is available is Debian, that is an heavily patched 3.8 version of OpenSSH that implements Kerberos authentication. The patched version is not the latest version of OpenSSH, and apparently adapting the patches to the latest version will be a lot of work according to the author. I therefore consider that this solution is not reliable in the long term, and therefore I have not tested it.

OpenSSH 3.9 now supports Kerberos authentication. However, the Debian packages openssh-client and openssh-server have Kerberos authentication support disabled, so one would need to rebuild them to enable GSSAPI authentication. Also, this package is incompatible with ssh-krb5, and the set of supported authentication mechanisms seems to be changing rapidly. I have therefore not tested it, although this package works well for starting Erlang slave nodes without Kerberos authentication, so it should also work with Kerberos authentication enabled.

Remote Kerberos-authenticated command execution (remctl)

remctl is a kind of replacement for rsh that provides Kerberos authentication. It also adds a way to restrict which commands can be run by every user, very much like a “configurable remote restricted shell”, or sudo. Version 1.9 of remctl (Debian package version 1.9-1) must be installed, since it corrects an important bug from the previous version and it makes configuration easier.

To install it, on the client and the server hosts:

sudo apt-get install remctl

On the server host, to make the remctld server automatically start through the inetd daemon, uncomment the line in the/etc/inetd.conf file that starts the /usr/sbin/remctld command.

On the server host, make sure that a host/server_host_name principal has been created in kerberos (where server_host_name is the server host name, including domain name), and the key has been copied into the/etc/krb5.keytab Kerberos private key file: (how to configure Kerberos and especially how create Kerberos principals and manipulate keytab files using kadmin is out of the scope of this post)

sudo klist -k /etc/krb5.keytab

On the server host, the /etc/remctl/remctl.conf configuration file defines the services that can be executed through the remctld daemon. According to remctld(8), every line in this file has the following fields, separated by spaces, in that order:

  • type: a category of services, that can be freely specified.
  • service: the service that is requested, that can be freely specified.
  • executable: the absolute path to the command to run for performing the service.
  • logmask: an option to mask certain actual parameters to invoked services (e.g. passwords), so that they do not appear in the logs. We do not use such options in our case.
  • aclfile: the full paths to one or more files that contain the list of Kerberos principals of users that can execute the service.

The default /etc/remctl/remctl.conf file provided in the Debian packages includes all files in the /etc/remctl/conf.d/directory that have no period in their name (this is a new feature of remctl version 1.9). Those files have the same format as remctl.conf. In our case, we create a /etc/remctl/conf.d/erlang file that contains one line to define the erlang category of services, and only the startslave service:

cat <<EOF >/tmp/erlang
erlang startslave /etc/remctl/conf.d/ /etc/remctl/acl/lenglet
chmod 644 /tmp/erlang
sudo mv /tmp/erlang /etc/remctl/conf.d/erlang
sudo chown root:root /etc/remctl/conf.d/erlang

Since the remctld daemon passes the invoked service name as the first parameter to the executable, the /usr/bin/erl command cannot be directly specified as an executable in the /etc/remctl.conf configuration file. Instead, on the server host, create a /etc/remctl/conf.d/ shell script to run the erl command properly:


if [ $# -lt 1 ]; then
    echo "This script should only be started by remctld"
    exit 1


case "${SERVICE}" in
        exec /usr/bin/erl $*
        echo "Invalid service"
        exit 1

Set the rights of this file:

sudo chmod 755 /etc/remctl/conf.d/
sudo chown root:root /etc/remctl/conf.d/

On the server host, create the ACL file for the erlang category of services. A remctld ACL file is a text file which every line is a Kerberos principal of a user that is allowed to execute a remctl service. For instance, my ACL file contains the Kerberos principals for user lenglet in Kerberos realm CSG.IS.TITECH.AC.JP:

cat <<EOF >/tmp/lenglet
chmod 644 /tmp/lenglet
sudo mv /tmp/lenglet /etc/remctl/acl/lenglet
sudo chown root:root /etc/remctl/acl/lenglet

On the client host, the remctl command can now be used to run services remotely through the remctld server. However, the syntax of the remctl command is different from that of rsh. The remctl command takes the following arguments, in that order:

  • hostname: the name of the server host that runs the remctld server.
  • type: the category of services, in our case it is erlang.
  • service: the service to execute, in our case it is startslave.
  • additional parameters: parameters that are passed to the executable run by the remctld server for the service, after the service name parameter.

On the client host, it is therefore necessary to write a shell script that has a syntax compatible with rsh, to run remctl:


if [ $# -lt 2 ]; then
    echo "Invalid number of parameters"
    exit 1


if [ "${COMMAND}" != "erl" ]; then
    echo "Invalid command"
    exit 1

shift 2

exec /usr/bin/remctl "${REMOTEHOST}" erlang startslave $*

Set the rights of this file:

sudo chmod 755

On the client host, before running erl, check that you have a cached valid Kerberos Ticket Granting Ticket:


The shell script must be accessible from the PATH on the client host. It can then be passed as the rsh-like command to erl, to start remote slave nodes, e.g.:

erl -name yourpreferredname -rsh

To test if the creation of slave nodes works, you can use my example slavetest module (downloadable source code file: slavetest.erl). For instance, in the client host Erlang shell instantiated above:

Hostname = "".