Kerberized remote start of Erlang/OTP slave nodes

This document describes how to configure remote Kerberos-authenticated remote command execution to start remote slave Erlang nodes, on Debian GNU/Linux.

Introduction

Starting slave nodes requires to use a rsh-compatible command, which name is specified as an argument to the erl command. For instance, ssh is commonly used to start remote slave nodes:

erl -rsh ssh

When starting a slave node, using the functions of the standard slave module, that specified rsh-like command (for instance, ssh) is executed to start an Erlang VM Unix process on the remote host. That Unix process detaches itself from the rsh-like server process, to run in the background, and the rsh connection is terminated. Then, the slave node exclusively communicates using standard Erlang messages with the master node and the other nodes.

There are several ways to execute commands on a remote host, using Kerberos authentication:

  • remote shells commands: MIT kerberized telnet, MIT kerberized rlogin.
  • rsh-like commands: MIT kerberized rsh, kerberized OpenSSH, remctl.

In the following, I evaluate the several alternatives to rsh and ssh, that are based on Kerberos authentication. The advantage of Kerberos-based authentication vs. ssh key-pair-based authentication, in this case, is that it does not require to copy public keys on every node on which one wants to start slave nodes (like when using ssh): authentication is centralized. In addition, Kerberos is a single-sign-on (SSO) method for authentication, i.e. if your client system is well configured (e.g. using the libpam-krb5 package), there is no need for a user to enter any password in order to start slave nodes. Using ssh, it is necessary to use a ssh-agent, and to call ssh-add at least once for every client login session.

Currently, I have successfully used only remctl (see below).

MIT kerberized rsh

MIT kerberized rsh (Debian packages krb5-clients and krb5-rsh-server) is a drop-in replacement for the traditionnal rsh, adding strong authentication and encryption to it. However, it appears that it cannot be used to start remote slave Erlang nodes, because when the Erlang VM process detaches from the kshd server process after being started, the kshd process kills it and the rsh-like command terminates without any error message. I have found no way to remotely start a Unix command and put it in the background, using MIT kerberized rsh.

MIT kerberized rsh cannot therefore be used to start slave nodes.

Kerberized ssh

A ssh-krb5 is available is Debian, that is an heavily patched 3.8 version of OpenSSH, to implement Kerberos authentication. The patched version is not the latest version of OpenSSH (3.9), and apparently adapting the patches to the latest version will be a lot of work according to the author. I therefore consider that this solution is not reliable in the long term, and therefore I have not tested it.

OpenSSH 3.9 now supports Kerberos authentication. However, the Debian packages openssh-client and openssh-server have Kerberos authentication support disabled, so one would need to rebuild them to enable GSSAPI authentication. Also, this package is incompatible with ssh-krb5, and the set of supported authentication mechanisms seems to be changing rapidly. I have therefore not tested it, although this package works well for starting Erlang slave nodes without Kerberos authentication, so it should also work with Kerberos authentication enabled.

Remote Kerberos-authenticated command execution (remctl)

The remctl is a kind of replacement for rsh, with Kerberos authentication. It additionnally adds a way to restrict which commands can be run by every user, very much like a "configurable remote restricted shell", or sudo. Version 1.9 of remctl (Debian package version 1.9-1) must be installed, since it corrects an important bug from the previous version and it makes configuration easier.

To install it, on the client and the server hosts:

sudo apt-get install remctl

On the server host, to make the remctld server automatically start through the inetd daemon, uncomment the line in the /etc/inetd.conf file that starts the /usr/sbin/remctld command.

On the server host, make sure that a host/server_host_name principal has been created in kerberos (where server_host_name is the server host name, including domain name), and the key has been copied into the /etc/krb5.keytab Kerberos private key file: (how to configure Kerberos and especially how create Kerberos principals and manipulate keytab files using kadmin is out of the scope of this document)

sudo klist -k /etc/krb5.keytab

On the server host, the /etc/remctl/remctl.conf configuration file defines the services that can be executed through the remctld daemon. According to remctld(8), every line in this file has the following fields, separated by spaces, in that order:

  • type: a category of services, that can be freely specified.
  • service: the service that is requested, that can be freely specified.
  • executable: the absolute path to the command to run for performing the service.
  • logmask: an option to mask certain actual parameters to invoked services (e.g. passwords), so that they do not appear in the logs. We do not use such options in our case.
  • aclfile: the full paths to one or more files that contain the list of Kerberos principals of users that can execute the service.

The default /etc/remctl/remctl.conf file provided in the Debian packages includes all files in the /etc/remctl/conf.d/ directory that have no period in their name (this is a new feature of remctl version 1.9). Those files have the same format as remctl.conf. In our case, we create a /etc/remctl/conf.d/erlang file that contains one line to define the erlang category of services, and only the startslave service:

cat <<EOF >/tmp/erlang
erlang startslave /etc/remctl/conf.d/erlang.sh /etc/remctl/acl/lenglet
EOF
chmod 644 /tmp/erlang
sudo mv /tmp/erlang /etc/remctl/conf.d/erlang
sudo chown root:root /etc/remctl/conf.d/erlang

Since the remctld daemon passes the invoked service name as the first parameter to the executable, the /usr/bin/erl command cannot be directly specified as an executable in the /etc/remctl.conf configuration file. Instead, on the server host, create a /etc/remctl/conf.d/erlang.sh shell script to run the erl command properly:

#!/bin/sh

if [ $# -lt 1 ]; then
    echo "This script should only be started by remctld"
    exit 1
fi

SERVICE="$1"
shift

case "${SERVICE}" in
    startslave)
        exec /usr/bin/erl $*
        ;;
    *)
        echo "Invalid service"
        exit 1
        ;;
esac

Set the rights of this file:

sudo chmod 755 /etc/remctl/conf.d/erlang.sh
sudo chown root:root /etc/remctl/conf.d/erlang.sh

On the server host, create the ACL file for the erlang category of services. A remctld ACL file is a text file which every line is a Kerberos principal of a user that is allowed to execute a remctl service. For instance, my ACL file contains the Kerberos principals for user lenglet in Kerberos realm CSG.IS.TITECH.AC.JP:

cat <<EOF >/tmp/lenglet
lenglet@CSG.IS.TITECH.AC.JP
lenglet/admin@CSG.IS.TITECH.AC.JP
EOF
chmod 644 /tmp/lenglet
sudo mv /tmp/lenglet /etc/remctl/acl/lenglet
sudo chown root:root /etc/remctl/acl/lenglet

On the client host, the remctl command can now be used to run services remotely through the remctld server. However, the syntax of the remctl command is different from that of rsh. The remctl command takes the following arguments, in that order:

  • hostname: the name of the server host that runs the remctld server.
  • type: the category of services, in our case it is erlang.
  • service: the service to execute, in our case it is startslave.
  • additional parameters: parameters that are passed to the executable run by the remctld server for the service, after the service name parameter.

On the client host, it is therefore necessary to write a erlang_startslave_remctlrsh.sh shell script that has a syntax compatible with rsh, to run remctl:

#!/bin/sh

if [ $# -lt 2 ]; then
    echo "Invalid number of parameters"
    exit 1
fi

REMOTEHOST="$1"
COMMAND="$2"

if [ "${COMMAND}" != "erl" ]; then
    echo "Invalid command"
    exit 1
fi

shift 2

exec /usr/bin/remctl "${REMOTEHOST}" erlang startslave $*

Set the rights of this file:

sudo chmod 755 erlang_startslave_remctlrsh.sh

On the client host, before running erl, check that you have a cached valid Kerberos Ticket Granting Ticket:

klist

The erlang_startslave_remctlrsh.sh shell script must be accessible from the PATH on the client host. It can then be passed as the rsh-like command to erl, to start remote slave nodes, e.g.:

erl -name yourpreferredname -rsh erlang_startslave_remctlrsh.sh

To test if the creation of slave nodes works, you can use my example slavetest module (downloadable source code file: slavetest.erl). For instance, in the client host Erlang shell instantiated above:

Hostname = "put.here.your.server.hostname".
c(slavetest).
slavetest:start(Hostname).

History of this document

  • 2005-11-08: changed the license of this document to the GNU Free Documentation License
  • 2005-08-09: modified
  • 2005-05-15: updated to refer to remctl version 1.9
  • 2005-04-20: first version of this document, that refers to remctl version 1.7

Copyright

Copyright © 2005 Romain Lenglet.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free S oftware Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is available from the GNU project web site at: http:// www.gnu.org/copyleft/fdl.html.