Installation of Xenomai real-time Linux

This post describes how to compile, package and install the Xenomai real time Linux extension (previously called RTAI/Fusion) version 2.0 on the Debian GNU/Linux sid distribution on an IA-32 computer.

Xenomai is the continuation of RTAI‘s previous development branch called Fusion, that has been split from RTAI on 2005-10-08. Fusion / Xenomai is now independent from RTAI, hence this article does not address the installation of any RTAI branch (e.g. Magma), although the installation processes should be very similar. For more information about the two projects, see:

Apply the kernel patch for the Adeos real-time microkernel

Xenomai relies on the Adeos real-time microkernel, to handle real time interruption dispatching. Adeos is implemented as a patch to the Linux kernel.

Package the Adeos patch

In Debian GNU/Linux, the Adeos patch is normally provided in package kernel-patch-adeos, but this package has not been updated for Linux kernels versions more recent than 2.6.11 (as of 2005-11-01) while the current version of Linux in Debian (and the one that I is considered in this article) is 2.6.14. This can be checked by looking for the line that defines the KVERSIONS variable in the /usr/src/kernel-patches/i386/apply/adeos script file that is installed by the kernel-patch-adeos package. It is therefore necessary to download the patches directly from the Adeos website, at http://download.gna.org/adeos/, and to build a new Debian GNU/Linux package for it.

Download the Adeos patch for Linux kernel version 2.6.14, that is now named I-pipe, using the following commands:

> mkdir -p /tmp/rtai/adeos
> cd /tmp/rtai/adeos
> wget http://download.gna.org/adeos/patches/v2.6/adeos/i386/adeos-ipipe-2.6.14-i386-1.0-09.patch

Get the sources of the current Debian GNU/Linux kernel-patch-adeos package (version 20050809-1 as of 2005-11-01):

> mkdir -p /tmp/rtai/adeos/srcpackage
> cd /tmp/rtai/adeos/srcpackage
> apt-get source adeos
> cd adeos-20050809

Add the downloaded patch into the source package, which is very easy because this package uses Debian’s kpatches helper tools for building the package:

> cp ../../adeos-ipipe-2.6.14-i386-1.0-09.patch .
> cat >> debian/kernel-patch-adeos.kpatches <<EOF
Patch-file: adeos-ipipe-2.6.14-i386-1.0-09.patch
Kernel-version: 2.6.14
Architecture: i386
Path-strip-level: 1

EOF

Change the package version number (you can put the extension text you like after the revision number -1, but there are a few restrictions as specified in the Debian Policy):

> dch --newversion=20050809-1.rlenglet

When the dch command makes you edit the changelog file, add a comment line like:

 * added patch for kernel version 2.6.14 from upstream)

Install packages required for building this package:

> sudo apt-get build-dep kernel-patch-adeos

Build the package:

> debuild

Install the newly generated package:

> cd ..
> sudo dpkg -i kernel-patch-adeos_20050809-1.rlenglet_i386.deb

Now, check that the installed package is OK. The command:

> grep 2.6.14 /usr/src/kernel-patches/i386/apply/adeos

should output the two lines:

KVERSIONS=(blahblahblah 2.6.14)
PATCHFILES=(blahblahblah/adeos-ipipe-2.6.14-i386-1.0-09.patch.gz")

Check that the patch file has been correctly installed by the package:

> ls /usr/src/kernel-patches/diffs/adeos/adeos-ipipe-2.6.14-i386-1.0-09.patch.gz

Patch, compile and package the Linux kernel

It is possible to use either Debian’s patched Linux kernel sources, or the vanilla Linux kernel sources. Since Debian’s patched kernel works fine with Xenomai and it is more natural to use it on Debian, we will use it in the following instead of the vanilla sources. Install Debian’s patched Linux kernel sources:

> sudo apt-get install linux-source-2.6.14
> cd /usr/src
> sudo tar xjf linux-source-2.6.14.tar.bz2
> sudo ln -s linux-source-2.6.14 linux
> cd linux-source-2.6.14

If you are already running a 2.6.14 version of the Linux kernel from an official Debian binary package, e.g. package linux-image-2.6.14-1-686, you should use the kernel configuration from that kernel (that activates everything as modules, etc. as always in Debian) as the initial configuration file for the Xenomai-enhanced kernel:

> sudo cp /boot/config-2.6.14-1-686 ./.config

Otherwise, edit a new initial configuration (although it may be difficult to create such a hand-configured Linux kernel work with Debian the first time: it must have initrds enabled, etc.):

> sudo make menuconfig

To build the kernel, any GCC compiler available in Debian’s sid distribution is OK: either 3.3, 3.4 or 4.x versions have been tested to work fine (and all Debian kernel images are built using GCC 4.0 without trouble). Build the kernel image package with the Adeos patch applied, using Debian’s kernel-package commands:

> sudo apt-get install kernel-package
> sudo make-kpkg --added-patches adeos \
    --config menuconfig --initrd \
    --append-to-version -1-686-ipipe --stem linux \
    kernel_image

When this command makes you edit the Linux kernel configuration, it is necessary to activate I-pipe and to deactivate some kernel features (essentially some power management features) that would ruin the latencies in the real-time kernel:

  • In Processor type and features:
    • Check that Interrupt pipeline is enabled (this is to enable the Adeos implementation – I-pipe).
    • Check that Local APIC support on uniprocessors and IO-APIC support on uniprocessors are enabled.
  • In Power management options (ACPI, APM):
    • In ACPI (Advanced Configuration and Power Interface) Support:
      • Disable Processor.
    • In APM (Advanced Power Management) BIOS Support:
      • Disable APM (Advanced Power Management) BIOS support.
    • In CPU Frequency scaling:
      • Disable CPU Frequency scaling.

Once the package has been created, install it:

> cd ..
> sudo dpkg -i linux-image-2.6.14-1-686-ipipe_10.00.Custom_i386.deb

If you use either LILO or Grub for bootloading, you need not do anything to make this kernel selectable at boot time: this is done automatically thanks to the Debian GNU/Linux kernel packaging mechanisms. Otherwise, you need to configure your bootloader properly. Reboot your computer, and boot with the new kernel.

Check that Adeos is working. The command:

> dmesg | grep -i I-pipe

should output the line:

I-pipe 1.0-09: pipeline enabled.

And the command:

> cat /proc/ipipe/Linux

should show that Linux has been registered into the Adeos/I-pipe interrupt pipeline. If the /proc/ipipe directory does not exist, Adeos/I-pipe has not been integrated and enabled into the currently running kernel: you must check that you have performed all the steps above.

Install Xenomai

Packages for RTAI already exist in Debian GNU/Linux sid (rtai, rtai-doc and rtai-source), but there is no package yet for Xenomai (or for RTAI’s previous Fusion branch it is the continuation of). Until a package is created for Xenomai, the following will guide you to install it “by hand”, not in a package-managed way.

Download and configure

Download the latest version of the Xenomai sources (as of 2005-11-01, the latest and first official release is 2.0):

> mkdir -p /tmp/xenomai
> cd /tmp/xenomai
> wget http://download.gna.org/xenomai/stable/xenomai-2.0.tar.bz2
> tar xjf xenomai-2.0.tar.bz2
> cd xenomai-2.0

Configure Xenomai:

> make menuconfig

and in the configuration application:

  • In General:
    • Set the Installation directory to /usr/lib/xenomai.
    • Set the Linux build tree to /usr/src/linux.
    • Enable Enable expert configuration mode.
  • In Nucleus:
    • Optionally, if needed by your applications, enable Interrupt shield support.
    • Enable Watchdog support (this is sometimes useful).
    • In Scalability:
      • Optionally, if your applications create many RT threads, enable O(1) scheduler.
  • In Machine (x86):
    • Only if you use NPTL threads (see below), enable Enable SEP instructions for syscalls to improve latency.
    • In SMI workaround:
      • Only if your system uses SMI, AND you encouter high maximum latencies (see below), enable Enable SMI workaround to improve latency.
  • In APIs, enable all options.
  • In Drivers, enable all options.

NPTL setup

To increase performance, you may enable the Enable SEP instructions for syscalls option in the configuration application, ONLY IF two conditions are met: your CPU supports the SEP instruction, AND your system has NPTL enabled and your RTAI applications use it.

To check if your CPU supports the SEP instruction, the command:

> cat /proc/cpuinfo | grep '^flags.*sep'

must output one line that contains the sep keyword if it supports it.

To make applications use the NPTL threads library instead of the older LinuxThreads library, you must run a Linux kernel of the 2.6 branch (which is the case if you run the Adeos-patched 2.6.14 kernel installed as described above), AND you must install NPTL-enabled C libraries:

> sudo apt-get install libc6-i686

I have tested that this works in practice.

Compile and install

Compile Xenomai:

> make all

Install Xenomai:

> sudo make install-nodev

You should run Udev to automatically manage Xenomai’s device files in /dev. However, if Xenomai is installed with theinstall target instead of install-nodev, Xenomai’s Udev rules are appended to to the /etc/udev/udev.rules file, which is not the “Debian way of doing”. In Debian, Udev rules should instead be added by creating new files in the/etc/udev/rules.d/ directory. Therefore, to correctly install Xenomai’s Udev configuration files you must instead run:

> sudo install --mode 644 nucleus/udev/*.rules /etc/udev/rules.d/

Alternatively, only if you don’t run Udev, you can create static device files by hand in /dev:

> for n in `seq 0 31`; do f="/dev/rtp$n"; sudo mknod -m 666 $f c 150 $n; done
> sudo mknod -m 666 /dev/rtheap c 10 254

Xenomai requires to have local APIC enabled for the CPUs by the kernel, which is always the case for SMP systems, but is not always the case by default for single-CPU systems. For instance, local APIC is not automatically enabled on my (single-CPU) IBM ThinkPad X31 notebook. To force the enabling of the local APIC at kernel startup, the lapic parameter must be passed to the kernel at boot time.

For instance, using Grub as a bootloader, only the lines starting with # kopt in the /boot/grub/menu.lst file must be modified, to append the lapic keyword (preceded by a space). For instance, on my system, I have modified the line:

# kopt=root=/dev/hda4 ro

to become:

# kopt=root=/dev/hda4 ro lapic

Then, update-grub must be called to update the boot loader:

> sudo /sbin/update-grub

Then, you must reboot to activate this option.

To solve the paths to Xenomai’s libraries, modify LD’s configuration:

> sudo echo "/usr/lib/xenomai/lib" >> /etc/ld.so.conf
> sudo ldconfig

Test Xenomai

Load the Xenomai modules into the kernel:

> sudo insmod /usr/lib/xenomai/modules/xeno_hal.ko
> sudo insmod /usr/lib/xenomai/modules/xeno_nucleus.ko
> sudo insmod /usr/lib/xenomai/modules/xeno_native.ko

Those modules print kernel logs when starting. The command:

> dmesg

should at least output the following three lines if nothing went wrong:

I-pipe: Domain Xenomai registered.
Xenomai: hal/x86 loaded.
Xenomai: real-time nucleus v2.0 (Surfing With The Alien) loaded.
Xenomai: starting native API services.

To measure the real-time interrupt latencies of your setup of Linux / Adeos and Xenomai, Xenomai provides the xeno-testcommand:

> /usr/lib/xenomai/bin/xeno-test

See the man page for options:

> man /usr/lib/xenomai/man/man1/xeno-test.1

This application uses sudo(8) to execute latency test real-time applications. You must therefore properly configure sudo for your user, to let it execute Xenomai’s test applications as root.

If xeno-test execute without problem, any Xenomai application should execute fine also.

SMI workaround

If when loading the xeno_hal.ko module, you get the following line in dmesg‘s output:

Xenomai: Intel chipset found and SMI workaround not enabled,
       you may encounter high interrupt latencies.

your computer has SMI enabled, which could significantly increase latency: SMIs are non-maskable interrupts that can happen even if a real-time interrupt handler is being executed, hence increasing its execution time / latency. Xenomai provides a workaround to disable SMI interrupts, which must be enabled at configuration time: enable Enable SMI workaround in the Machine (x86) / SMI workaround configuration section.

If some devices do not work anymore when loading Xenomai modules with the SMI workaround enabled, it probably means that those devices rely on SMI. You should try to re-enable some SMI interrupts in Xenomai’s SMI workaround configuration, one by one. In the configuration application:

  • In Machine (x86):
    • In SMI workaround:
      • Disable Globally disable SMI.
      • Try to enable Enable Intel-Specific USB2 SMI (disable if it has no effect).
      • Try to enable Enable legacy USB2 SMI (disable if it has no effect).
      • Try to enable Enable periodic SMI (disable if it has no effect).
      • Try to enable Enable TCO SMI (disable if it has no effect).
      • Try to enable Enable microcontroller SMI (disable if it has no effect).
      • Try to enable Enable APM SMI (disable if it has no effect).
      • Try to enable Enable legacy USB SMI (disable if it has no effect).
      • Try to enable Enable ACPI BIOS SMI (disable if it has no effect).