debian-lts news sig-io

Sig-I/O now 9 years old, debian-lts sponsor for 3

This week marks the 9th year in Sig-I/O's existance. Looking back over the past nine years shows a nice growth in the number of clients and a collection of interesting assignments Sig-I/O has been involved in.

While the services that Sig-I/O provides have shifted somewhat over these 9 years, some have also remained the same, with some clients already with us from the very beginning. The past few years have been mostly about Managed Hosting, Linux-, Ansible- and Security-Consulting and since a few years also Training, via a partnership with IT-Gilde which has been going steady for 3 years now.


This month also marks the 3rd complete year that Sig-I/O has been a sponsor of the Debian LTS project. This sponsorship has also been renewed for the coming year. The Debian LTS project could use some more sponsors, so if your organisation uses Debian servers extensively, it might be beneficial to sponsor them. Debian LTS will support Debian 7 until May of 2019, and will then continue with LTS support for Debian 8 into the 2020's

The next few years will undoubtedly bring many more interesting challenges and opportunities.

migration nikola site static

Migrated website to a static site powered by Nikola

Migrated the website

As you might have noticed if you visited my site before, the entire look and feel has been changed. The site is now powered by the Nikola static-site-generator. The most-recent articles have been migated over, the older articles from the previous website will be restored when they are still relevant.

I had been thinking about using a Static-Site-Generator before, but wordpress was working quite well for me. Recently however I managed to lose my wordpress database, and this privided me with a good opportunity to re-do the site using Nikola.

I'm still getting the hang of writing reStucturedText, and still need to update some pages, but at least the website is back from the abyss. Most articles have been restored from the WayBack-Machine operated by

Using Nikola

In case you are interested in using a static-site generator, this is my workflow:

  • Stream some nice relaxing music
  • Install python3, setup a virtualenv for Nikola
  • Pip install "Nikola[Extras]"
  • nikola init mywebsite
  • Version the newly created site in git
  • Make some changes to the config-file
  • Choose and download a theme
  • Write some posts and pages (nikola new_post -e)
  • Git add all your changes, push to a remote server
  • Nikola build
  • Rsync the output directory to a webserver
code hex3 lede openwrt rb750gr3

OpenWRT/LEDE On a Routerboard RB750Gr3 (Hex3)

The Routerboard RB750Gr3 (aka Hex3) is a nice and very affordable (~$60) hardware platform based on the MediaTek MT7621AT. It features gigabit ethernet ports, and a relatively fast multi-core CPU. However, out of the box it runs RouterOS. While this is a feature-full platform, I found configuration difficult and not very pleasant. Hoping that a build of OpenWRT/LEDE would soon become available, I bought an RB750Gr3 a couple of months ago.

A build became available, but it couldn’t be flashed on the RB750Gr3 without external hardware like a bus-pirate. But this has recently changed, and it’s now possible to TFTP boot a LEDE runtime, which allows access to write to the flash/mtd. Using this method it’s now possible to install OpenWRT/LEDE and upgrade uboot on the device without external hardware.

The OpenWRT site documents this procedure nicely, however, it’s still quite a lot of work to build all the required images and files. So I’ve deciced to host the images that I’ve created here, so other people can skip this rather tedious step of building the image.

The files can be found on

Please update factory.bin with your own MAC address at offset E000. Currently it’s set to 64:D1:54:AA:BB:CC

  • You can then netboot/tftpboot using the file ‘vmlinux-initramfs.elf’ (boot the RB750 and hold RESET until the leds stop flashing)
  • Wait for the system to come up on
  • Login as root
  • Copy the factory.bin uboot.bin uboot-env.bin and lede-17.01.2-ramips-mt7621-rb750gr3-squashfs-sysupgrade.bin to the system (in /tmp for example)
  • mtd write (without reboot) the 4 files to the relevant partitions (cat /proc/mtd to see the names/devices)
icinga nagios smart ssd

Monitoring S.M.A.R.T. attributes in Nagios/Icinga

One of my customers is using various (Samsung) SSD’s in their servers, and the first of these have started reaching their end-of-life. SSD’s have a somewhat different failure scenario then spinning metal disks, so monitoring their life-expectancy can be interesting.

Besides just logging and graphing the SMART attributes, it is also handy to have some alerting on when certain thresholds are crossed. To do this, I’ve written a simple nagios/icinga script which will alert on interesting SMART attributes, and will also calculate the percentage of total guaranteed writes on the SSD’s. Since the guaranteed TBW value will differ between various SSD vendors and product-ranges, this value needs to be specified on the command-line by the user.


I’ve integrated this check-script into my normal monitoring-scripts, but it can off-course also be used as a stand-alone tool. If has options to specify the device to smartctl, so disks behind raid-controllers can also be monitored.

The script can be found in my sysadmin repository on github: check_ssd_attribs

cooperatie itgilde news sig-io

Sig-I/O joins up with ITGilde Coöperatie

Sig-I/O joins forces with IT-Gilde Coöperatie

Today, september 2nd 2015 Sig-I/O Automatisering has joined the ITGilde Coöperatie


By joining ITGilde Coöperatie the customers of Sig-I/O Automatisering can make use of the knowledge and skills of a large (~50) group of highly skilled freelance Unix/Linux specialists.

The following (dutch) article was posted on the ITGilde website:

Vandaag verwelkomt ITGilde Mark Janssen als 49ste lid van ITGilde. Naast
ITGilde is Mark actief bij de NLUUG en RevSpace . Het motto van Mark
is: “over 5 minuten is het opgelost”.

De reden voor Mark om aan te sluiten bij ITGilde is een kwalitatief
hoogwaardig netwerk met een hoge frequentie van vakgerelateerde
bijeenkomsten; gericht op Linux/Unix ondernemers.

Daarnaast werkt Mark meer dan fulltime en wil toch meer klanten kunnen
helpen dan hij nu aankan. ITGilde is daar de oplossing voor: “ik kan
mezelf maar 1 keer inzetten maar met ITGilde beschik ik nu over een pool
van goede technische Unix/Linux specialisten!”

Welkom Mark!
ansible pass passwordstore

Read passwords from the 'pass' passwordstore into ansible

Ansible is a great orchestration-tool, and while it has it’s own secure password storage system (Vault), I prefer to use ‘pass’ from

There is no specific ‘pass’ plugin for ansible, but using the ‘pipe’ lookup plugin works quite well

  - name: Debug
      debug: msg={{lookup('pipe', 'pass some/password/i/need') }}

This can then be used to store API keys, passwords for various accounts, or other information that needs to remain secret while still being able to share and version your ansible plays.

Update 2017: There is now a passwordstore lookup-plugin for ansible

Tenshi log monitor now supports Redis inputs

Sig-I/O has been using Tenshi for quite a while, as it’s one of the easier and more flexible log monitoring tools available. It’s also quite light-weight and has only a few perl modules as requirements.

However, tenshi has been showing it’s age, as it only supported syslog, flat files or fifo’s as inputs. These days json based logging with graylog2, logstash or other tools seems to be all the rage.

Since we needed to setup a new log monitoring solution for a customer and they didn’t have a central syslog server, but were using logstash and redis, it was a perfect time to add Redis support to Tenshi.

Patching a Redis input to Tenshi turned out to be quite easy using the perl Redis module. The patch has been sent to the upstream developer and will most likely be included into a next release.

For those who can’t wait, and want to try out the Redis support, the code can be found at my github repository in the ‘redis’ branch of tenshi

centos php rhel

Installing php 5.4 or 5.5 on CentOS 6.x / RHEL 6.x / SL 6.x

There are many posts on the internet about people wanting to install a newer PHP release on their EL6 boxes. Most of these posts will tell you to either install the ‘remi’ repository, or packages from ‘webtatic’. However, there is a newer, and in my opinion better, method now. Software Collections

Redhat has created the concept of software collections, in which they can provide newer or additional packages to the base OS. These packages come with a more limited support package, but they are at least a somewhat standardised way of installing additional functionality without impacting the base OS. Red Hat Enterprise Linux 6

In RHEL systems, collections can be enabled with:

  • Enable the redhat collections channel
    • rhn-channel –add –channel=rhel-x86_64-server-6-rhscl-1
  • Then install software from it:
    • yum install php54-php

More info can be found on CentOS 6 / SL 6 / OEL 6

For the community EL6 systems, the following procedure can be used:

The list of available collections and their package-url’s can be found on

centos centos7 changes rhel

Testing Red Hat Enterprise Linux 7 and CentOS 7 (preview)

Last week, 'Red Hat'_ released the final version of Red Hat Enterprise Linux version 7. A few days later the CentOS project made a first preview version of CentOS 7 available. Since many of our customers are running on RHEL 6 and/or CentOS 6, now was a good time to look into the newly release 7.0 version.

Both the CentOS-7 and RHEL-7 installations completed without any problems, something that was still giving more then enough issues during the beta and release-candidate stages of RHEL-7. We tested the ‘default’ graphical install, the text-based install and kickstart installs in both graphical and text-modes. Currently we’re fine-tuning our kickstart configuration for the 7 releases, so installs can be fully automated and fast. Kickstart Configuration

At this time, our kickstart looks somewhat like this (censored to protect sensitive data):

# System authorization information
auth --enableshadow --passalgo=sha512

# Use network installation
url --url=""
# Use text mode install
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8

# Network information
network  --bootproto=dhcp --device=eth0 --ipv6=auto --activate
network  --hostname=centos7previewkickstarttest
# Root password
rootpw some-password
# Do not configure the X Window System
# System timezone
timezone Europe/Amsterdam --isUtc
#user --groups=wheel --name=useraccount --password=some-password --gecos="User"
# Skip EULA
eula --agreed
# Disable firewall
firewall --disabled
# Don't run the Setup Agent on first boot
firstboot --disabled
# Selinux (ENFORCING|permissive|disabled)
selinux --enforcing
# Reboot the machine when the installation is finished, eject CD
reboot --eject
# Enable SSH services
services --enabled sshd
# Include auto-generated disk-config
%include /tmp/

# Core and base are default, just specify them anyway to make this clear
# Then unselect the 'default' marked packages from core and base, which we don't need
# default from core
# default from base


# Check physical and virtio disks
for disk in /sys/block/sd* /sys/block/vd*
        dsk=$(basename $disk)

        if [[ `cat $disk/ro` -eq 1 ]];
                echo "Skipping disk $dsk: READONLY"

        if [[ `cat $disk/removable` -eq 1 ]];
                echo "Skipping disk $dsk: REMOVABLE"

        if [[ `cat $disk/size` -lt 20971520 ]];
                echo "Skipping disk $dsk: Smaller then 10G"
                echo "Using disk $dsk"

> $incfile

if [[ -n $chosen ]];
        echo "zerombr" >> $incfile
        echo "bootloader --location=mbr --driveorder=$chosen --append=\"nomodeset console=tty0 console=ttyS0,115200n8\"" >> $incfile
        echo "ignoredisk --only-use=$chosen" >> $incfile
        echo "clearpart --all --initlabel --drives=$chosen" >> $incfile
        echo "part /boot --fstype=ext3 --asprimary --size=256" >> $incfile
        echo "part pv.$chosen --grow --size=15000" >> $incfile
        echo "volgroup vg00 --pesize=32768 pv.$chosen" >> $incfile
        echo "logvol / --fstype=xfs --name=root --vgname=vg00 --size=4096" >> $incfile
        echo "logvol swap --name=swap --vgname=vg00 --size=256" >> $incfile
        echo "" > $incfile


# PostInstall stuff
%post --log=/root/anaconda-postinstall.log
cd /
echo "GRUB_TERMINAL=\"serial console\"" >> /etc/default/grub
echo "GRUB_TERMINAL_OUTPUT=\"serial console\"" >> /etc/default/grub
echo "GRUB_SERIAL_COMMAND=\"serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1\"" >> /etc/default/grub

echo "#!/bin/sh" > /usr/local/sbin/update-grub
echo "grub2-mkconfig -o /boot/grub2/grub.cfg" >> /usr/local/sbin/update-grub
chmod +x /usr/local/sbin/update-grub


echo "[base-c7-preview]" > /etc/yum.repos.d/c7-preview.repo
echo "name=CentOS-7-Preview" >> /etc/yum.repos.d/c7-preview.repo
echo "baseurl=" >> /etc/yum.repos.d/c7-preview.repo
echo "enabled=1" >> /etc/yum.repos.d/c7-preview.repo
echo "gpgcheck=0" >> /etc/yum.repos.d/c7-preview.repo

This kickstart configuration will try to install a minimal CentOS 7 (preview) system, on the first available disk that is non-removable and bigger then 10GB. It will use LVM and create a 4GB root filesystem, leaving the remaining diskspace free for later use.

It will also configure grub to react on serial input and the vga console, and also configure the kernel and getty’s to work on both serial and vga consoles. It will configure the centos7-preview repository, so installing extra software should be easy. The current install is optimized for virtual machines, and doesn’t install anything related to sound, wifi, network-manager etc.

The entire install will end up being about 840MB, and includes all requirements for running ansible playbooks, so ansible can be used to further configure the system after initial installation. Most notable changes

The major changes that you will run into, if you are used to CentOS/RHEL 5 and 6 are:

  • Systemd is now used as init system, with all required changes that come with it
  • BTRFS and XFS are supported, and the system can use either as a root filesystem (xfs was available in centos/redhat 6, but not usable as root or boot filesystem)
  • On installs with X and a desktop, you will now get a Gnome 3 Classic based desktop. Luckily a tweak-tool is available and installed by default, which will allow you to tune the desktop somewhat.

Post heartbleed, looking at alternatives to OpenSSL

With the bug of the year (Heartbleed) patched on all my systems, I decided to look into alternative SSL implementations. I mostly use Lighttpd and Nginx, and try to stay away from Apache, unless I really need something that will only work with it.


Apache seems to have 2 possible SSL implementations available. The first one, mod_ssl, which is based on OpenSSL, is used by ~99% of the users. mod_nss is the second SSL implementation for Apache, based on the netscape/mozilla NSS library.

A quick google-search for alternative SSL implementations or modules for Nginx and Lighttpd returned no actual working code, only some requests for functionality, so it seems that this is (at this time) a dead end. Looking further on google and wikipedia I looked at the alternative SSL implementations, specifically axTLS, PolarSSL and MatrixSSL.


AxTLS was the first library I looked at, within the source distribution is a minimalistic webserver with SSL/TLS support and a tool called axtlswrap, a simple stunnel like wrapper for the AxTLS library. I quickly configured the minimalistic webserver (axhttpd) and gave it a temporary StartSSL certificate. The configuration was quite simple after configuring the library to actually use my provided certificate and key instead of the built-in certificate. I then decided to run the SSLLabs tests against this webserver, to see how it would compare against my regular Lighttpd/OpenSSL secured websites.

The library provided the minimal required features, but at this time doesn’t support TLS1.2, and had only limited support for the various popular ciphers. This would result in a usable webserver, but with limited options in ciphers and limited support for strong encryption. Since I didn’t feel like dropping back to a ‘B’ score in SSLLabs tests, AxTLS was not a sufficient solution.


PolarSSL is another open-source SSL library, which is available under the Gnu GPLv2 or a commercial license. It’s currently being used in various well-known products (PowerDNS and OpenVPN-NL, a dutch-government approved version of OpenVPN). PolarSSL seemed to support all the latest ciphers and TLS standards and there were at least 2 httpd servers that supported using PolarSSL.


Hiawatha was the first webserver software I tried with PolarSSL. The code for PolarSSL is actually included in the downloads from Hiawatha, so you don’t need to download this seperately. The hiawatha source-tree includes a script to download a newer version of PolarSSL when needed.

Hiawatha was also quite easy to configure, and I had it running with my certificate in a matter of minutes. Another test-run of the Qualys SSLLabs tests gave a very positive result. The only downside was a choice for a 1024 bit Diffie–Hellman key exchange. I struggled a bit to get this to 2048 or 4096 bits, but after a few failed attempts found a Hiawatha configuration option to set this (DHsize = 4096).

With this setting I got my beloved A score in Qualys SSLLabs. Support for TLS versions 1.0, 1.1 and 1.2, with support for SSLv2 and v3 completely disabled.

I didn’t do much further testing with Hiawatha, as I only configured it to do SSL/TLS and then reverse-proxy everything to a lighttpd server running plain http.


The monkey webserver is another http daemon with support for PolarSSL. It’s focussed and optimized for running only on Linux systems, and aims to have a good performance while also having all the standard and required features (ipv6, virtual-hosting, fastcgi)

To enable the SSL module with Monkey, it’s required to run ‘configure’ with the ‘–enable-plugins=polarssl’ option. This doesn’t seem to be documented in either configure or the INSTALL or README files. It’s also funny to see that compiling jemalloc.c takes about as much time as all the other files combined.

My first build of Monkey was against the systems PolarSSL, which resulted in an A- score in the SSLLabs tests. I spotted that the Debian polarssl package was still stuck at version 1.2.9, so I downloaded the latest PolarSSL and built that version (1.3.6) (make lib SHARED=1; make install)

When running with PolarSSL 1.3.6, the Monkey webserver also got an ‘A’ score in SSLLabs. The only ‘red’ is the session resumption, which for some reason isn’t working. This is still something I need to look into. Also, it appears that monkey can only run over a single transport, so either HTTP or HTTPS. This would mean running two seperate monkey-instances for http+https support. This might be something that will be fixed in a future monkey version.

For as long as my testing system is still running, you can check the live SSLLabs score on it. Please ignore the ‘axtls’ in the hostname, it’s currently running Monkey, but the certificate I created for testing was first used for testing axtls.

Feature-wise monkey seems like a nice alternative to Apache, Nginx and Lighttpd, and looking at the http performance, it also gives some very nice results. My testing with HTTPS however quickly triggered various bugs and crashes, so it seems that the PolarSSL support for Monkey is still somewhat buggy. I could succesfully run my benchmarks against the server in HTTP mode, at up to 18000 requests per second (on a dual core VM), but running with SSL enabled would crash the monkey process when running with more then a hand full of threads. The system would be stable for hours with only 2-6 threads, but when running with 10+ threads it would crash within seconds. I hope these bugs will be fixed soon, so Monkey with PolarSSL will prove itself to be a worthy competitor in the SSL serverspace.

Update 2014/04/22:

It seems my libpolarssl was compiled with some incorrect settings. I’ve reconfigured/recompiled polarssl and both monkey and hiawatha. The servers are now stable as expected, performance-data will be updated as well, as soon as my new tests have completed.

Basically: make sure you enable POLARSSL_THREADING_PTHREAD and disable POLARSSL_DEBUG_C