- - - By CrazyStat - - -

17. August 2016

Linux Software Raid: Enlarge Raid Array of Hetzner EX41

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 21:15

Recently, I ordered an EX41 server at Hetzner, which by default comes with two 4 TB HDDs in a software RAID1. I chose the minimal Debian Jessie Image. From Hetzner’s EX40, which has two 2 TB drives in a Software RAID1, I was used to find the whole drive formatted as one partition by Hetzner. But with the EX41, it turned out that the 4 TB were split: The root partition / was 2 TB of size and 1.7 TB were allocated to /home. As the application that should run on this server needs more than 2 TB space in one folder, I repartitioned the server. If somebody else has a similar issue, here is how it can be done:

  1. Edit /etc/fstab and remove or comment (by placing a # at the beginning) the line that mounts /home from /dev/md3
  2. Reboot into the Rescue system or some other system that is not on the same drive. (The Hetzner rescue system boots from network. On a local system, you would boot a linux from CD or USB.)
  3. Check the Raid status and setup:
    cat /proc/mdstat
    md3 : active raid1 sda4[0] sdb4[1]
          1777751872 blocks super 1.2 [2/2] [UU]
          bitmap: 0/14 pages [0KB], 65536KB chunk
    md2 : active raid1 sda3[0] sdb3[1]
          2111700992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/16 pages [0KB], 65536KB chunk
    md1 : active raid1 sda2[0] sdb2[1]
          523712 blocks super 1.2 [2/2] [UU]
    md0 : active raid1 sda1[0] sdb1[1]
          16760832 blocks super 1.2 [2/2] [UU]

    So here md3 is the raid array that we want to get rid of together with its partitions sda4 and sdb4. md2 the raid array that we want to grow in size.

  4. First let’s get rid of the md3:
    mdadm --stop /dev/md3
    mdadm --zero-superblock /dev/sda4
    mdadm --zero-superblock /dev/sdb4
    parted /dev/sda rm 4
    parted /dev/sdb rm 4
  5. Resize the partitions of md2 (sda3 and sdb3):
    (Adjust the end if necessary)

    parted /dev/sda
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
    parted /dev/sdb
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
  6. Let the raid array md2 grow:
    mdadm --grow /dev/md2 --size=max
  7. Check the raid status:
    cat /proc/mdstat
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          [===================>.]  resync = 97.8% (3807048128/3888815376) finish=13.3min speed=101952K/sec
          bitmap: 1/15 pages [4KB], 131072KB chunk
  8. Wait until the resync finished (took a few hours for 2 TB):
    cat /proc/mdstat                                                                                                                                        
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 131072KB chunk
  9. Check the filesystem:
    e2fsck -fv /dev/md2
  10. Resize the filesystem:
    resize2fs /dev/md2
  11. Reboot into your normal system and you are done:
  12. Check the free disk space:
    df -h

    It should give the new size of / (3.6 TB) and a lot more free space 🙂

Hope this helps somebody with the same or similar issue.



Try my Open Source PHP visitor analytics script CrazyStat.

15. July 2016

Request-Tracker4: Enabling fast fulltext search with Sphinxsearch on Debian Jessie

I just successfully made fulltext search in our request tracker super fast. This is a guide for Debian Jessie and request tracker set up to use MySQL (or MariaDB).

  • Don’t forget to backup your RT database:
    mysqldump -p -u root rtdb > rtdb.sql
  • Log out of request tracker. I had some strange issue because I did not 😉
  • Make sure you are using request-tracker4 version 4.2.12 from jessie-backports. If you don’t already, this is how you update:
    – add this to the end of your /etc/apt/sources.list:

    deb jessie-backports main

    – run

    apt-get update

    – install request-tracker4 from jessie-backports:

    apt-get -t jessie-backports install request-tracker4

    – follow the update procedure, make sure your database is upgraded
    (We need the package from backports as jessie’s rt4-db-mysql package depends on mysql-client, whereas the backports version depends on virtual-mysql-client and thus can also be easily used with MariaDB.)

  • Second, make sure you are using MariaDB instead of MySQL. MariaDB is a drop-in replacement for MySQL. The big advantage of MariaDB here is that it comes with the SphinxSE storage engine that we will use for the fulltext search. SphinxSE is also available for MySQL, but you would need to compile it yourself whereas MariaDB includes SphinxSE by default. MariaDB has also a lot more advantages. Switching to MariaDB is very simple, no need to convert databases or configuration. I used aptitude and selected to install mariadb-client and maridb-server. Then I checked the dependency resolutions and choose the option to uninstall mysql-client, mysql-server and related packages. Then everything went smooth automatically, I just needed to enter the root password again, everything else was updated automatically.
  • Now check if your RT is still working well with MariaDB.
  • Now enable the SphinxSE storage engine in MariaDB:
    mysql -u root -p
    [enter password]
    INSTALL SONAME 'ha_sphinx';
  • Now install the Sphinxsearch daemon:
    apt-get install sphinxsearch
  • Now set up indexed full text search in RT:
    rt-setup-fulltext-index --dba root --dba-password secret

    (Let me know if you find a way without passing the password as a parameter)
    This asks you what type of index you want to use. Enter “sphinx”. Then keep the defaults.

  • The command above gave you code for /etc/request-tracker4/ – like this:
    Set( %FullTextSearch,
        Enable     => 1,
        Indexed    => 1,
        MaxMatches => '10000',
        Table      => 'AttachmentsIndex',

    Add / adjust this in

  • Then you need to configure the index in sphinx. The above tool also gave you a config file for this, but it does not work with the current sphinxsearch version of jessie. So here is what works. Create an /etc/sphinxsearch/sphinx.conf like this:
    source rt {
        type            = mysql
        sql_host        = localhost
        sql_db          = rtdb
        sql_user        = rtuser
        sql_pass        = RT_PASSWORD
        sql_query_pre   = SET NAMES utf8
        sql_query       = \
            SELECT, t.Subject, a.content FROM Attachments a \
            JOIN Transactions txn ON a.TransactionId = AND txn.ObjectType = 'RT::Ticket' \
            JOIN Tickets t ON txn.ObjectId = \
            WHERE a.ContentType LIKE 'text/%' AND t.Status != 'deleted'
    #    sql_query_info  = SELECT * FROM Attachments WHERE id=$id
    index rt {
        source                  = rt
        path                    = /var/lib/sphinxsearch/data/rt
        docinfo                 = extern
    #    charset_type            = utf-8
    indexer {
        mem_limit               = 32M
    searchd {
        listen                  = 3312
        log                     = /var/log/sphinxsearch/searchd.log
        query_log               = /var/log/sphinxsearch/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinxsearch/
    #    max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 0
        unlink_old              = 1
        # For sphinx >= 1.10:
        binlog_path             = /var/lib/sphinxsearch/data

    Make sure to adjust the mysql database, user and password.
    Note that there are some differences to the config provided by rt-setup-fulltext-index. Especially, you need to use listen instead of port. If you don’t adjust this, sphinxsearch will start on a different port and RT will not find any results in fulltext searches! If your RT is not behind a firewall, better also set listen to . Also, I used the Debian style paths from the Debian sample. The fields commented out are not necessary anymore in current sphinxsearch.
    UPDATE 26.08.2016: I changed the SQL Query to also index the Subject of the tickets and index all ‘text/%’ types and not only ‘text/plain’. This change is based on a comment by Claes Merin in the Request Tracker Mailing List. Thanks a lot. Without this change, the fulltext search will only search within ticket content, but not within ticket titles… Also, before it would not index text/html attachments.

  • Now run the indexer to build the initial index. This will take some time, for a 6 GB Attachments Table, it took not more than 30 minutes. Maybe run this in a screen in case your ssh connection breaks:
    indexer rt
  • Enable the Sphinx daemon. To do so, adjust START to yes in /etc/default/sphinxsearch
  • Finally, start the sphinxsearch daemon:
    /etc/init.d/sphinxsearch start
  • Now you can log into RT and do a fulltext search by entering in the search field:
  • In my case, this made the fulltext searches run within a second instead of 5 minutes 🙂
  • Make sure the indexer is run regularily to update the index. To do so, enter
    crontab -e

    And add a line like this:

    13 * * * * /usr/bin/indexer rt --rotate

I hope this helps somebody to setup this a little faster.




14. March 2015

Bind nameserver: view cache entries

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 15:35

If you want to view the cache entries of your bind nameserver, this is how it works on bind9 on Debian:

rndc dumpdb
less /var/cache/bind/named_dump.db

Tested on Debian Wheezy. Hope this helps somebody.

13. August 2014

Icinga: Monitor refused mails in postfix mailqueue

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , — Christopher Kramer @ 12:09

In case your server gets listed on blacklists, mails will get refused by destination servers and stick in the deferred mail queue for some time until the sender finally gets a mailer daemon.

As it takes some time until the sender gets the mailer daemon and informs the server admin, it would be better if you could directly get notified by Icinga/Nagios when a mail is in the deferred queue because the destination server refused it.

Therefore I wrote a small shell script which I want to share with you here. I am assuming Debian Wheezy with Icinga and a postfix mailserver.

Create the shell script with the actual plugin in

/usr/lib/nagios/plugins/check_mailq_blacklist :
# detects if mails in mail queue were refused by destination server (because of blacklist?)
# From
# Version: 2015-04-27

if mailq | -qP "(refused to talk to me)|(unsolicited mail originating from your IP)"
  mails=`mailq | grep -oP "(refused to talk to me)|(unsolicited mail originating from your IP)" | wc -l`
  echo "$mails mail(s) were refused, check mailq!"
  if [ "$mails" -le 10 ]; then
    # 10 or less mails -> warning
    return 1;
    # more than 10 mails -> critical
    return 2;
  return 1;
  echo "Ok, there seems to be no refused mail in the mailq"
  exit 0;

This will check for the texts “refused to talk to me” and “unsolicited mail originating from your IP” in the mailq output. These are the most common errors you get when the destination server has your server’s IP blacklisted.  In case at least one mail was refused, this causes a warning state in icinga. If more than 10 mails were refused, it causes a critical state.

Now you need to make this script executable:

chmod +x /usr/lib/nagios/plugins/check_mailq_blacklist

Now create the config file for the plugin in

/etc/nagios-plugins/config/mailq_blacklist.cfg :
# 'check_mailq_blacklist' command definition
define command{
        command_name    check_mailq_blacklist
        command_line    /usr/lib/nagios/plugins/check_mailq_blacklist

So now we have the command and need to define a service that uses it. Let’s say we use this locally for localhost. In



define service{
        use                             generic-service
        host_name                       localhost
        service_description             Mail Queue Refused Mail
        check_command                   check_mailq_blacklist

This is it, just restart icinga and you are done:

service icinga restart

I hope this is of use to somebody.

Of course it is also useful to monitor in Icinga, if you are on some of the most used blacklists. A script to do this can be found here.

27. May 2014

Debian Linux: Update packages automatically

Filed under: Linux,Server Administration — Tags: , , , , , — Christopher Kramer @ 20:50

Here is how you configure automatic (security) updates on Debian:

aptitude install unattended-upgrades

Here you can configure it:

nano /etc/apt/apt.conf.d/50unattended-upgrades

This could look like this:

Unattended-Upgrade::Origins-Pattern {
Unattended-Upgrade::Package-Blacklist {
// add packages here that need manual steps like this:
//        "vim";
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::InstallOnShutdown "false";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
Acquire::http::Dl-Limit "200";

Now create the following file:

nano /etc/apt/apt.conf.d/02periodic

With this content:

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "7";

Of course the server needs to be able to send mails so it can send mails in case of problems.

You can test it like this:

# mail -s test
My testmail

Hope this helps somebody.

Update: Some updates caused dpkg questions about changed config files and therefore failed.

For example php5-fpm did ask this:

Setting up php5-fpm (5.4.4-14+deb7u10) ...

Configuration file `/etc/php5/fpm/pool.d/www.conf'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** www.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing php5-fpm (--configure):
 EOF on stdin at conffile prompt

This caused PHP5-FPM to stop and all PHP sites to show an Internal Server Errror…

So you don’t run into this problem, create /etc/apt/apt.conf.d/local with this content:

Dpkg::Options {

This tells DPKG to keep the old config file. It will create .dpkg-dist files with the package distributer’s version. More information on this can be found here.

18. January 2014

Linux: get members of a group / get groups of a user

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 00:23

This might sound easy with /etc/group and /etc/passwd, but what if you use libnss-mysql for example and need to try if it works?

So just forget grepping config files. Here are the real commands:

Members of a group:

getent group GROUPNAME

Replace GROUPNAME with the group you want to check.

Groups of a user:


Replace USERNAME with the name of the user to check.


Checked on Debian, not sure if it works on any Linux/Unix.

21. March 2013

Linux: ACPI disabled, won’t shut down, /sys/class/rtc/rtc0/wakealarm vanished

Filed under: Linux — Tags: , , , , , , , , , , , — Christopher Kramer @ 00:16

MythTV server won’t shut down automatically any longer

Today my mythTV based media homeserver suddenly would not automatically shutdown any longer after a kernel update.

Logs by mythbackend (/var/log/mythtv/mythbackend ) that triggers the shutdown  looked like this:

localhost mythbackend[1674]: I Scheduler scheduler.cpp:2729 (CheckShutdownServer) CheckShutdownServer returned - OK to shutdown
localhost mythbackend[1674]: N Scheduler scheduler.cpp:2814 (ShutdownServer) Running the command to set the next scheduled wakeup time [...]
localhost mythbackend[1674]: E Scheduler scheduler.cpp:2820 (ShutdownServer) SetWakeuptimeCommand failed, shutdown aborted

Problem: /sys/class/rtc/rtc0/wakealarm vanished after kernel update

So I ran the SetWakeupTime command myself and go this problem:

/sys/class/rtc/rtc0/wakealarm: No such file or directory

And in fact the file was missing.

System won’t power off any longer

I tried to shut down the pc completely (manually) and start it again. But it wouldn’t power off, it just freezed with the shutdown bar on the screen. Pressing the power button once (only short, not holding it) made the PC power off.

So this looked like some big ACPI problem.

ACPI disabled

After some more search, I found this in the syslog (/var/log/syslog):

localhost kernel: [    0.000000] ACPI: no DMI BIOS year, acpi=force is required to enable ACPI
localhost kernel: [    0.000000] ACPI: Disabling ACPI support

So no ACPI support explains why wakealarm and shutdown is broken.

I am wondering why this happens suddenly after a kernel update (I have not changed my BIOS or something and ACPI was perfectly working before).

Solution: force ACPI

Forcing ACPI solved the problem.

If you use grub2, this is how to do it:

Open /etc/default/grub (e.g. “sudo nano /etc/default/grub”) and update this line:


Into this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force"

Save the file and update the grub configuration:

sudo update-grub

Then restart and you’ll be done.

sudo shutdown -r now

I hope this could help somebody.

30. January 2013

Nagios / Icinga: Monitor (local) memory usage

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , , , , — Christopher Kramer @ 17:55

Nagios and its fork icinga are great monitoring tools. They come with a bundle of plugins to monitor standard services such as HTTP, SMTP, POP3, load and stuff like that. And there are lots of 3rd party plugins available for almost everything else you can think of.

But one standard thing that is missing in the official nagios-plugins package is a plugin to check memory usage (of the local machine).

So here is how to install one. I assume a Debian system with Icinga running – you might want to adjust paths for other distros or nagios.

  1. Download the plugin here
    e.g. from the shell:

  2. Then move the file to the other plugins
    mv /usr/lib/nagios/plugins/
  3. Make it executable
    chmod +x /usr/lib/nagios/plugins/
  4. Try to run it:
    perl /usr/lib/nagios/plugins/ -w 50% -c 25%
  5. This should give something like “CHECK_MEMORY OK – […] free […]”. If an error occurs, you probably need to install the perl module Nagios::Plugin. On Debian, the easiest way is:
    apt-get install libnagios-plugin-perl

    On other distros, you might use CPAN:

    perl -MCPAN -e 'install Nagios::Plugin'

    This will ask you lots of questions and install lots of dependencies (where you should say “yes”).

  6. Configure the check_memory command. To do this, create a file /etc/nagios-plugins/config/memory.cfg with this content:
    # 'check_memory' command definition
    define command{
            command_name    check_memory
            command_line    perl /usr/lib/nagios/plugins/ -w $ARG1$ -c $ARG2$
  7. Now you can use the check_memory command to define a service. For example, add this to /etc/icinga/objects/localhost_icinga.cfg (assuming you define localhost-services there):
    define service{
            use                             generic-service
            host_name                       localhost
            service_description             Memory
            check_command                   check_memory!50%!25%

    This will send you a warning when memory usage is 50% and critical when only 25% is free. You might want to adjust these values of course depending on what is normal on your system and how early you want to be notified.

  8. Check your configuration:
    /usr/local/icinga/bin/icinga -v /etc/icinga/icinga.cfg
  9. Restart Icinga / Nagios if the preflight-check was okay:
    /etc/init.d/icinga restart

This should be it.

I hope this helped somebody.

To monitor memory usage of a remote server, you’ll need SNMP for example. Maybe I’ll post another blog post on this soon.

14. February 2012

Horde language selection does not work

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 13:22

When selecting a language at login, Horde webmailer does not change the language?

Here is what I found out what helps:

On Debian, run the following command:

dpkg-reconfigure locales

Then select the correct languages. I had only selected the UTF8 languages for German, but Horde needs the following ones:

de_DE ISO-8859-1
de_DE@euro ISO-8859-15

If you have the problem with another language, select the corresponding language.

On Ubuntu, the chosen languages are stored here:


I had a file named “de” in there where my chosen languages where listed and I added the ISO-versions above. You can find all supported languages here:

less /usr/share/i18n/SUPPORTED

On Ubuntu, after you included your languages, you have to run the following command:

dpkg-reconfigure locales

Afterwards, you need to restart apache:

apache2ctl -k graceful

That’s the smoothest way. In case it does not work, use one of those:

apache2ctl restart
service apache2 restart
/etc/init.d/apache2 restart

Now refresh Horde and everything should work.

Another problem is the following: if you chose a language in your Horde settings (login, Global Options, Locale and time, Select your preferred language), this overwrites the language you chose on login. So select “default” there to be able to chose language on login.

Hope this helps somebody.