- - - By CrazyStat - - -

17. August 2016

Linux Software Raid: Enlarge Raid Array of Hetzner EX41

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 21:15

Recently, I ordered an EX41 server at Hetzner, which by default comes with two 4 TB HDDs in a software RAID1. I chose the minimal Debian Jessie Image. From Hetzner’s EX40, which has two 2 TB drives in a Software RAID1, I was used to find the whole drive formatted as one partition by Hetzner. But with the EX41, it turned out that the 4 TB were split: The root partition / was 2 TB of size and 1.7 TB were allocated to /home. As the application that should run on this server needs more than 2 TB space in one folder, I repartitioned the server. If somebody else has a similar issue, here is how it can be done:

  1. Edit /etc/fstab and remove or comment (by placing a # at the beginning) the line that mounts /home from /dev/md3
  2. Reboot into the Rescue system or some other system that is not on the same drive. (The Hetzner rescue system boots from network. On a local system, you would boot a linux from CD or USB.)
  3. Check the Raid status and setup:
    cat /proc/mdstat
    md3 : active raid1 sda4[0] sdb4[1]
          1777751872 blocks super 1.2 [2/2] [UU]
          bitmap: 0/14 pages [0KB], 65536KB chunk
    md2 : active raid1 sda3[0] sdb3[1]
          2111700992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/16 pages [0KB], 65536KB chunk
    md1 : active raid1 sda2[0] sdb2[1]
          523712 blocks super 1.2 [2/2] [UU]
    md0 : active raid1 sda1[0] sdb1[1]
          16760832 blocks super 1.2 [2/2] [UU]

    So here md3 is the raid array that we want to get rid of together with its partitions sda4 and sdb4. md2 the raid array that we want to grow in size.

  4. First let’s get rid of the md3:
    mdadm --stop /dev/md3
    mdadm --zero-superblock /dev/sda4
    mdadm --zero-superblock /dev/sdb4
    parted /dev/sda rm 4
    parted /dev/sdb rm 4
  5. Resize the partitions of md2 (sda3 and sdb3):
    (Adjust the end if necessary)

    parted /dev/sda
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
    parted /dev/sdb
    (parted) resize 3
    (END?) 4000GB
    (parted) quit

    Note: Tomas pointed out in the comments that the resize-command was removed and replaced with resizepart in current versions of parted. So the command is now:

    parted /dev/sda
    (parted) resizepart 3 4000GB
    (parted) quit

    (then do the same for sdb)

  6. Let the raid array md2 grow:
    mdadm --grow /dev/md2 --size=max
  7. Check the raid status:
    cat /proc/mdstat
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          [===================>.]  resync = 97.8% (3807048128/3888815376) finish=13.3min speed=101952K/sec
          bitmap: 1/15 pages [4KB], 131072KB chunk
  8. Wait until the resync finished (took a few hours for 2 TB):
    cat /proc/mdstat                                                                                                                                        
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 131072KB chunk
  9. Check the filesystem:
    e2fsck -fv /dev/md2
  10. Resize the filesystem:
    resize2fs /dev/md2
  11. Reboot into your normal system and you are done:
  12. Check the free disk space:
    df -h

    It should give the new size of / (3.6 TB) and a lot more free space 🙂

Hope this helps somebody with the same or similar issue.



Try my Open Source PHP visitor analytics script CrazyStat.

28. June 2016

flashplugin-installer cannot download files using unattended-upgrades on Ubuntu

Filed under: Linux — Tags: , , , , , , — Christopher Kramer @ 10:29

Using unattended-upgrades on Ubuntu  16.04 (and previous versions) set up to install updates on shutdown, flashplugin-installer always failed when downloading the new version. The reason probably is that on shutdown, the Internet connection is not available anymore. Up to now, I always updated flashplugin-installer manually. But today, I found a nice and simple solution:

Just get rid of the installer! In the Cononical partner repository, there is a package that directly contains the flash plugin. First, make sure you include the partner repository in your /etc/apt/sources.list, e.g. for xenial:

deb xenial partner
deb-src xenial partner

Then, update, get rid of the flashplugin-installer, and install the adobe-flashplugin instead:

sudo apt-get update
sudo apt-get remove flashplugin-installer
sudo apt-get install adobe-flashplugin

Hope this helps someone who also has problems with unattended upgrades and falshplugin-installer.

17. June 2016

Multimedia Keys “stuck” after Ubuntu 16.04 upgrade

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 21:19

After upgrading Ubuntu Gnome to 16.04, the multimedia keys / extra keys (like play/pause, volume up/down, mute, calculator, forward, backward, etc.) of my old Cherry G86-21050 DEAAAA keyboard showed a strange behavior: Whenever one of the buttons was pressed, its effect was repeated over and over, just as if the button was stuck. So pressing pause would play and pause the song over and over, pressing calculator would open hundreds of calculator windows. Pressing another special keys stopped the previous effect from repeating, but repeated the effect of the last special key over and over. The normal keys (letter, numbers etc.) were not affected.

As the effect appeared after the upgrade for all special keys, I was sure it was a software problem and not a hardware problem.

Digging around with evtest (no, it is not a good idea to run evtest –grab on your keyboard in a tty :D), I found out that the key events are not repeated multiple times, but the keys are not released.

Annoyed, I tried another keyboard (Logitech Elite Keyboard) for comparison. The special keys of that one worked well.

Finally, I accidentally plugged the “faulty” Cherry Keyboard in the USB port that I had the working Logitech in between and voila: The Cherry’s special keys worked again.

So lesson learned: Have you tried plugging it out and in again? Is the first question you should ask (yourself) whenever some USB device behaves strangely.

19. May 2016

Ubuntu Gnome: Upgrading to 16.04 Xenial Xerus fails: ubuntu-gnome-desktop (gdm, gnome-session, adwaita-icon-theme and adwaita-icon-theme-full)

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 13:26

Today I upgraded my Ubuntu Gnome 15.10 to 16.04 Xenial Xerus. I was surprised to find in the details that it removes gdm. I started the update and it went normal until it said ubuntu-gnome-desktop was broken due to unresolved dependencies. The GUI upgrade process that I had used closed and I was left with a system upgraded by half. But it was still running, so I started a terminal and started aptitude. It suggested to install various packages to resolve dependencies, especially gdm3, gnome-shell, gnome-shell-extensions and gnome-session. It also proposed to remove unused packages, which were not removed as the upgrade process had stopped half way. I followed aptitudes suggestions, but it left two packages not upgraded: adwaita-icon-theme and gnome-session.

And this took me quite a while to solve. There are two packages: adwaita-icon-theme and adwaita-icon-theme-full. The first one contains a subset of icons, whereas the second one contains all icons. The ubuntu-gnome-desktop requires the adwaita-icon-theme-full package, which in turn requires the adwaita-icon-theme package. The adwaita-icon-theme had an upgrade from version 3.18.0-2ubuntu3 to 3.18.0-2ubuntu3.1. The adwaita-icon-theme-full package requires that it is exactly the same version as adwaita-icon-theme, but the only version that apt found was 3.18.0-2ubuntu3.

Then I found out that adwaita-icon-theme-full is in the universe repository, whereas adwaita-icon-theme is in main. In my sources.list, I had the following:

deb xenial-proposed main restricted
deb xenial-proposed main restricted

Therefore, the new version of adwaita-icon-theme was found in the main repository. But universe (and multiverse) repositories were missing  the proposed level, and this was the reason why the new version of adwaita-icon-theme-full was not found by apt. So the solution was simple: Just changing the above lines in sources.list like this, adding universe and multiverse repositories:

deb xenial-proposed main restricted universe multiverse
deb xenial-proposed main restricted universe multiverse

Then it updated the remaining packages normally using:

aptitude update
aptitude safe-upgrade

So lesson learned is:
If you add the proposed level in your sources.list, make sure you add it for all repositories you use, e.g. including universe and multiverse, not only main.

Hope this helps somebody who runs into the same issue.

19. April 2016

Linux: Find the MAC address of your wireless adapter like wlan0

Filed under: Linux — Tags: , , , , , , , , , , , , — Christopher Kramer @ 14:57

You tried ifconfig and iwconfig, but they all don’t show you the MAC address of your wireless adapter under Linux? For me, on Arch Linux and Ubuntu. This is what gives you what you want and its so beautifully short and simple:

ip addr

The MAC address comes after “link/ether”. This works for wired ethernet adapters (eth0 etc.), wireless adapters (wlan0 etc.) and any other ethernet device  (e.g. IEEE 802.11s mesh point devices).

If you are on Windows, read my old blog post about getting the MAC address on Windows.

Hope this helps somebody to find it a little faster.

14. July 2015

Ubuntu Gnome: Sort folders first in Nautilus

Filed under: Linux — Tags: , , , — Christopher Kramer @ 14:56

If you prefer to have the folders at the beginning in Nautilus, just type this command in a terminal:

gsettings set org.gnome.nautilus.preferences sort-directories-first true

Works in Ubuntu Gnome 15.04

Ubuntu Gnome: Find file starting with the letter you type in Nautilus

Filed under: Linux — Tags: , , , — Christopher Kramer @ 14:49

New nautilus versions start searching for what you start typing inside a folder. If you instead prefer to jump to the files starting with what you type, just like Windows Explorer behaves, type this in a terminal:

gsettings set org.gnome.nautilus.preferences enable-interactive-search true

Works on Ubuntu Gnome 15.04

23. June 2015

Bind9 on Debian Wheezy: journal rollforward failed: journal out of sync with zone

Filed under: Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 20:29

Getting this error when restarting bind9?

journal rollforward failed: journal out of sync with zone

You probably changed the zonefile while bind was running without freezing it.

To fix this, first stop bind:

service bind9 stop

Then remove the journal file for the zone. On Debian Wheezy, it is in the same folder as the zone:

rm /etc/bind/zones/

Then start bind again:

service bind9 start

To avoid the problem next time, before changing a zonefile, freeze the zone:

rndc freeze

Then edit your zonefile and finally thaw it:

rndc thaw

6. March 2015

Spamassasin: MISSING_SUBJECT in every mail even though mail has a subject

Filed under: Linux,Server Administration — Tags: , , , — Christopher Kramer @ 09:48

I noticed the MISSING_SUBJECT rule matched for every mail recently even though the mails contained a subject header. Finally I found this in the spamd.log:

warn: Possible unintended interpolation of @yahoo in string at /etc/spamassassin/, rule MY_SPAMMY_YAHOO, line 1.
warn: rules: failed to compile Mail::SpamAssassin::Plugin::Check::_head_tests_0_4, skipping:
warn:  (Global symbol "@yahoo" requires explicit package name at /etc/spamassassin/, rule MY_SPAMMY_YAHOO, line 1.)

The reason was a rule like this:

header          MY_SPAMMY_YAHOO        To =~ /
score           MY_SPAMMY_YAHOO        3
describe        MY_SPAMMY_YAHOO        Mail to is mostly spam

Escaping the @ in the rule fixed the Problem:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress\

Hope this helps someone to spot the error faster.

The problem is that spamassasin does not autolearn ham if the ham messages match the MISSING_SUBJECT rule. So an error like this basically not only breaks lots of tests but also autolearning of ham. So lesson learned: better always look at the spamd.log after creating a new rule, even if you think you know what you are doing 😉

13. August 2014

Icinga: Monitor refused mails in postfix mailqueue

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , — Christopher Kramer @ 12:09

In case your server gets listed on blacklists, mails will get refused by destination servers and stick in the deferred mail queue for some time until the sender finally gets a mailer daemon.

As it takes some time until the sender gets the mailer daemon and informs the server admin, it would be better if you could directly get notified by Icinga/Nagios when a mail is in the deferred queue because the destination server refused it.

Therefore I wrote a small shell script which I want to share with you here. I am assuming Debian Wheezy with Icinga and a postfix mailserver.

Create the shell script with the actual plugin in

/usr/lib/nagios/plugins/check_mailq_blacklist :
# detects if mails in mail queue were refused by destination server (because of blacklist?)
# From
# Version: 2017-03-07

if mailq | grep -qP "(refused to talk to me(?!(.*out of connection slots)))|(unsolicited mail originating from your IP)|(temporarily deferred due to user complaints)"
  mails=`mailq | grep -oP "(refused to talk to me(?!(.*out of connection slots)))|(unsolicited mail originating from your IP)|(temporarily deferred due to user complai$
  echo "$mails mail(s) were refused, check mailq!"
  if [ "$mails" -le 10 ] && [ "$mails" -gt 1 ]; then
    # 2-10 mails -> warning
    echo "\nWarning. | refused=$mails;2;11;0"
    return 1;
  if [ "$mails" -gt 10 ]; then
    # more than 10 mails -> critical
    echo "\nCriticial! | refused=$mails;2;11;0"
    return 2;
  return 1;
  echo "Ok, there seems to be no refused mail in the mailq | refused=0;2;11;0"
  exit 0;

This will check for the texts “refused to talk to me” (not followed by “out of connection slots”) and “unsolicited mail originating from your IP” in the mailq output. These are the most common errors you get when the destination server has your server’s IP blacklisted.  In case at least one mail was refused, this causes a warning state in icinga. If more than 10 mails were refused, it causes a critical state.

Now you need to make this script executable:

chmod +x /usr/lib/nagios/plugins/check_mailq_blacklist

Now create the config file for the plugin in

/etc/nagios-plugins/config/mailq_blacklist.cfg :
# 'check_mailq_blacklist' command definition
define command{
        command_name    check_mailq_blacklist
        command_line    /usr/lib/nagios/plugins/check_mailq_blacklist

So now we have the command and need to define a service that uses it. Let’s say we use this locally for localhost. In



define service{
        use                             generic-service
        host_name                       localhost
        service_description             Mail Queue Refused Mail
        check_command                   check_mailq_blacklist

This is it, just restart icinga and you are done:

service icinga restart

I hope this is of use to somebody.

Of course it is also useful to monitor in Icinga, if you are on some of the most used blacklists. A script to do this can be found here.

Older Posts »