- - - By CrazyStat - - -

17. August 2016

Linux Software Raid: Enlarge Raid Array of Hetzner EX41

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 21:15

Recently, I ordered an EX41 server at Hetzner, which by default comes with two 4 TB HDDs in a software RAID1. I chose the minimal Debian Jessie Image. From Hetzner’s EX40, which has two 2 TB drives in a Software RAID1, I was used to find the whole drive formatted as one partition by Hetzner. But with the EX41, it turned out that the 4 TB were split: The root partition / was 2 TB of size and 1.7 TB were allocated to /home. As the application that should run on this server needs more than 2 TB space in one folder, I repartitioned the server. If somebody else has a similar issue, here is how it can be done:

  1. Edit /etc/fstab and remove or comment (by placing a # at the beginning) the line that mounts /home from /dev/md3
  2. Reboot into the Rescue system or some other system that is not on the same drive. (The Hetzner rescue system boots from network. On a local system, you would boot a linux from CD or USB.)
  3. Check the Raid status and setup:
    cat /proc/mdstat
    md3 : active raid1 sda4[0] sdb4[1]
          1777751872 blocks super 1.2 [2/2] [UU]
          bitmap: 0/14 pages [0KB], 65536KB chunk
    md2 : active raid1 sda3[0] sdb3[1]
          2111700992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/16 pages [0KB], 65536KB chunk
    md1 : active raid1 sda2[0] sdb2[1]
          523712 blocks super 1.2 [2/2] [UU]
    md0 : active raid1 sda1[0] sdb1[1]
          16760832 blocks super 1.2 [2/2] [UU]

    So here md3 is the raid array that we want to get rid of together with its partitions sda4 and sdb4. md2 the raid array that we want to grow in size.

  4. First let’s get rid of the md3:
    mdadm --stop /dev/md3
    mdadm --zero-superblock /dev/sda4
    mdadm --zero-superblock /dev/sdb4
    parted /dev/sda rm 4
    parted /dev/sdb rm 4
  5. Resize the partitions of md2 (sda3 and sdb3):
    (Adjust the end if necessary)

    parted /dev/sda
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
    parted /dev/sdb
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
  6. Let the raid array md2 grow:
    mdadm --grow /dev/md2 --size=max
  7. Check the raid status:
    cat /proc/mdstat
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          [===================>.]  resync = 97.8% (3807048128/3888815376) finish=13.3min speed=101952K/sec
          bitmap: 1/15 pages [4KB], 131072KB chunk
  8. Wait until the resync finished (took a few hours for 2 TB):
    cat /proc/mdstat                                                                                                                                        
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 131072KB chunk
  9. Check the filesystem:
    e2fsck -fv /dev/md2
  10. Resize the filesystem:
    resize2fs /dev/md2
  11. Reboot into your normal system and you are done:
  12. Check the free disk space:
    df -h

    It should give the new size of / (3.6 TB) and a lot more free space 🙂

Hope this helps somebody with the same or similar issue.



Try my Open Source PHP visitor analytics script CrazyStat.

15. July 2016

Request-Tracker4: Enabling fast fulltext search with Sphinxsearch on Debian Jessie

I just successfully made fulltext search in our request tracker super fast. This is a guide for Debian Jessie and request tracker set up to use MySQL (or MariaDB).

  • Don’t forget to backup your RT database:
    mysqldump -p -u root rtdb > rtdb.sql
  • Log out of request tracker. I had some strange issue because I did not 😉
  • Make sure you are using request-tracker4 version 4.2.12 from jessie-backports. If you don’t already, this is how you update:
    – add this to the end of your /etc/apt/sources.list:

    deb jessie-backports main

    – run

    apt-get update

    – install request-tracker4 from jessie-backports:

    apt-get -t jessie-backports install request-tracker4

    – follow the update procedure, make sure your database is upgraded
    (We need the package from backports as jessie’s rt4-db-mysql package depends on mysql-client, whereas the backports version depends on virtual-mysql-client and thus can also be easily used with MariaDB.)

  • Second, make sure you are using MariaDB instead of MySQL. MariaDB is a drop-in replacement for MySQL. The big advantage of MariaDB here is that it comes with the SphinxSE storage engine that we will use for the fulltext search. SphinxSE is also available for MySQL, but you would need to compile it yourself whereas MariaDB includes SphinxSE by default. MariaDB has also a lot more advantages. Switching to MariaDB is very simple, no need to convert databases or configuration. I used aptitude and selected to install mariadb-client and maridb-server. Then I checked the dependency resolutions and choose the option to uninstall mysql-client, mysql-server and related packages. Then everything went smooth automatically, I just needed to enter the root password again, everything else was updated automatically.
  • Now check if your RT is still working well with MariaDB.
  • Now enable the SphinxSE storage engine in MariaDB:
    mysql -u root -p
    [enter password]
    INSTALL SONAME 'ha_sphinx';
  • Now install the Sphinxsearch daemon:
    apt-get install sphinxsearch
  • Now set up indexed full text search in RT:
    rt-setup-fulltext-index --dba root --dba-password secret

    (Let me know if you find a way without passing the password as a parameter)
    This asks you what type of index you want to use. Enter “sphinx”. Then keep the defaults.

  • The command above gave you code for /etc/request-tracker4/ – like this:
    Set( %FullTextSearch,
        Enable     => 1,
        Indexed    => 1,
        MaxMatches => '10000',
        Table      => 'AttachmentsIndex',

    Add / adjust this in

  • Then you need to configure the index in sphinx. The above tool also gave you a config file for this, but it does not work with the current sphinxsearch version of jessie. So here is what works. Create an /etc/sphinxsearch/sphinx.conf like this:
    source rt {
        type            = mysql
        sql_host        = localhost
        sql_db          = rtdb
        sql_user        = rtuser
        sql_pass        = RT_PASSWORD
        sql_query_pre   = SET NAMES utf8
        sql_query       = \
            SELECT, t.Subject, a.content FROM Attachments a \
            JOIN Transactions txn ON a.TransactionId = AND txn.ObjectType = 'RT::Ticket' \
            JOIN Tickets t ON txn.ObjectId = \
            WHERE a.ContentType LIKE 'text/%' AND t.Status != 'deleted'
    #    sql_query_info  = SELECT * FROM Attachments WHERE id=$id
    index rt {
        source                  = rt
        path                    = /var/lib/sphinxsearch/data/rt
        docinfo                 = extern
    #    charset_type            = utf-8
    indexer {
        mem_limit               = 32M
    searchd {
        listen                  = 3312
        log                     = /var/log/sphinxsearch/searchd.log
        query_log               = /var/log/sphinxsearch/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinxsearch/
    #    max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 0
        unlink_old              = 1
        # For sphinx >= 1.10:
        binlog_path             = /var/lib/sphinxsearch/data

    Make sure to adjust the mysql database, user and password.
    Note that there are some differences to the config provided by rt-setup-fulltext-index. Especially, you need to use listen instead of port. If you don’t adjust this, sphinxsearch will start on a different port and RT will not find any results in fulltext searches! If your RT is not behind a firewall, better also set listen to . Also, I used the Debian style paths from the Debian sample. The fields commented out are not necessary anymore in current sphinxsearch.
    UPDATE 26.08.2016: I changed the SQL Query to also index the Subject of the tickets and index all ‘text/%’ types and not only ‘text/plain’. This change is based on a comment by Claes Merin in the Request Tracker Mailing List. Thanks a lot. Without this change, the fulltext search will only search within ticket content, but not within ticket titles… Also, before it would not index text/html attachments.

  • Now run the indexer to build the initial index. This will take some time, for a 6 GB Attachments Table, it took not more than 30 minutes. Maybe run this in a screen in case your ssh connection breaks:
    indexer rt
  • Enable the Sphinx daemon. To do so, adjust START to yes in /etc/default/sphinxsearch
  • Finally, start the sphinxsearch daemon:
    /etc/init.d/sphinxsearch start
  • Now you can log into RT and do a fulltext search by entering in the search field:
  • In my case, this made the fulltext searches run within a second instead of 5 minutes 🙂
  • Make sure the indexer is run regularily to update the index. To do so, enter
    crontab -e

    And add a line like this:

    13 * * * * /usr/bin/indexer rt --rotate

I hope this helps somebody to setup this a little faster.




28. June 2016

flashplugin-installer cannot download files using unattended-upgrades on Ubuntu

Filed under: Linux — Tags: , , , , , , — Christopher Kramer @ 10:29

Using unattended-upgrades on Ubuntu  16.04 (and previous versions) set up to install updates on shutdown, flashplugin-installer always failed when downloading the new version. The reason probably is that on shutdown, the Internet connection is not available anymore. Up to now, I always updated flashplugin-installer manually. But today, I found a nice and simple solution:

Just get rid of the installer! In the Cononical partner repository, there is a package that directly contains the flash plugin. First, make sure you include the partner repository in your /etc/apt/sources.list, e.g. for xenial:

deb xenial partner
deb-src xenial partner

Then, update, get rid of the flashplugin-installer, and install the adobe-flashplugin instead:

sudo apt-get update
sudo apt-get remove flashplugin-installer
sudo apt-get install adobe-flashplugin

Hope this helps someone who also has problems with unattended upgrades and falshplugin-installer.

17. June 2016

Multimedia Keys “stuck” after Ubuntu 16.04 upgrade

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 21:19

After upgrading Ubuntu Gnome to 16.04, the multimedia keys / extra keys (like play/pause, volume up/down, mute, calculator, forward, backward, etc.) of my old Cherry G86-21050 DEAAAA keyboard showed a strange behavior: Whenever one of the buttons was pressed, its effect was repeated over and over, just as if the button was stuck. So pressing pause would play and pause the song over and over, pressing calculator would open hundreds of calculator windows. Pressing another special keys stopped the previous effect from repeating, but repeated the effect of the last special key over and over. The normal keys (letter, numbers etc.) were not affected.

As the effect appeared after the upgrade for all special keys, I was sure it was a software problem and not a hardware problem.

Digging around with evtest (no, it is not a good idea to run evtest –grab on your keyboard in a tty :D), I found out that the key events are not repeated multiple times, but the keys are not released.

Annoyed, I tried another keyboard (Logitech Elite Keyboard) for comparison. The special keys of that one worked well.

Finally, I accidentally plugged the “faulty” Cherry Keyboard in the USB port that I had the working Logitech in between and voila: The Cherry’s special keys worked again.

So lesson learned: Have you tried plugging it out and in again? Is the first question you should ask (yourself) whenever some USB device behaves strangely.

19. May 2016

Ubuntu Gnome: Upgrading to 16.04 Xenial Xerus fails: ubuntu-gnome-desktop (gdm, gnome-session, adwaita-icon-theme and adwaita-icon-theme-full)

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 13:26

Today I upgraded my Ubuntu Gnome 15.10 to 16.04 Xenial Xerus. I was surprised to find in the details that it removes gdm. I started the update and it went normal until it said ubuntu-gnome-desktop was broken due to unresolved dependencies. The GUI upgrade process that I had used closed and I was left with a system upgraded by half. But it was still running, so I started a terminal and started aptitude. It suggested to install various packages to resolve dependencies, especially gdm3, gnome-shell, gnome-shell-extensions and gnome-session. It also proposed to remove unused packages, which were not removed as the upgrade process had stopped half way. I followed aptitudes suggestions, but it left two packages not upgraded: adwaita-icon-theme and gnome-session.

And this took me quite a while to solve. There are two packages: adwaita-icon-theme and adwaita-icon-theme-full. The first one contains a subset of icons, whereas the second one contains all icons. The ubuntu-gnome-desktop requires the adwaita-icon-theme-full package, which in turn requires the adwaita-icon-theme package. The adwaita-icon-theme had an upgrade from version 3.18.0-2ubuntu3 to 3.18.0-2ubuntu3.1. The adwaita-icon-theme-full package requires that it is exactly the same version as adwaita-icon-theme, but the only version that apt found was 3.18.0-2ubuntu3.

Then I found out that adwaita-icon-theme-full is in the universe repository, whereas adwaita-icon-theme is in main. In my sources.list, I had the following:

deb xenial-proposed main restricted
deb xenial-proposed main restricted

Therefore, the new version of adwaita-icon-theme was found in the main repository. But universe (and multiverse) repositories were missing  the proposed level, and this was the reason why the new version of adwaita-icon-theme-full was not found by apt. So the solution was simple: Just changing the above lines in sources.list like this, adding universe and multiverse repositories:

deb xenial-proposed main restricted universe multiverse
deb xenial-proposed main restricted universe multiverse

Then it updated the remaining packages normally using:

aptitude update
aptitude safe-upgrade

So lesson learned is:
If you add the proposed level in your sources.list, make sure you add it for all repositories you use, e.g. including universe and multiverse, not only main.

Hope this helps somebody who runs into the same issue.

19. April 2016

Linux: Find the MAC address of your wireless adapter like wlan0

Filed under: Linux — Tags: , , , , , , , , , , , , — Christopher Kramer @ 14:57

You tried ifconfig and iwconfig, but they all don’t show you the MAC address of your wireless adapter under Linux? For me, on Arch Linux and Ubuntu. This is what gives you what you want and its so beautifully short and simple:

ip addr

The MAC address comes after “link/ether”. This works for wired ethernet adapters (eth0 etc.), wireless adapters (wlan0 etc.) and any other ethernet device  (e.g. IEEE 802.11s mesh point devices).

If you are on Windows, read my old blog post about getting the MAC address on Windows.

Hope this helps somebody to find it a little faster.

14. July 2015

Ubuntu Gnome 15.04: Hide Hidden files in Nautilus

Filed under: Linux — Tags: , , , — Christopher Kramer @ 15:14

To hide hidden files and folders in Nautilus in Ubuntu Gnome 15.05, type this in a terminal:

gsettings set org.gtk.Settings.FileChooser show-hidden false

If you want to see hidden files from time to time, press CTRL + H in a nautilus window.

Ubuntu Gnome: Sort folders first in Nautilus

Filed under: Linux — Tags: , , , — Christopher Kramer @ 14:56

If you prefer to have the folders at the beginning in Nautilus, just type this command in a terminal:

gsettings set org.gnome.nautilus.preferences sort-directories-first true

Works in Ubuntu Gnome 15.04

Ubuntu Gnome: Find file starting with the letter you type in Nautilus

Filed under: Linux — Tags: , , , — Christopher Kramer @ 14:49

New nautilus versions start searching for what you start typing inside a folder. If you instead prefer to jump to the files starting with what you type, just like Windows Explorer behaves, type this in a terminal:

gsettings set org.gnome.nautilus.preferences enable-interactive-search true

Works on Ubuntu Gnome 15.04

23. June 2015

Bind9 on Debian Wheezy: journal rollforward failed: journal out of sync with zone

Filed under: Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 20:29

Getting this error when restarting bind9?

journal rollforward failed: journal out of sync with zone

You probably changed the zonefile while bind was running without freezing it.

To fix this, first stop bind:

service bind9 stop

Then remove the journal file for the zone. On Debian Wheezy, it is in the same folder as the zone:

rm /etc/bind/zones/

Then start bind again:

service bind9 start

To avoid the problem next time, before changing a zonefile, freeze the zone:

rndc freeze

Then edit your zonefile and finally thaw it:

rndc thaw
Older Posts »