- - - By CrazyStat - - -

26. February 2017

Postgrey: let nagios/icinga tell you what domains might need whitelisting

Filed under: Linux,Server Administration — Tags: , , , , , — Christopher Kramer @ 16:58

The problem: greylisting significantly delays mail from big mail providers with many servers

I still think greylisting with postgrey is very effective against spam. But now and then, mail from some senders get delayed for hours. This especially is the case for big mail service providers like Amazon SES, Microsoft Exchange Online Protection or Mailchimp. These providers have lots of servers and IP-addresses and the first attempt to deliver an email usually comes from a different server than the second. Thus, postgrey will not find a matching triplet and block the second attempt again. This will continue until one of the mail servers tried twice and postgrey found a matching tripled. This might take hours and thus cause significant delay of mails.

Therefore, if you want to use postgrey without delaying lots of mail for several hours, you need to whitelist these big email services with many servers. As new services come up and others go, you need some way to find out which servers try to send mail to your server and frequently get denied because no triplet is found.

The solution: Check your mail logs

To get an idea of what is happening and which servers might need whitelisting, I wrote a small command that analyzes your mail.log-files. Here it is:

grep -ohP 'reason=new, client_name=[^,.]+\.[^,.]+\.\K[^,.]+\.[^,]+' /var/log/mail.log* | sort | uniq -c | sort -nr | less

So what does this do?

It scans your /var/log/mail.log* files  for occurrences containing reason=new, which is what postgrey logs when no triplet was found. It then takes the domain of the server that is trying to send the mail to you, stripping the first subdomain, as this might vary from server to server that the mail service provider uses. It then sorts, counts and orders the result.

The result of this command might look like this (stripped after some lines):


You should check especially all the lines that have a lot of occurrences. They might be domains from big email service providers that need whitelisting for the reasons explained above. For example, the mail from and is mail sent by customers of these mail providers that usually is not spam. Especially, the mail servers of these providers do retry delivery lots of times, so greylisting would not block spam anyway, it would just cause significant delays. So these domains should be whitelisted.

But the domains in this list might also only be domains that are used by an ISP for dynamic IP addresses that are actually sending spam. For example, seems to be a big ISP from Thailand and multiple dynamic IP-addresses from this ISP tried to send spam, resulting in 17 occurrences where no triplet was found. This is mail that you want to be blocked/delayed.

Once you decided which domains you want to whitelist, you can do so by adding them in your whitelist-file (on Debian this is usually /etc/postgrey/whitelist_clients).

The entries should look like this:


Note: You should escape all dots with a backslash, as done in the example. Otherwise, a dot matches any character, so would also whitelist, which might be a domain that spammers register on purpose to get through servers that use inaccurate whitelists.

Automation with nagios/icinga

You could run the above command from time to time to check if new services need whitelisting. But it would be more easy, if you get notified when a new domain pops up in this list that you have not checked yet, right?

Therefore, I wrote a small nagios/icinga plugin that checks if any new domains appear on this list. It is a bit quick and dirty but does its job.

The main part is a bash script, that you could place in /usr/lib/nagios/plugins/check_postgrey_whitelist :


hosts=`grep -ohP 'reason=(new|early-retry[^,]*), client_name=[^,.]+\.[^,.]+\.\K[^,.]+(\.co|\.com)\.?[^,]+' $log | sort | uniq -c | awk '$1>=15{print $2}' | sort`;
noWhitelist=`cat $ignoreFile | sort`;

diff=`comm -23 <(echo "$hosts") <(echo "$noWhitelist") | sed ':a;N;$!ba;s/\n/ /g'`;


if [ -z "$diff" ]; then
        echo "Okay, the top domains are whitelisted or ignored";
        exit $OK;
        echo "Whitelist or ignore these senders: $diff";
        exit $WARNING;

This script ignores all domains listed in /etc/nagios-plugins/postgrey_no_whitelist  (one per line), assuming you checked that you do not want to whitelist these. So create this file as well and put the domains in that you checked and do not want to whitelist:

Now we need to define the command for this plugin:

Create the configuration where your other plugins are defined, e.g. /etc/nagios-plugins/config/postgrey_whitelist.cfg

define command {
        command_name    check_postgrey_whitelist
        command_line    /usr/lib/nagios/plugins/check_postgrey_whitelist

Now you need to create a service for your localhost that runs this command.

In your localhost service definition, e.g. /etc/icinga/objects/localhost_icinga.cfg , add this:

define service{
        use                             generic-service
        host_name                       localhost
        service_description             Postgrey Whitelist
        check_command                   check_postgrey_whitelist

Finally, make sure that nagios/icinga has read-access to your mail-logfile. You can adjust which file to check in the script above, I chose mail.log.1. So you might chown this file to the nagios user like this:

chown nagios:adm mail.log.1

Of course logrotation needs to know that new rotated files should have this owner. On a debian system, you can configure logrotate to do so in /etc/logrotate.d/rsyslog. Adjust it similar to this one:

        rotate 4
                invoke-rc.d rsyslog rotate > /dev/null

        rotate 4
        create 440 nagios adm
                invoke-rc.d rsyslog rotate > /dev/null

So I removed the mail.log from the first logrotate definition and created a new one below which is exactly the same but contains an extra line create 440 nagios adm to tell logrotate which user and permission to assign the logfiles to.

You might need to adjust some stuff depending on your distro, but I hope this helps somebody to set this up.

Update 06.03.2017: Adjusted script so it does not consider second-level domains like or as single senders anymore.


Try my Open Source PHP visitor analytics script CrazyStat.

14. December 2016

Microsoft and AOL sending DMARC aggregate reports (RUA) without DKIM signature

Filed under: Server Administration — Tags: , , , , , , , — Christopher Kramer @ 19:18

Just noticed that Microsoft and AOL are sending their DMARC aggregate reports without a DKIM signature. This alone is not uncommon, lots of small mail providers don’t use DKIM for their DMARC aggregate report mails. But the domains that Microsoft and AOL use as sender of their aggregate report mails have DMARC policies themselves. AOL even sets p=reject in their policy, which means that a mail server checking DMARC policies would reject aggregate reports sent by AOL if SPF fails, e.g. because the mail was relayed in between.

This is their DMARC policies:         2083    IN      TXT     
     "v=DMARC1\; p=reject\; pct=100\;\;\;"   391     IN      TXT     
     "v=DMARC1\; p=quarantine\; pct=100\;\;\; fo=1"

And this are the SPF records:                3190    IN      TXT     
     "v=spf1 ~all"          227     IN      TXT     
     "v=spf1 ip4: ip4: -all"          1683    IN      TXT     
 "v=spf1 ip4: ip4: ip4: ip4: ip4: -all"

AOL is sending its aggregate reports from, and the SPF records of this subdomain has a strict “-all” (which doesn’t). Microsoft sends the reports from, and also uses “-all”.

Google for example sends its aggregate reports with DKIM signature, so DMARC evaluation runs smooth. SPF is relaxed “~all” at, but DMARC nowadays is “p=reject”, which I was a little surprised to see.

So in conclusion, if you really want to get your DMARC aggregate reports, you should not strictly follow the DMARC policies of the others, otherwise you might miss some reports…

And if you set up a mail server yourself that sends DMARC reports, don’t forget to sign those mails with DKIM as well.

15. November 2016

Typo3 Mask – Javascript and CSS includes of extensions missing

Filed under: JavaScript,Typo3 — Tags: , , , , , , — Christopher Kramer @ 11:36

I was using Mask 2.1.1 with Typo3 7.6.11 for a site as described in the Mask manual with this typoscript:

page <
page.10.file.stdWrap.cObject.default.value = fileadmin/templates/index.html
page {
        includeCSS.styles = fileadmin/templates/style.css

The problem with this was that the CSS and JS files includes from extensions where missing, even though the static templates of this where included. The reason is that this typoscript overwrites the page.include* stuff defined by the templates. To solve this, I copied the page.include* stuff defined by the extensions in a temporary variable and restored it after loading the mask object into page:

tempJSFooterlibs < page.includeJSFooterlibs
tempJSFooter < page.includeJSFooter
tempCSS < page.includeCSS

page <
page.10.file.stdWrap.cObject.default.value = fileadmin/templates/index.html
page {
        includeJSFooterlibs < tempJSFooterlibs
        includeJSFooter < tempJSFooter
        includeCSS < tempCSS

        includeCSS.styles = fileadmin/templates/style.css

There are more includes you might need to consider if your extensions use these:


Hope this helps someone with the same problem! Please let me know if there is an easier solution.

17. August 2016

Linux Software Raid: Enlarge Raid Array of Hetzner EX41

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 21:15

Recently, I ordered an EX41 server at Hetzner, which by default comes with two 4 TB HDDs in a software RAID1. I chose the minimal Debian Jessie Image. From Hetzner’s EX40, which has two 2 TB drives in a Software RAID1, I was used to find the whole drive formatted as one partition by Hetzner. But with the EX41, it turned out that the 4 TB were split: The root partition / was 2 TB of size and 1.7 TB were allocated to /home. As the application that should run on this server needs more than 2 TB space in one folder, I repartitioned the server. If somebody else has a similar issue, here is how it can be done:

  1. Edit /etc/fstab and remove or comment (by placing a # at the beginning) the line that mounts /home from /dev/md3
  2. Reboot into the Rescue system or some other system that is not on the same drive. (The Hetzner rescue system boots from network. On a local system, you would boot a linux from CD or USB.)
  3. Check the Raid status and setup:
    cat /proc/mdstat
    md3 : active raid1 sda4[0] sdb4[1]
          1777751872 blocks super 1.2 [2/2] [UU]
          bitmap: 0/14 pages [0KB], 65536KB chunk
    md2 : active raid1 sda3[0] sdb3[1]
          2111700992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/16 pages [0KB], 65536KB chunk
    md1 : active raid1 sda2[0] sdb2[1]
          523712 blocks super 1.2 [2/2] [UU]
    md0 : active raid1 sda1[0] sdb1[1]
          16760832 blocks super 1.2 [2/2] [UU]

    So here md3 is the raid array that we want to get rid of together with its partitions sda4 and sdb4. md2 the raid array that we want to grow in size.

  4. First let’s get rid of the md3:
    mdadm --stop /dev/md3
    mdadm --zero-superblock /dev/sda4
    mdadm --zero-superblock /dev/sdb4
    parted /dev/sda rm 4
    parted /dev/sdb rm 4
  5. Resize the partitions of md2 (sda3 and sdb3):
    (Adjust the end if necessary)

    parted /dev/sda
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
    parted /dev/sdb
    (parted) resize 3
    (END?) 4000GB
    (parted) quit

    Note: Tomas pointed out in the comments that the resize-command was removed and replaced with resizepart in current versions of parted. So the command is now:

    parted /dev/sda
    (parted) resizepart 3 4000GB
    (parted) quit

    (then do the same for sdb)

  6. Let the raid array md2 grow:
    mdadm --grow /dev/md2 --size=max
  7. Check the raid status:
    cat /proc/mdstat
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          [===================>.]  resync = 97.8% (3807048128/3888815376) finish=13.3min speed=101952K/sec
          bitmap: 1/15 pages [4KB], 131072KB chunk
  8. Wait until the resync finished (took a few hours for 2 TB):
    cat /proc/mdstat                                                                                                                                        
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 131072KB chunk
  9. Check the filesystem:
    e2fsck -fv /dev/md2
  10. Resize the filesystem:
    resize2fs /dev/md2
  11. Reboot into your normal system and you are done:
  12. Check the free disk space:
    df -h

    It should give the new size of / (3.6 TB) and a lot more free space 🙂

Hope this helps somebody with the same or similar issue.


25. July 2016

Roundcube: Multiple identities (mail_domains) automatically created by default when user is created

The problem

Roundcube is a popular webmailer written in PHP. Today I configured it on a host where each user has multiple identities, one for each domain: E.g., user bob has the three identities:, and

The same for all other users.

When the user logs in for the first time in roundcube, all three identities should be automatically created. There is a config variable mail_domain, which can be used like this:

$config['mail_domain'] = '';

This will make sure that when user bob logs in, the created identity will be . You can give an array here, but only one domain per IMAP host. So using mail_domain, it seems that you currently cannot define multiple domains that will create different identities.

The solution

The solution that I found uses the plugin virtuser_query. This plugin lets you query mail identities from an SQL database. But within the SQL query, you can also statically configure all the domains like this:

$config['virtuser_query'] = array('email' => 
'user' => '', 'host' => '', 'alias' => '');

The plugin replaces %u with the username, e.g. bob. So this query will always return three rows, one for each domain. It is not really querying the database, it is just a workaround to configure multiple default domains in roundcube, as mail_domain currently does not allow us to do this.

For the plugin to work, you need to enable it like this:

$config['plugins'] = array(...,'virtuser_query'); 
// note: there is an underscore after virtuser

In my case the virtuser_query plugin was already included in Roundcube as I have installed the Debian package roundcube-plugins . If it is not already part of your roundcube installation, you can get in here.

I hope this helps somebody to solve this issue faster 🙂
Please leave a comment if it helped you, if you have problems with this, or if you found another way.

15. July 2016

Request-Tracker4: Enabling fast fulltext search with Sphinxsearch on Debian Jessie

I just successfully made fulltext search in our request tracker super fast. This is a guide for Debian Jessie and request tracker set up to use MySQL (or MariaDB).

  • Don’t forget to backup your RT database:
    mysqldump -p -u root rtdb > rtdb.sql
  • Log out of request tracker. I had some strange issue because I did not 😉
  • Make sure you are using request-tracker4 version 4.2.12 from jessie-backports. If you don’t already, this is how you update:
    – add this to the end of your /etc/apt/sources.list:

    deb jessie-backports main

    – run

    apt-get update

    – install request-tracker4 from jessie-backports:

    apt-get -t jessie-backports install request-tracker4

    – follow the update procedure, make sure your database is upgraded
    (We need the package from backports as jessie’s rt4-db-mysql package depends on mysql-client, whereas the backports version depends on virtual-mysql-client and thus can also be easily used with MariaDB.)

  • Second, make sure you are using MariaDB instead of MySQL. MariaDB is a drop-in replacement for MySQL. The big advantage of MariaDB here is that it comes with the SphinxSE storage engine that we will use for the fulltext search. SphinxSE is also available for MySQL, but you would need to compile it yourself whereas MariaDB includes SphinxSE by default. MariaDB has also a lot more advantages. Switching to MariaDB is very simple, no need to convert databases or configuration. I used aptitude and selected to install mariadb-client and maridb-server. Then I checked the dependency resolutions and choose the option to uninstall mysql-client, mysql-server and related packages. Then everything went smooth automatically, I just needed to enter the root password again, everything else was updated automatically.
  • Now check if your RT is still working well with MariaDB.
  • Now enable the SphinxSE storage engine in MariaDB:
    mysql -u root -p
    [enter password]
    INSTALL SONAME 'ha_sphinx';
  • Now install the Sphinxsearch daemon:
    apt-get install sphinxsearch
  • Now set up indexed full text search in RT:
    rt-setup-fulltext-index --dba root --dba-password secret

    (Let me know if you find a way without passing the password as a parameter)
    This asks you what type of index you want to use. Enter “sphinx”. Then keep the defaults.

  • The command above gave you code for /etc/request-tracker4/ – like this:
    Set( %FullTextSearch,
        Enable     => 1,
        Indexed    => 1,
        MaxMatches => '10000',
        Table      => 'AttachmentsIndex',

    Add / adjust this in

  • Then you need to configure the index in sphinx. The above tool also gave you a config file for this, but it does not work with the current sphinxsearch version of jessie. So here is what works. Create an /etc/sphinxsearch/sphinx.conf like this:
    source rt {
        type            = mysql
        sql_host        = localhost
        sql_db          = rtdb
        sql_user        = rtuser
        sql_pass        = RT_PASSWORD
        sql_query_pre   = SET NAMES utf8
        sql_query       = \
            SELECT, t.Subject, a.content FROM Attachments a \
            JOIN Transactions txn ON a.TransactionId = AND txn.ObjectType = 'RT::Ticket' \
            JOIN Tickets t ON txn.ObjectId = \
            WHERE a.ContentType LIKE 'text/%' AND t.Status != 'deleted'
    #    sql_query_info  = SELECT * FROM Attachments WHERE id=$id
    index rt {
        source                  = rt
        path                    = /var/lib/sphinxsearch/data/rt
        docinfo                 = extern
    #    charset_type            = utf-8
    indexer {
        mem_limit               = 32M
    searchd {
        listen                  = 3312
        log                     = /var/log/sphinxsearch/searchd.log
        query_log               = /var/log/sphinxsearch/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinxsearch/
    #    max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 0
        unlink_old              = 1
        # For sphinx >= 1.10:
        binlog_path             = /var/lib/sphinxsearch/data

    Make sure to adjust the mysql database, user and password.
    Note that there are some differences to the config provided by rt-setup-fulltext-index. Especially, you need to use listen instead of port. If you don’t adjust this, sphinxsearch will start on a different port and RT will not find any results in fulltext searches! If your RT is not behind a firewall, better also set listen to . Also, I used the Debian style paths from the Debian sample. The fields commented out are not necessary anymore in current sphinxsearch.
    UPDATE 26.08.2016: I changed the SQL Query to also index the Subject of the tickets and index all ‘text/%’ types and not only ‘text/plain’. This change is based on a comment by Claes Merin in the Request Tracker Mailing List. Thanks a lot. Without this change, the fulltext search will only search within ticket content, but not within ticket titles… Also, before it would not index text/html attachments.

  • Now run the indexer to build the initial index. This will take some time, for a 6 GB Attachments Table, it took not more than 30 minutes. Maybe run this in a screen in case your ssh connection breaks:
    indexer rt
  • Enable the Sphinx daemon. To do so, adjust START to yes in /etc/default/sphinxsearch
  • Finally, start the sphinxsearch daemon:
    /etc/init.d/sphinxsearch start
  • Now you can log into RT and do a fulltext search by entering in the search field:
  • In my case, this made the fulltext searches run within a second instead of 5 minutes 🙂
  • Make sure the indexer is run regularily to update the index. To do so, enter
    crontab -e

    And add a line like this:

    13 * * * * /usr/bin/indexer rt --rotate

I hope this helps somebody to setup this a little faster.




28. June 2016

flashplugin-installer cannot download files using unattended-upgrades on Ubuntu

Filed under: Linux — Tags: , , , , , , — Christopher Kramer @ 10:29

Using unattended-upgrades on Ubuntu  16.04 (and previous versions) set up to install updates on shutdown, flashplugin-installer always failed when downloading the new version. The reason probably is that on shutdown, the Internet connection is not available anymore. Up to now, I always updated flashplugin-installer manually. But today, I found a nice and simple solution:

Just get rid of the installer! In the Cononical partner repository, there is a package that directly contains the flash plugin. First, make sure you include the partner repository in your /etc/apt/sources.list, e.g. for xenial:

deb xenial partner
deb-src xenial partner

Then, update, get rid of the flashplugin-installer, and install the adobe-flashplugin instead:

sudo apt-get update
sudo apt-get remove flashplugin-installer
sudo apt-get install adobe-flashplugin

Hope this helps someone who also has problems with unattended upgrades and falshplugin-installer.

17. June 2016

Multimedia Keys “stuck” after Ubuntu 16.04 upgrade

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 21:19

After upgrading Ubuntu Gnome to 16.04, the multimedia keys / extra keys (like play/pause, volume up/down, mute, calculator, forward, backward, etc.) of my old Cherry G86-21050 DEAAAA keyboard showed a strange behavior: Whenever one of the buttons was pressed, its effect was repeated over and over, just as if the button was stuck. So pressing pause would play and pause the song over and over, pressing calculator would open hundreds of calculator windows. Pressing another special keys stopped the previous effect from repeating, but repeated the effect of the last special key over and over. The normal keys (letter, numbers etc.) were not affected.

As the effect appeared after the upgrade for all special keys, I was sure it was a software problem and not a hardware problem.

Digging around with evtest (no, it is not a good idea to run evtest –grab on your keyboard in a tty :D), I found out that the key events are not repeated multiple times, but the keys are not released.

Annoyed, I tried another keyboard (Logitech Elite Keyboard) for comparison. The special keys of that one worked well.

Finally, I accidentally plugged the “faulty” Cherry Keyboard in the USB port that I had the working Logitech in between and voila: The Cherry’s special keys worked again.

So lesson learned: Have you tried plugging it out and in again? Is the first question you should ask (yourself) whenever some USB device behaves strangely.

19. May 2016

Ubuntu Gnome: Upgrading to 16.04 Xenial Xerus fails: ubuntu-gnome-desktop (gdm, gnome-session, adwaita-icon-theme and adwaita-icon-theme-full)

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 13:26

Today I upgraded my Ubuntu Gnome 15.10 to 16.04 Xenial Xerus. I was surprised to find in the details that it removes gdm. I started the update and it went normal until it said ubuntu-gnome-desktop was broken due to unresolved dependencies. The GUI upgrade process that I had used closed and I was left with a system upgraded by half. But it was still running, so I started a terminal and started aptitude. It suggested to install various packages to resolve dependencies, especially gdm3, gnome-shell, gnome-shell-extensions and gnome-session. It also proposed to remove unused packages, which were not removed as the upgrade process had stopped half way. I followed aptitudes suggestions, but it left two packages not upgraded: adwaita-icon-theme and gnome-session.

And this took me quite a while to solve. There are two packages: adwaita-icon-theme and adwaita-icon-theme-full. The first one contains a subset of icons, whereas the second one contains all icons. The ubuntu-gnome-desktop requires the adwaita-icon-theme-full package, which in turn requires the adwaita-icon-theme package. The adwaita-icon-theme had an upgrade from version 3.18.0-2ubuntu3 to 3.18.0-2ubuntu3.1. The adwaita-icon-theme-full package requires that it is exactly the same version as adwaita-icon-theme, but the only version that apt found was 3.18.0-2ubuntu3.

Then I found out that adwaita-icon-theme-full is in the universe repository, whereas adwaita-icon-theme is in main. In my sources.list, I had the following:

deb xenial-proposed main restricted
deb xenial-proposed main restricted

Therefore, the new version of adwaita-icon-theme was found in the main repository. But universe (and multiverse) repositories were missing  the proposed level, and this was the reason why the new version of adwaita-icon-theme-full was not found by apt. So the solution was simple: Just changing the above lines in sources.list like this, adding universe and multiverse repositories:

deb xenial-proposed main restricted universe multiverse
deb xenial-proposed main restricted universe multiverse

Then it updated the remaining packages normally using:

aptitude update
aptitude safe-upgrade

So lesson learned is:
If you add the proposed level in your sources.list, make sure you add it for all repositories you use, e.g. including universe and multiverse, not only main.

Hope this helps somebody who runs into the same issue.

19. April 2016

Linux: Find the MAC address of your wireless adapter like wlan0

Filed under: Linux — Tags: , , , , , , , , , , , , — Christopher Kramer @ 14:57

You tried ifconfig and iwconfig, but they all don’t show you the MAC address of your wireless adapter under Linux? For me, on Arch Linux and Ubuntu. This is what gives you what you want and its so beautifully short and simple:

ip addr

The MAC address comes after “link/ether”. This works for wired ethernet adapters (eth0 etc.), wireless adapters (wlan0 etc.) and any other ethernet device  (e.g. IEEE 802.11s mesh point devices).

If you are on Windows, read my old blog post about getting the MAC address on Windows.

Hope this helps somebody to find it a little faster.

Older Posts »