DeutschEnglish

Submenu

 - - - By CrazyStat - - -

14. December 2016

Microsoft and AOL sending DMARC aggregate reports (RUA) without DKIM signature

Filed under: Server Administration — Tags: , , , , , , , — Christopher Kramer @ 19:18

Just noticed that Microsoft and AOL are sending their DMARC aggregate reports without a DKIM signature. This alone is not uncommon, lots of small mail providers don’t use DKIM for their DMARC aggregate report mails. But the domains that Microsoft and AOL use as sender of their aggregate report mails have DMARC policies themselves. AOL even sets p=reject in their policy, which means that a mail server checking DMARC policies would reject aggregate reports sent by AOL if SPF fails, e.g. because the mail was relayed in between.

This is their DMARC policies:

_dmarc.aol.com.         2083    IN      TXT     
     "v=DMARC1\; p=reject\; pct=100\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\;"

_dmarc.microsoft.com.   391     IN      TXT     
     "v=DMARC1\; p=quarantine\; pct=100\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\; fo=1"

And this are the SPF records:

aol.com.                3190    IN      TXT     
     "v=spf1 ptr:mx.aol.com include:spf.constantcontact.com include:aspmx.sailthru.com include:mail.zendesk.com ~all"

abuse.aol.com.          227     IN      TXT     
     "v=spf1 ip4:204.29.186.192/26 ip4:204.29.187.0/27 -all"

microsoft.com.          1683    IN      TXT     
 "v=spf1 include:_spf-a.microsoft.com include:_spf-b.microsoft.com include:_spf-c.microsoft.com include:_spf-ssg-a.microsoft.com include:spf-a.hotmail.com ip4:147.243.128.24 ip4:147.243.128.26 ip4:147.243.1.153 ip4:147.243.1.47 ip4:147.243.1.48 -all"

AOL is sending its aggregate reports from abuse_dmarc@abuse.aol.com, and the SPF records of this subdomain has a strict “-all” (which aol.com doesn’t). Microsoft sends the reports from dmarcrep@microsoft.com, and microsoft.com also uses “-all”.

Google for example sends its aggregate reports with DKIM signature, so DMARC evaluation runs smooth. SPF is relaxed “~all” at google.com, but DMARC nowadays is “p=reject”, which I was a little surprised to see.

So in conclusion, if you really want to get your DMARC aggregate reports, you should not strictly follow the DMARC policies of the others, otherwise you might miss some reports…

And if you set up a mail server yourself that sends DMARC reports, don’t forget to sign those mails with DKIM as well.

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

15. November 2016

Typo3 Mask – Javascript and CSS includes of extensions missing

Filed under: JavaScript,Typo3 — Tags: , , , , , , — Christopher Kramer @ 11:36

I was using Mask 2.1.1 with Typo3 7.6.11 for a site as described in the Mask manual with this typoscript:

page < temp.mask.page
page.10.file.stdWrap.cObject.default.value = fileadmin/templates/index.html
page {
        includeCSS.styles = fileadmin/templates/style.css
}

The problem with this was that the CSS and JS files includes from extensions where missing, even though the static templates of this where included. The reason is that this typoscript overwrites the page.include* stuff defined by the templates.

A workaround

To solve this, I copied the page.include* stuff defined by the extensions in a temporary variable and restored it after loading the mask object into page:

tempJSFooterlibs < page.includeJSFooterlibs
tempJSFooter < page.includeJSFooter
tempCSS < page.includeCSS

page < temp.mask.page
page.10.file.stdWrap.cObject.default.value = fileadmin/templates/index.html
page {
        includeJSFooterlibs < tempJSFooterlibs
        includeJSFooter < tempJSFooter
        includeCSS < tempCSS

        includeCSS.styles = fileadmin/templates/style.css
}

There are more includes you might need to consider if your extensions use these:

page.includeJS
page.includeJSlibs
page.jsInline
page.jsFooterInline

Hope this helps someone with the same problem! Please let me know if there is an easier solution.

Update 06.07.2017: The clean way

So the workaround described above works, but it has a disadvantage: You might install a plugin that defines some other page.* property that gets lost. For example, I just had this problem with mindshape_cookie_hint: It defines page.9877 which gets lost when you overwrite page with temp.mask.page. It took quite some time until I found the reason for this problem. And then I found another way which is much simpler than the one described before:

page = PAGE
page.10 < temp.mask.page.10

Just don’t copy the whole temp.mask.page, only temp.mask.page.10. This will preserve all the stuff defined by plugins. The solution is so simple and easy!

So I think this is not really a bug in mask, it is rather a mistake in the documentation.

Please let me know in the comments if this saved your day or if you have suggestions.

17. August 2016

Linux Software Raid: Enlarge Raid Array of Hetzner EX41

Filed under: Linux,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 21:15

Recently, I ordered an EX41 server at Hetzner, which by default comes with two 4 TB HDDs in a software RAID1. I chose the minimal Debian Jessie Image. From Hetzner’s EX40, which has two 2 TB drives in a Software RAID1, I was used to find the whole drive formatted as one partition by Hetzner. But with the EX41, it turned out that the 4 TB were split: The root partition / was 2 TB of size and 1.7 TB were allocated to /home. As the application that should run on this server needs more than 2 TB space in one folder, I repartitioned the server. If somebody else has a similar issue, here is how it can be done:

  1. Edit /etc/fstab and remove or comment (by placing a # at the beginning) the line that mounts /home from /dev/md3
  2. Reboot into the Rescue system or some other system that is not on the same drive. (The Hetzner rescue system boots from network. On a local system, you would boot a linux from CD or USB.)
  3. Check the Raid status and setup:
    cat /proc/mdstat
    
    md3 : active raid1 sda4[0] sdb4[1]
          1777751872 blocks super 1.2 [2/2] [UU]
          bitmap: 0/14 pages [0KB], 65536KB chunk
    
    md2 : active raid1 sda3[0] sdb3[1]
          2111700992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/16 pages [0KB], 65536KB chunk
    
    md1 : active raid1 sda2[0] sdb2[1]
          523712 blocks super 1.2 [2/2] [UU]
    
    md0 : active raid1 sda1[0] sdb1[1]
          16760832 blocks super 1.2 [2/2] [UU]

    So here md3 is the raid array that we want to get rid of together with its partitions sda4 and sdb4. md2 the raid array that we want to grow in size.

  4. First let’s get rid of the md3:
    mdadm --stop /dev/md3
    mdadm --zero-superblock /dev/sda4
    mdadm --zero-superblock /dev/sdb4
    parted /dev/sda rm 4
    parted /dev/sdb rm 4
  5. Resize the partitions of md2 (sda3 and sdb3):
    (Adjust the end if necessary)

    parted /dev/sda
    (parted) resize 3
    (END?) 4000GB
    (parted) quit
    
    parted /dev/sdb
    (parted) resize 3
    (END?) 4000GB
    (parted) quit

    Note: Tomas pointed out in the comments that the resize-command was removed and replaced with resizepart in current versions of parted. So the command is now:

    parted /dev/sda
    (parted) resizepart 3 4000GB
    (parted) quit

    (then do the same for sdb)

  6. Let the raid array md2 grow:
    mdadm --grow /dev/md2 --size=max
  7. Check the raid status:
    cat /proc/mdstat
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          [===================>.]  resync = 97.8% (3807048128/3888815376) finish=13.3min speed=101952K/sec
          bitmap: 1/15 pages [4KB], 131072KB chunk
  8. Wait until the resync finished (took a few hours for 2 TB):
    cat /proc/mdstat                                                                                                                                        
    Personalities : [raid1]
    md2 : active raid1 sda3[0] sdb3[1]
          3888815376 blocks super 1.2 [2/2] [UU]
          bitmap: 0/15 pages [0KB], 131072KB chunk
  9. Check the filesystem:
    e2fsck -fv /dev/md2
  10. Resize the filesystem:
    resize2fs /dev/md2
  11. Reboot into your normal system and you are done:
    reboot
  12. Check the free disk space:
    df -h

    It should give the new size of / (3.6 TB) and a lot more free space 🙂

Hope this helps somebody with the same or similar issue.

 

25. July 2016

Roundcube: Multiple identities (mail_domains) automatically created by default when user is created

The problem

Roundcube is a popular webmailer written in PHP. Today I configured it on a host where each user has multiple identities, one for each domain: E.g., user bob has the three identities:
bob@example1.org, bob@example2.org and bob@example3.org

The same for all other users.

When the user logs in for the first time in roundcube, all three identities should be automatically created. There is a config variable mail_domain, which can be used like this:

$config['mail_domain'] = 'example1.org';

This will make sure that when user bob logs in, the created identity will be bob@example1.org . You can give an array here, but only one domain per IMAP host. So using mail_domain, it seems that you currently cannot define multiple domains that will create different identities.

The solution

The solution that I found uses the plugin virtuser_query. This plugin lets you query mail identities from an SQL database. But within the SQL query, you can also statically configure all the domains like this:

$config['virtuser_query'] = array('email' => 
'SELECT "%u@example1.org" UNION 
SELECT "%u@example2.org" UNION 
SELECT "%u@example3.org"',
'user' => '', 'host' => '', 'alias' => '');

The plugin replaces %u with the username, e.g. bob. So this query will always return three rows, one for each domain. It is not really querying the database, it is just a workaround to configure multiple default domains in roundcube, as mail_domain currently does not allow us to do this.

For the plugin to work, you need to enable it like this:

$config['plugins'] = array(...,'virtuser_query'); 
// note: there is an underscore after virtuser

In my case the virtuser_query plugin was already included in Roundcube as I have installed the Debian package roundcube-plugins . If it is not already part of your roundcube installation, you can get in here.

I hope this helps somebody to solve this issue faster 🙂
Please leave a comment if it helped you, if you have problems with this, or if you found another way.

15. July 2016

Request-Tracker4: Enabling fast fulltext search with Sphinxsearch on Debian Jessie

I just successfully made fulltext search in our request tracker super fast. This is a guide for Debian Jessie and request tracker set up to use MySQL (or MariaDB).

  • Don’t forget to backup your RT database:
    mysqldump -p -u root rtdb > rtdb.sql
  • Log out of request tracker. I had some strange issue because I did not 😉
  • Make sure you are using request-tracker4 version 4.2.12 from jessie-backports. If you don’t already, this is how you update:
    – add this to the end of your /etc/apt/sources.list:

    deb http://ftp.debian.org/debian jessie-backports main

    – run

    apt-get update

    – install request-tracker4 from jessie-backports:

    apt-get -t jessie-backports install request-tracker4

    – follow the update procedure, make sure your database is upgraded
    (We need the package from backports as jessie’s rt4-db-mysql package depends on mysql-client, whereas the backports version depends on virtual-mysql-client and thus can also be easily used with MariaDB.)

  • Second, make sure you are using MariaDB instead of MySQL. MariaDB is a drop-in replacement for MySQL. The big advantage of MariaDB here is that it comes with the SphinxSE storage engine that we will use for the fulltext search. SphinxSE is also available for MySQL, but you would need to compile it yourself whereas MariaDB includes SphinxSE by default. MariaDB has also a lot more advantages. Switching to MariaDB is very simple, no need to convert databases or configuration. I used aptitude and selected to install mariadb-client and maridb-server. Then I checked the dependency resolutions and choose the option to uninstall mysql-client, mysql-server and related packages. Then everything went smooth automatically, I just needed to enter the root password again, everything else was updated automatically.
  • Now check if your RT is still working well with MariaDB.
  • Now enable the SphinxSE storage engine in MariaDB:
    mysql -u root -p
    [enter password]
    INSTALL SONAME 'ha_sphinx';
    SHOW ENGINES;
    EXIT;
  • Now install the Sphinxsearch daemon:
    apt-get install sphinxsearch
  • Now set up indexed full text search in RT:
    rt-setup-fulltext-index --dba root --dba-password secret

    (Let me know if you find a way without passing the password as a parameter)
    This asks you what type of index you want to use. Enter “sphinx”. Then keep the defaults.

  • The command above gave you code for /etc/request-tracker4/RT_SiteConfig.pm – like this:
    Set( %FullTextSearch,
        Enable     => 1,
        Indexed    => 1,
        MaxMatches => '10000',
        Table      => 'AttachmentsIndex',
    );

    Add / adjust this in RT_SiteConfig.pm.

  • Then you need to configure the index in sphinx. The above tool also gave you a config file for this, but it does not work with the current sphinxsearch version of jessie. So here is what works. Create an /etc/sphinxsearch/sphinx.conf like this:
    source rt {
        type            = mysql
    
        sql_host        = localhost
        sql_db          = rtdb
        sql_user        = rtuser
        sql_pass        = RT_PASSWORD
    
        sql_query_pre   = SET NAMES utf8
        sql_query       = \
            SELECT a.id, t.Subject, a.content FROM Attachments a \
            JOIN Transactions txn ON a.TransactionId = txn.id AND txn.ObjectType = 'RT::Ticket' \
            JOIN Tickets t ON txn.ObjectId = t.id \
            WHERE a.ContentType LIKE 'text/%' AND t.Status != 'deleted'
    
    #    sql_query_info  = SELECT * FROM Attachments WHERE id=$id
    }
    
    index rt {
        source                  = rt
        path                    = /var/lib/sphinxsearch/data/rt
        docinfo                 = extern
    #    charset_type            = utf-8
    }
    
    indexer {
        mem_limit               = 32M
    }
    
    searchd {
        listen                  = 3312
        log                     = /var/log/sphinxsearch/searchd.log
        query_log               = /var/log/sphinxsearch/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinxsearch/searchd.pid
    #    max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 0
        unlink_old              = 1
        # For sphinx >= 1.10:
        binlog_path             = /var/lib/sphinxsearch/data
    }

    Make sure to adjust the mysql database, user and password.
    Note that there are some differences to the config provided by rt-setup-fulltext-index. Especially, you need to use listen instead of port. If you don’t adjust this, sphinxsearch will start on a different port and RT will not find any results in fulltext searches! If your RT is not behind a firewall, better also set listen to 127.0.0.1:3312 . Also, I used the Debian style paths from the Debian sample. The fields commented out are not necessary anymore in current sphinxsearch.
    UPDATE 26.08.2016: I changed the SQL Query to also index the Subject of the tickets and index all ‘text/%’ types and not only ‘text/plain’. This change is based on a comment by Claes Merin in the Request Tracker Mailing List. Thanks a lot. Without this change, the fulltext search will only search within ticket content, but not within ticket titles… Also, before it would not index text/html attachments.

  • Now run the indexer to build the initial index. This will take some time, for a 6 GB Attachments Table, it took not more than 30 minutes. Maybe run this in a screen in case your ssh connection breaks:
    indexer rt
  • Enable the Sphinx daemon. To do so, adjust START to yes in /etc/default/sphinxsearch
  • Finally, start the sphinxsearch daemon:
    /etc/init.d/sphinxsearch start
  • Now you can log into RT and do a fulltext search by entering in the search field:
    fulltext:search_word
  • In my case, this made the fulltext searches run within a second instead of 5 minutes 🙂
  • Make sure the indexer is run regularily to update the index. To do so, enter
    crontab -e

    And add a line like this:

    13 * * * * /usr/bin/indexer rt --rotate

I hope this helps somebody to setup this a little faster.

 

 

 

28. June 2016

flashplugin-installer cannot download files using unattended-upgrades on Ubuntu

Filed under: Linux — Tags: , , , , , , — Christopher Kramer @ 10:29

Using unattended-upgrades on Ubuntu  16.04 (and previous versions) set up to install updates on shutdown, flashplugin-installer always failed when downloading the new version. The reason probably is that on shutdown, the Internet connection is not available anymore. Up to now, I always updated flashplugin-installer manually. But today, I found a nice and simple solution:

Just get rid of the installer! In the Cononical partner repository, there is a package that directly contains the flash plugin. First, make sure you include the partner repository in your /etc/apt/sources.list, e.g. for xenial:

deb http://archive.canonical.com/ubuntu xenial partner
deb-src http://archive.canonical.com/ubuntu xenial partner

Then, update, get rid of the flashplugin-installer, and install the adobe-flashplugin instead:

sudo apt-get update
sudo apt-get remove flashplugin-installer
sudo apt-get install adobe-flashplugin

Hope this helps someone who also has problems with unattended upgrades and falshplugin-installer.

17. June 2016

Multimedia Keys “stuck” after Ubuntu 16.04 upgrade

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 21:19

After upgrading Ubuntu Gnome to 16.04, the multimedia keys / extra keys (like play/pause, volume up/down, mute, calculator, forward, backward, etc.) of my old Cherry G86-21050 DEAAAA keyboard showed a strange behavior: Whenever one of the buttons was pressed, its effect was repeated over and over, just as if the button was stuck. So pressing pause would play and pause the song over and over, pressing calculator would open hundreds of calculator windows. Pressing another special keys stopped the previous effect from repeating, but repeated the effect of the last special key over and over. The normal keys (letter, numbers etc.) were not affected.

As the effect appeared after the upgrade for all special keys, I was sure it was a software problem and not a hardware problem.

Digging around with evtest (no, it is not a good idea to run evtest –grab on your keyboard in a tty :D), I found out that the key events are not repeated multiple times, but the keys are not released.

Annoyed, I tried another keyboard (Logitech Elite Keyboard) for comparison. The special keys of that one worked well.

Finally, I accidentally plugged the “faulty” Cherry Keyboard in the USB port that I had the working Logitech in between and voila: The Cherry’s special keys worked again.

So lesson learned: Have you tried plugging it out and in again? Is the first question you should ask (yourself) whenever some USB device behaves strangely.

19. May 2016

Ubuntu Gnome: Upgrading to 16.04 Xenial Xerus fails: ubuntu-gnome-desktop (gdm, gnome-session, adwaita-icon-theme and adwaita-icon-theme-full)

Filed under: Linux — Tags: , , , , , , , — Christopher Kramer @ 13:26

Today I upgraded my Ubuntu Gnome 15.10 to 16.04 Xenial Xerus. I was surprised to find in the details that it removes gdm. I started the update and it went normal until it said ubuntu-gnome-desktop was broken due to unresolved dependencies. The GUI upgrade process that I had used closed and I was left with a system upgraded by half. But it was still running, so I started a terminal and started aptitude. It suggested to install various packages to resolve dependencies, especially gdm3, gnome-shell, gnome-shell-extensions and gnome-session. It also proposed to remove unused packages, which were not removed as the upgrade process had stopped half way. I followed aptitudes suggestions, but it left two packages not upgraded: adwaita-icon-theme and gnome-session.

And this took me quite a while to solve. There are two packages: adwaita-icon-theme and adwaita-icon-theme-full. The first one contains a subset of icons, whereas the second one contains all icons. The ubuntu-gnome-desktop requires the adwaita-icon-theme-full package, which in turn requires the adwaita-icon-theme package. The adwaita-icon-theme had an upgrade from version 3.18.0-2ubuntu3 to 3.18.0-2ubuntu3.1. The adwaita-icon-theme-full package requires that it is exactly the same version as adwaita-icon-theme, but the only version that apt found was 3.18.0-2ubuntu3.

Then I found out that adwaita-icon-theme-full is in the universe repository, whereas adwaita-icon-theme is in main. In my sources.list, I had the following:

deb http://de.archive.ubuntu.com/ubuntu/ xenial-proposed main restricted
deb http://de.archive.ubuntu.com/ubuntu/ xenial-proposed main restricted

Therefore, the new version of adwaita-icon-theme was found in the main repository. But universe (and multiverse) repositories were missing  the proposed level, and this was the reason why the new version of adwaita-icon-theme-full was not found by apt. So the solution was simple: Just changing the above lines in sources.list like this, adding universe and multiverse repositories:

deb http://de.archive.ubuntu.com/ubuntu/ xenial-proposed main restricted universe multiverse
deb http://de.archive.ubuntu.com/ubuntu/ xenial-proposed main restricted universe multiverse

Then it updated the remaining packages normally using:

aptitude update
aptitude safe-upgrade

So lesson learned is:
If you add the proposed level in your sources.list, make sure you add it for all repositories you use, e.g. including universe and multiverse, not only main.

Hope this helps somebody who runs into the same issue.

19. April 2016

Linux: Find the MAC address of your wireless adapter like wlan0

Filed under: Linux — Tags: , , , , , , , , , , , , — Christopher Kramer @ 14:57

You tried ifconfig and iwconfig, but they all don’t show you the MAC address of your wireless adapter under Linux? For me, on Arch Linux and Ubuntu. This is what gives you what you want and its so beautifully short and simple:

ip addr

The MAC address comes after “link/ether”. This works for wired ethernet adapters (eth0 etc.), wireless adapters (wlan0 etc.) and any other ethernet device  (e.g. IEEE 802.11s mesh point devices).

If you are on Windows, read my old blog post about getting the MAC address on Windows.

Hope this helps somebody to find it a little faster.

16. March 2016

Typo3: Redirect backend to https and frontend to http

Filed under: PHP,Security,Server Administration,Typo3 — Tags: , , , , , , , , , — Christopher Kramer @ 20:02

If you are using SNI to secure the access to your site through SSL, you might decide that you do not want to use SSL for the frontend, as users of old clients such as Android 2 or Internet Explorer 8 on Windows XP won’t be able to access the site. But for the backend, old clients should not be a problem.

So this solves the redirection in two ways:

  1. Frontend gets redirected to HTTP if accessed through https (optional)
  2. Backend gets redirected to HTTPS if accessed through http

Put this in your .htaccess after “RewriteEngine On” (assuming Apache webserver):

# 1. optionally: redirect FE to http
RewriteCond %{HTTPS} on
RewriteCond %{REQUEST_URI} !^/?typo3
RewriteRule (.*) http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

# 2. redirect BE to https
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} ^/?typo3
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

And hopefully, your backend access is now securely encrypted…

https

 

« Newer PostsOlder Posts »