DeutschEnglish

Submenu

 - - - By CrazyStat - - -

8. November 2019

rspamd and redis: Random Connection refused errors

Filed under: Server Administration — Tags: , , , , , — Christopher Kramer @ 19:35

For some time, I am using rspamd as Spamfilter. By the way, I can really recommend it over Spamassasin / Amavisd-new solutions. I configured rspamd to use a redis database for the bayes filter, neural network etc.

Everything worked well until after some time, probably due to an update, it started spitting out Connection refused errors like these:

2019-11-07 06:25:10 #32043(normal) rspamd_redis_stat_keys: cannot get keys to gather stat: Connection refused
2019-11-07 06:25:15 #32042(normal) <4n3xhp>; lua; neural.lua:855: cannot get ANN data from key: rn_default_default_t66yaxz4_0; Connection refused
2019-11-07 06:25:15 #32044(normal) <4n3xhp>; lua; neural.lua:1101: cannot get ANNs list from redis: Connection refused
2019-11-07 06:25:15 #32040(controller) <4n3xhp>; lua; neural.lua:1135: cannot exec invalidate script in redis: Connection refused

Not every connection to redis failed, but quite a lot. This is a Debian Stretch system with a local redis 3.2.6 database and rspamd 2.1 running on the same host. So network problems couldn’t be it. The redis logs did not even mention the failed connections, the MONITOR command of redis also did not show anything for the refused connections. Restarting redis didn’t change anything, but restarting rspamd solved the problem for about an hour, when the problem started again. I tried adjusting the linux open files limit, the number of maximum redis connections and timeout settings without any luck.

Finally, I found the setting that solved the problem. My redis configuration in /etc/rspamd/local.d/redis.conf had looked like this:

password = "somePassword";
write_servers = "localhost";
read_servers = "localhost";

Changing it to this solved the problem:

password = "somePassword";
write_servers = "127.0.0.1:6379";
read_servers = "127.0.0.1:6379";

The problem seems to be that localhost both resolves to an IPv4 address and an IPv6 address. Redis, however, is configured to only listen on the IPv4 address in /etc/redis/redis.conf:

bind 127.0.0.1

Rspamd seems to sometimes use the IPv4 address, which works, and sometimes the IPv6 address, which doesn’t. This is why the problem seems to occur randomly.

Hope this helps somebody to find the issue faster. If this saved your day, please drop a comment.

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

19. March 2019

PHP: DateTime::createFromFormat fails for string read from CSV

Filed under: PHP — Tags: , , , , , — Christopher Kramer @ 19:31

I wrote a small PHP script to import a dozen csv files exported from Excel into a database. The CSV import basically looked like this:

<?php
$f = "file.csv";	
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
    }
} ?>

And now what happened is that $date was false. So I fetched the errors like this:

var_dump(DateTime::getLastErrors());

And this returned:

array(4) {
  ["warning_count"]=>
  int(0)
  ["warnings"]=>
  array(0) {
  }
  ["error_count"]=>
  int(1)
  ["errors"]=>
  array(1) {
    [0]=>
    string(22) "Unexpected data found."
  }
}

So I added var_dump($date), but it gave string(13) "31.01.2019", which looked right. But looking closely, the string length of 13 seems a bit long for a 10 character date, right? I tried trim() , but without luck. And then I remembered that I had a similar problem before where invisible empty space was due to the UTF-8 Byte Order Mark (BOM). This is a sequence of “inivisible” bytes at the beginning of a textfile that define in which unicode encoding the file is (UTF-8, UTF-16, …) and its endianess (big-endian or little-endian). Microsoft Office programs such as Excel or Word like to write this to the beginning of a file, but other programs may do so as well.

So the solution is simple: In the first line, strip the BOM if it is there:

<?php
$f = "file.csv";
$bom = pack('CCC', 0xEF, 0xBB, 0xBF);
$firstline=true;
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         if ($firstline and substr($data[0], 0, 3) === $bom)
             $data[0] = substr($data[0], 3);
         $firstline=false;
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
    }
} ?>

So this just checks whether the first three bytes in the file match the UTF-8 BOM added by Excel and in case it detects them, it remove these bytes. Now the date parses fine. If your file has a different BOM, e.g. for UTF-16, you may need to change the definition of $bom. Just check your file in a hex-editor to find the first three bytes. This is PSPad, a great text-editor that includes a HEX-editor:

Note the first three bytes EF BB BF, which are the BOM.

If this helped you to solve your problem faster, please drop a comment below. This motivates me to keep writing these articles.

4. January 2019

Mit dem Navi Diesel-Fahrverbote umfahren

Filed under: Uncategorized — Tags: , , , , , — Christopher Kramer @ 20:14

Hamburg hat als erste deutsche Stadt Fahrverbote für Diesel (Euro 4 und älter) eingeführt. Betroffen sind zunächst nur ca. 600 Meter der Max-Brauer-Allee und ca. 1,6km der Stresemannstraße. Wer mit seinem Euro 4 Diesel in Hamburg unterwegs ist, muss also den Umleitungsschildern folgen. Praktischer wäre es, wenn das Navi die gesperrten Straßen direkt von sich aus vermeiden würde. Ich habe mir die Eigenschaften der betroffenen Straßen in OpenStreeMap angesehen. Sie haben aktuell das Attribut mit dem Schlüssel motor_vehicle:conditional und dem Wert no @ (fuel=diesel), wie hier ein Abschnitt der Max-Brauer-Allee:

Dieselfahrverbot in OpenStreetMap

Da die Daten also schon in OpenStreeMap vorliegen, kann man nun Navis, welche auf OpenStreetMap-Karten basieren, beibringen diese Straßen auf Wunsch zu umfahren.

Das beliebte OsmAnd bietet die Möglichkeit, die Routing-Profile leicht selbst anzupassen. Dazu muss nur eine routing.xml Datei angepasst und auf dem Smartphone abgelegt werden.

Ich habe mir die routing.xml von github heruntergeladen. Innerhalb des Routing-Profils für Autos (car) habe ich zunächst einen neuen Parameter eingefügt, mit dem man in der Oberfläche einstellen kann, ob Dieselfahrverbote gemeidet werden sollen oder nicht:

Als nächstes muss definiert werden, was geschehen soll, wenn der Benutzer die Option aktiviert hat. Dazu habe ich folgende Regel eingefügt, welche die Straßen mit dem oben genannten Parameter meidet:

Diese Regel lässt OsmAnd für Diesel gesperrte Straßen vermeiden.

Die angepasste routing.xml hier downloaden. Wie in der Datei selbst steht, muss man sie i.d.R. an eine der folgenden Orte ablegen:

  • /sdcard/Android/data/net.osmand.plus/files/
  • /storage/emulated/0/Android/data/net.osmand.plus/files
  • /data/user/0/net.osmand.plus/no_backup

Ich habe sie unter /storage/emulated/0/Android/data/net.osmand.plus/files abgelegt, da dieser Ordner bei mir existierte. OsmAnd dann schließen (zur Seite wischen) und neu öffnen. Nun sollte in den Routingoptionen die neue Option erscheinen:

Die Option in den Routingeinstellungen

Wenn man nun Routen innerhalb Hamburgs berechnet, die normalerweise die für Diesel gesperrten Routen nutzen würden, kann man sehen, dass es funktioniert. Hier eine Route, die beide Straßen nutzt:

Route in Hamburg, auf der Dieselfahrverbote gelten

Aktiviert man nun die Option zum Meiden von Dieselfahrverboten, sieht die Route deutlich anders aus und meidet die gesperrten Straßen:

Alternative Route durch Hamburg ohne Dieselfahrverbote

Fahrverbote in Stuttgart und anderen Regionen

Ab dem 01.01.2019 gelten nun auch großflächige Dieselfahrverbote in Stuttgart. Leider funktioniert dieser Ansatz (aktuell) nicht für Stuttgart. Er vermeidet Straßen, welche das oben genannte Attribut aufweisen. In Stuttgart sind aber nicht einzelne Straßen, sondern das gesamte Stadtgebiet (genauer: die ausgewiesene Umweltzone) gesperrt. Ich habe erst heute in OpenStreetMap die bereits vorhandene Markierung der Umweltzone mit dem obigen Attribut versehen. Bis dies in den OsmAnd-Karten landet, wird es noch bis zu einen Monat dauern (außer man hat das Karten-Abo). Da es sich bei der Stuttgarter Umweltzone aber um eine Fläche und keine Straße handelt, werden wahrscheinlich noch Anpassungen an der routing.xml nötig, damit OsmAnd die Dieselfahrverbote in Stuttgart berücksichtigen kann. Sobald es hierzu neues gibt, werde ich diesen Artikel aktualisieren.

29. August 2018

phpBB: Upgrade fails (timeout) – solution and how to update on the CLI

Filed under: PHP,Server Administration — Tags: , , , , , , , — Christopher Kramer @ 10:46

So if you are getting a Timeout when updating phpBB to 3.2.2, it tells you to either increase the maximum time limit or do the update on the CLI.

How to update on the CLI

Of course, this requires that you have access to the CLI.

  • On the CLI, go into the install folder of phpBB
  • Create a file named config.yml that contains this:
    updater:
        type: db_only
  • Make sure the above file does not end with newlines
  • Run:
    php ./phpbbcli.php update config.yml

Fixing the update timeout problem

In my case, the CLI update gave this error:

PHP Fatal error:  Call to a member function fetch_array() on resource in [...]/install/update/new/phpbb/db/migration/data/v32x/fix_user_styles.php on line 42

This is a bug in the migration script. The most easy way to fix it, is to go into your config.php and change the $dbms from mysql to mysqli. This is recommended anyway.

If this is not possible for you, open the mentioned file and search for:

$enabled_styles = $result->fetch_array();

And replace this with:

$enabled_styles = $this->db->sql_fetchrowset($result);

Thanks to RMcGirr83 and Marc on this thread.

24. August 2018

Clementine Music Player needs to be started twice, no window on first start

Filed under: Linux — Tags: , , , , , , , , — Christopher Kramer @ 15:01

clementine music player logoIn Ubuntu 18.04, I start Clementine Music Player, but no window is coming up. But once I start Clementine another time, it comes up.

This is not a bug, it is a setting. So this is what you want:

Tools -> Preferences -> Behaviour -> When Clementine starts -> Always show the main window

No, you are not alone, it is you, me, and a bunch of other people.

9. August 2018

Debian Unattended Upgrades: Upgrade Third Party mono packages (sury.org, icinga)

When using the official third-party repository of mono in Debian, unattended Upgrades will not upgrade the mono packages unless you allow the origin. To do so, edit your /etc/apt/apt.conf.d/50unattended-upgrades , which may look like this:

Unattended-Upgrade::Origins-Pattern {
      "o=Debian,n=${distro_codename}";
      "o=Debian,n=${distro_codename}-updates";
      "o=Debian,n=${distro_codename}-proposed-updates";
      "o=Debian,n=${distro_codename},l=Debian-Security";
};

To also update mono packages on Debian Stretch, add XamarinStretch as Origin:

Unattended-Upgrade::Origins-Pattern {
      "o=Debian,n=${distro_codename}";
      "o=Debian,n=${distro_codename}-updates";
      "o=Debian,n=${distro_codename}-proposed-updates";
      "o=Debian,n=${distro_codename},l=Debian-Security";
      "o=XamarinStretch";
};

If you use another Debian version or Distribution, like Jessie or Ubuntu, search for the file /var/lib/apt/lists/download.mono-project.com_repo_*_InRelease and check the “Origin: ” line and adjust the origin in the config file accordingly.

Repositories with empty Origin like packages.sury.org

If you find that the Origin of the packages in the repository is not given, then you can also tell unattended upgrade to select them based on the given site. For example, this works for the packages from sury.org:

"site=packages.sury.org";

Icinga repository

The packages.icinga.com Repository has the Origin “debian icinga-stretch”, so this is what you need to add to /etc/apt/apt.conf.d/50unattended-upgrades:

"o=debian icinga-${distro_codename}";

Test your changes

To test if it works, you can run unattended updates manually like this (as root or with sudo):

unattended-upgrade -d

If this made your day or you still have problems, just drop a comment.

3. August 2018

iptables: Accept IP address of current ssh client

Filed under: Linux,Security,Server Administration — Tags: , , , , — Christopher Kramer @ 20:39

You have some service, e.g. webmin, that should not be accessible to the public and block access with iptables? And sometimes you connect from a client that is not yet whitelisted in iptables, and always need to look up its IP and add an iptables rule by ssh. Here is a small shell one-liner that makes your life easier:

iptables -I INPUT -p tcp -s `echo $SSH_CLIENT | awk '{ print $1}'` --dport 10000 -j ACCEPT

This just adds an accept rule to iptables that accepts requests from the IP address of the ssh client to port 10000. Of course, you need to adjust the port. You can just paste this in a bash-script, add a bash alias for it or whatever you want to access it fast.

To remove the iptables rule, just replace -I with -D:

iptables -D INPUT -p tcp -s `echo $SSH_CLIENT | awk '{ print $1}'` --dport 10000 -j ACCEPT

You can create one script or shell alias or for each one for easy access.

If this made your day, just leave a comment.

17. July 2018

Debian Stretch: Adjust the CPU priority (nice level) of systemd daemons like spamassasin, amavisd-new, clamd etc.

The good old days: init.d

When still using Debian Wheezy, I configured my spamassasin service to run at a nice level of 10 so it does not slow down apache, mysql, php etc. To do so, I noticed in /etc/init.d/spamassasin a parameter called NICE with a comment saying I should not touch this and instead, go into /etc/default/spamassassin. So I adjusted this file, and on my Debian Stretch, it still looks like this:

 

# /etc/default/spamassassin
# Duncan Findlay

# WARNING: please read README.spamd before using.
# There may be security risks.

# Change to one to enable spamd
ENABLED=1

# Options
# See man spamd for possible options. The -d option is automatically added.

# SpamAssassin uses a preforking model, so be careful! You need to
# make sure --max-children is not set to anything higher than 5,
# unless you know what you're doing.

OPTIONS="--create-prefs --max-children 5 --helper-home-dir"

# Pid file
# Where should spamd write its PID to file? If you use the -u or
# --username option above, this needs to be writable by that user.
# Otherwise, the init script will not be able to shut spamd down.
PIDFILE="/var/run/spamd.pid"

# Set nice level of spamd
NICE="--nicelevel 10"

# Cronjob
# Set to anything but 0 to enable the cron job to automatically update
# spamassassin's rules on a nightly basis
CRON=1

 

This had worked well on Debian Wheezy. Now I noticed that on Debian Stretch, the spamassasin service is running with a nice level of 0, so basically this has no effect. This is because Debian now uses systemd and these init.d files are magically transformed into systemd units and there, the nice level cannot be adjusted like this. The complete story is a lot longer and complex, but I don’t want to bother you here.

How to adjust the nice level of services on Debian Stretch

First, check the location of the unit file of your service. To do so, run:

systemctl status spamassassin.service

It will output something like:

● spamassassin.service - Perl-based spam filter using text analysis
   Loaded: loaded (/etc/systemd/system/spamassassin.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-07-17 16:07:26 CEST; 19min ago
 Main PID: 946 (/usr/sbin/spamd)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/spamassassin.service
           ├─946 /usr/sbin/spamd -d --pidfile=/var/run/spamd.pid --create-prefs --max-children 5 --helper-home-dir
           ├─949 spamd child
           └─950 spamd child

Jul 17 16:07:23 systemd[1]: Starting Perl-based spam filter using text analysis...
Jul 17 16:07:26 systemd[1]: Started Perl-based spam filter using text analysis.

 

In the line “Loaded”, it gives the location of the unit file, in this case:

/etc/systemd/system/spamassassin.service

In your case, it is probably:

/lib/systemd/system/spamassassin.service

If it gives the path to /etc/init.d/ and says “generated” under loaded, as it does e.g. for amavisd-new, then you will probably find the unit file automatically generated by systemd-sysv-generator in one of these paths:

/run/systemd/generator.late
/run/systemd/generator.early
/run/systemd/generator

If your unit file is not already in etc, copy the file from where it is now to /etc, e.g. like this:

cp /lib/systemd/system/spamassassin.service /etc/systemd/system/

If your file is already in /etc, then just go ahead and edit this.

Now open the file /etc/systemd/system/spamassassin.service – it will look like this:

[Unit]
Description=Perl-based spam filter using text analysis
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/var/run/spamd.pid
EnvironmentFile=-/etc/default/spamassassin
ExecStart=/usr/sbin/spamd -d --pidfile=/var/run/spamd.pid $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
StandardError=null
Restart=always

[Install]
WantedBy=multi-user.target

Now adjust it so after [Service] and before [Install], it sets the nice level like this:

[Unit]
Description=Perl-based spam filter using text analysis
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/var/run/spamd.pid
EnvironmentFile=-/etc/default/spamassassin
ExecStart=/usr/sbin/spamd -d --pidfile=/var/run/spamd.pid $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
StandardOutput=null
StandardError=null
Restart=always
Nice=10

[Install]
WantedBy=multi-user.target

 

Now reload the systemd-daemon:

systemctl daemon-reload

And restart your service:

systemctl restart spamassassin.service

Now check the nice level of your spamd processes, it should be 10 now.

 

If this made your day or you still have problems, please let me know in the comments.

5. July 2018

PHP: trying to catch an Exception, but it never gets caught?

Filed under: PHP — Tags: , , , , , — Christopher Kramer @ 20:24

You are trying to catch an Exception like this and it never gets caught?

<?php
try {
// whatever you do here
// probably, you use some library
} catch(Exception $e) {
// this never works
}
?>

Most likely, your app is in some namespace. This means with the above code you only catch exceptions of the namespace of your app, not any exception. The solution is easy, just add a backslash:

try {
// whatever
} catch(\Exception $e) {
// this should work
// note the backslash!
}

Hope this helps somebody.

20. September 2017

How to find out the real name belonging to an email address for 1 cent through paypal

Filed under: Security,Uncategorized — Christopher Kramer @ 18:36

Paypal allows you to transfer money to your friends and family. To do so, you just need the email or phone number of the recipient. I just did this and noticed something interesting: I just gave the email address of the recipient, but in the list of transactions, paypal shows the full real name of my friend as recipient, and also sent me a mail that contains the full name. This means, you can easily find out the real name assigned to some mail address that is used as a paypal account by just sending a cent to the address. Of course the recipient will notice it, but probably will not complain.

This is especially interesting as paypal itself says in the footer of every email that scam mails will usually not include your real name. But scammers could first send you a cent to find out your real name and then greet you with your real name in the scam mail.

It seems to be a good idea to use an email address for paypal that you use nowhere else. This way, it is not possible to find out your real name through your normal mail address. Additionally, this has the advantage that you can ignore any phishing mail that you receive in the name of paypal, but through your normal mail address and not your paypal mail address.

Older Posts »