- - - By CrazyStat - - -

19. March 2019

PHP: DateTime::createFromFormat fails for string read from CSV

Filed under: PHP — Tags: , , , , , — Christopher Kramer @ 19:31

I wrote a small PHP script to import a dozen csv files exported from Excel into a database. The CSV import basically looked like this:

$f = "file.csv";	
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
} ?>

And now what happened is that $date was false. So I fetched the errors like this:


And this returned:

array(4) {
  array(0) {
  array(1) {
    string(22) "Unexpected data found."

So I added var_dump($date), but it gave string(13) "31.01.2019", which looked right. But looking closely, the string length of 13 seems a bit long for a 10 character date, right? I tried trim() , but without luck. And then I remembered that I had a similar problem before where invisible empty space was due to the UTF-8 Byte Order Mark (BOM). This is a sequence of “inivisible” bytes at the beginning of a textfile that define in which unicode encoding the file is (UTF-8, UTF-16, …) and its endianess (big-endian or little-endian). Microsoft Office programs such as Excel or Word like to write this to the beginning of a file, but other programs may do so as well.

So the solution is simple: In the first line, strip the BOM if it is there:

$f = "file.csv";
$bom = pack('CCC', 0xEF, 0xBB, 0xBF);
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         if ($firstline and substr($data[0], 0, 3) === $bom)
             $data[0] = substr($data[0], 3);
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
} ?>

So this just checks whether the first three bytes in the file match the UTF-8 BOM added by Excel and in case it detects them, it remove these bytes. Now the date parses fine. If your file has a different BOM, e.g. for UTF-16, you may need to change the definition of $bom. Just check your file in a hex-editor to find the first three bytes. This is PSPad, a great text-editor that includes a HEX-editor:

Note the first three bytes EF BB BF, which are the BOM.

If this helped you to solve your problem faster, please drop a comment below. This motivates me to keep writing these articles.


Try my Open Source PHP visitor analytics script CrazyStat.

4. January 2019

Mit dem Navi Diesel-Fahrverbote umfahren

Filed under: Uncategorized — Tags: , , , , , — Christopher Kramer @ 20:14

Hamburg hat als erste deutsche Stadt Fahrverbote für Diesel (Euro 4 und älter) eingeführt. Betroffen sind zunächst nur ca. 600 Meter der Max-Brauer-Allee und ca. 1,6km der Stresemannstraße. Wer mit seinem Euro 4 Diesel in Hamburg unterwegs ist, muss also den Umleitungsschildern folgen. Praktischer wäre es, wenn das Navi die gesperrten Straßen direkt von sich aus vermeiden würde. Ich habe mir die Eigenschaften der betroffenen Straßen in OpenStreeMap angesehen. Sie haben aktuell das Attribut mit dem Schlüssel motor_vehicle:conditional und dem Wert no @ (fuel=diesel), wie hier ein Abschnitt der Max-Brauer-Allee:

Dieselfahrverbot in OpenStreetMap

Da die Daten also schon in OpenStreeMap vorliegen, kann man nun Navis, welche auf OpenStreetMap-Karten basieren, beibringen diese Straßen auf Wunsch zu umfahren.

Das beliebte OsmAnd bietet die Möglichkeit, die Routing-Profile leicht selbst anzupassen. Dazu muss nur eine routing.xml Datei angepasst und auf dem Smartphone abgelegt werden.

Ich habe mir die routing.xml von github heruntergeladen. Innerhalb des Routing-Profils für Autos (car) habe ich zunächst einen neuen Parameter eingefügt, mit dem man in der Oberfläche einstellen kann, ob Dieselfahrverbote gemeidet werden sollen oder nicht:

Als nächstes muss definiert werden, was geschehen soll, wenn der Benutzer die Option aktiviert hat. Dazu habe ich folgende Regel eingefügt, welche die Straßen mit dem oben genannten Parameter meidet:

Diese Regel lässt OsmAnd für Diesel gesperrte Straßen vermeiden.

Die angepasste routing.xml hier downloaden. Wie in der Datei selbst steht, muss man sie i.d.R. an eine der folgenden Orte ablegen:

  • /sdcard/Android/data/
  • /storage/emulated/0/Android/data/
  • /data/user/0/

Ich habe sie unter /storage/emulated/0/Android/data/ abgelegt, da dieser Ordner bei mir existierte. OsmAnd dann schließen (zur Seite wischen) und neu öffnen. Nun sollte in den Routingoptionen die neue Option erscheinen:

Die Option in den Routingeinstellungen

Wenn man nun Routen innerhalb Hamburgs berechnet, die normalerweise die für Diesel gesperrten Routen nutzen würden, kann man sehen, dass es funktioniert. Hier eine Route, die beide Straßen nutzt:

Route in Hamburg, auf der Dieselfahrverbote gelten

Aktiviert man nun die Option zum Meiden von Dieselfahrverboten, sieht die Route deutlich anders aus und meidet die gesperrten Straßen:

Alternative Route durch Hamburg ohne Dieselfahrverbote

Fahrverbote in Stuttgart und anderen Regionen

Ab dem 01.01.2019 gelten nun auch großflächige Dieselfahrverbote in Stuttgart. Leider funktioniert dieser Ansatz (aktuell) nicht für Stuttgart. Er vermeidet Straßen, welche das oben genannte Attribut aufweisen. In Stuttgart sind aber nicht einzelne Straßen, sondern das gesamte Stadtgebiet (genauer: die ausgewiesene Umweltzone) gesperrt. Ich habe erst heute in OpenStreetMap die bereits vorhandene Markierung der Umweltzone mit dem obigen Attribut versehen. Bis dies in den OsmAnd-Karten landet, wird es noch bis zu einen Monat dauern (außer man hat das Karten-Abo). Da es sich bei der Stuttgarter Umweltzone aber um eine Fläche und keine Straße handelt, werden wahrscheinlich noch Anpassungen an der routing.xml nötig, damit OsmAnd die Dieselfahrverbote in Stuttgart berücksichtigen kann. Sobald es hierzu neues gibt, werde ich diesen Artikel aktualisieren.

29. August 2018

phpBB: Upgrade fails (timeout) – solution and how to update on the CLI

Filed under: PHP,Server Administration — Tags: , , , , , , , — Christopher Kramer @ 10:46

So if you are getting a Timeout when updating phpBB to 3.2.2, it tells you to either increase the maximum time limit or do the update on the CLI.

How to update on the CLI

Of course, this requires that you have access to the CLI.

  • On the CLI, go into the install folder of phpBB
  • Create a file named config.yml that contains this:
        type: db_only
  • Make sure the above file does not end with newlines
  • Run:
    php ./phpbbcli.php update config.yml

Fixing the update timeout problem

In my case, the CLI update gave this error:

PHP Fatal error:  Call to a member function fetch_array() on resource in [...]/install/update/new/phpbb/db/migration/data/v32x/fix_user_styles.php on line 42

This is a bug in the migration script. The most easy way to fix it, is to go into your config.php and change the $dbms from mysql to mysqli. This is recommended anyway.

If this is not possible for you, open the mentioned file and search for:

$enabled_styles = $result->fetch_array();

And replace this with:

$enabled_styles = $this->db->sql_fetchrowset($result);

Thanks to RMcGirr83 and Marc on this thread.

24. August 2018

Clementine Music Player needs to be started twice, no window on first start

Filed under: Linux — Tags: , , , , , , , , — Christopher Kramer @ 15:01

clementine music player logoIn Ubuntu 18.04, I start Clementine Music Player, but no window is coming up. But once I start Clementine another time, it comes up.

This is not a bug, it is a setting. So this is what you want:

Tools -> Preferences -> Behaviour -> When Clementine starts -> Always show the main window

No, you are not alone, it is you, me, and a bunch of other people.

9. August 2018

Debian Unattended Upgrades: Upgrade Third Party mono packages

When using the official third-party repository of mono in Debian, unattended Upgrades will not upgrade the mono packages unless you allow the origin. To do so, edit your /etc/apt/apt.conf.d/50unattended-upgrades , which may look like this:

Unattended-Upgrade::Origins-Pattern {

To also update mono packages on Debian Stretch, add XamarinStretch as Origin:

Unattended-Upgrade::Origins-Pattern {

If you use another Debian version or Distribution, like Jessie or Ubuntu, search for the file /var/lib/apt/lists/download.mono-project.com_repo_*_InRelease and check the “Origin: ” line and adjust the origin in the config file accordingly.

Repositories with empty Origin like

If you find that the Origin of the packages in the repository is not given, then you can also tell unattended upgrade to select them based on the given site. For example, this works for the packages from


Test your changes

To test if it works, you can run unattended updates manually like this (as root or with sudo):

unattended-upgrade -d

If this made your day or you still have problems, just drop a comment.

3. August 2018

iptables: Accept IP address of current ssh client

Filed under: Linux,Security,Server Administration — Tags: , , , , — Christopher Kramer @ 20:39

You have some service, e.g. webmin, that should not be accessible to the public and block access with iptables? And sometimes you connect from a client that is not yet whitelisted in iptables, and always need to look up its IP and add an iptables rule by ssh. Here is a small shell one-liner that makes your life easier:

iptables -I INPUT -p tcp -s `echo $SSH_CLIENT | awk '{ print $1}'` --dport 10000 -j ACCEPT

This just adds an accept rule to iptables that accepts requests from the IP address of the ssh client to port 10000. Of course, you need to adjust the port. You can just paste this in a bash-script, add a bash alias for it or whatever you want to access it fast.

To remove the iptables rule, just replace -I with -D:

iptables -D INPUT -p tcp -s `echo $SSH_CLIENT | awk '{ print $1}'` --dport 10000 -j ACCEPT

You can create one script or shell alias or for each one for easy access.

If this made your day, just leave a comment.

17. July 2018

Debian Stretch: Adjust the CPU priority (nice level) of systemd daemons like spamassasin, amavisd-new, clamd etc.

The good old days: init.d

When still using Debian Wheezy, I configured my spamassasin service to run at a nice level of 10 so it does not slow down apache, mysql, php etc. To do so, I noticed in /etc/init.d/spamassasin a parameter called NICE with a comment saying I should not touch this and instead, go into /etc/default/spamassassin. So I adjusted this file, and on my Debian Stretch, it still looks like this:


# /etc/default/spamassassin
# Duncan Findlay

# WARNING: please read README.spamd before using.
# There may be security risks.

# Change to one to enable spamd

# Options
# See man spamd for possible options. The -d option is automatically added.

# SpamAssassin uses a preforking model, so be careful! You need to
# make sure --max-children is not set to anything higher than 5,
# unless you know what you're doing.

OPTIONS="--create-prefs --max-children 5 --helper-home-dir"

# Pid file
# Where should spamd write its PID to file? If you use the -u or
# --username option above, this needs to be writable by that user.
# Otherwise, the init script will not be able to shut spamd down.

# Set nice level of spamd
NICE="--nicelevel 10"

# Cronjob
# Set to anything but 0 to enable the cron job to automatically update
# spamassassin's rules on a nightly basis


This had worked well on Debian Wheezy. Now I noticed that on Debian Stretch, the spamassasin service is running with a nice level of 0, so basically this has no effect. This is because Debian now uses systemd and these init.d files are magically transformed into systemd units and there, the nice level cannot be adjusted like this. The complete story is a lot longer and complex, but I don’t want to bother you here.

How to adjust the nice level of services on Debian Stretch

First, check the location of the unit file of your service. To do so, run:

systemctl status spamassassin.service

It will output something like:

● spamassassin.service - Perl-based spam filter using text analysis
   Loaded: loaded (/etc/systemd/system/spamassassin.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-07-17 16:07:26 CEST; 19min ago
 Main PID: 946 (/usr/sbin/spamd)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/spamassassin.service
           ├─946 /usr/sbin/spamd -d --pidfile=/var/run/ --create-prefs --max-children 5 --helper-home-dir
           ├─949 spamd child
           └─950 spamd child

Jul 17 16:07:23 systemd[1]: Starting Perl-based spam filter using text analysis...
Jul 17 16:07:26 systemd[1]: Started Perl-based spam filter using text analysis.


In the line “Loaded”, it gives the location of the unit file, in this case:


In your case, it is probably:


If it gives the path to /etc/init.d/ and says “generated” under loaded, as it does e.g. for amavisd-new, then you will probably find the unit file automatically generated by systemd-sysv-generator in one of these paths:


If your unit file is not already in etc, copy the file from where it is now to /etc, e.g. like this:

cp /lib/systemd/system/spamassassin.service /etc/systemd/system/

If your file is already in /etc, then just go ahead and edit this.

Now open the file /etc/systemd/system/spamassassin.service – it will look like this:

Description=Perl-based spam filter using text analysis

ExecStart=/usr/sbin/spamd -d --pidfile=/var/run/ $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID


Now adjust it so after [Service] and before [Install], it sets the nice level like this:

Description=Perl-based spam filter using text analysis

ExecStart=/usr/sbin/spamd -d --pidfile=/var/run/ $OPTIONS
ExecReload=/bin/kill -HUP $MAINPID



Now reload the systemd-daemon:

systemctl daemon-reload

And restart your service:

systemctl restart spamassassin.service

Now check the nice level of your spamd processes, it should be 10 now.


If this made your day or you still have problems, please let me know in the comments.

5. July 2018

PHP: trying to catch an Exception, but it never gets caught?

Filed under: PHP — Tags: , , , , , — Christopher Kramer @ 20:24

You are trying to catch an Exception like this and it never gets caught?

try {
// whatever you do here
// probably, you use some library
} catch(Exception $e) {
// this never works

Most likely, your app is in some namespace. This means with the above code you only catch exceptions of the namespace of your app, not any exception. The solution is easy, just add a backslash:

try {
// whatever
} catch(\Exception $e) {
// this should work
// note the backslash!

Hope this helps somebody.

20. September 2017

How to find out the real name belonging to an email address for 1 cent through paypal

Filed under: Security,Uncategorized — Christopher Kramer @ 18:36

Paypal allows you to transfer money to your friends and family. To do so, you just need the email or phone number of the recipient. I just did this and noticed something interesting: I just gave the email address of the recipient, but in the list of transactions, paypal shows the full real name of my friend as recipient, and also sent me a mail that contains the full name. This means, you can easily find out the real name assigned to some mail address that is used as a paypal account by just sending a cent to the address. Of course the recipient will notice it, but probably will not complain.

This is especially interesting as paypal itself says in the footer of every email that scam mails will usually not include your real name. But scammers could first send you a cent to find out your real name and then greet you with your real name in the scam mail.

It seems to be a good idea to use an email address for paypal that you use nowhere else. This way, it is not possible to find out your real name through your normal mail address. Additionally, this has the advantage that you can ignore any phishing mail that you receive in the name of paypal, but through your normal mail address and not your paypal mail address.

13. June 2017

Putty: Automatically start an SSH tunnel without a window (headless)

Filed under: Windows — Tags: , , , , , , , — Christopher Kramer @ 21:23

You use Putty to tunnel some port through SSH to a foreign network but manually starting it again and again annoys you and the window gets in the way? Here is your solution. It requires admin rights and is tested on Windows 7 Professional only. Please let us know in the comments whether it works the same in your windows version.

  1. Set up your SSH tunnel in putty and save it as a session (e.g. with the name mytunnel). Make sure to use keys for authentication and that the key does not require a passphrase. Without a window there is no way to enter the passphrase of course ;-). If you worry about security, best restrict the rights of the user that you connect to as much as possible.
  2. Press WINDOWS + R and enter Taskschd.msc to start the Windows Task Scheduler. Accept the UAC prompt, if any.
  3. In the tree on the left, select the folder Task Scheduler Library.
  4. On the right, click Create task…
  5. Enter a name like SSH Tunnel.
  6. Choose Run whether user is logged on or not (this is important to make sure the window does not pop up).
  7. Click the checkbox Do not store the password. The process will only have access to local resources.
  8. Switch to the Triggers tab.
  9. Click the New… button
  10. Select a trigger that suits your needs. I chose Daily,  recur every 1 days, Repeat task every 30 minutes for the duration of 1 day and stop all running tasks at the end of repetition interval.  Click OK. This will restart the SSH tunnel every 30 minutes, so you might be left without connection for at most 30 minutes. You could add another trigger at startup or login if you need that.
  11. Switch to the Actions tab.
  12. Click the New… button.
  13. Choose Start a program as action.
  14. Use the Browse button to find plink.exe, which belongs to putty, you can download it here.
  15. Next to Add arguments, enter the name of your putty session (e.g. mytunnel).
  16. Click OK.
  17. You can adjust more settings like in the Conditions tab, unselect Start the task only if the computer is on AC power to make sure your tunnel also starts when your laptop is operating on battery, if necessary.
  18. Click OK.
  19. Right click your task and select Run to see if it works as expected. No window should come up but your ssh tunnel should be working.

Let me know if this saved your day and what you are using it for. I am using this to create an SSH tunnel for Sophos Endpoint Security and Control to fetch its updates from a server that is only accessible from an internal network.

Older Posts »