DeutschEnglish

Submenu

 - - - By CrazyStat - - -

14. March 2015

Bind nameserver: view cache entries

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 15:35

If you want to view the cache entries of your bind nameserver, this is how it works on bind9 on Debian:

rndc dumpdb
less /var/cache/bind/named_dump.db

Tested on Debian Wheezy. Hope this helps somebody.

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

6. March 2015

Owncloud: Upgrading to owncloud 8 fails (Integrity constraint violation in oc_filecache)

Filed under: Linux,PHP,Server Administration — Tags: , — Christopher Kramer @ 21:33

First, the upgrade to owncloud 8 in the web GUI failed. Then, I performed the upgrade in the console like this:

sudo -u www-data php /var/www/owncloud/occ upgrade

This gave an error like this (I don’t remember exactly):

An exception occurred while executing 'INSERT INTO "oc_filecache" ... Integrity constraint violation: key s_storage_path_hash is not unique
...
Upgrade failed

This owncloud is using a MySQL DB.

So what in the end solved the problem: First put Owncloud in maintainance mode:

sudo -u www-data php /var/www/owncloud/occ maintenance:mode --on

Then make sure no occ processes of old upgrade attempts are running. If there are, kill them. Then clear the oc_filecache using this MySQL command:

TRUNCATE oc_filecache;

Don’t worry, it will populate itself again during the upgrade. Then restart the upgrade:

sudo -u www-data php /var/www/owncloud/occ upgrade

This might take a lot of time! Better run this on a screen (see screen tutorial if you don’t know how) so it does not stop when your SSH connection breaks.

Spamassasin: MISSING_SUBJECT in every mail even though mail has a subject

Filed under: Linux,Server Administration — Tags: , , , — Christopher Kramer @ 09:48

I noticed the MISSING_SUBJECT rule matched for every mail recently even though the mails contained a subject header. Finally I found this in the spamd.log:

warn: Possible unintended interpolation of @yahoo in string at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.
warn: rules: failed to compile Mail::SpamAssassin::Plugin::Check::_head_tests_0_4, skipping:
warn:  (Global symbol "@yahoo" requires explicit package name at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.)

The reason was a rule like this:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress@yahoo.de/
score           MY_SPAMMY_YAHOO        3
describe        MY_SPAMMY_YAHOO        Mail to myspammyaddress@yahoo.de is mostly spam

Escaping the @ in the rule fixed the Problem:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress\@yahoo.de/

Hope this helps someone to spot the error faster.

The problem is that spamassasin does not autolearn ham if the ham messages match the MISSING_SUBJECT rule. So an error like this basically not only breaks lots of tests but also autolearning of ham. So lesson learned: better always look at the spamd.log after creating a new rule, even if you think you know what you are doing ๐Ÿ˜‰

9. December 2014

Icinga: Group all services in a servicegroup instead of using a wildcard

Filed under: Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 16:15

At some places you can use the * wildcard as a service description (which requires use_regexp_matching=0), but sometimes it does not seem to work:

Error: Could not expand services specified

Therefore, I simply wanted to group all services in one servicegroup. That’s quite easy if you use a generic service template as a basis for all services. First, create a servicegroup “allservices”:

define servicegroup {
        servicegroup_name               allservices
        alias                           All Services
}

Then edit your generic service template (see servicegroups line):

# generic service template definition
define service{
        name                            generic-service ; The 'name' of this service template
        active_checks_enabled           1       ; Active service checks are enabled
        passive_checks_enabled          1       ; Passive service checks are enabled/accepted
        parallelize_check               1       ; Active service checks should be parallelized (disabling this can lead to major performance probl$
        obsess_over_service             1       ; We should obsess over this service (if necessary)
        check_freshness                 0       ; Default is to NOT check service 'freshness'
        notifications_enabled           1       ; Service notifications are enabled
        event_handler_enabled           1       ; Service event handler is enabled
        flap_detection_enabled          1       ; Flap detection is enabled
        failure_prediction_enabled      1       ; Failure prediction is enabled
        process_perf_data               1       ; Process performance data
        retain_status_information       1       ; Retain status information across program restarts
        retain_nonstatus_information    1       ; Retain non-status information across program restarts
        notification_interval           0               ; Only send notifications on status change by default.
        is_volatile                     0
        check_period                    24x7
        normal_check_interval           5
        retry_check_interval            1
        max_check_attempts              4
        notification_period             24x7
        notification_options            w,u,c,r
        contact_groups                  admins
        servicegroups                   allservices ; ADD THIS TO ADD ALL SERVICES INTO THE allservices GROUP
        register                        0       ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL SERVICE, JUST A TEMPLATE!
        }

This assumes all your services use this template like this (see use line):

define service {
        hostgroup_name                  ssh-servers
        service_description             SSH
        check_command                   check_ssh
        use                             generic-service  ; USE THE TEMPLATE ABOVE
        notification_interval           0 ; set > 0 if you want to be renotified
}

 

Now you can easily use this servicegroup for example in serviceescalations (see servicegroup_name line):

 define serviceescalation{
        hostgroup_name          intranet-servers
        servicegroup_name       allservices
        first_notification      1
        last_notification       0
        notification_interval   1440
        contact_groups          intranet-admins
        }

Hope this helps somebody. I guess it works the same way in Nagios.

13. August 2014

Icinga: Monitor refused mails in postfix mailqueue

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , — Christopher Kramer @ 12:09

In case your server gets listed on blacklists, mails will get refused by destination servers and stick in the deferred mail queue for some time until the sender finally gets a mailer daemon.

As it takes some time until the sender gets the mailer daemon and informs the server admin, it would be better if you could directly get notified by Icinga/Nagios when a mail is in the deferred queue because the destination server refused it.

Therefore I wrote a small shell script which I want to share with you here. I am assuming Debian Wheezy with Icinga and a postfix mailserver.

Create the shell script with the actual plugin in

/usr/lib/nagios/plugins/check_mailq_blacklist :
#!/bin/sh
# detects if mails in mail queue were refused by destination server (because of blacklist?)
# From http://blog.christosoft.de/2014/08/icinga-monitor-refused-mails-postfix-mailqueue/
# Version: 2015-04-27

if mailq | -qP "(refused to talk to me)|(unsolicited mail originating from your IP)"
then
  mails=`mailq | grep -oP "(refused to talk to me)|(unsolicited mail originating from your IP)" | wc -l`
  echo "$mails mail(s) were refused, check mailq!"
  if [ "$mails" -le 10 ]; then
    # 10 or less mails -> warning
    return 1;
  else
    # more than 10 mails -> critical
    return 2;
  fi
  return 1;
else
  echo "Ok, there seems to be no refused mail in the mailq"
  exit 0;
fi

This will check for the texts “refused to talk to me” and “unsolicited mail originating from your IP” in the mailq output. These are the most common errors you get when the destination server has your server’s IP blacklisted.ย  In case at least one mail was refused, this causes a warning state in icinga. If more than 10 mails were refused, it causes a critical state.

Now you need to make this script executable:

chmod +x /usr/lib/nagios/plugins/check_mailq_blacklist

Now create the config file for the plugin in

/etc/nagios-plugins/config/mailq_blacklist.cfg :
# 'check_mailq_blacklist' command definition
define command{
        command_name    check_mailq_blacklist
        command_line    /usr/lib/nagios/plugins/check_mailq_blacklist
}

So now we have the command and need to define a service that uses it. Let’s say we use this locally for localhost. In

/etc/icinga/objects/localhost_icinga.cfg

add:

define service{
        use                             generic-service
        host_name                       localhost
        service_description             Mail Queue Refused Mail
        check_command                   check_mailq_blacklist
        }

This is it, just restart icinga and you are done:

service icinga restart

I hope this is of use to somebody.

Of course it is also useful to monitor in Icinga, if you are on some of the most used blacklists. A script to do this can be found here.

27. May 2014

Debian Linux: Update packages automatically

Filed under: Linux,Server Administration — Tags: , , , , , — Christopher Kramer @ 20:50

Here is how you configure automatic (security) updates on Debian:

aptitude install unattended-upgrades

Here you can configure it:

nano /etc/apt/apt.conf.d/50unattended-upgrades

This could look like this:

Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,archive=stable,label=Debian-Security";
};
Unattended-Upgrade::Package-Blacklist {
// add packages here that need manual steps like this:
//        "vim";
}
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::InstallOnShutdown "false";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
Acquire::http::Dl-Limit "200";

Now create the following file:

nano /etc/apt/apt.conf.d/02periodic

With this content:

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "7";

Of course the server needs to be able to send mails so it can send mails in case of problems.

You can test it like this:

# mail -s test mail@example.com
My testmail
.
EOT

Hope this helps somebody.

Update: Some updates caused dpkg questions about changed config files and therefore failed.

For example php5-fpm did ask this:

Setting up php5-fpm (5.4.4-14+deb7u10) ...

Configuration file `/etc/php5/fpm/pool.d/www.conf'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** www.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing php5-fpm (--configure):
 EOF on stdin at conffile prompt

This caused PHP5-FPM to stop and all PHP sites to show an Internal Server Errror…

So you don’t run into this problem, create /etc/apt/apt.conf.d/local with this content:

Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

This tells DPKG to keep the old config file. It will create .dpkg-dist files with the package distributer’s version. More information on this can be found here.

9. May 2014

Updating to PHP 5.4 causes missing Text

Filed under: PHP,Server Administration — Tags: , , , , , , — Christopher Kramer @ 14:29

After updating from PHP5.3 to PHP 5.4, on some sites text was missing. No error could be found in the error log so I had to dig into the code to find out what was going on.

The root cause is that with PHP5.4, the default character set expected by htmlentites(), htmlspecialcharacters() and html_entity_decode() changed from ISO-8859-1 to UTF-8. So if a script passes ISO-8859-1 characters like German “Umlaute” (รถรครผร–ร„รœรŸ) to one of these functions without specifying the charset with the corresponding parameter, these functions will return an empty string. And unfortunately, with PHP 5.4, they also removed the error message that PHP 5.3 recorded in the logfile in this case. This makes finding the problem a lot more difficult.

So what can you do about it? You could

  1. Use PHP 5.3 ๐Ÿ˜‰
    Here is a blog post on downgrading to PHP 5.3. on Debian Wheezy
  2. change the used charset to UTF-8
    This might require changing the character set in files, databases or config files, depending on what is used on the site.
    I explained in a blog post how to change the charset in Typo3 to UTF-8 back in 2012.
  3. Provide ISO-8859-1 as a parameter to all calls of htmlspecialcharacters() etc.

So for the third option, what you have to do is find places like this:

htmlspecialchars($string);

And replace them with something like:

htmlspecialchars($string, ENT_COMPAT | ENT_XHML, 'ISO-8859-1');

The problem is that it’s hard to do this automatically. What is easy to do, is replace all htmlspecialchars()-calls with calls to htmlspecialchars_PHP5-3() etc. and place these functions there:

function htmlspecialchars_PHP5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
    return htmlspecialchars($string, $ent, $charset);
}

function htmlentities_PHP-5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
    return htmlentities($string, $ent, $charset);
}

function html_entity_decode_PHP-5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
    return html_entity_decode($string, $ent, $charset);
}

So just do a search & replace over all files and make sure that all scripts have a file included that contains these functions.

18. January 2014

Linux: get members of a group / get groups of a user

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 00:23

This might sound easy with /etc/group and /etc/passwd, but what if you use libnss-mysql for example and need to try if it works?

So just forget grepping config files. Here are the real commands:

Members of a group:

getent group GROUPNAME

Replace GROUPNAME with the group you want to check.

Groups of a user:

groups USERNAME

Replace USERNAME with the name of the user to check.

 

Checked on Debian, not sure if it works on any Linux/Unix.

1. September 2013

MySQL: reset a forgotten MySQL root password

Filed under: DBMS,Linux,Security,Server Administration — Christopher Kramer @ 14:37

Just in case you don’t know the password of your MySQL root user anylonger, this is how you can set a new one.

Windows users, follow this guide.

Linux / Unix users: This is for Debian (wheezy) and MySQL 5.5, but should work more or less the same on any linux/unix:

  1. Stop the MySQL server
    service mysql stop
  2. Create a init file:
    touch /etc/mysql/mysql-init
  3. Write the following into this file (adjust the PW):
    SET PASSWORD FOR 'root'@'localhost' = PASSWORD('MyNewPassword');
  4. Restart MySQL with this init-file:
    mysqld_safe --init-file=/etc/mysql/mysql-init
  5. The new password should work now. Try it:
    mysql -u root -p
  6. Restart MySQL normally:
    service mysql restart
  7. Remove the init-file
    rm /etc/mysql/mysql-init

And you’re done. I hope this helps somebody.

Note: if it does not work, it is most likely because the MySQL process cannot access the init-file (e.g. because of missing access rights). In this case check your (sys)logs for stuff like this:

mysqld: 130901 14:20:30 [ERROR] /usr/sbin/mysqld: File '/root/mysql-init' not found (Errcode: 13)

I had this problem first because I tried creating the init-file in /root where the mysql-process was not able to read it.

11. August 2013

RT Request Tracker: Migrate From SQlite to mySQL

Filed under: DBMS,Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 23:07

I lately had to migrate an RT installation (version 4.0.4) from SQLite to MySQL. In case anyone else has to do this, here is a brief description of how it worked out.

  1. Setup a working MySQL server in case you have not already
  2. Create a MySQL user for RT (e.g. rt4)
  3. Configure RT to use MySQL using this username and a dbname (e.g. rt4) of a not yet existing db (See /etc/request-tracker4/RT_SiteConfig.pm and RT_SiteConfig.d/)
    I’d recommend to keep a copy of the SQlite-Config…
  4. Run
    rt-setup-database --action init

    to create a blank RT DB in MySQL. Check if it worked.

  5. Delete all rows from the MySQL DB, only keep the schema.
  6. Create a copy of your SQLite db
    cp rtdb mydbcopy
  7. Open the copy of the db in the sqlite shell
    sqlite3 mydbcopy
  8. For each table in the DB:
    1. Set the Output file to something like this (“Attachments” is the table name)

    .out data_Attachments

    2. Set the mode to insert (“Attachments” is again the table name)

    .mode insert Attachments

    3. Get all the data

    SELECT * FROM Attachments;

    4. Now you have a file with lots of INSERT statements for the Attachments table. Try to run it in mysql:

    mysql -u rt4 -p rt4 < data_Attachments

    (On a linux shell, not the sqlite shell of course. Here the first rt4 is the username and the second the dbname. data_Attachments is the dump file created before)
    5. In case mysql complains some NOT NULL constraints are violated:
    Go back to the sqlite shell and set these cells to the default-value (0):

    UPDATE Attachments SET someColumn=0 WHERE someColumn IS NULL;

    (Only do this on the copy, not the original db ๐Ÿ˜‰ )
    Now recreate the dump and retry to insert this in mysql. Do it with all columns where NOT NULL constraints are violated.

This works table by table, there are no foreign keys that would get in the way. You can also do several tables in one output file, but you might run into problems when NOT NULL constraints are violated by one table. After all your tables are filled with your data, RT should work. Maybe restart Apache.

apache2ctl restart

This worked without problems so far for me. I first tried pumping the whole SQLite dump into MySQL (using this conversion script) but the schema that this ended up in was different, missed indexes and RT only liked it until I restarted Apache (which then refused to start). Better start with a schema created by rt-setup-database, not with one that originates from SQLite.

I hope this is of some help for somebody. Please let me know in case it helped you or if you have any comments or questions.

« Newer PostsOlder Posts »