DeutschEnglish

Submenu

 - - - By CrazyStat - - -

23. June 2015

Bind9 on Debian Wheezy: journal rollforward failed: journal out of sync with zone

Filed under: Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 20:29

Getting this error when restarting bind9?

journal rollforward failed: journal out of sync with zone

You probably changed the zonefile while bind was running without freezing it.

To fix this, first stop bind:

service bind9 stop

Then remove the journal file for the zone. On Debian Wheezy, it is in the same folder as the zone:

rm /etc/bind/zones/example.com.jnl

Then start bind again:

service bind9 start

To avoid the problem next time, before changing a zonefile, freeze the zone:

rndc freeze example.com

Then edit your zonefile and finally thaw it:

rndc thaw example.com

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

6. March 2015

Spamassasin: MISSING_SUBJECT in every mail even though mail has a subject

Filed under: Linux,Server Administration — Tags: , , , — Christopher Kramer @ 09:48

I noticed the MISSING_SUBJECT rule matched for every mail recently even though the mails contained a subject header. Finally I found this in the spamd.log:

warn: Possible unintended interpolation of @yahoo in string at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.
warn: rules: failed to compile Mail::SpamAssassin::Plugin::Check::_head_tests_0_4, skipping:
warn:  (Global symbol "@yahoo" requires explicit package name at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.)

The reason was a rule like this:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress@yahoo.de/
score           MY_SPAMMY_YAHOO        3
describe        MY_SPAMMY_YAHOO        Mail to myspammyaddress@yahoo.de is mostly spam

Escaping the @ in the rule fixed the Problem:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress\@yahoo.de/

Hope this helps someone to spot the error faster.

The problem is that spamassasin does not autolearn ham if the ham messages match the MISSING_SUBJECT rule. So an error like this basically not only breaks lots of tests but also autolearning of ham. So lesson learned: better always look at the spamd.log after creating a new rule, even if you think you know what you are doing 😉

13. August 2014

Icinga: Monitor refused mails in postfix mailqueue

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , — Christopher Kramer @ 12:09

In case your server gets listed on blacklists, mails will get refused by destination servers and stick in the deferred mail queue for some time until the sender finally gets a mailer daemon.

As it takes some time until the sender gets the mailer daemon and informs the server admin, it would be better if you could directly get notified by Icinga/Nagios when a mail is in the deferred queue because the destination server refused it.

Therefore I wrote a small shell script which I want to share with you here. I am assuming Debian Wheezy with Icinga and a postfix mailserver.

Create the shell script with the actual plugin in

/usr/lib/nagios/plugins/check_mailq_blacklist :
#!/bin/sh
# detects if mails in mail queue were refused by destination server (because of blacklist?)
# From https://blog.christosoft.de/2014/08/icinga-monitor-refused-mails-postfix-mailqueue/
# Version: 2017-03-07

if mailq | grep -qP "(refused to talk to me(?!(.*out of connection slots)))|(unsolicited mail originating from your IP)|(temporarily deferred due to user complaints)"
then
  mails=`mailq | grep -oP "(refused to talk to me(?!(.*out of connection slots)))|(unsolicited mail originating from your IP)|(temporarily deferred due to user complai$
  echo "$mails mail(s) were refused, check mailq!"
  if [ "$mails" -le 10 ] && [ "$mails" -gt 1 ]; then
    # 2-10 mails -> warning
    echo "\nWarning. | refused=$mails;2;11;0"
    return 1;
  fi
  if [ "$mails" -gt 10 ]; then
    # more than 10 mails -> critical
    echo "\nCriticial! | refused=$mails;2;11;0"
    return 2;
  fi
  return 1;
else
  echo "Ok, there seems to be no refused mail in the mailq | refused=0;2;11;0"
  exit 0;
fi

This will check for the texts “refused to talk to me” (not followed by “out of connection slots”) and “unsolicited mail originating from your IP” in the mailq output. These are the most common errors you get when the destination server has your server’s IP blacklisted.  In case at least one mail was refused, this causes a warning state in icinga. If more than 10 mails were refused, it causes a critical state.

Now you need to make this script executable:

chmod +x /usr/lib/nagios/plugins/check_mailq_blacklist

Now create the config file for the plugin in

/etc/nagios-plugins/config/mailq_blacklist.cfg :
# 'check_mailq_blacklist' command definition
define command{
        command_name    check_mailq_blacklist
        command_line    /usr/lib/nagios/plugins/check_mailq_blacklist
}

So now we have the command and need to define a service that uses it. Let’s say we use this locally for localhost. In

/etc/icinga/objects/localhost_icinga.cfg

add:

define service{
        use                             generic-service
        host_name                       localhost
        service_description             Mail Queue Refused Mail
        check_command                   check_mailq_blacklist
        }

This is it, just restart icinga and you are done:

service icinga restart

I hope this is of use to somebody.

Of course it is also useful to monitor in Icinga, if you are on some of the most used blacklists. A script to do this can be found here.

27. May 2014

Debian Linux: Update packages automatically

Filed under: Linux,Server Administration — Tags: , , , , , — Christopher Kramer @ 20:50

Here is how you configure automatic (security) updates on Debian:

aptitude install unattended-upgrades

Here you can configure it:

nano /etc/apt/apt.conf.d/50unattended-upgrades

This could look like this:

Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,archive=stable,label=Debian-Security";
};
Unattended-Upgrade::Package-Blacklist {
// add packages here that need manual steps like this:
//        "vim";
}
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::InstallOnShutdown "false";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
Acquire::http::Dl-Limit "200";

Now create the following file:

nano /etc/apt/apt.conf.d/02periodic

With this content:

// Enable the update/upgrade script (0=disable)
APT::Periodic::Enable "1";

// Do "apt-get update" automatically every n-days (0=disable)
APT::Periodic::Update-Package-Lists "1";

// Do "apt-get upgrade --download-only" every n-days (0=disable)
APT::Periodic::Download-Upgradeable-Packages "1";

// Run the "unattended-upgrade" security upgrade script
// every n-days (0=disabled)
// Requires the package "unattended-upgrades" and will write
// a log in /var/log/unattended-upgrades
APT::Periodic::Unattended-Upgrade "1";

// Do "apt-get autoclean" every n-days (0=disable)
APT::Periodic::AutocleanInterval "7";

Of course the server needs to be able to send mails so it can send mails in case of problems.

You can test it like this:

# mail -s test mail@example.com
My testmail
.
EOT

Hope this helps somebody.

Update: Some updates caused dpkg questions about changed config files and therefore failed.

For example php5-fpm did ask this:

Setting up php5-fpm (5.4.4-14+deb7u10) ...

Configuration file `/etc/php5/fpm/pool.d/www.conf'
 ==> Deleted (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** www.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing php5-fpm (--configure):
 EOF on stdin at conffile prompt

This caused PHP5-FPM to stop and all PHP sites to show an Internal Server Errror…

So you don’t run into this problem, create /etc/apt/apt.conf.d/local with this content:

Dpkg::Options {
   "--force-confdef";
   "--force-confold";
}

This tells DPKG to keep the old config file. It will create .dpkg-dist files with the package distributer’s version. More information on this can be found here.

18. January 2014

Linux: get members of a group / get groups of a user

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 00:23

This might sound easy with /etc/group and /etc/passwd, but what if you use libnss-mysql for example and need to try if it works?

So just forget grepping config files. Here are the real commands:

Members of a group:

getent group GROUPNAME

Replace GROUPNAME with the group you want to check.

Groups of a user:

groups USERNAME

Replace USERNAME with the name of the user to check.

 

Checked on Debian, not sure if it works on any Linux/Unix.

11. August 2013

RT Request Tracker: Migrate From SQlite to mySQL

Filed under: DBMS,Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 23:07

I lately had to migrate an RT installation (version 4.0.4) from SQLite to MySQL. In case anyone else has to do this, here is a brief description of how it worked out.

  1. Setup a working MySQL server in case you have not already
  2. Create a MySQL user for RT (e.g. rt4)
  3. Configure RT to use MySQL using this username and a dbname (e.g. rt4) of a not yet existing db (See /etc/request-tracker4/RT_SiteConfig.pm and RT_SiteConfig.d/)
    I’d recommend to keep a copy of the SQlite-Config…
  4. Run
    rt-setup-database --action init

    to create a blank RT DB in MySQL. Check if it worked.

  5. Delete all rows from the MySQL DB, only keep the schema.
  6. Create a copy of your SQLite db
    cp rtdb mydbcopy
  7. Open the copy of the db in the sqlite shell
    sqlite3 mydbcopy
  8. For each table in the DB:
    1. Set the Output file to something like this (“Attachments” is the table name)

    .out data_Attachments

    2. Set the mode to insert (“Attachments” is again the table name)

    .mode insert Attachments

    3. Get all the data

    SELECT * FROM Attachments;

    4. Now you have a file with lots of INSERT statements for the Attachments table. Try to run it in mysql:

    mysql -u rt4 -p rt4 < data_Attachments

    (On a linux shell, not the sqlite shell of course. Here the first rt4 is the username and the second the dbname. data_Attachments is the dump file created before)
    5. In case mysql complains some NOT NULL constraints are violated:
    Go back to the sqlite shell and set these cells to the default-value (0):

    UPDATE Attachments SET someColumn=0 WHERE someColumn IS NULL;

    (Only do this on the copy, not the original db 😉 )
    Now recreate the dump and retry to insert this in mysql. Do it with all columns where NOT NULL constraints are violated.

This works table by table, there are no foreign keys that would get in the way. You can also do several tables in one output file, but you might run into problems when NOT NULL constraints are violated by one table. After all your tables are filled with your data, RT should work. Maybe restart Apache.

apache2ctl restart

This worked without problems so far for me. I first tried pumping the whole SQLite dump into MySQL (using this conversion script) but the schema that this ended up in was different, missed indexes and RT only liked it until I restarted Apache (which then refused to start). Better start with a schema created by rt-setup-database, not with one that originates from SQLite.

I hope this is of some help for somebody. Please let me know in case it helped you or if you have any comments or questions.

21. March 2013

Linux: ACPI disabled, won’t shut down, /sys/class/rtc/rtc0/wakealarm vanished

Filed under: Linux — Tags: , , , , , , , , , , , — Christopher Kramer @ 00:16

MythTV server won’t shut down automatically any longer

Today my mythTV based media homeserver suddenly would not automatically shutdown any longer after a kernel update.

Logs by mythbackend (/var/log/mythtv/mythbackend ) that triggers the shutdown  looked like this:

localhost mythbackend[1674]: I Scheduler scheduler.cpp:2729 (CheckShutdownServer) CheckShutdownServer returned - OK to shutdown
localhost mythbackend[1674]: N Scheduler scheduler.cpp:2814 (ShutdownServer) Running the command to set the next scheduled wakeup time [...]
localhost mythbackend[1674]: E Scheduler scheduler.cpp:2820 (ShutdownServer) SetWakeuptimeCommand failed, shutdown aborted

Problem: /sys/class/rtc/rtc0/wakealarm vanished after kernel update

So I ran the SetWakeupTime command myself and go this problem:

/sys/class/rtc/rtc0/wakealarm: No such file or directory

And in fact the file was missing.

System won’t power off any longer

I tried to shut down the pc completely (manually) and start it again. But it wouldn’t power off, it just freezed with the shutdown bar on the screen. Pressing the power button once (only short, not holding it) made the PC power off.

So this looked like some big ACPI problem.

ACPI disabled

After some more search, I found this in the syslog (/var/log/syslog):

localhost kernel: [    0.000000] ACPI: no DMI BIOS year, acpi=force is required to enable ACPI
localhost kernel: [    0.000000] ACPI: Disabling ACPI support

So no ACPI support explains why wakealarm and shutdown is broken.

I am wondering why this happens suddenly after a kernel update (I have not changed my BIOS or something and ACPI was perfectly working before).

Solution: force ACPI

Forcing ACPI solved the problem.

If you use grub2, this is how to do it:

Open /etc/default/grub (e.g. “sudo nano /etc/default/grub”) and update this line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

Into this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=force"

Save the file and update the grub configuration:

sudo update-grub

Then restart and you’ll be done.

sudo shutdown -r now

I hope this could help somebody.

30. January 2013

Nagios / Icinga: Monitor (local) memory usage

Filed under: Linux,Server Administration — Tags: , , , , , , , , , , , , , — Christopher Kramer @ 17:55

Nagios and its fork icinga are great monitoring tools. They come with a bundle of plugins to monitor standard services such as HTTP, SMTP, POP3, load and stuff like that. And there are lots of 3rd party plugins available for almost everything else you can think of.

But one standard thing that is missing in the official nagios-plugins package is a plugin to check memory usage (of the local machine).

So here is how to install one. I assume a Debian system with Icinga running – you might want to adjust paths for other distros or nagios.

  1. Download the plugin here
    e.g. from the shell:

    wget https://exchange.icinga.com/exchange/check_memory/files/784/check_memory.pl
  2. Then move the file to the other plugins
    mv check_memory.pl /usr/lib/nagios/plugins/check_memory.pl
  3. Make it executable
    chmod +x /usr/lib/nagios/plugins/check_memory.pl
  4. Try to run it:
    perl /usr/lib/nagios/plugins/check_memory.pl -w 50% -c 25%
  5. This should give something like “CHECK_MEMORY OK – […] free […]”. If an error occurs, you probably need to install the perl module Nagios::Plugin. On Debian, the easiest way is:
    apt-get install libnagios-plugin-perl

    On other distros, you might use CPAN:

    perl -MCPAN -e 'install Nagios::Plugin'

    This will ask you lots of questions and install lots of dependencies (where you should say “yes”).

  6. Configure the check_memory command. To do this, create a file /etc/nagios-plugins/config/memory.cfg with this content:
    # 'check_memory' command definition
    define command{
            command_name    check_memory
            command_line    perl /usr/lib/nagios/plugins/check_memory.pl -w $ARG1$ -c $ARG2$
            }
  7. Now you can use the check_memory command to define a service. For example, add this to /etc/icinga/objects/localhost_icinga.cfg (assuming you define localhost-services there):
    define service{
            use                             generic-service
            host_name                       localhost
            service_description             Memory
            check_command                   check_memory!50%!25%
            }

    This will send you a warning when memory usage is 50% and critical when only 25% is free. You might want to adjust these values of course depending on what is normal on your system and how early you want to be notified.

  8. Check your configuration:
    /usr/local/icinga/bin/icinga -v /etc/icinga/icinga.cfg
  9. Restart Icinga / Nagios if the preflight-check was okay:
    /etc/init.d/icinga restart

This should be it.

I hope this helped somebody.

To monitor memory usage of a remote server, you’ll need SNMP for example. Maybe I’ll post another blog post on this soon.

11. September 2012

Zimbra: Setting up a free (real) “commercial” SSL certificate

Filed under: Linux,Security,Server Administration — Tags: , , , , , — Christopher Kramer @ 20:40

I recently wrote about how to configure a new self-signed certificate for Zimbra.

Today I want to explain you how you can do even better and setup a real SSL certificate by Startcom which will make those annoying browser warning messages go away 😉 And the best: It is all for free, thanks to Startcom! It is also important to have a real SSL-certificate for use with most smartphones.

Please note that you have to respect Startcom’s certificate policy, which might require a non-free class 2 certificate for your commercial use. See the policy for details. Thanks to Thomas for this remark. With “commercial” certificate, I here mean a “real” CA-signed certificate, which Zimbra calls “commercial”, no matter whether used in a commercial context or not.

First, create a Certificate Request (CSR) in Zimbra. To do so, ssh into your server, login as root and issue a command like this:

/opt/zimbra/bin/zmcertmgr createcsr comm -new -keysize 2048 -digest sha256
-subject "/C=US/ST=CA/L=Sunnyvale/O=Yahoo/OU=Zimbra Collaboration Suite/CN=host.example.com"
-subjectAltNames host.example.com

This is one line. What is important here, is that you use the domain name for which the certificate should be issued at the end (instead of host.example.com). Startcom will ignore everything else anyway, so you can leave country, state, organization and so on as is or change it, doesn’t matter. What is also important is that you define a keysize of 2048 as Startcom won’t accept 1024 bit keys anymore. This parameter is not documented in the wiki yet, and seems to be available for newer versions of Zimbra only. With a little trick, it should also be also possible for older versions of zimbra such as 5 (but better update to the new version anyway…).

Next thing you do is register at startcom if you do not have an account yet. You’ll get an S/MIME certificate by Startcom for free which you need to login to their control panel. Your browser will generate the secret certificate and store it in its internal storage. I’d recommend you to backup this certificate – you will not be able to login into your startcom account if you loose it.

Then at startcom, you use the validation wizard to validate your domain. This will send a mail to postmaster/hostmaster/webmaster@host.example.com (you can choose which one) with host.example.com being your domain. So you need access to one of these mailboxes to prove that you own the domain.

Then use the certificate wizard at Startcom to create a new certificate. Skip the certificate creation step! Instead, past the CSR created by Zimbra ( /opt/zimbra/ssl/zimbra/commercial/commercial.csr ) into the webform. (Better always create private certificates yourself, never use certificate generators by somebody else, not even the CA.)

Once the certificate is created by Startcom (usually takes some minutes), install it as described in the zimbra wiki:

  1. Store the new (public) certificate you get from Startcom somewhere (e.g. /root/commercial.crt )
  2. Download the root CA certificate
    wget -O /root/ca.pem https://www.startssl.com/certs/ca.pem
  3. Download the intermediary certificate from startcom
    # If your certificate is class 1:
    wget -O /root/ca_intermediary.crt https://www.startssl.com/certs/class1/sha2/pem/sub.class1.server.sha2.ca.pem
    # If your certificate is class 2:
    wget -O /root/ca_intermediary.crt https://www.startssl.com/certs/class2/sha2/pem/sub.class2.server.sha2.ca.pem
  4. Combine them:
    cat /root/ca_intermediary.crt /root/ca.crt > /root/ca_chain.crt
  5. Verify your certificate:
    /opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /root/commercial.crt /root/ca_chain.crt
  6. Deploy your certificate:
    /opt/zimbra/bin/zmcertmgr deploycrt comm /root/commercial.crt /root/ca_chain.crt
  7. Check:
    /opt/zimbra/bin/zmcertmgr viewdeployedcrt
  8. Restart Zimbra
    su – zimbra
    zmcontrol stop
    zmcontrol start

I hope this post was useful so some of you. If you have problems with one of the steps, just ask in the comments.

Read here how to redirect http to https to enforce the use of https.

9. September 2012

Zimbra: Creating a new self-signed SSL certificate

Filed under: Linux,Server Administration — Tags: , , , , , , , , , — Christopher Kramer @ 10:04

I recently had to recreate the SSL certificate of a Zimbra server and surprisingly it was not as easy as the documentation looked like, so I’d like to document how it is done and make comments on some difficulties that might come up.

So this is how it is done (on a Ubuntu Server running Zimbra Network edition 6.0.16 GA):

  1. SSH into the server, login as root
  2. Switch to the zimbra-user using
    su - zimbra
  3. Then run the following commands:
     sudo /opt/zimbra/bin/zmcertmgr createca -new
     sudo /opt/zimbra/bin/zmcertmgr deployca
     sudo /opt/zimbra/bin/zmcertmgr deploycrt self
  4. Restart Zimbra. To do so, as user zimbra, issue these commands (no sudo here):
    /opt/zimbra/bin/zmcontrol stop
    /opt/zimbra/bin/zmcontrol start

So the difficulties I had and some remarks:

  • sudo kept asking me for a password when I typed in
    sudo zmcertmgr createca -new

    Seems I am not the only one with this problem. The zmcertmgr command is white-listed in /etc/sudoers so you should normally not be asked for a password. Run the following command to edit /etc/sudoers (do not edit it in any other way!)

    visudo

    So make sure in this file the following line is included:

    %zimbra ALL=NOPASSWD:/opt/zimbra/bin/zmcertmgr

    The % at the beginning seems to belong there. Note that the zimbra wiki has typo (zmvertmgr) in this line.
    But although I had this line in there, sudo kept asking me for the password. So what finally worked was invoking zmcertmgr with the complete path (as done above).
    Update: It seems I had a typo in here myself. Make sure it is “zmcertmgr”  and not “zmzertmgr” 😉
    Thanks to the comment by erolha!

  • In the Zimbra Release notes, the last command for updating the certificate is
    sudo zmcertmgr deploycrt self -new

    I got this error:

    Can't deploy cert for -new.  Unknown service.

    Without -new (and the complete path), it went through well.

  • No zimbra documentation I found mentions that a restart of zimbra is required, but without a restart, the old certificate was still used when opening the webmailer or the admin interface via https.

 

I hope I could help some of you that run into one of these problems.

« Newer PostsOlder Posts »