DeutschEnglish

Submenu

 - - - By CrazyStat - - -

15. July 2016

Request-Tracker4: Enabling fast fulltext search with Sphinxsearch on Debian Jessie

I just successfully made fulltext search in our request tracker super fast. This is a guide for Debian Jessie and request tracker set up to use MySQL (or MariaDB).

  • Don’t forget to backup your RT database:
    mysqldump -p -u root rtdb > rtdb.sql
  • Log out of request tracker. I had some strange issue because I did not 😉
  • Make sure you are using request-tracker4 version 4.2.12 from jessie-backports. If you don’t already, this is how you update:
    – add this to the end of your /etc/apt/sources.list:

    deb http://ftp.debian.org/debian jessie-backports main

    – run

    apt-get update

    – install request-tracker4 from jessie-backports:

    apt-get -t jessie-backports install request-tracker4

    – follow the update procedure, make sure your database is upgraded
    (We need the package from backports as jessie’s rt4-db-mysql package depends on mysql-client, whereas the backports version depends on virtual-mysql-client and thus can also be easily used with MariaDB.)

  • Second, make sure you are using MariaDB instead of MySQL. MariaDB is a drop-in replacement for MySQL. The big advantage of MariaDB here is that it comes with the SphinxSE storage engine that we will use for the fulltext search. SphinxSE is also available for MySQL, but you would need to compile it yourself whereas MariaDB includes SphinxSE by default. MariaDB has also a lot more advantages. Switching to MariaDB is very simple, no need to convert databases or configuration. I used aptitude and selected to install mariadb-client and maridb-server. Then I checked the dependency resolutions and choose the option to uninstall mysql-client, mysql-server and related packages. Then everything went smooth automatically, I just needed to enter the root password again, everything else was updated automatically.
  • Now check if your RT is still working well with MariaDB.
  • Now enable the SphinxSE storage engine in MariaDB:
    mysql -u root -p
    [enter password]
    INSTALL SONAME 'ha_sphinx';
    SHOW ENGINES;
    EXIT;
  • Now install the Sphinxsearch daemon:
    apt-get install sphinxsearch
  • Now set up indexed full text search in RT:
    rt-setup-fulltext-index --dba root --dba-password secret

    (Let me know if you find a way without passing the password as a parameter)
    This asks you what type of index you want to use. Enter “sphinx”. Then keep the defaults.

  • The command above gave you code for /etc/request-tracker4/RT_SiteConfig.pm – like this:
    Set( %FullTextSearch,
        Enable     => 1,
        Indexed    => 1,
        MaxMatches => '10000',
        Table      => 'AttachmentsIndex',
    );

    Add / adjust this in RT_SiteConfig.pm.

  • Then you need to configure the index in sphinx. The above tool also gave you a config file for this, but it does not work with the current sphinxsearch version of jessie. So here is what works. Create an /etc/sphinxsearch/sphinx.conf like this:
    source rt {
        type            = mysql
    
        sql_host        = localhost
        sql_db          = rtdb
        sql_user        = rtuser
        sql_pass        = RT_PASSWORD
    
        sql_query_pre   = SET NAMES utf8
        sql_query       = \
            SELECT a.id, t.Subject, a.content FROM Attachments a \
            JOIN Transactions txn ON a.TransactionId = txn.id AND txn.ObjectType = 'RT::Ticket' \
            JOIN Tickets t ON txn.ObjectId = t.id \
            WHERE a.ContentType LIKE 'text/%' AND t.Status != 'deleted'
    
    #    sql_query_info  = SELECT * FROM Attachments WHERE id=$id
    }
    
    index rt {
        source                  = rt
        path                    = /var/lib/sphinxsearch/data/rt
        docinfo                 = extern
    #    charset_type            = utf-8
    }
    
    indexer {
        mem_limit               = 32M
    }
    
    searchd {
        listen                  = 3312
        log                     = /var/log/sphinxsearch/searchd.log
        query_log               = /var/log/sphinxsearch/query.log
        read_timeout            = 5
        max_children            = 30
        pid_file                = /var/run/sphinxsearch/searchd.pid
    #    max_matches             = 10000
        seamless_rotate         = 1
        preopen_indexes         = 0
        unlink_old              = 1
        # For sphinx >= 1.10:
        binlog_path             = /var/lib/sphinxsearch/data
    }

    Make sure to adjust the mysql database, user and password.
    Note that there are some differences to the config provided by rt-setup-fulltext-index. Especially, you need to use listen instead of port. If you don’t adjust this, sphinxsearch will start on a different port and RT will not find any results in fulltext searches! If your RT is not behind a firewall, better also set listen to 127.0.0.1:3312 . Also, I used the Debian style paths from the Debian sample. The fields commented out are not necessary anymore in current sphinxsearch.
    UPDATE 26.08.2016: I changed the SQL Query to also index the Subject of the tickets and index all ‘text/%’ types and not only ‘text/plain’. This change is based on a comment by Claes Merin in the Request Tracker Mailing List. Thanks a lot. Without this change, the fulltext search will only search within ticket content, but not within ticket titles… Also, before it would not index text/html attachments.

  • Now run the indexer to build the initial index. This will take some time, for a 6 GB Attachments Table, it took not more than 30 minutes. Maybe run this in a screen in case your ssh connection breaks:
    indexer rt
  • Enable the Sphinx daemon. To do so, adjust START to yes in /etc/default/sphinxsearch
  • Finally, start the sphinxsearch daemon:
    /etc/init.d/sphinxsearch start
  • Now you can log into RT and do a fulltext search by entering in the search field:
    fulltext:search_word
  • In my case, this made the fulltext searches run within a second instead of 5 minutes 🙂
  • Make sure the indexer is run regularily to update the index. To do so, enter
    crontab -e

    And add a line like this:

    13 * * * * /usr/bin/indexer rt --rotate

I hope this helps somebody to setup this a little faster.

 

 

 

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

16. March 2016

Typo3: Redirect backend to https and frontend to http

Filed under: PHP,Security,Server Administration,Typo3 — Tags: , , , , , , , , , — Christopher Kramer @ 20:02

If you are using SNI to secure the access to your site through SSL, you might decide that you do not want to use SSL for the frontend, as users of old clients such as Android 2 or Internet Explorer 8 on Windows XP won’t be able to access the site. But for the backend, old clients should not be a problem.

So this solves the redirection in two ways:

  1. Frontend gets redirected to HTTP if accessed through https (optional)
  2. Backend gets redirected to HTTPS if accessed through http

Put this in your .htaccess after “RewriteEngine On” (assuming Apache webserver):

# 1. optionally: redirect FE to http
RewriteCond %{HTTPS} on
RewriteCond %{REQUEST_URI} !^/?typo3
RewriteRule (.*) http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

# 2. redirect BE to https
RewriteCond %{HTTPS} off
RewriteCond %{REQUEST_URI} ^/?typo3
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]

And hopefully, your backend access is now securely encrypted…

https

 

23. June 2015

Bind9 on Debian Wheezy: journal rollforward failed: journal out of sync with zone

Filed under: Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 20:29

Getting this error when restarting bind9?

journal rollforward failed: journal out of sync with zone

You probably changed the zonefile while bind was running without freezing it.

To fix this, first stop bind:

service bind9 stop

Then remove the journal file for the zone. On Debian Wheezy, it is in the same folder as the zone:

rm /etc/bind/zones/example.com.jnl

Then start bind again:

service bind9 start

To avoid the problem next time, before changing a zonefile, freeze the zone:

rndc freeze example.com

Then edit your zonefile and finally thaw it:

rndc thaw example.com

5. June 2015

Zimbra: Redirect http to https

Filed under: Linux,Security,Server Administration — Tags: , , , , — Christopher Kramer @ 11:15

Zimbra without Proxy (pre 8.5)

That’s the easy way how you can enforce https encrpytion by redirecting http to https:

su – zimbra
zmtlsctl redirect
zmcontrol stop
zmcontrol start

Works at least on Zimbra 8.0 and I think should also work on 7

Zimbra with Proxy (required from 8.5+)

With Zimbra 8.5+, a Proxy is required. This makes the configuration a little different. To configure the proxy to redirect http to https, run:

su zimbra
~/libexec/zmproxyconfig -e -w -o -a 8080:80:8443:443 -x both  -H `zmhostname`
# if your proxy is local:
zmprov ms `zmhostname` zimbraReverseProxyMailMode redirect
# if your proxy is proxy.server.name
zmprov ms proxy.server.name zimbraReverseProxyMailMode redirect
zmcontrol restart

With the latest Zimbra versions, the restart is not even necessary, it automatically detects the change within 2 minutes.

Hope this helps somebody!

2. June 2015

Zimbra: Using zmpostfixpolicyd to stop backscatter

Filed under: Linux,Server Administration — Tags: , , , — Christopher Kramer @ 19:20

Your Zimbra is backscattering? This is what may help you:

Update 2018-08-21:

Before doing anything else, make sure you don’t have a CatchAll defined for your domain. To do so:

su - zimbra
zmprov gd example.com | grep zimbraMailCatchAllAddress 
# should be empty. NOT @example.com
zmprov gd example.com | grep zimbraMailCatchAllForwardingAddress
# should be empty. NOT @example.com
zmprov gd example.com | grep zimbraMailTransport 
# should be lmtp:zimbra.example.com:7025 . NOT e.g. smtp:relayhost.com:587

If any of this is not as should be, then change it:

zmprov md domain.com zimbraMailCatchAllAddress ""
zmprov md domain.com zimbraMailCatchAllForwardingAddress ""
zmprov md domain.com zimbraMailTransport lmtp:zimbra.example.com:7025

Note that in the last line, you need to give the FQDN of your Zimbra server, instead of zimbra.example.com.

You may also read the Zimbra wiki about catchalls.

Original article:

A Zimbra 8.0 Server (on Ubuntu) was backscattering. Unfortunately Zimbra servers cannot use the postfix setting

smtpd_reject_unlisted_sender=yes

when an alias domain is setup. Everybody says it is easy, just install the policyd as a recipient_restriction and that’s it. All you need to do is:

su - zimbra
zmlocalconfig -e postfix_enable_smtpd_policyd=yes
zmprov mcf +zimbraMtaRestriction "check_policy_service unix:private/policy"
zmmtactl stop
zmmtactl start

Unfortunately, this did not work for me with Zimbra 8.0. The server was still backscattering.

First, I enabled verbose output for the policyd. To do so, open /opt/zimbra/postfix/conf/master.cf.in (the .in is important) in a texteditor and search for this part:

%%uncomment LOCAL:postfix_enable_smtpd_policyd%%policy    unix  -       n       n       -       -       spawn
%%uncomment LOCAL:postfix_enable_smtpd_policyd%%        user=zimbra argv=/usr/bin/perl /opt/zimbra/libexec/zmpostfixpolicyd

Add a -v ad the end of the command:

%%uncomment LOCAL:postfix_enable_smtpd_policyd%%policy    unix  -       n       n       -       -       spawn
%%uncomment LOCAL:postfix_enable_smtpd_policyd%%        user=zimbra argv=/usr/bin/perl /opt/zimbra/libexec/zmpostfixpolicyd -v

Then (as zimbra user), recreate the master.cf and restart postfix with these commands:

zmmtactl stop
zmmtactl start

Now when a mail is received (from an external sender), you will see output like this in your mail log (on Debian/Ubuntu: /var/log/mail.log):

/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: ccert_fingerprint=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: sasl_method=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: sasl_sender=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: size=3428
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: helo_name=[...]
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: reverse_client_name=[...]
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: queue_id=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: encryption_cipher=AECDH-AES256-SHA
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: encryption_protocol=TLSv1.2
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: etrn_domain=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: ccert_subject=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: request=smtpd_access_policy
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: protocol_state=RCPT
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: stress=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: recipient=does-not-exist@example.com
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: sasl_username=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: ccert_pubkey_fingerprint=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: instance=7289.556db23f.b6b7d.0
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: protocol_name=ESMTP
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: encryption_keysize=256
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: recipient_count=0
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: ccert_issuer=
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: sender=[...]
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: client_name=[...]
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Attribute: client_address=[...]
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Recipient Domain: example.com
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Recipient userid: does-not-exist
/opt/zimbra/libexec/zmpostfixpolicyd[29326]: Action: dunno

If you see these logs, then zmpostfixpolicyd is set up correctly in postfix. If you don’t, check the postfix configuration.

But the policy daemon always returned “dunno”, even if the recipient did not exist. Therefore, I looked into the perl code of the policy daemon (/opt/zimbra/libexec/zmpostfixpolicyd):

The original code includes this part:

    if (lc(($ldap->get_values("zimbraDomainType"))[0]) eq "alias") {
      my $robject = ($ldap->get_values("zimbraMailCatchAllForwardingAddress"))[0];
      syslog $syslog_priority, "Real Domain: %s", $robject if $verbose;
      @attrs=('1.1');
      $mesg = $ldap->search_s(
                "",
                LDAP_SCOPE_SUBTREE,
                "(&(|(zimbraMailDeliveryAddress=$user"."$robject)(zimbraMailDeliveryAddress=$daddr)(zimbraMailAlias=$user".
                "$robject)(zimbraMailAlias=$daddr)(zimbraMailCatchAllAddress=$user"."$robject)(zimbraMailCatchAllAddress=$robject)".
                "(zimbraMailCatchAllAddress=$daddr))(zimbraMailStatus=enabled))",
                \@attrs,
                0,
                $result
              );
      $ent = $ldap->first_entry();
      $ldap->unbind;
      if ($ent != 0) {
        return "dunno";
      } else {
        return "reject 5.1.1 Mailbox unavailable";
      }
    } else {
      $ldap->unbind;
      return "dunno";
    }

So basically, this code only rejects mail to recipients for alias-domains, but accepts all mail to domains that are not alias-domains. I adjusted the code as follows:

    if (lc(($ldap->get_values("zimbraDomainType"))[0]) eq "alias") {
      my $robject = ($ldap->get_values("zimbraMailCatchAllForwardingAddress"))[0];
      syslog $syslog_priority, "Real Domain: %s", $robject if $verbose;
      @attrs=('1.1');
      $mesg = $ldap->search_s(
                "",
                LDAP_SCOPE_SUBTREE,
                "(&(|(zimbraMailDeliveryAddress=$user"."$robject)(zimbraMailDeliveryAddress=$daddr)(zimbraMailAlias=$user".
                "$robject)(zimbraMailAlias=$daddr)(zimbraMailCatchAllAddress=$user"."$robject)(zimbraMailCatchAllAddress=$robject)".
                "(zimbraMailCatchAllAddress=$daddr))(zimbraMailStatus=enabled))",
                \@attrs,
                0,
                $result
              );
      $ent = $ldap->first_entry();
      $ldap->unbind;
      if ($ent != 0) {
        syslog $syslog_priority, "Existing Alias-Recipient";
        return "dunno";
      } else {
        syslog $syslog_priority, "Non-Existing Alias-Recipient";
        return "reject 5.1.1 Mailbox unavailable";
      }
    } else {


      @attrs=('1.1');
      $mesg = $ldap->search_s(
                "",
                LDAP_SCOPE_SUBTREE,
                "(&(|(zimbraMailDeliveryAddress=$user"."$domain)(zimbraMailDeliveryAddress=$daddr)(zimbraMailAlias=$user".
                "$domain)(zimbraMailAlias=$daddr)(zimbraMailCatchAllAddress=$user"."$domain)(zimbraMailCatchAllAddress=$domain)".
                "(zimbraMailCatchAllAddress=$daddr))(zimbraMailStatus=enabled))",
                \@attrs,
                0,
                $result
              );
      $ent = $ldap->first_entry();
      $ldap->unbind;
      if ($ent != 0) {
        syslog $syslog_priority, "Existing Recipient";
        return "dunno";
      } else {
        syslog $syslog_priority, "Non-Existing Recipient";
        return "reject 5.1.1 Mailbox unavailable";
      }
    }

 

Then I restarted postfix as a zimbra user with:

zmmtactl stop
zmmtactl start

And finally, the policy daemon rejected mails to unexisting recipients 🙂

/opt/zimbra/libexec/zmpostfixpolicyd[5799]: Recipient Domain: example.com
/opt/zimbra/libexec/zmpostfixpolicyd[5799]: Recipient userid: does-not-exist
/opt/zimbra/libexec/zmpostfixpolicyd[5799]: Non-Existing Recipient
/opt/zimbra/libexec/zmpostfixpolicyd[5799]: Action: reject 5.1.1 Mailbox unavailable
postfix/smtpd[5791]: NOQUEUE: reject: RCPT from [...]: 554 5.1.1 <does-not-exist@example.com>: Recipient address rejected: Mailbox unavailable; [...]

Please use the described solution with care. It is my workaround, it is working on the system in question. It might not work with another setup. If you use it, test thoroughly if only the correct mails get rejected.

A Zimbra update will overwrite the zmpostfixpolicyd code and thus revert the change, so you might need to readjust the code after an update.

This drove me nuts and I hope this information is of help for somebody. Please comment if you have a better solution or find the described solution useful.

21. May 2015

wp-cli: syntax error, unexpected ‘?’, error connecting to the database

Filed under: Server Administration,Windows,Wordpress — Christopher Kramer @ 10:55

Running wp-cli on a website that was set up by somebody else resulted in a strange error:

PHP Parse error:  syntax error, unexpected '?' in phar:///usr/local/wp-cli.phar/php/wp-cli.php(23) : eval()'d code on line 1
Error:
<h1>Fehler beim Aufbau der Datenbank-Verbindung</h1>
...

The database settings in wp-config.php were all correct and WordPress was also working, so this did not seem to be the problem. In the end, I found out the reason was a byte order mark at the beginning of wp-config.php that had been inserted probably by Windows Editor. To solve the issue, the easiest way is to open the file in a good editor, for Windows I recommend PSPad, copy everything into the clipboard (CTRL +A, CTRL + C), create a new empty file (CTRL + N), paste everything (CTRL + V), save it (CTRL + S) as wp-config.php and upload the file.

 

20. March 2015

Remove duplicate mails from a maildir

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 19:30

In case somehow, you got duplicate mails in a maildir, like me, maybe this is helpful to you:

find /var/mail/user -type d -name cur -print0 | xargs -0 /usr/bin/fdupes -n | less

Of course you need fdupes installed. On debian:

apt-get install fdupes

The above command is the dry run, showing you the duplicates.

Then to delete the duplicates, leaving only the first found file:

find /var/mail/user -type d -name cur -print0 | xargs -0 /usr/bin/fdupes -ndN | less

Use the same thing again with “new” instead of “cur” if you have duplicate unread mails. This only looks at duplicates within the same folder, so you can also safely run it on /var/mail for all users.

Note that this assumes the duplicates have exactly the same content. If you want to clean duplicates which are a bit different, consider this tool.

Update 2018: Three years later, I had the same problem, googled it and found my own blog post 😉 And it again solved the problem 🙂

14. March 2015

Bind nameserver: view cache entries

Filed under: Linux,Server Administration — Tags: , , , , — Christopher Kramer @ 15:35

If you want to view the cache entries of your bind nameserver, this is how it works on bind9 on Debian:

rndc dumpdb
less /var/cache/bind/named_dump.db

Tested on Debian Wheezy. Hope this helps somebody.

6. March 2015

Owncloud: Upgrading to owncloud 8 fails (Integrity constraint violation in oc_filecache)

Filed under: Linux,PHP,Server Administration — Tags: , — Christopher Kramer @ 21:33

First, the upgrade to owncloud 8 in the web GUI failed. Then, I performed the upgrade in the console like this:

sudo -u www-data php /var/www/owncloud/occ upgrade

This gave an error like this (I don’t remember exactly):

An exception occurred while executing 'INSERT INTO "oc_filecache" ... Integrity constraint violation: key s_storage_path_hash is not unique
...
Upgrade failed

This owncloud is using a MySQL DB.

So what in the end solved the problem: First put Owncloud in maintainance mode:

sudo -u www-data php /var/www/owncloud/occ maintenance:mode --on

Then make sure no occ processes of old upgrade attempts are running. If there are, kill them. Then clear the oc_filecache using this MySQL command:

TRUNCATE oc_filecache;

Don’t worry, it will populate itself again during the upgrade. Then restart the upgrade:

sudo -u www-data php /var/www/owncloud/occ upgrade

This might take a lot of time! Better run this on a screen (see screen tutorial if you don’t know how) so it does not stop when your SSH connection breaks.

Spamassasin: MISSING_SUBJECT in every mail even though mail has a subject

Filed under: Linux,Server Administration — Tags: , , , — Christopher Kramer @ 09:48

I noticed the MISSING_SUBJECT rule matched for every mail recently even though the mails contained a subject header. Finally I found this in the spamd.log:

warn: Possible unintended interpolation of @yahoo in string at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.
warn: rules: failed to compile Mail::SpamAssassin::Plugin::Check::_head_tests_0_4, skipping:
warn:  (Global symbol "@yahoo" requires explicit package name at /etc/spamassassin/local.cf, rule MY_SPAMMY_YAHOO, line 1.)

The reason was a rule like this:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress@yahoo.de/
score           MY_SPAMMY_YAHOO        3
describe        MY_SPAMMY_YAHOO        Mail to myspammyaddress@yahoo.de is mostly spam

Escaping the @ in the rule fixed the Problem:

header          MY_SPAMMY_YAHOO        To =~ /myspammyaddress\@yahoo.de/

Hope this helps someone to spot the error faster.

The problem is that spamassasin does not autolearn ham if the ham messages match the MISSING_SUBJECT rule. So an error like this basically not only breaks lots of tests but also autolearning of ham. So lesson learned: better always look at the spamd.log after creating a new rule, even if you think you know what you are doing 😉

« Newer PostsOlder Posts »