After updating from PHP5.3 to PHP 5.4, on some sites text was missing. No error could be found in the error log so I had to dig into the code to find out what was going on.
The root cause is that with PHP5.4, the default character set expected by htmlentites(), htmlspecialcharacters() and html_entity_decode() changed from ISO-8859-1 to UTF-8. So if a script passes ISO-8859-1 characters like German “Umlaute” (öäüÖÄÜß) to one of these functions without specifying the charset with the corresponding parameter, these functions will return an empty string. And unfortunately, with PHP 5.4, they also removed the error message that PHP 5.3 recorded in the logfile in this case. This makes finding the problem a lot more difficult.
So what can you do about it? You could
- Use PHP 5.3 😉
Here is a blog post on downgrading to PHP 5.3. on Debian Wheezy - change the used charset to UTF-8
This might require changing the character set in files, databases or config files, depending on what is used on the site.
I explained in a blog post how to change the charset in Typo3 to UTF-8 back in 2012. - Provide ISO-8859-1 as a parameter to all calls of htmlspecialcharacters() etc.
So for the third option, what you have to do is find places like this:
htmlspecialchars($string);
And replace them with something like:
htmlspecialchars($string, ENT_COMPAT | ENT_XHML, 'ISO-8859-1');
The problem is that it’s hard to do this automatically. What is easy to do, is replace all htmlspecialchars()-calls with calls to htmlspecialchars_PHP5-3() etc. and place these functions there:
function htmlspecialchars_PHP5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
return htmlspecialchars($string, $ent, $charset);
}
function htmlentities_PHP-5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
return htmlentities($string, $ent, $charset);
}
function html_entity_decode_PHP-5-3($string, $ent=ENT_COMPAT, $charset='ISO-8859-1') {
return html_entity_decode($string, $ent, $charset);
}
So just do a search & replace over all files and make sure that all scripts have a file included that contains these functions.