DeutschEnglish

Submenu

 - - - By CrazyStat - - -

19. March 2019

PHP: DateTime::createFromFormat fails for string read from CSV

Filed under: PHP — Tags: , , , , , — Christopher Kramer @ 19:31

I wrote a small PHP script to import a dozen csv files exported from Excel into a database. The CSV import basically looked like this:

<?php
$f = "file.csv";	
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
    }
} ?>

And now what happened is that $date was false. So I fetched the errors like this:

var_dump(DateTime::getLastErrors());

And this returned:

array(4) {
  ["warning_count"]=>
  int(0)
  ["warnings"]=>
  array(0) {
  }
  ["error_count"]=>
  int(1)
  ["errors"]=>
  array(1) {
    [0]=>
    string(22) "Unexpected data found."
  }
}

So I added var_dump($date), but it gave string(13) "31.01.2019", which looked right. But looking closely, the string length of 13 seems a bit long for a 10 character date, right? I tried trim() , but without luck. And then I remembered that I had a similar problem before where invisible empty space was due to the UTF-8 Byte Order Mark (BOM). This is a sequence of “inivisible” bytes at the beginning of a textfile that define in which unicode encoding the file is (UTF-8, UTF-16, …) and its endianess (big-endian or little-endian). Microsoft Office programs such as Excel or Word like to write this to the beginning of a file, but other programs may do so as well.

So the solution is simple: In the first line, strip the BOM if it is there:

<?php
$f = "file.csv";
$bom = pack('CCC', 0xEF, 0xBB, 0xBF);
$firstline=true;
if (($handle = fopen($f, "r")) !== FALSE) {
    while (($data = fgetcsv($handle, 1000, ";")) !== FALSE) {
         if ($firstline and substr($data[0], 0, 3) === $bom)
             $data[0] = substr($data[0], 3);
         $firstline=false;
         $date = DateTime::createFromFormat('d.m.Y', $data[0]);
         // Inserting this along with other data into a DB
    }
} ?>

So this just checks whether the first three bytes in the file match the UTF-8 BOM added by Excel and in case it detects them, it remove these bytes. Now the date parses fine. If your file has a different BOM, e.g. for UTF-16, you may need to change the definition of $bom. Just check your file in a hex-editor to find the first three bytes. This is PSPad, a great text-editor that includes a HEX-editor:

Note the first three bytes EF BB BF, which are the BOM.

If this helped you to solve your problem faster, please drop a comment below. This motivates me to keep writing these articles.

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

30. May 2012

Released: phpLiteAdmin 1.9.2 includes CSV import/export

Filed under: DBMS,PHP,phpLiteAdmin,Server Administration — Tags: , , , , , , — Christopher Kramer @ 15:54

As I wrote in March, I implemented CSV import and fixed export issues of phpLiteAdmin. This fixed phpLiteAdmin bug #71. I also wrote a small fix for bug #75. Today, new version 1.9.2 of phpLiteAdmin was released including both fixes. You can download it here.

I’d like to thank the phpLiteAdmin team for including my work and allowing me to join the team. I plan to address more issues of phpLiteAdmin in the future to push phpLiteAdmin a little further. There is still some more work to be done which I will have a look at once I find the time.

I recommend the new version to anybody using phpLiteAdmin (and also everybody who doesn’t yet ;-)). Please use the bugtracker in case you find any issues.

I hope some of you find the new features useful or are happy to see those bugs fixed.

Thanks again to the phpLiteAdmin team for the great tool and the opportunity to contribute to the project. Fortunately, I do not have to create a fork to improve the tool.

 

10. March 2012

phpliteadmin: CSV import and export

Filed under: DBMS,PHP,phpLiteAdmin,Server Administration — Tags: , , , , , , , , — Christopher Kramer @ 12:24

In case you use sqlite, a serverless, transactional SQL-complete database engine, you might also know phpliteadmin. It is a web-based DB admin tool just like phpMyAdmin is for MySQL. It is small and does not require complex installation just like sqlite. It is great to do most simple operations on a sqlite database, but in my opinion it still has some important limitations and bugs.

One thing I recently missed was the possibility to import CSV files into a SQLite database. The SQLite shell makes this very easy (if the file has the correct syntax). But this has also some limitations and you have to leave the phpLiteAdmin GUI and start a SQLite shell. So I thought it should be part of phpLiteAdmin. It already had some GUI options for CSV import, but the actual import implementation was missing, so I implemented CSV import myself. On the way testing the exported csv and importing it again I noticed some export bugs of phpliteadmin like this one and fixed them on the way.

So those are the changes I did to phpLiteAdmin:

Import:

  • implemented CSV import with some features like:
  • you can define Enclosure, Escaper, Null and first-row-field-description just like when exporting
  • using PHP’s fgetcsv()
  • can import files that were exported with phpliteadmin using the same settings!
  • can even import files exported with phpliteadmin without my corrections and improvements (see below), although they are not “standard CSV”.

Export:

  • removed line terminator option because:
    • this was not implemented
    • in my opinion anything else than a linebreak does not make sense, so useless feature
  • Fix: Tables with indexes were exported multiple times
  • Fix: Last column was terminated which is not usual in CSV
  • NULLs are not enclosed anymore to allow distinction between “NULL”-text
  • FIX: Removing CRLF was not implemented
  • FIX: Tables that were not exported caused additional linebreaks in output

You can find anything related to these changes in phpLiteAdmin’s bug tracker.

You can also get my version with the improvements from the bugtracker or download it here.

I really hope this will make it into the next release of phpLiteAdmin, but unfortunately I did not get any feedback by the developers of phpLiteAdmin yet. Hopefully this is not getting a fork 😉

I hope you like these features and bugfixes. You can give me feedback here or in the phpliteadmin bug tracker.

I really think a good database management tool for SQLite is needed and phpLiteAdmin has the potential to become one. We will see.

You can also tell me if you use some other management tool for SQLite which you think is way better 😉

Update 2012/05/13: The phpliteadmin developers just announced that an official new version including these improvements will come up soon. Great news, I am looking forward to this new version!