DeutschEnglish

Submenu

 - - - By CrazyStat - - -

11. August 2013

RT Request Tracker: Migrate From SQlite to mySQL

Filed under: DBMS,Linux,Server Administration — Tags: , , , , , , — Christopher Kramer @ 23:07

I lately had to migrate an RT installation (version 4.0.4) from SQLite to MySQL. In case anyone else has to do this, here is a brief description of how it worked out.

  1. Setup a working MySQL server in case you have not already
  2. Create a MySQL user for RT (e.g. rt4)
  3. Configure RT to use MySQL using this username and a dbname (e.g. rt4) of a not yet existing db (See /etc/request-tracker4/RT_SiteConfig.pm and RT_SiteConfig.d/)
    I’d recommend to keep a copy of the SQlite-Config…
  4. Run
    rt-setup-database --action init

    to create a blank RT DB in MySQL. Check if it worked.

  5. Delete all rows from the MySQL DB, only keep the schema.
  6. Create a copy of your SQLite db
    cp rtdb mydbcopy
  7. Open the copy of the db in the sqlite shell
    sqlite3 mydbcopy
  8. For each table in the DB:
    1. Set the Output file to something like this (“Attachments” is the table name)

    .out data_Attachments

    2. Set the mode to insert (“Attachments” is again the table name)

    .mode insert Attachments

    3. Get all the data

    SELECT * FROM Attachments;

    4. Now you have a file with lots of INSERT statements for the Attachments table. Try to run it in mysql:

    mysql -u rt4 -p rt4 < data_Attachments

    (On a linux shell, not the sqlite shell of course. Here the first rt4 is the username and the second the dbname. data_Attachments is the dump file created before)
    5. In case mysql complains some NOT NULL constraints are violated:
    Go back to the sqlite shell and set these cells to the default-value (0):

    UPDATE Attachments SET someColumn=0 WHERE someColumn IS NULL;

    (Only do this on the copy, not the original db 😉 )
    Now recreate the dump and retry to insert this in mysql. Do it with all columns where NOT NULL constraints are violated.

This works table by table, there are no foreign keys that would get in the way. You can also do several tables in one output file, but you might run into problems when NOT NULL constraints are violated by one table. After all your tables are filled with your data, RT should work. Maybe restart Apache.

apache2ctl restart

This worked without problems so far for me. I first tried pumping the whole SQLite dump into MySQL (using this conversion script) but the schema that this ended up in was different, missed indexes and RT only liked it until I restarted Apache (which then refused to start). Better start with a schema created by rt-setup-database, not with one that originates from SQLite.

I hope this is of some help for somebody. Please let me know in case it helped you or if you have any comments or questions.

Recommendation

Try my Open Source PHP visitor analytics script CrazyStat.

18. March 2013

phpLiteAdmin 1.9.4 released

Filed under: DBMS,PHP,phpLiteAdmin — Tags: , , , , , , , , , , — Christopher Kramer @ 18:53

I just released phpLiteAdmin 1.9.4. phpLiteAdmin 1.9.4

phpLiteAdmin is for SQLite what phpMyAdmin is for MySQL: A web GUI to manage your databases.

A lot of work has again gone into this release. It fixes bugs and introduces new features. No security issues fixed (compared to 1.9.3.3).

Every user of phpLiteAdmin is recommended to update.

New features of phpLiteAdmin 1.9.4 include:

  • Multi-Language support
  • external configuration possible
  • empty password -> no login required
  • easy backup of db files
  • edit and delete possible from search results
  • search function: added “LIKE %…%”
  • css and Js now served as separate, cacheable and compressed resources to speed up page loading

Important bug fixes include fixes in the ALTER TABLE support. We have again spent quite a lot of work to improve phpLiteAdmin. Thanks to everybody who reported issues and especially to the team for your work on phpLiteAdmin – especially Dreadnaut and Teryaki helped me a lot in this release. Thanks guys.

Download the new version here.

15. January 2013

phpLiteAdmin 1.9.3.3 released fixing an XSS vulnerability

Filed under: PHP,phpLiteAdmin,Security — Tags: , , , , , , — Christopher Kramer @ 11:53

It seems currently people have a very close look at security of phpLiteAdmin. This is really good. We immediately fix any security issue we get aware of. Therefore, we yesterday released another security patch with version 1.9.3.3. The security issue fixed in this version is an XSS vulnerability. The risk of this particular issue is considered medium. All users of phpLiteAdmin < 1.9.3.3 are advised to update to the new version. Users of the development version of 1.9.4 should please update to the latest revision from svn, at least revision 317 (2013-01-14).

We are really sorry for those users who needed to update phpLiteAdmin three times in the last days. But the only way to solve security issues is to patch, just ignoring them will not help.

Thanks a lot to Urd for making us aware of this issue. If anybody thinks he found a security issue, please do as Urd did and contact us.

To update phpLiteAdmin, just download the new version, adjust the configuration and replace the phpliteadmin.php with the new one.

Christopher Kramer,

member of the phpLiteAdmin team

12. January 2013

phpLiteAdmin: Another security release 1.9.3.2

Filed under: PHP,phpLiteAdmin,Security — Tags: , , , , , , — Christopher Kramer @ 01:27

Yesterdays security release for phpLiteAdmin unfortunately did not fix the security issues described by an exploit published by “l@usch” completely. Therefore, today a new release 1.9.3.2 was published to cover the remaining issues.

All users of phpLiteAdmin < 1.9.3.2 are advised to update their installation to 1.9.3.2. The fixed security issues can only be used by users with access to phpLiteAdmin, i.e. users that know the password. As a general recommendation, you should always use a secure password an keep it secret. Never use the default password on a publicly accessible installation.

I’d like to thank l@usch for reporting the issue and his cooperation to resolve it.

To update, just download the new version, adjust the configuration and replace your old phpliteadmin.php with the new one.

11. January 2013

phpLiteAdmin: Release 1.9.3.1 fixes a security issue

Filed under: PHP,phpLiteAdmin,Security — Tags: , , , , , — Christopher Kramer @ 00:07

Today, a security issue of phpLiteAdmin was discovered and published. The new version 1.9.3.1, which has just been released, fixes this issue. All users of phpLiteAdmin are recommended to update their installation to the current version 1.9.3.1.
The security issue can only be used by users with access to phpLiteAdmin, i.e. users that know the password, to gain more rights on the system. Therefore, the issue cannot be exploited if you use a strong password an keep it secret. For this reason, this is always recommended. Please never use the default password if phpLiteAdmin is publicly accessible over the internet.

To update phpLiteAdmin, just adjust the configuration in the new version and replace the phpliteadmin.php with the new one. Done in a minute or two.

You can download the new version here.

3. November 2012

phpLiteAdmin 1.9.3 released (security-update)

Filed under: DBMS,PHP,phpLiteAdmin,Security,Server Administration — Tags: , , , , , , , , , , — Christopher Kramer @ 00:45
Screenshot of phpLiteAdmin 1.9.3

Screenshot of phpLiteAdmin 1.9.3

Some minutes ago, I released the new version of phpLiteAdmin, a web management GUI for SQLite databases written in PHP. You can download it from our project site.

The new version addresses and mostly fixes lots of issues. Among these, one security issue has been fixed. Therefore, I’d recommend anybody using phpLiteAdmin to update.

A lot of work has gone into this release, fixing lots of bugs to make phpLiteAdmin more robust. For example, you can now have tables or columns containing special characters. The ALTER TABLE features have been partly rewritten so they now work a lot more reliable. And lots of other issues have been fixed. Thanks to anybody who reported bugs to the bug tracker.

If you still have any problems or suggestions, please let us know on our issue tracker.

7. October 2012

SQLite: escaping table and column names correctly

Filed under: DBMS,PHP,phpLiteAdmin — Tags: , , , , , , , , , , — Christopher Kramer @ 21:44

Summary:

Use double quotes around column names, table names, trigger names etc.

SELECT "column name", "my column", FROM "table name"

Use single quotes around string values

SELECT * FROM "table name" WHERE "column name" = 'string value'

Use two double quotes if e.g. your column name contains a double quote (e.g. my”column)

SELECT "my""column" FROM "my""table"

Details:

I joined the development team of phpliteadmin some months ago and am currently fixing several issues. One problem is caused by SQLite’s flexibility of column- and table names. A column named “SELECT”, “my column”, “this, that”, or even “this contains ‘single quotes'” is completely legal in SQLite. And what is legal is being used. So phpliteadmin needs to be able to handle these types of names as well, which is not really the case at the moment.

Let’s assume you query a column called “select”, you are likely to get errors like that in PHP:

Warning: SQLiteDatabase::query() [sqlitedatabase.query]: near “select”: syntax error in …

Or using PDO:

HY000 / 1 / near “select”: syntax error

From MySQL I knew backticks around table or column identifiers like in this example:

SELECT `select`, `my column`, `this, that` FROM `group by table`

(By the way, you see my syntax highlighter has problems with reserved words as column names as well ;-))

I tried this in SQLite and it worked, so I went that way. But there are different versions of SQLite and for the old version that, in PHP, is implemented by SQLiteDatabase, would give the following error:

Warning: SQLiteDatabase::query() [sqlitedatabase.query]: unrecognized token: “`” in …

So I found out that instead of backticks, normal single quotes work:

SELECT * FROM 'group by table'

This works well, but there is a problem here with column names:

SELECT 'select' FROM 'group by table'

This will not work as expected. It will not return the values of the column ‘select’, but create a column filled with the text ‘select’. That is of course standard SQL behaviour and shows that it is a bad idea to use single quotes around column or table names. The bug report by maitredede on the phpliteadmin issue tracker made me aware of another way to escape column and table names:

SELECT [select], [my column], [this, that] FROM [group by table]

This works very well, both with SQLiteDatabase, SQLite3 and PDO when accessing SQLite from PHP. I have not tried it using the SQLite C library directly, but it should work as well.

So I’d say this is the best way to do this as it works the same way with all versions of SQLite / all PHP SQLite extensions.

 

Update:

Well, probably I should have read the manual first:

SQlite supports ` for compatibility with MySQL and [] for compatibility with MS Access and SQL Server. But it also supports double quotes, which is standard SQL. So I’d say this is the best way to do it:

SELECT "select", "my column", "this, that" FROM "group by table"

It works with all SQLite versions / extensions just like the [] syntax. (And my syntax highlighter supports it as well ;-).)

Update 2:

Another thing that might be of interest: What to do, if a column/table name contains double quotes? Well, in this case duplicate them and you are done. Let’s assume you have a column called like this:

my “column

Then you query it like this:

SELECT "my""column" FROM "my_table"

4. October 2012

PDO / sqlite: database table is locked

Filed under: DBMS,PHP,phpLiteAdmin — Tags: , , , , , , , — Christopher Kramer @ 22:06

At the moment I am working again on phpliteadmin, a  php-based web GUI for database administration of sqlite databases. While debugging, I stumbled across a problem that only occurred with the PDO extension (not with SQLiteDatabase or SQLite3). I got the following error message while trying to drop a table:

HY000 / 6 / database table is locked

By the way, you can use PDO::errorInfo() to get these error messages. So as the error correctly explains, the table I tried to drop seemed to be in use. PDO documentation for PDO::query() also explains the problem (even though DROP TABLE statements are fired using PDO::exec()):

If you do not fetch all of the data in a result set before issuing your next call to PDO::query(), your call may fail. Call PDOStatement::closeCursor() to release the database resources associated with the PDOStatement object before issuing your next call to PDO::query().

So I hunted for the open cursor on a resultset of the table in question and could not find any. Finally, I found the SQL statement which still had an open cursor:

SELECT * FROM sqlite_master

Thinking about this, it is obvious: When querying sqlite_master, you request information about database tables. If one of the table gets altered or dropped, this might change the data listed in the resultset on which I still have a cursor.

Maybe this is a special case as usually you do not query sqlite_master a lot. But in case you do, this might be useful information.

To solve the problem, as the manual says, simply release the cursor using PDOStatement::closeCursor() before dropping/altering tables.

30. May 2012

Released: phpLiteAdmin 1.9.2 includes CSV import/export

Filed under: DBMS,PHP,phpLiteAdmin,Server Administration — Tags: , , , , , , — Christopher Kramer @ 15:54

As I wrote in March, I implemented CSV import and fixed export issues of phpLiteAdmin. This fixed phpLiteAdmin bug #71. I also wrote a small fix for bug #75. Today, new version 1.9.2 of phpLiteAdmin was released including both fixes. You can download it here.

I’d like to thank the phpLiteAdmin team for including my work and allowing me to join the team. I plan to address more issues of phpLiteAdmin in the future to push phpLiteAdmin a little further. There is still some more work to be done which I will have a look at once I find the time.

I recommend the new version to anybody using phpLiteAdmin (and also everybody who doesn’t yet ;-)). Please use the bugtracker in case you find any issues.

I hope some of you find the new features useful or are happy to see those bugs fixed.

Thanks again to the phpLiteAdmin team for the great tool and the opportunity to contribute to the project. Fortunately, I do not have to create a fork to improve the tool.

 

13. March 2012

sqlite.js: SQLite for Javascript!

Filed under: DBMS,JavaScript — Tags: , , , , , — Christopher Kramer @ 12:01

I just found something quite cool which I thought might be interesting to some of you.

Alon Zaka created emskripten, a LLVM-to-JavaScript converter. It can convert LLVM-bytecode such as provided by compilation from C/C++ code into JavaScript. This approach seems quite cool and it already proved to produce some cool results:

He converted the SQLite-library, which is written in C, into JavaScript. You can find (and fork) the project on github or just try the demo.

The first thing that I was curious about was: How does it store the database file? Well, it creates a new database when you do SQL.open(). But you can pass data to SQL.open(data) to start with a pre-filled database. So I think using this library would normally look like this:

  1. fetch the initial database from the server (e.g using AJAX). At least DB schema and some important data.
  2. open the database with SQL.js
  3. do some operations on the database, mostly SELECTs probably
  4. maybe fetch some more data using AJAX from the server when needed
  5. maybe send some data to the server to save changes using form or ajax

So this would mean we move some more stuff to the client. It would allow us to use SQL on the client side just like we do on the server-side. So we could use the same queries for javascript-based clients on the client and for html-only-clients on the server. (But exposing SQL-queries used in the server might make SQL injection as easy as never before…)

I see two other interesting ways of using this:

  1. In combination with HTML5 local storage – this would allow to store a persistent client-side database that could be accessed using SQL. Sounds pretty cool and the main usage scenario of SQL.js for me.
  2. On the server using node.js – but when I think about it: There are better ways to access a database within node.js, so this is probably only showing that it works, but no real usage scenario.

Tell me what you think about it.

Older Posts »