{"id":602,"date":"2009-03-23T02:09:28","date_gmt":"2009-03-23T06:09:28","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=602"},"modified":"2009-03-23T02:09:28","modified_gmt":"2009-03-23T06:09:28","slug":"multithreaded-madness","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/programming\/multithreaded-madness\/","title":{"rendered":"Multithreaded madness"},"content":{"rendered":"

An application I wrote long ago that used File_DB for short-term persistent multithreaded storage had a few issues with concurrency.\u00a0 I debated rewriting the script to use BerkeleyDB which included proper file locking, but, decided to use Sqlite3 instead as it was closer to SQL and would eliminate a bit of code.<\/p>\n

The transition was relatively easy.\u00a0 Writing self-test functions worked well and a few bugs were dealt with along the way.\u00a0 Most of the issues were getting used to Sqlite3’s quirks, but, all in all the code worked fine.\u00a0 Multiple tests with multiple terminal windows resulted in everything working as expected including locking tables, concurrency issues and removing a logic error on the prior application.<\/p>\n

First startup of the application resulted in a rather strange result which didn’t make a lot of sense.\u00a0 I chalked that up to something I had done during testing, deleted the sqlite3 database file and restarted the application.<\/p>\n

Success.<\/p>\n

The application started, set it self as a daemon and detached from the terminal.\u00a0 I sent a task to the daemon, and bam.\u00a0 It seemed to work, it complained of a length error in the unpack which meant there was some data that didn’t get retrieved correctly from the database.\u00a0 A second task was sent and the error received was even stranger.\u00a0 Trying to connect to sqlite3 through the command line resulted in:<\/p>\n

sqlite> select * from tasks;
\nSQL error: database disk image is malformed<\/p><\/blockquote>\n

Ok, something broke badly.<\/p>\n

I checked and doublechecked my code with perl running in strict mode and could find nothing that would cause this.\u00a0 It seems that the packaged version of sqlite3 in debian’s packaged perl is not compiled with threading enabled.<\/p>\n

Oops.<\/p>\n

I missed that when I was digging through the library configs and will have to build that package for testing.\u00a0 I did want to avoid using the BerkeleyDB library and move to Sqlite3, but, I think in the interest of finishing this project a little sooner, I will rewrite the code and adjust the locking and use Sqlite3 in the future.<\/p>\n

Sqlite3 works very well with SQLAlchemy and TurboGears, but, in this case, it didn’t quite solve the problem that I needed solved.<\/p>\n

\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"excerpt":{"rendered":"

An application I wrote long ago that used File_DB for short-term persistent multithreaded storage had a few issues with concurrency.\u00a0 I debated rewriting the script to use BerkeleyDB which included proper file locking, but, decided to use Sqlite3 instead as it was closer to SQL and would eliminate a bit of code. The transition was […]<\/p>\n

\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[18,19,17,20],"class_list":["post-602","post","type-post","status-publish","format-standard","hentry","category-programming","tag-berkeleydb","tag-file_db","tag-perl","tag-sqlite3"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/602","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=602"}],"version-history":[{"count":1,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/602\/revisions"}],"predecessor-version":[{"id":603,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/602\/revisions\/603"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=602"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=602"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=602"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}