File_DB lacked decent file locking and concurrence.\u00a0 I wasn’t really willing to move to MySQL which would have solved the problem, but, added a few minor inconveniences along the way.\u00a0 I needed to store a few thousand bytes for a number of seconds.\u00a0 While File_DB was wrapped with file locking and assisted by my own lock routine, it lacked truly concurrent access which I felt was leading to some of the issues we were seeing.<\/p>\n
However, after a relatively painless conversion from File_DB to BerkeleyDB, it did not solve the problem completely.\u00a0 The error I was addressing is now much harder to get to occur in normal use, but, I am able to reproduce it with a small test script.<\/p>\n
The documentation for the perl methods to access BerkeleyDB are a bit sparse for setting up CDB, but, after digging through the documentation, and a few examples on the net, I ended up with some code that did indeed work consistently.<\/p>\n
Since CDB isn’t documented very well, I ended up with the following script to test file locking and ensure things worked as expected.<\/p>\n
\r\n#!\/usr\/bin\/perl\r\n\r\nuse Data::Dumper;\r\nuse BerkeleyDB;\r\n\r\nmy %hash;\r\nmy $filename = \"filt.db\";\r\nunlink $filename;\r\n\r\nmy $env = new BerkeleyDB::Env\r\n -Flags => DB_INIT_CDB|DB_INIT_MPOOL|DB_CREATE;\r\n\r\nmy $db = tie %hash, 'BerkeleyDB::Hash',\r\n -Filename\u00a0\u00a0 => $filename,\r\n -Flags\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 => DB_CREATE,\r\n -Env\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 => $env\r\nor die \"Cannot open $filename: $!\\n\" ;\r\n\r\nmy $lock = $db->cds_lock();\r\n\r\n$hash{\"abc\"} = \"def\" ;\r\nmy $a = $hash{\"ABC\"} ;\r\n# ...\r\nsleep(10);\r\n\r\nprint Dumper(%hash);\r\n$lock->cds_unlock();\r\nundef $db ;\r\nuntie %hash ;\r\n<\/pre>\nPath issues caused most of the issues as did previous tests not actually clearing out the _db* and filt.db file. One test got CDB working, I modified a few things and didn’t realize I had actually broken CDB creation because the other files were still present. Once I moved the script to another location, it failed to work. A few quick modifications and I was back in business.<\/p>\n
Perhaps this will save someone a few minutes of time debugging BerkeleyDB and Perl.<\/p>\n
—–<\/p>\n
Due to a logic error in the way I handled deletions to work around the fact that BerkeleyDB doesn’t allow you to delete a single record when you have a duplicate key, my code didn’t work properly in production. After diagnosing that and fixing it with a little bit of code, 125 successive tests resulted in 100% completion. I’ve pushed it to a few machines and will monitor it, but, I do believe that BerkeleyDB fixed the issues I was having with File_DB.<\/p>\n
\n<\/div>\n<\/fb:like>\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"File_DB lacked decent file locking and concurrence.\u00a0 I wasn’t really willing to move to MySQL which would have solved the problem, but, added a few minor inconveniences along the way.\u00a0 I needed to store a few thousand bytes for a number of seconds.\u00a0 While File_DB was wrapped with file locking and assisted by my own […]<\/p>\n
\n<\/div>\n<\/fb:like>\n<\/div><\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[18,19,17],"class_list":["post-609","post","type-post","status-publish","format-standard","hentry","category-programming","tag-berkeleydb","tag-file_db","tag-perl"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=609"}],"version-history":[{"count":3,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/609\/revisions"}],"predecessor-version":[{"id":611,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/609\/revisions\/611"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}