{"id":604,"date":"2009-03-28T02:27:27","date_gmt":"2009-03-28T06:27:27","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=604"},"modified":"2009-03-28T02:27:27","modified_gmt":"2009-03-28T06:27:27","slug":"embedded-indexing-versus-clientserver","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/programming\/embedded-indexing-versus-clientserver\/","title":{"rendered":"Embedded indexing versus Client\/Server"},"content":{"rendered":"

For a particular application, I require temporary persistent storage of some data.\u00a0 That data consists of a key value and a payload.\u00a0 That key value can be a dupe, which is what causes the problem.<\/p>\n

File_DB in perl handles duplicates and I can delete a key\/value pair without too much difficulty.\u00a0 However, file locking is not handled very well with File_DB which created concurrency issues with the threaded daemon.<\/p>\n

Sqlite3 had no problem with duplicates, and could be compiled with the delete from\/limit clause to easily handle duplicate keys.\u00a0 Rather than recompile the packaged Sqlite3 in Debian, I made a slight modification to the code on my side so that I could do further testing.\u00a0 Due to a few issues with threading and a potential issue with storing binary data and retrieving it in perl, I needed to reevaluate.<\/p>\n

BerkeleyDB solves a few problems.\u00a0 It supports concurrency, it supports proper file locking, but, a minor limitation is that duplicate keys are not handled well when you want to delete a key.\u00a0 It’ll require a rewrite of some functionality to use BerkeleyDB, but, I believe that solution will provide the least potential for failures.<\/p>\n

I could have use MySQL which I am very comfortable with, but, the storage of the data really only needs to be there for a few minutes in most cases, and the amount of data stored is 10-20K at most.\u00a0 With MySQL’s client timeout, I couldn’t really guarantee everything would work every time without writing in considerable error checking.\u00a0 While MySQL would handle everything perfectly, it was overkill for the task at hand.<\/p>\n

I’m rewriting the File_DB methods to use BerkeleyDB and modifying the saved data slightly to work around the key delete issue.<\/p>\n

It should work and should raise the reliability of this process from 99.2% to 99.9% which will be a considerable improvement.<\/p>\n

\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"excerpt":{"rendered":"

For a particular application, I require temporary persistent storage of some data.\u00a0 That data consists of a key value and a payload.\u00a0 That key value can be a dupe, which is what causes the problem. File_DB in perl handles duplicates and I can delete a key\/value pair without too much difficulty.\u00a0 However, file locking is […]<\/p>\n

\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[18,19,16,20],"class_list":["post-604","post","type-post","status-publish","format-standard","hentry","category-programming","tag-berkeleydb","tag-file_db","tag-mysql","tag-sqlite3"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/604","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=604"}],"version-history":[{"count":1,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/604\/revisions"}],"predecessor-version":[{"id":605,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/604\/revisions\/605"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=604"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=604"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=604"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}