Recently one of our servers had some filesystem corruption – corruption that has occurred more than once over time. As we use hardlinks a lot with link-dest and rsync, I’m reasonably sure the issue occurs due to the massive number of hardlinks and deletions that take place on that system.<\/p>\n
I’ve written a small script to repeatedly test things and started it running a few minutes ago. My guess is that the problem should show up in a few days.<\/p>\n
\r\n#!\/bin\/bash\r\n\r\nRSYNC=\/usr\/bin\/rsync\r\nREVISIONS=10\r\n\r\nfunction rsync_kernel () {\r\n DATE=`date +%Y%m%d%H%M%S`\r\n\r\n BDATES=\"\"\r\n loop=0\r\n for f in `ls -d1 \/tmp\/2011*`\r\n do\r\n BDATES[$loop]=$f\r\n loop=$(($loop+1))\r\n done\r\n\r\n CT=${#BDATES[*]}\r\n\r\n if (( $CT > 0 ))\r\n then\r\n RECENT=${BDATES[$(($CT-1))]}\r\n LINKDEST=\" --link-dest=$RECENT\"\r\n else\r\n RECENT=\"\/tmp\/linux-3.0.3\"\r\n LINKDEST=\" --link-dest=\/tmp\/linux-3.0.3\"\r\n fi\r\n\r\n $RSYNC -aplxo $LINKDEST $RECENT\/ $DATE\/\r\n\r\n if (( ${#BDATES[*]} >= $REVISIONS ))\r\n then\r\n DELFIRST=$(( ${#BDATES[*]} - $REVISIONS ))\r\n loop=0\r\n for d in ${BDATES[*]}\r\n do\r\n if (( $loop < = $DELFIRST ))\r\n then\r\n `rm -rf $d`\r\n fi\r\n loop=$(($loop+1))\r\n done\r\n fi\r\n}\r\n\r\nwhile [ 1==1 ]\r\ndo\r\n rsync_kernel\r\n echo .\r\n sleep 1\r\ndone\r\n<\/pre>\n\n<\/div>\n<\/fb:like>\n<\/div><\/div>","protected":false},"excerpt":{"rendered":"Recently one of our servers had some filesystem corruption – corruption that has occurred more than once over time. As we use hardlinks a lot with link-dest and rsync, I’m reasonably sure the issue occurs due to the massive number of hardlinks and deletions that take place on that system. I’ve written a small script […]<\/p>\n
\n<\/div>\n<\/fb:like>\n<\/div><\/div>","protected":false},"author":15,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[78,100,193],"class_list":["post-1225","post","type-post","status-publish","format-standard","hentry","category-infrastructure","tag-kernel","tag-linux","tag-xfs"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=1225"}],"version-history":[{"count":2,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1225\/revisions"}],"predecessor-version":[{"id":1227,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1225\/revisions\/1227"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=1225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=1225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=1225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}