{"id":1260,"date":"2011-11-01T23:34:57","date_gmt":"2011-11-02T03:34:57","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=1260"},"modified":"2011-11-01T23:34:57","modified_gmt":"2011-11-02T03:34:57","slug":"ext4-xfs-and-btrfs-benchmarks-and-testing","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/infrastructure\/ext4-xfs-and-btrfs-benchmarks-and-testing\/","title":{"rendered":"Ext4, XFS and BtrFS benchmarks and testing"},"content":{"rendered":"

Recently I talked about versioning filesystems<\/a> available for OSS systems. While most of our server farms use XFS, we have been moving to Ext4 on a number of machines. This wasn’t done as a precursor to BtrFS but problems we’ve been having with XFS on very large filesystems. The fact that we can migrate Ext4 to BtrFS in-place is just a coincidental bonus.<\/p>\n

While ZFS is still a consideration if we move to FreeBSD (I was not suitably impressed with Debian’s K*BSD project enough to consider it stable enough for production), I felt that looking at BtrFS might be worth a look. There is also CephFS but that requires a little more infrastructure as you need to run a cluster and it isn’t really made for single machine deployments.<\/p>\n

We’re also going to make some assumptions and do things you might not want to do on a home system. Since the data center we’re in has a 100% power SLA, we can be sure that we won’t lose power and can be a little more aggressive. We disable atime which may negatively impact clients if you are dealing with a disk that handles your mailspool. Also, recent versions of XFS handle atime updates much differently, so, the performance boost from noatime is negligible.<\/p>\n

Command used to test:<\/p>\n

\r\n\/usr\/sbin\/bonnie++ -s 8g -n 512\r\n<\/pre>\n
Ext4<\/h2>\n
\r\nmkfs -t ext4 \/dev\/sda5\r\nmount -o noatime \/dev\/sda5 \/mnt\r\n<\/pre>\n
Results:<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   332  99 54570  12 23512   5  1615  98 62905   6 131.8   3\r\nLatency             24224us     471ms     370ms   13739us     110ms    5257ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 24764  69 267757  98  2359   6 25084  67 388005  98  1403   3\r\nLatency              1258ms    1402us   11767ms    1187ms      66us   11682ms\r\n\r\n1.96,1.96,version,1,1320193244,8G,,332,99,54570,12,23512,5,1615,98,62905,6,131.8,3,512,,,,,24764,69,267757,98,2359,6,25084,67,388005,98,1403,3,24224us,471ms,370ms,13739us,110ms,5257ms,1258ms,1402us,11767ms,1187ms,66us,11682ms\r\n<\/pre>\n
Ext4 with journal conversion and mount options<\/h2>\n
\r\nmkfs -t ext4 \/dev\/sda5\r\ntune2fs -o journal_data_writeback \/dev\/sda5\r\nmount -o rw,noatime,data=writeback,barrier=0,nobh,commit=60 \/dev\/sda5 \/mnt\r\n<\/pre>\n
Results:<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   335  99 53396  11 25240   6  1619  99 62724   6 130.9   5\r\nLatency             23955us     380ms     231ms   15962us     143ms   16261ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 24253  65 266963  98  2341   6 24567  65 389243  98  1392   3\r\nLatency              1232ms    1405us   11500ms    1232ms     130us   11543ms\r\n\r\n1.96,1.96,version,1,1320192213,8G,,335,99,53396,11,25240,6,1619,99,62724,6,130.9,5,512,,,,,24253,65,266963,98,2341,6,24567,65,389243,98,1392,3,23955us,380ms,231ms,15962us,143ms,16261ms,1232ms,1405us,11500ms,1232ms,130us,11543ms\r\n<\/pre>\n
XFS:<\/h2>\n
\r\nmount -t xfs -f \/dev\/sda5\r\nmount -o noatime \/dev\/sda5 \/mnt\r\n<\/pre>\n
Results:<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   558  98 55174   9 26660   6  1278  96 62598   6 131.4   5\r\nLatency             14264us     227ms     253ms   77527us   85140us     773ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512  2468  19 386301  99  4311  25  2971  22 375624  99   546   3\r\nLatency              1986ms     346us    1341ms    1580ms      82us    5904ms\r\n1.96,1.96,version,1,1320194740,8G,,558,98,55174,9,26660,6,1278,96,62598,6,131.4,5,512,,,,,2468,19,386301,99,4311,25,2971,22,375624,99,546,3,14264us,227ms,253ms,77527us,85140us,773ms,1986ms,346us,1341ms,1580ms,82us,5904ms\r\n<\/pre>\n
XFS, mount options:<\/h2>\n
\r\nmkfs -t xfs -f \/dev\/sda5\r\nmount -o noatime,logbsize=262144,logbufs=8 \/dev\/sda5 \/mnt\r\n<\/pre>\n
Results:<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   563  98 55423   9 26710   6  1328  99 62650   6 129.5   5\r\nLatency             14401us     345ms     298ms   20328us     119ms     357ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512  3454  26 385552 100  5966  35  4459  34 375917  99   571   3\r\nLatency              1625ms     360us    1323ms    1243ms      67us    5060ms\r\n\r\n1.96,1.96,version,1,1320196498,8G,,563,98,55423,9,26710,6,1328,99,62650,6,129.5,5,512,,,,,3454,26,385552,100,5966,35,4459,34,375917,99,571,3,14401us,345ms,298ms,20328us,119ms,357ms,1625ms,360us,1323ms,1243ms,67us,5060ms\r\n<\/pre>\n
XFS, file system creation options and mount options:<\/h2>\n
\r\nmkfs -t xfs -d agcount=32 -l size=64m -f \/dev\/sda5\r\nmount -o noatime,logbsize=262144,logbufs=8 \/dev\/sda5 \/mnt\r\n<\/pre>\n
Results:<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   561  97 54674   9 26502   6  1235  95 62613   6 131.4   5\r\nLatency             14119us     346ms     247ms   94238us   76841us     697ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512  9576  73 383305 100 14398  85  9156  70 373557  99  2375  14\r\nLatency              1110ms     375us     301ms     850ms      36us    5772ms\r\n1.96,1.96,version,1,1320198613,8G,,561,97,54674,9,26502,6,1235,95,62613,6,131.4,5,512,,,,,9576,73,383305,100,14398,85,9156,70,373557,99,2375,14,14119us,346ms,247ms,94238us,76841us,697ms,1110ms,375us,301ms,850ms,36us,5772ms\r\n<\/pre>\n
BtrFS:<\/h2>\n
\r\nmkfs -t btrfs \/dev\/sda5\r\nmount -o noatime \/dev\/sda5 \/mnt\r\n<\/pre>\n
Also, make sure CONFIG_CRYPTO_CRC32C_INTEL is set in the kernel, or loaded as a module and use an Intel CPU.<\/p>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\nversion          8G   254  99 54778   9 23070   8  1407  92 59932  13 131.2   5\r\nLatency             31553us     264ms     826ms   94269us     180ms   17963ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\nversion             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 17034  83 256486 100 13485  97 15282  76 38942  73  1472  23\r\nLatency               126ms    2162us   11295us   71992us   20713us   28647ms\r\n\r\n1.96,1.96,version,1,1320204006,8G,,254,99,54778,9,23070,8,1407,92,59932,13,131.2,5,512,,,,,17034,83,256486,100,13485,97,15282,76,38942,73,1472,23,31553us,264ms,826ms,94269us,180ms,17963ms,126ms,2162us,11295us,71992us,20713us,28647ms\r\n<\/pre>\n
Analysis<\/h2>\n
Ext4 is considerably better than Ext3 was last time we ran the check. Even with the allocation group tweaks and mount options we use, Ext4 isn’t a bad alternative and shows some improvements over XFS. However, BtrFS even with the Intel CRC hardware acceleration, the Random Create Read benchmark shows a significant drop.<\/p>\n
I believe our recent conversion to Ext4 isn’t negatively impacting things based on the typical workload machines see.<\/p>\n
I’ll continue to work with BtrFS and see if I can figure out why that one particular benchmark performs so poorly, but, some of the other options present in BtrFS since it is a versioning filesystem will be quite useful.<\/p>\n
Machine specs:<\/p>\n
* Linux version 3.1.0 #3 SMP Tue Nov 1 16:23:42 EDT 2011 i686 GNU\/Linux
\n* P4\/3.0ghz, 2gb RAM
\n* Western Digital SATA2 320gb 7200 RPM Drive<\/p>\n
\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"excerpt":{"rendered":"
Recently I talked about versioning filesystems available for OSS systems. While most of our server farms use XFS, we have been moving to Ext4 on a number of machines. This wasn’t done as a precursor to BtrFS but problems we’ve been having with XFS on very large filesystems. The fact that we can migrate Ext4 […]<\/p>\n
\n
<\/div>\n<\/fb:like>\n<\/div>
<\/div>","protected":false},"author":15,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-1260","post","type-post","status-publish","format-standard","hentry","category-infrastructure"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=1260"}],"version-history":[{"count":5,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1260\/revisions"}],"predecessor-version":[{"id":1265,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1260\/revisions\/1265"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=1260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=1260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=1260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}