{"id":1446,"date":"2012-05-22T02:52:54","date_gmt":"2012-05-22T06:52:54","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=1446"},"modified":"2012-05-30T19:27:34","modified_gmt":"2012-05-30T23:27:34","slug":"ext4-xfs-and-btrfs-benchmark-redux","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/scalability\/ext4-xfs-and-btrfs-benchmark-redux\/","title":{"rendered":"Ext4, XFS and Btrfs benchmark redux"},"content":{"rendered":"

As Linux 3.4 was just released and it includes a number of btrfs filesystem changes, I felt it was worth retesting to see if btrfs had better performance.<\/p>\n

\r\n$ \/usr\/sbin\/bonnie++ -s 8g -n 512\r\n<\/pre>\n
ext4<\/h2>\n
mkfs -t ext4 \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   582  98 59268   6 30754   3  3515  99 104817   4 306.1   1\r\nLatency             15867us    1456ms     340ms    8997us   50112us     323ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 35092  55 520637  91  1054   1 35182  54 791080 100  1664   2\r\nLatency              1232ms     541us   14112ms    1189ms      41us   11701ms\r\n1.96,1.96,colo7,1,1337657098,8G,,582,98,59268,6,30754,3,3515,99,104817,4,306.1,1,512,,,,,35092,55,520637,91,1054,1,35182,54,791080,100,1664,2,15867us,1456ms,340ms,8997us,50112us,323ms,1232ms,541us,14112ms,1189ms,41us,11701ms\r\n<\/pre>\n
ext4 with tuning and mount options<\/h2>\n
\r\nmkfs -t ext4 \/dev\/sda9\r\ntune2fs -o journal_data_writeback \/dev\/sda9\r\nmount -o rw,noatime,data=writeback,barrier=0,nobh,commit=60 \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   587  97 64875   6 34046   4  3149  96 105157   4 317.2   4\r\nLatency             13877us     562ms    1351ms   18692us   54835us     287ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 38127  59 525459  92  2118   2 37746  58 792967  99  1433   2\r\nLatency               980ms     525us   14018ms    1056ms      46us   12355ms\r\n1.96,1.96,colo7,1,1337661756,8G,,587,97,64875,6,34046,4,3149,96,105157,4,317.2,4,512,,,,,38127,59,525459,92,2118,2,37746,58,792967,99,1433,2,13877us,562ms,1351ms,18692us,54835us,287ms,980ms,525us,14018ms,1056ms,46us,12355ms\r\n<\/pre>\n
btrfs from ext4 partition<\/h2>\n
\r\numount \/mnt\r\nfsck.ext3 -f \/dev\/sda9\r\nbtrfs-convert \/dev\/sda9\r\nmount -t btrfs -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   462  98 62854   5 30782   4  3065  88 88883   8 313.1   7\r\nLatency             63644us     272ms     206ms   38178us     241ms     409ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 36868  85 598431  98 26007  93 32002  73 756164  99 21975  84\r\nLatency             15858us     427us    1003us     471us     157us    2161us\r\n1.96,1.96,colo7,1,1337660385,8G,,462,98,62854,5,30782,4,3065,88,88883,8,313.1,7,512,,,,,36868,85,598431,98,26007,93,32002,73,756164,99,21975,84,63644us,272ms,206ms,38178us,241ms,409ms,15858us,427us,1003us,471us,157us,2161us\r\n<\/pre>\n
btrfs without conversion<\/h2>\n
\r\nmkfs -t btrfs \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   468  98 60274   4 29605   4  3629 100 89250   8 301.5   7\r\nLatency             55633us     345ms     196ms    3767us     229ms    1119ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 26078  60 603783  99 26027  92 25617  58 754598  99 21935  84\r\nLatency               452us     423us    1029us     426us      16us    2314us\r\n1.96,1.96,colo7,1,1337661202,8G,,468,98,60274,4,29605,4,3629,100,89250,8,301.5,7,512,,,,,26078,60,603783,99,26027,92,25617,58,754598,99,21935,84,55633us,345ms,196ms,3767us,229ms,1119ms,452us,423us,1029us,426us,16us,2314us\r\n<\/pre>\n
xfs defaults<\/h2>\n
\r\nmkfs -t xfs \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G  1391  96 65559   5 31315   3  2984  99 103339   4 255.8   3\r\nLatency              5625us   33224us     221ms   10524us     103ms     198ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512  7834  37 807425  99 14627  50  8612  41 790321 100  1169   4\r\nLatency              1182ms     123us     837ms    2037ms      18us    7031ms\r\n1.96,1.96,colo7,1,1337660479,8G,,1391,96,65559,5,31315,3,2984,99,103339,4,255.8,3,512,,,,,7834,37,807425,99,14627,50,8612,41,790321,100,1169,4,5625us,33224us,221ms,10524us,103ms,198ms,1182ms,123us,837ms,2037ms,18us,7031ms\r\n<\/pre>\n
xfs tuned<\/h2>\n
\r\nmkfs -t xfs -d agcount=32 -l size=64m \/dev\/sda9\r\nmount -o noatime,logbsize=262144,logbufs=8 \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G  1413  96 64640   5 31226   3  2977  99 104762   4 246.8   3\r\nLatency              5616us     370ms     235ms   10530us   62654us     206ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 14763  70 793694  98 23959  81 15104  72 790204  99  2290   8\r\nLatency               482ms     118us     274ms     683ms      17us    5201ms\r\n1.96,1.96,colo7,1,1337666959,8G,,1413,96,64640,5,31226,3,2977,99,104762,4,246.8,3,512,,,,,14763,70,793694,98,23959,81,15104,72,790204,99,2290,8,5616us,370ms,235ms,10530us,62654us,206ms,482ms,118us,274ms,683ms,17us,5201ms\r\n<\/pre>\n
btrfs with a snapshot<\/h2>\n
\r\nmkfs -t btrfs \/dev\/sda9\r\nmount -o noatime,subvolid=0 \/dev\/sda9 \/mnt\r\nwget http:\/\/www.kernel.org\/pub\/linux\/kernel\/v3.0\/linux-3.4.tar.bz2\r\ntar xjf linux-3.4.tar.bz2\r\nbtrfs subvolume snapshot \/mnt\/ \/mnt\/@_2012_05_22\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   469  98 58400   5 30092   4  2999  85 89761   8 321.1   3\r\nLatency             17017us     267ms     240ms   22907us     300ms     359ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 31715  72 598360  98 25780  92 26411  59 756110  99 22058  84\r\nLatency               102ms     424us     844us     472us      20us    2171us\r\n1.96,1.96,colo7,1,1337664006,8G,,469,98,58400,5,30092,4,2999,85,89761,8,321.1,3,512,,,,,31715,72,598360,98,25780,92,26411,59,756110,99,22058,84,17017us,267ms,240ms,22907us,300ms,359ms,102ms,424us,844us,472us,20us,2171us\r\n<\/pre>\n
Deleted kernel, left it in snapshot, reran test<\/h3>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   469  98 63934   5 31244   4  3208  94 90227   8 296.3   7\r\nLatency             17009us     282ms     217ms    3746us     224ms    1269ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 28758  66 596074  97 26185  93 25714  59 755464  99 21893  84\r\nLatency             42108us     424us     993us     445us      17us    2245us\r\n1.96,1.96,colo7,1,1337671128,8G,,469,98,63934,5,31244,4,3208,94,90227,8,296.3,7,512,,,,,28758,66,596074,97,26185,93,25714,59,755464,99,21893,84,17009us,282ms,217ms,3746us,224ms,1269ms,42108us,424us,993us,445us,17us,2245us\r\n<\/pre>\n
Updated results using some different parameters. Same hardware, same hard drive.<\/h2>\n
leaf and btree size of 16384<\/h3>\n
\r\nmkfs -t btrfs -l 16384 -n 16384 \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   472  98 37514   2 14395   2  3135  89 80600   7 294.0   6\r\nLatency             16820us     781ms     383ms   19736us     230ms     379ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 17447  46 621480  99 24345  94 14984  39 754999  99 19873  82\r\nLatency               303us     494us     900us     412us     107us    3127us\r\n1.96,1.96,colo7,1,1338411461,8G,,472,98,37514,2,14395,2,3135,89,80600,7,294.0,6,512,,,,,17447,46,621480,99,24345,94,14984,39,754999,99,19873,82,16820us,781ms,383ms,19736us,230ms,379ms,303us,494us,900us,412us,107us,3127us\r\n<\/pre>\n
leaf and btree size of 32768<\/h3>\n
\r\nmkfs -t btrfs -l 32768 -n 32768 \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   468  97 26136   2 17256   2  3135  89 84450   7 306.5   7\r\nLatency             43238us     923ms     330ms   12632us     367ms     986ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 17958  61 624570  99 19930  95 14506  50 753354  99 15976  80\r\nLatency             15384us     514us     937us     431us     144us    4782us\r\n1.96,1.96,colo7,1,1338409200,8G,,468,97,26136,2,17256,2,3135,89,84450,7,306.5,7,512,,,,,17958,61,624570,99,19930,95,14506,50,753354,99,15976,80,43238us,923ms,330ms,12632us,367ms,986ms,15384us,514us,937us,431us,144us,4782us\r\n<\/pre>\n
leaf and btree size of 65536<\/h3>\n
\r\nmkfs -t btrfs -l 65536 -n 65536 \/dev\/sda9\r\nmount -o noatime \/dev\/sda9 \/mnt\r\n<\/pre>\n
\r\nVersion  1.96       ------Sequential Output------ --Sequential Input- --Random-\r\nConcurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--\r\nMachine        Size K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP K\/sec %CP  \/sec %CP\r\ncolo7            8G   467  97 25097   2 17349   2  2845  87 86653   8 300.2   7\r\nLatency             56046us     772ms     414ms    4101us     249ms     241ms\r\nVersion  1.96       ------Sequential Create------ --------Random Create--------\r\ncolo7               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--\r\n              files  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP  \/sec %CP\r\n                512 15372  68 626336  98 14723  96 13137  58 753463 100 11652  80\r\nLatency             15825us     395us   77890us     428us      19us   15727us\r\n1.96,1.96,colo7,1,1338410439,8G,,467,97,25097,2,17349,2,2845,87,86653,8,300.2,7,512,,,,,15372,68,626336,98,14723,96,13137,58,753463,100,11652,80,56046us,772ms,414ms,4101us,249ms,241ms,15825us,395us,77890us,428us,19us,15727us\r\n<\/pre>\n
Analysis<\/h2>\n
Last time I tested ext4, xfs and btrfs<\/a>, deletions really lagged behind. Now, it looks like btrfs is quite a bit more robust. Additionally, there are better repair and recovery tools which were basically missing before. btrfs doesn’t lag behind like it used to, and while it is a little slower in some cases, it’s only a few percent. However, it makes up for that with some of the random and sequential creation and deletions.<\/p>\n
Rough analysis at this point – if you need a versioning filesystem and don’t mind being a bit on the bleeding edge, btrfs has made substantial strides.<\/p>\n
Updated Analysis<\/h3>\n
For the hardware in question, it appears that the larger block sizes with Bonnie++ don’t benefit things, but, make sure you test with your workload.<\/p>\n
Test Equipment<\/h2>\n
    \n
  • Linux colo7 3.4.0 #1 SMP Mon May 21 00:29:58 EDT 2012 x86_64 GNU\/Linux<\/li>\n
  • Intel(R) Xeon(R) CPU           X3220  @ 2.40GHz<\/li>\n
  • WDC WD7500AACS-0 01.0 PQ: 0 ANSI: 5<\/li>\n
  • ahci enabled<\/li>\n
  • 100gb partition<\/li>\n
  • machine rebooted between each test<\/li>\n<\/ul>\n
    \n
    <\/div>\n<\/fb:like>\n<\/div>
    <\/div>","protected":false},"excerpt":{"rendered":"
    As Linux 3.4 was just released and it includes a number of btrfs filesystem changes, I felt it was worth retesting to see if btrfs had better performance. $ \/usr\/sbin\/bonnie++ -s 8g -n 512 ext4 mkfs -t ext4 \/dev\/sda9 mount -o noatime \/dev\/sda9 \/mnt Version 1.96 ——Sequential Output—— –Sequential Input- –Random- Concurrency 1 -Per Chr- […]<\/p>\n
    \n
    <\/div>\n<\/fb:like>\n<\/div>
    <\/div>","protected":false},"author":15,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[205,204,193],"class_list":["post-1446","post","type-post","status-publish","format-standard","hentry","category-scalability","tag-btrfs","tag-ext4","tag-xfs"],"_links":{"self":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1446","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/comments?post=1446"}],"version-history":[{"count":5,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1446\/revisions"}],"predecessor-version":[{"id":1451,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/posts\/1446\/revisions\/1451"}],"wp:attachment":[{"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/media?parent=1446"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/categories?post=1446"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cd34.com\/blog\/wp-json\/wp\/v2\/tags?post=1446"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}