Pages

Friday, December 6, 2013

GlusterFS performance on different frameworks

A couple months ago, I did a comparison of different distributed filesystems. It came out that GlusterFS was the easiest and most feature full, but it was slow. Since I would really like to use it, I decided to give another chance. Instead of doing raw benchmarks using sysbench, I decided to stress test a basic installation of the three PHP frameworks/CMS I use the most using siege.

My test environment:

  • MacBook Pro (Late 2003, Retina, i7 2.66 Ghz)
  • PCIe-based Flash Storage
  • 2-4 virtuals machines using VMware Fusion 4, each with 2 GB of RAM.
  • Ubuntu 13.10 server edition with PHP 5.5 and OPCache enabled
  • GlusterFS running on all VMs with a volume in replica mode
  • The volume was mounted using nodiratime,noatime using GlusterFS native driver (NFS was slower)
The test:
  1. siege -c 20 -r 5 http://localhost/foo  # Cache warming
  2. siege -c 20 -r 100 http://localhost/foo  # Actual test
I then compared the local filesystem (inside the VM) vs the Gluster volume using these setups:
  • 2 nodes, 4 cores per node
  • 2 nodes, 2 cores per node
  • 4 nodes, 2 cores per node
The compared value is the total time to serve 20 x 100 requests in parallel.
All tests were ran 2-3 times while my computer was doing nothing and the results were very consistent.

Symfony Wordpress Drupal Average
2 nodes
4 cores
Local 2.91 s 9.92 s 5.39 s 6.07 s
Gluster 10.84 s 23.94 s 7.81 s 14.20 s
2 nodes
2 cores
Local 5.41 s 19.14 s 9.67 s 11.41 s
Gluster 25.05 s 31.91 s 15.17 s 24.04 s
4 nodes
2 cores
Local 5.57 s 19.6 s 9.79 s 11.65 s
Gluster 30.56 s 35.92 s 18.36 s 28.28 s
Local vs
Gluster
2 nodes, 4 cores 273 % 141 % 45 % 153 %
2 nodes, 2 cores 363 % 67 % 57 % 162 %
4 nodes, 2 cores 449 % 83 % 88 % 206 %
Average 361 % 97 % 63 % 174 %
2 nodes vs
4 nodes
Local 3 % 2 % 1 % 2 %
Gluster 22 % 13 % 21 % 19 %
4 cores vs
2 cores
Local 86 % 93 % 79 % 86 %
Gluster 131 % 33 % 94 % 86 %


Observations:
  1. Red — Wordpress and Drupal have an acceptable loss in performance under Gluster, but Symfony is catastrophic.
  2. Blue — The local tests are slightly slower when using 4 nodes vs 2 nodes. This is normal, my computer had 4 VMs running.
  3. Green — The gluster tests are 20% slower on a 4 node setup because there is more communication between the nodes to keep them all in sync. 20% overhead for double the nodes isn’t that bad.
  4. Purple — The local tests are 85% quicker using 4 cores vs 2 cores. A bit under 100% is normal, there is always some overhead to parallel processing.
  5. Yellow — For the Gluster tests, Symfony and Drupal scale very well with the number of nodes, but Wordpress is stalling, I am not sure why.

I am still not sure why Symfony is so much slower on GlusterFS, but really, I can’t use it in production for the moment because I/O is already the weak point of my infrastructure. I am in the process of looking for a different hosting solution, maybe it will be better then.