GlusterFS performance on different frameworks

A couple months ago, I did a comparison of different distributed filesystems. It came out that GlusterFS was the easiest and most feature full, but it was slow. Since I would really like to use it, I decided to give another chance. Instead of doing raw benchmarks using sysbench, I decided to stress test a basic installation of the three PHP frameworks/CMS I use the most using siege.

My test environment:

• MacBook Pro (Late 2003, Retina, i7 2.66 Ghz)
• PCIe-based Flash Storage
• 2-4 virtuals machines using VMware Fusion 4, each with 2 GB of RAM.
• Ubuntu 13.10 server edition with PHP 5.5 and OPCache enabled
• GlusterFS running on all VMs with a volume in replica mode
• The volume was mounted using nodiratime,noatime using GlusterFS native driver (NFS was slower)

The test:

1. siege -c 20 -r 5 http://localhost/foo  # Cache warming
2. siege -c 20 -r 100 http://localhost/foo  # Actual test

I then compared the local filesystem (inside the VM) vs the Gluster volume using these setups:

• 2 nodes, 4 cores per node
• 2 nodes, 2 cores per node
• 4 nodes, 2 cores per node

The compared value is the total time to serve 20 x 100 requests in parallel.
All tests were ran 2-3 times while my computer was doing nothing and the results were very consistent.

		Symfony	Wordpress	Drupal	Average
2 nodes 4 cores	Local	2.91 s	9.92 s	5.39 s	6.07 s
	Gluster	10.84 s	23.94 s	7.81 s	14.20 s
2 nodes 2 cores	Local	5.41 s	19.14 s	9.67 s	11.41 s
	Gluster	25.05 s	31.91 s	15.17 s	24.04 s
4 nodes 2 cores	Local	5.57 s	19.6 s	9.79 s	11.65 s
	Gluster	30.56 s	35.92 s	18.36 s	28.28 s
Local vs Gluster	2 nodes, 4 cores	273 %	141 %	45 %	153 %
	2 nodes, 2 cores	363 %	67 %	57 %	162 %
	4 nodes, 2 cores	449 %	83 %	88 %	206 %
	Average	361 %	97 %	63 %	174 %
2 nodes vs 4 nodes	Local	3 %	2 %	1 %	2 %
	Gluster	22 %	13 %	21 %	19 %
4 cores vs 2 cores	Local	86 %	93 %	79 %	86 %
	Gluster	131 %	33 %	94 %	86 %

Google spreadsheet

Observations:

1. Red — Wordpress and Drupal have an acceptable loss in performance under Gluster, but Symfony is catastrophic.
2. Blue — The local tests are slightly slower when using 4 nodes vs 2 nodes. This is normal, my computer had 4 VMs running.
3. Green — The gluster tests are 20% slower on a 4 node setup because there is more communication between the nodes to keep them all in sync. 20% overhead for double the nodes isn’t that bad.
4. Purple — The local tests are 85% quicker using 4 cores vs 2 cores. A bit under 100% is normal, there is always some overhead to parallel processing.
5. Yellow — For the Gluster tests, Symfony and Drupal scale very well with the number of nodes, but Wordpress is stalling, I am not sure why.

I am still not sure why Symfony is so much slower on GlusterFS, but really, I can’t use it in production for the moment because I/O is already the weak point of my infrastructure. I am in the process of looking for a different hosting solution, maybe it will be better then.