Preventing cache stampede when using Doctrine Cache

Wikipedia has short and clear article on the matter, cache stampede can be quite deadly, especially when you are rebooting your server, clearing your cache or having a midnight maintenance (cronjob).

Cache stampede, put simply, is when your process is trying to fetch an expensive cache entry, it is not there, and tries to build it, but a simultaneous process comes and does the same thing. This results in resource waste because of processes all trying to do the same thing, uselessly.

The trick is to put a lock on the cache entry and make the other processes wait. I use the trick of having a TTL for the cache calculation. This way, even if the server crashes or there is an exception, the lock will expire and another process will take over. This estimation is also used to calculate how much time to sleep while waiting.

This example uses Doctrine Cache as a global variable, but this should definitely be refactored (or change to another system).

GlusterFS performance on different frameworks

A couple months ago, I did a comparison of different distributed filesystems. It came out that GlusterFS was the easiest and most feature full, but it was slow. Since I would really like to use it, I decided to give another chance. Instead of doing raw benchmarks using sysbench, I decided to stress test a basic installation of the three PHP frameworks/CMS I use the most using siege.

My test environment:

• MacBook Pro (Late 2003, Retina, i7 2.66 Ghz)
• PCIe-based Flash Storage
• 2-4 virtuals machines using VMware Fusion 4, each with 2 GB of RAM.
• Ubuntu 13.10 server edition with PHP 5.5 and OPCache enabled
• GlusterFS running on all VMs with a volume in replica mode
• The volume was mounted using nodiratime,noatime using GlusterFS native driver (NFS was slower)

The test:

1. siege -c 20 -r 5 http://localhost/foo  # Cache warming
2. siege -c 20 -r 100 http://localhost/foo  # Actual test

I then compared the local filesystem (inside the VM) vs the Gluster volume using these setups:

• 2 nodes, 4 cores per node
• 2 nodes, 2 cores per node
• 4 nodes, 2 cores per node

The compared value is the total time to serve 20 x 100 requests in parallel.
All tests were ran 2-3 times while my computer was doing nothing and the results were very consistent.

		Symfony	Wordpress	Drupal	Average
2 nodes 4 cores	Local	2.91 s	9.92 s	5.39 s	6.07 s
	Gluster	10.84 s	23.94 s	7.81 s	14.20 s
2 nodes 2 cores	Local	5.41 s	19.14 s	9.67 s	11.41 s
	Gluster	25.05 s	31.91 s	15.17 s	24.04 s
4 nodes 2 cores	Local	5.57 s	19.6 s	9.79 s	11.65 s
	Gluster	30.56 s	35.92 s	18.36 s	28.28 s
Local vs Gluster	2 nodes, 4 cores	273 %	141 %	45 %	153 %
	2 nodes, 2 cores	363 %	67 %	57 %	162 %
	4 nodes, 2 cores	449 %	83 %	88 %	206 %
	Average	361 %	97 %	63 %	174 %
2 nodes vs 4 nodes	Local	3 %	2 %	1 %	2 %
	Gluster	22 %	13 %	21 %	19 %
4 cores vs 2 cores	Local	86 %	93 %	79 %	86 %
	Gluster	131 %	33 %	94 %	86 %

Google spreadsheet

Observations:

1. Red — Wordpress and Drupal have an acceptable loss in performance under Gluster, but Symfony is catastrophic.
2. Blue — The local tests are slightly slower when using 4 nodes vs 2 nodes. This is normal, my computer had 4 VMs running.
3. Green — The gluster tests are 20% slower on a 4 node setup because there is more communication between the nodes to keep them all in sync. 20% overhead for double the nodes isn’t that bad.
4. Purple — The local tests are 85% quicker using 4 cores vs 2 cores. A bit under 100% is normal, there is always some overhead to parallel processing.
5. Yellow — For the Gluster tests, Symfony and Drupal scale very well with the number of nodes, but Wordpress is stalling, I am not sure why.

I am still not sure why Symfony is so much slower on GlusterFS, but really, I can’t use it in production for the moment because I/O is already the weak point of my infrastructure. I am in the process of looking for a different hosting solution, maybe it will be better then.

Organizing Javascript and CSS assets for optimal loading

On Reddit, I recently stumbled upon DynoSRC which allows to serve only a differential Javascript file to your users. The concept is pretty amazing, but I find it a bit overkill. I believe this would only be useful for very high traffic websites with a lot of Javascript with small parts changing often. For example: Facebook, Asana, WolframAlpha, Google Maps, etc.

For common websites however, you will probably be deploying a bunch of files at the same time. If you modified 10% of 40% of your files, the overhead of computing all those diffs (server side and client side) and having this system to manage is probably not worth it. If you are already compressing and grouping your assets and you have a CDN with proper HTTP caching, you are already pretty good. That can be hard though, especially the proper part.

Separate assets in 3 groups

When I have complete control over my assets, I usually like to split all the assets in 3 groups:

1. Very common libraries (Bootstrap, jQuery, Modernizr). I serve them using public CDN like cdnjs. This is because it is very likely that the user will already have it in cache.
2. External assets specific to my project (Custom Bootstrap build, jQuery plugins, lightbox plugin, etc.). I bundle them all in a big libs.js/css.
3. Global custom assets written for this project, bundled in a single global.js/css. It does not need to be all the custom assets, but stuff you will need on 80% of your requests. Specific code for specific pages can be included individually.

Cache busting

I mostly use Amazon CloudFront as a CDN which handles query parameters so I set the expire date to one year and append a query parameter with the last git commit. (git rev-parse --short HEAD). That way, a fresh file is used each time there is any change in the project’s code. See Automatic cache busting using Git commit in Symfony2 for an example.

About combining

People often talk about combining how it saves HTTP requests but consider that it also compresses a lot better if the files are all grouped together. You may be adding an overhead on the first page load but the rest of the website will be blazing fast.

However, be careful not to overload the browser. Keep it mind that the Javascript will be executed on each page load. For example, to not try to initialize every modal window just in case one might pop up. See Optimizing page loads by reducing the impact of the Javascript initialization for more details.

Using Edge Side Includes with Varnish to cache partial pages

Caching full pages with Varnish can be hard, most applications use sessions, which sets a Cookie, which makes Varnish ignore all caching.

When sessions are needed and full page cache is not available, you can resort to ESI (Edge Side Includes).

ESI has a markup language of its own, but the subset that Varnish supports is fairly simple: it is basically a placeholder that gets replaced by the referenced URL. They generate a subrequest that will mimic the original request, but for another URL. Using this system, you can cache parts of your website that do not change frequently or that are hard to generate. Since this is internal to Varnish, it will honour the cache system, including Cache-Control, If-Modified-Since, ETag, etc.

However, since the subrequest is built on top of the original, it will contain the original Cookie header, so we must ignore it.

The solution includes:

• The original script must add a "X-Esi" header to activate ESI parsing (performance).
• Cookies are removed from ESI requests unless "esi-cookies=1" is present is the URL.
• A "X-Esi-Level" header gets added when the current request is a ESI. Otherwise, it is removed.

Various scenarios where this technique can be used:

• A navigation menu with the current URL in parameter.
• A footer
• A user profile box (popup box) with the user id in parameter.
• Widgets of a sidebar

Since the X-Esi-Level header is enforced to be only present for ESI requests, you can trust it and safely ignore any security check as they would have already been done in the original request.

Here is the VCL used for Varnish and a simple example to illustrate the ?esi-cookies=1 trick.

How to properly create and destroy a PHP session

When working with shared caches like Varnish or Nginx, cookies will kill everything you are trying to do.

The idea is that since a cookie can be used by the backend to modify the reply, like being logged in as a user, the shared does not take the chance and refuse to cache it. This behaviour can be modified, especially in the case of Google Analytics cookies, but for the PHP session cookie, you will typically want it.

However, it is important, for a useful usage of your shared cache, to only start a session when you really need it and destroy it when it is not needed anymore. PHP’s session_start specifically mentions that is does not unset the associated cookie.

So the idea is to start a session when you need it; for example, in a login page. This will send a header to the client, setting a cookie and the cookie will get sent back on every request. Therefore, in some global file, you can detect this cookie and reload the session. At last, on the logout, clear everything.