Pages

Tuesday, September 18, 2012

Protect Webserver against DOS attacks using UFW

Ubuntu comes bundled with UFW, which is an interface to iptables. This is basically a very lightweight router/firewall inside the Linux kernel that runs way before any other application.

Typical setup of ufw is to allow HTTP(S), limit SSH and shut everything else. This is not a UFW or iptables tutorial, you may find a lot of online help to guide you through all your needs. However, I personally had a lot of difficulties to find good documentation on how to protect yourself against HTTP attacks.

A lot of HTTP requests is normal

The problem is that HTTP can get very noisy. A typical Web page can easily have up to a hundred of assets but usually, if you receive 100 requests in a second, it means you are under siege. If you really need to have 100 assets on a single Web page, you need a CDN, not as better server.

Rate limiting

These rules have been mostly guessed through trial-and-error and some search around the Web, tweak to fit your needs. A rate limit of x connections per y seconds means that if x connections has been initiated in the last y seconds by this profile, it will be dropped. Dropping is actually a nice protection against flooding because the sender won't know that you dropped it. He might think the packet was lost, that the port is closed or even better, the server is overloaded. Imagine how nice, your attacker thinks he succeeded, but in fact you are up and running, him being blocked.

Connections per IP
A connection is an open channel. A typical browser will open around 5 connections per page load and they should last under 5 seconds each. Firefox, for example, has a default max of 15 connections per server and 256 total.

I decided to go for 20 connections / 10 seconds / IP. 

Connections per Class C
Same a above, but this time we apply the rule to the whole Class C of the IP because it is quite common for someone to have a bunch of available IPs. This means for example all IPs looking like 11.12.13.*

I decided to go for 50 simultaneous connections.

Packets per IP
This is the challenging part. Due to a limitation that is not easy to circumvent, it is only possible to keep track of the last 20 packets. At the same time, it might add a considerable overhead to track 100 packets for each IPs. While big website may eventually need more than this, like I said, you should take a look in a proper CDN.

I decided to go for 20 packets / second / IP

Configuring UFW

The following instructions are targeted at UFW, but it is really just a wrapper so it should be easy to adapt them for a generic system.

Edit /etc/ufw/before.rules, putting each part where it belongs

Make sure ufw runs and reload everything using ufw reload.

Testing the results

Make sure everything runs smoothly by refreshing your browser like a mad-man. You should start getting timeout after ~15 refreshes and it should come back in less than 30 seconds. This is good.

But if you want to get serious on your tests, some tools may help you putting your server to its knees. It is highly discouraged to use this on a production server, but it is still better if you do it yourself than if you wait for someone to try.

Try those with UFW enabled and disabled to see the difference but be careful, some machines may downright crash on you or fill all available space with logs.
  • http://ha.ckers.org/slowloris/
    Written in Perl, features a lot of common attacks, including HTTPS
  • http://www.sectorix.com/2012/05/17/hulk-web-server-dos-tool/
    Written in Python, basic multi-threaded attack, very easy to use.
  • http://www.joedog.org/siege-home/
    Compiled, available in Ubuntu repositories, very good to benchmark
  • http://blitz.io/
    Online service when you can test freely with up to 250 concurrent users
To confirm that everything works perfectly, SSH into your machine and start a tail -f /var/log/ufw.log to see the packets being dropped and htop to watch the CPU have fun. 

SSH into another machine and start a script. You should see the CPU sky-rocket for a few seconds and then go back to normal. Logs will start to appear and your stress-tool will have some problems. While all this is going on, you should be able to browse normally your website using your computer. 

Great success.

11 comments:

  1. Hi,

    Quite interesting article. I have a few comments and my answer does not fit in the comment section :P.

    Please feel free to read and provide with any additional comment or answer at: Re: Protect Webserver against DOS attacks using UFW

    ReplyDelete
  2. While I do understand your point of not limiting too much, I don’t think it is. Like I said, if you omit websites with a lot of assets, which should use a CDN, 20 requests per second per IP is huge. Users are not on 3 tabs at the same time and clicking everywhere. Also, if you are building the next Facebook where people have 5+ tabs opened which each make an AJAX request/sec, you fall a bit outside of my scope.

    For bots, it goes about the same, I expect bots not to crawl faster than 2-3 requests per seconds. Since they won't crawl all the sites on my system at once, it won't be a problem soon. Moreover, the packet rule is only on a window of 1 second so it gets reset rather quickly compared to the connection rule which is a better indicator of an attack.

    As for connections, 20 connections / 10sec and 50 parallel connections / Class C is pretty high as well. I understand the Class C rule could be dropped/modified, but the idea remains. And as for the precedent point, packets are simply drop so the client will retry. It will slow him a bit but will not crash or reply with a 50x. Also, I know browsers like Chrome and Firefox and a maximum parallel connections per server, global to all tabs.

    Lastly, for connection starvation and other types of attacks, UFW already includes a decent setting. There is simply no setting for HTTP request other than allow because it is more complicated.

    Thanks for your complete reply, did I forget anything ?

    ReplyDelete
  3. And for anyone interested, this blogpost does a roundup of behaviours of all browsers
    http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/

    ReplyDelete
  4. problem running ufw-init

    ReplyDelete
  5. Can this configuration work in another ports ?
    Because i have a app in my server and i received multiple attacks from one ip..
    The aplication only need two open TCP ports.
    Thanks and sorry for my english.

    ReplyDelete
  6. Yes, totally. Simply change this line or add more, changing 80 for your custom port:
    -A ufw-before-input -p tcp --dport 80 -j ufw-http

    ReplyDelete
    Replies
    1. Thanks you ! Im running your script in my server :)

      Is there any way to view the logs in UFW to check if it work during a dos attack ?

      Delete
    2. Yes:

      sudo less /var/log/ufw.log
      or to follow in real time:
      sudo tail -f /var/log/ufw.log

      Delete
    3. I cant view "[UFW HTTP DROP]" is there any way to view if the script is working ?

      Delete
  7. What version of UFW are you using? Also Ubuntu? Thank you, --Dan

    ReplyDelete