warrenkonkel

Apache Tuning: MaxClients and Keep-Alive

May 14th, 2007

When a post about statisfy.net made it to digg's front page, we experienced an onslaught of traffic (shocker!). Within a few minutes of getting dugg, I noticed that Apache would sometimes disregard requests entirely. If I reloaded, it would work fine. This post goes through what I learned about how Apache handles requests and what I changed to allow more requests to be handled.

MaxClients

I started digging around on the server and found the Apache error log spitting out this message over and over:

[error] server reached MaxClients setting, consider raising the MaxClients setting

After some quick research (thanks google and Aaron from RHG), I learned a few things about Apache and how it handles requests. Out of the box, Apache is setup to have at most 16 child processes (ServerLimit). Each of these child processes will run with 25 threads (ThreadsPerChild). If you multiply these two numbers, you'll get that Apache should be able to handle 400 simultaneous requests... however Apache also has an out of the box limit of 150 simultaneous requests (MaxClients). At first, I changed just the MaxClients to 400.. but then I figured "more is better", right? So, I upped ServerLimit to 20 and set MaxClients to 500. My Apache config now had a section that looks like this:

<IfModule mpm_worker_module> ServerLimit 20 StartServers 5 MaxClients 500 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule>

After restarting Apache, the load average immediately doubled. My first reaction was that I broke something, but I went to statisfy.net, reloaded a few times, and (to my surprise) watched it load right away without any problems. Aaron suggested that "the load average doubled because you're now handling more requests... before you were just discarding them"... seems logical.

Keep-Alive optimization

After a while, I noticed that Apache would still hit the 500 MaxClients from time to time. I setup mod_status so I could watch what was actually happening at the Apache level and I noticed probably 70% of the threads were in a Keep-Alive state. This means that the server isn't actually processing a request but rather just waiting for another request from that user. I tried disabling Keep-Alive entirely and the simultaneous connections problem went away, but the browser experience was noticeably slower... so a better solution was needed. I tried playing around with the Keep-Alive timeout but I couldn't find a good balance between simultaneous connections and browser experience.

Statisfy has two distinct use cases for people requesting files from the server. The first use case is the user who is actually on the statisfy.net site watching statistics go by. The second use case is a user visiting a remote site with the statisfy.net javascript embedded and their browser hitting the statisfy.net server to record their statistics. In this second use case, there are (almost) always two requests made and no more beyond that. The first request is for the static file stats.js and the second request is for /stats/record?blah_blah_blah. I realized that if a client requests /stats/record, we can safely close their connection rather than doing Keep-Alive because chances are they won't be making another request for a little while.

To accomplish this, I enabled mod_headers and mod_setenvif and added the following settings to my Apache config:

SetEnvIf Request_URI "/stat/record" CloseConnectionAfterRequest Header set Connection "close" env=CloseConnectionAfterRequest

I restarted Apache and watched mod_status for a while and noticed we went from close to 500 simultaneous connections down to about 100. Furthermore, the browser experience wasn't affected at all because Keep-Alive would be active for the browsers that were actually going to use it. Success!!!