Web Server Performance Tuning Notes, Nov 2001 (c) EarthLink, Inc. 2001 I want to thank EarthLink for giving me time to prepare the talk. However as a sysadmin, I kept getting interrupts, but I was able to put the below together, I hope it's of use to you. Aleksey Tsalolikhin 1 Nov 2001 eesti@corp.earthlink.net Web server performance tuning philosophy: 1. use dedicated server ( stability/performance ) Max out on mem, CPU. 2. Set httpd controls (for things like max simultaneous connections) very high. Don't want to create an artificial bottleneck. This will put the bottleneck somewhere in the system (Like not enough memory or CPU, or some OS resource limitation.) When you hit some bottleneck, tune your website and your httpd config, then your OS, then upgrade server (or add more servers). 3. KNOW your performance characteristics. Track you server latency. Make graphs, etc. Recognize when your numbers start drifting up. Apache httpd.conf, example of things to tune: StartServers should be pretty high, so you start lots of servers when server is started. MaxClients - crank it up to 256 (make sure you have the memory) If you get lots of traffic, edit HARD_SERVER_LIMIT in src/include/httpd.h, and raise it to 1024 or higher, and recompile. MaxRequestsPerChild - set to 10000 to be safe from libraries leaking FancyIndexing - off (disable the pretty icons in directory listings) Timeout (timeout to get a TCP ACK or a complete GET request) default 300 seconds is too long on modern networks - try 60 or 120. StartServers default is 5, for a dedicated server, make higher (on a dedicated server, just set to MaxClients, so the parent doesn't have to tune it up as the requests come in; or half of MaxClients or more - definitely more than 5 though.) Links: http://httpd.apache.org/docs/misc/perf-tuning.html http://httpd.apache.org/docs/misc/perf.html http://perl.apache.org/guide/performance.html Other load-testing or perf measurement software: httperf http://www.hpl.hp.com/personal/David_Mosberger/httperf.html http_ping http://www.acme.com/software/http_ping/ torture http://stein.cshl.org/~lstein/torture/torture.html ab (from Apache distribution, in the bin directory - Apache Bench(mark).) == Two ways to run it: conventional (with -c and -n) and single-sample (ab -n 1 http://...). Single-sample way is good for minding server latency on production servers. (Can't load-test production servers.) Get a lot of samples. This way unusual spikes or dips become unweighted. Sample regularly - minutely. Look at avg and mean. When starting tuning, they'll be far apart (avg will be higher because of spikes). As you progress, and get rid of/reduce the spikes, the avg and mean will drift closer together. run ab -n 1 on same network segment to get server latency (otherwise you get server + network latency) -------------------------------------------------------- "truss httpd -X" runs httpd in single-process mode, so you can truss it easily. hit it with your Web browser and see if there are any system cools that look extra or unnecessary - config change can fix it sometimes. -------------------------------------------------------- "ndd /dev/tcp \?" shows all TCP stack tunables (same for /dev/ip and /dev/upd, BTW) tcp_conn_req_max_q0 - incomplete connections queue (SYN received) tcp_conn_req_max_q - complete connections queue (handshake done, waiting to be accept()'ed) the separation into two queues is a defense against SYN flood attacks. q0 is usually much larger than q defaults (in Solaris 8) are: q0: 4096 q: 128 One can increase these: example# ndd /dev/tcp tcp_conn_req_max_q0 131072 example# ndd /dev/tcp tcp_conn_req_max_q 1024 example# use "ndd /dev/tcp tcp_listen_hash" right-most column to see if you're hitting the queue max Other things to tune for a busy web server might be tcp_conn_hash_size (an /etc/system paramater - default is 512, can change to 1024 or more) rlim_fd_max (per-process file descriptor limit) Links: Solaris Tunable Parameters Manual - http://docs.sun.com:80/ab2/coll.707.1/SOLTUNEPARAMREF/@Ab2TocView?Ab2Lang=C&Ab2Enc=iso-8859-1 http://www.sean.de/Solaris http://www.sunperf.com/ Linux performance tuning links: http://linuxperf.nl.linux.org/ When doing performance tuning, it's important to have quantified measurement of performance. Use ab, don't fly blind. --- What to aim for, with regard to web server response time? >200 ms is potentially problematic, try and keep it under 100 ms, the lower the better. --- Time::HiRes module for benchmarking - useful for monitoring response latencies of back-end tiers. perlipc man page from perl distribution has nice example of perl TCP/IP client you can hack up and use with Time::HiRes to time TCP/IP client-server sessions.