varnish cache
-
Upload
khang-nguyen -
Category
Documents
-
view
49 -
download
1
description
Transcript of varnish cache
-
VARNISH-CACHE *Lessons Learnt from Performance Testing
* https://www.varnish-cache.orgTuesday, 25 October 11
-
WHY? FAST!
If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second.
Theoretical limits in hundreds of thousands per second.*
* http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11
-
GET OUT OF JAIL. FREE.
Crappy cache control?
Expensive CDNs? 20/GB?
Not enough hardware?
Backend struggling?
Tuesday, 25 October 11
-
IT WONT
Do HTTPS... terminate SSL with something else, hopefully your load balancers.
Tuesday, 25 October 11
-
INSTALL THE LATEST
RHEL / CentOS
Debian
rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpmyum install -y varnish
apt-get install varnish
Tuesday, 25 October 11
-
USE 64BIT*
Its all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, youre somewhat on your own. However, try to fit it within 2GB. I wouldnt recommend a cache larger than 1GB, and no more than a few hundred threads (Why are you on 32bit again?)
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
-
SYSTEM CONFIG
The minimum.
Do not set tw_reuse to 1 (sysctl). It will cause problems for NAT based clients.
Enable keepalives on load balancers.
echo 'fs.file-max=1000000' >> /etc/sysctl.confecho 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.confsysctl -pecho '* soft nofile 1000000' >> /etc/security/limits.confecho '* hard nofile 1000000' >> /etc/security/limits.confecho 'session required pam_limits.so' >> /etc/pam.d/loginulimit -n 1000000ulimit -n -H
Tuesday, 25 October 11
-
THREADS
Adjust default thread* settings in /etc/sysconfig/varnish
thread_pool_add_delay=2thread_pools = thread_pool_min = thread_pool_max = 4000session_linger = 50sess_workspace =
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesnt matter much. Our defaults are good.
Tuesday, 25 October 11
-
STORAGE
Memory is fast, duh.
Got enough RAM to store the cache?Yes! then use -s mallocUnsure? then use -s file
Tuesday, 25 October 11
-
VCL
Varnish Configuration Language
Looks like C and compiled into C but ...~ has regex tilda operatorAssignments use set keyword.Only has conditional if tests.Has a bunch of variables and some built in functions.
Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.
Tuesday, 25 October 11
-
VCL
Default VCL is defined in/etc/varnish/default.vcl
Then create your custom VCL in -f /var/custom.vclso varnish can fall through to the default.vclCan also include "/var/other.vcl";
Keep it simple within the VCL workflow*Do most of the work in vcl_recv and vcl_fetch
* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11
-
LOOKUP AND MISS
request received from client
hash request
lookup object and miss
fetch response from backend
deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
-
LOOKUP AND HIT
request received from client
hash request
lookup object and hit
deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
-
PASS AND DELIVER
request received from client
pass through to backend
fetch response
deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
-
PIPE
request received from client
pipe to backend until client or backend closes
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
-
PROBES
Define common health checks for multiple backends.
window is how many of the latest polls are probed for health.
threshold is how many polls must succeed to be healthy.
probe health {.url = "/healthCheck?onFailure=true";.expected_response = 200; .interval= 15s; .timeout = 5s;.window= 4; .threshold = 4; }
Tuesday, 25 October 11
-
BACKENDS
Define backend host, port and health checks inline.
Or named.backend mysite {.host = "www.example.com";.port = "http"; .probe = "health";}
{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }
Tuesday, 25 October 11
-
DIRECTORS
Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin Fallback
Tuesday, 25 October 11
-
YMMV
I found it better to balance in front of Varnish using a hardware load balancer ...
Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.
Tuesday, 25 October 11
-
DESIGN DECISIONS
Share nothing. Easy to scale horizontally.Great if your site is not transactional.Backend session affinity via LB set cookie.
LB
app
varnishvarnishvarnish
appapp
Set-Cookie=BackendID
Tuesday, 25 October 11
-
DESIGN DECISIONS
Share everything. Varnish loves RAM, scale vertically.Varnish cache becomes the shared source of truth.
LB
app
varnish
appapp
Tuesday, 25 October 11
-
SHARE NOTHING. FALLBACK.
LB
app
varnishvarnishvarnish
appapp
Set-Cookie=VarnishIDdirector varnishA fallback {{ .backend = { .host = "a";} } { .backend = { .host = "b";} } { .backend = { .host = "c";} } }
director varnishB fallback {{ .backend = { .host = "b";} } { .backend = { .host = "a";} } { .backend = { .host = "c";} } }
director varnishC fallback {{ .backend = { .host = "c";} } { .backend = { .host = "b";} } { .backend = { .host = "a";} } }
Tuesday, 25 October 11
-
TERMINATE SSL.
LB varnish appSet-Cookie=VarnishIDPort=https:80:443
if (req.http.Port ~ "https") { set req.backend = appa_80;} else {set req.backend = appa_443; }
Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to.
Tuesday, 25 October 11
-
VCL TRICKS
Rewrite headers to/from production.com.au to testdomain.com
vcl_recv
vcl_deliver
set req.http.Host = regsuball(req.http.Host, "testdomain\.com", "com\.au");set req.http.Referer = regsuball(req.http.Host, "testdomain\.com", "com\.au");
set resp.http.Host = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Referer = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Location = regsuball(resp.http.Location, "\.com\.au", "\.testdomain\.com");set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");
Tuesday, 25 October 11
-
VCL TRICKS
Add some debug headers for hit/miss in firebug in vcl_deliverif (obj.hits > 0) {set resp.http.X-Varnish-Hit = "++";set resp.http.X-Varnish-Hits = obj.hits;set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");} else {set resp.http.X-Varnish-Hit = "--";set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");}
Tuesday, 25 October 11
-
VCL TRICKS
Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv
if (req.http.Accept-Encoding) {if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {remove req.http.Accept-Encoding;} elsif (req.http.Accept-Encoding ~ "gzip") {set req.http.Accept-Encoding = "gzip";} elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate";} else {remove req.http.Accept-Encoding;}}
Tuesday, 25 October 11
-
VCL TRICKS
Remove Google Analytics cookies in vcl_recv
Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies.
Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
if (req.http.Authorization || req.http.Cookie) { return (lookup); }
Tuesday, 25 October 11
-
VCL TRICKS
Pass on the client IP to the backend servers using the X-Forwarded-For header in vcl_recv
remove req.http.X-Forwarded-For;setreq.http.X-Forwarded-For = client.ip;
Tuesday, 25 October 11
-
VCL TRICKS
Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch
Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv
Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.
set beresp.grace = 2h;
if (! req.backend.healthy) {set req.grace = 2h;} else {set req.grace = 30s;}
Tuesday, 25 October 11
-
VCL TRICKS
Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recvsub vcl_error {if (req.restarts == 0) {set req.http.X-Serve-Graced = "1";return(restart);}}
Tuesday, 25 October 11
-
TTL
Varnish will obey cache control headers from backendCache-Control: s-maxageCache-Control: max-age
Can also override TTL control in VCL, but best do it from the backendset obj.ttl = x
Tuesday, 25 October 11
-
REDUCING CACHE SIZE
Normalize / sanitize request headers
Set Content-Length on backends
Serve (static) digital assets from VFS cache
Pipe large objects like videos
Tuesday, 25 October 11
-
MONITORING VARNISH
Handy command line toolsvarnishstatvarnishlogvarnishtopvarnishadm
Tuesday, 25 October 11
-
VARNISHSTAT
varnishstatHit rate % (x100)Average per secondAverage/sec since bootSum total
Filteredvarnishstat -f \ client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail
Tuesday, 25 October 11
-
VARNISHSTAT
Log all statistics to disk ...
Then graph as you see fit ...
#!/bin/bashlogdir=/var/log/perf/`hostname`/latestmkdir -p $logdirwhile truedovarnishstat -x >> "$logdir/varnishstat.xml"sleep 60done
Tuesday, 25 October 11
-
VARNISHLOG
varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]
Tuesday, 25 October 11
-
VARNISHTOP
Monitor top URLs hitting backendvarnishtop -i txurl
Monitor top received URLsvarnishtop -i rxurl
Tuesday, 25 October 11
-
VARNISHADM
varnishadm for administration# varnishadmCLI connected to 127.0.0.1 6082200 -----------------------------Varnish Cache CLI 1.0-----------------------------Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbitType 'help' for command list.Type 'quit' to close CLI session.varnish> help200 help [command]ping [timestamp]auth responsequitbannerstatusstartstopvcl.load vcl.inline vcl.use vcl.discard vcl.listvcl.show param.show [-l] []param.set panic.showpanic.clearstorage.listban.url ban [&& ]...ban.list
Tuesday, 25 October 11
-
MORE HELP?
Talk to the devs, read their blogs.http://kristianlyng.wordpress.comhttp://ingvar.blog.redpill-linpro.com/
Get some online training.https://www.varnish-software.com/products-services/training/varnish-administration-course
Consider the Varnish Moral License.http://phk.freebsd.dk/VML/
Tuesday, 25 October 11