User Tools

Site Tools


misc:2015brainstormpress

Update Caching using Squid

What Stops updates from caching

  • fetching updates using HTTP POST and not HTTP GET
  • various HTTP headers
  • file size limits
  • URLs with “?” and parameters also cause problems

What updates can be cached

  • Windows Updates
  • iOS system updates and iOS App updates
  • Debian/Ubuntu updates (apt-get)
  • Steam game updates (storeid will be need to handle CDNS)
  • adobe (flash/shockwave/reader)
  • Oracle (java)

What updates can't be cached

  • Chrome updates
  • Android updates

Cache Setup

  • Ubuntu server 14.04 (VM or Bare Metal)
  • Squid 3.4 or higher (3.5.1 compiled/sorry no package:))
  • recommend setting up cache server as transparent proxy (enable IPv4 packet forwarding, forward port 80 to squid)
  • ZFS raidz works really well for cache store

Squid notable config changes

  • Enable Transparent Proxying
    http_port 3129 intercept
  • Set Max Object Size to something large
    maximum_object_size 5000 MB
    range_offset_limit -1
    quick_abort_min -1 KB
  • Set a large cache store
    cache_dir ufs /var/spool/squid 50000 16 256
  • Only allow caching from update sites
    acl updatesites dstdom_regex "/etc/squid/updatesites.txt"
    
    cache allow updatesites
    cache deny all
  • Setup store_id for CDN Matching
    store_id_access allow updatesites
    #store_id_access allow steamcdn
    store_id_access deny all
    store_id_program /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
    store_id_children 10 startup=3 idle=1 concurrency=0
  • Override Cache timeouts and other cache HTTP headers
    refresh_pattern ([^.]+.|)adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overr
    refresh_pattern ([^.]+.|)java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overri
    refresh_pattern ([^.]+.|)sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
    refresh_pattern ([^.]+.|)oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expi
    
    refresh_pattern ([^.]+.|)cs\.steampowered\.com 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
    
    refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
    refresh_pattern -i ([^.]+.|)apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
    
    refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims  i
    
    refresh_pattern -i ([^.]+.|)google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload r
    refresh_pattern -i ([^.]+.|)gstatic.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload
    
    refresh_pattern -i ([^.]+.|)ubuntu.com/.*\.(deb) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reloa
    
    refresh_pattern ^ftp: 144000 20% 1008000
    refresh_pattern -i \.(deb|rpm)$ 260000 100% 260009 override-expire

Other Squid configs

  • storeid_rewrite.conf
    ^http:\/\/.+?\.ws\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
    ^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
    ^http:\/\/[^\.]+\.cs\.steampowered\.com\/(.*)   http://steamupdates.squid.internal/$1
    ^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)      http://appupdates.apple.squid.internal/$1
  • updatesites.txt
    ([^.]+.|)adobe.com
    ([^.]+.|)java.com
    ([^.]+.|)sun.com
    ([^.]+.|)oracle.com
    ([^.]+.|)apple.com
    ([^.]+.|)apple.com
    ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com
    ([^.]+.|)ubuntu.com
    ([^.]+.|)steampowered.com
    ([^.]+.|)google.com
    ([^.]+.|)gstatic.com
  • rc.local
    iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 10.1.0.21:3129
    exit 0
  • sysctl.conf
    ...
    net.ipv4.ip_forward=1
    ...

Detailed Caching Install Notes

misc/2015brainstormpress.txt · Last modified: 2015/02/20 15:41 by tschulz