Update Caching using Squid

fetching updates using HTTP POST and not HTTP GET

various HTTP headers

file size limits

URLs with “?” and parameters also cause problems

Windows Updates

iOS system updates and iOS App updates

Debian/Ubuntu updates (apt-get)

Steam game updates (storeid will be need to handle CDNS)

adobe (flash/shockwave/reader)

Oracle (java)

Chrome updates

Android updates

Ubuntu server 14.04 (VM or Bare Metal)

Squid 3.4 or higher (3.5.1 compiled/sorry no package:))

recommend setting up cache server as transparent proxy (enable IPv4 packet forwarding, forward port 80 to squid)

ZFS raidz works really well for cache store

Enable Transparent Proxying

http_port 3129 intercept

Set Max Object Size to something large

maximum_object_size 5000 MB
range_offset_limit -1
quick_abort_min -1 KB

Set a large cache store

cache_dir ufs /var/spool/squid 50000 16 256

Only allow caching from update sites

acl updatesites dstdom_regex "/etc/squid/updatesites.txt"

cache allow updatesites
cache deny all

Setup store_id for CDN Matching

store_id_access allow updatesites
#store_id_access allow steamcdn
store_id_access deny all
store_id_program /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
store_id_children 10 startup=3 idle=1 concurrency=0

Override Cache timeouts and other cache HTTP headers

refresh_pattern ([^.]+.|)adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overr
refresh_pattern ([^.]+.|)java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overri
refresh_pattern ([^.]+.|)sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
refresh_pattern ([^.]+.|)oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expi

refresh_pattern ([^.]+.|)cs\.steampowered\.com 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+.|)apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims  i

refresh_pattern -i ([^.]+.|)google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload r
refresh_pattern -i ([^.]+.|)gstatic.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload

refresh_pattern -i ([^.]+.|)ubuntu.com/.*\.(deb) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reloa

refresh_pattern ^ftp: 144000 20% 1008000
refresh_pattern -i \.(deb|rpm)$ 260000 100% 260009 override-expire

storeid_rewrite.conf

^http:\/\/.+?\.ws\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/[^\.]+\.cs\.steampowered\.com\/(.*)   http://steamupdates.squid.internal/$1
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)      http://appupdates.apple.squid.internal/$1

updatesites.txt

([^.]+.|)adobe.com
([^.]+.|)java.com
([^.]+.|)sun.com
([^.]+.|)oracle.com
([^.]+.|)apple.com
([^.]+.|)apple.com
([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com
([^.]+.|)ubuntu.com
([^.]+.|)steampowered.com
([^.]+.|)google.com
([^.]+.|)gstatic.com

rc.local

iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 10.1.0.21:3129
exit 0

sysctl.conf

...
net.ipv4.ip_forward=1
...

Squid Based Update Cache (Ubuntu 14.04)

ZFS on Ubuntu