misc:2015brainstormpress
Update Caching using Squid
What Stops updates from caching
What updates can be cached
Windows Updates
iOS system updates and iOS App updates
Debian/Ubuntu updates (apt-get)
Steam game updates (storeid will be need to handle CDNS)
adobe (flash/shockwave/reader)
Oracle (java)
What updates can't be cached
Chrome updates
Android updates
Cache Setup
Ubuntu server 14.04 (VM or Bare Metal)
Squid 3.4 or higher (3.5.1 compiled/sorry no package:))
recommend setting up cache server as transparent proxy (enable IPv4 packet forwarding, forward port 80 to squid)
ZFS raidz works really well for cache store
Squid notable config changes
Enable Transparent Proxying
http_port 3129 intercept
Set Max Object Size to something large
maximum_object_size 5000 MB
range_offset_limit -1
quick_abort_min -1 KB
Set a large cache store
cache_dir ufs /var/spool/squid 50000 16 256
Only allow caching from update sites
acl updatesites dstdom_regex "/etc/squid/updatesites.txt"
cache allow updatesites
cache deny all
Setup store_id for CDN Matching
store_id_access allow updatesites
#store_id_access allow steamcdn
store_id_access deny all
store_id_program /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
store_id_children 10 startup=3 idle=1 concurrency=0
Override Cache timeouts and other cache HTTP headers
refresh_pattern ([^.]+.|)adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overr
refresh_pattern ([^.]+.|)java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overri
refresh_pattern ([^.]+.|)sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
refresh_pattern ([^.]+.|)oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expi
refresh_pattern ([^.]+.|)cs\.steampowered\.com 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+.|)apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims i
refresh_pattern -i ([^.]+.|)google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload r
refresh_pattern -i ([^.]+.|)gstatic.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload
refresh_pattern -i ([^.]+.|)ubuntu.com/.*\.(deb) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reloa
refresh_pattern ^ftp: 144000 20% 1008000
refresh_pattern -i \.(deb|rpm)$ 260000 100% 260009 override-expire
Other Squid configs
- storeid_rewrite.conf
^http:\/\/.+?\.ws\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/[^\.]+\.cs\.steampowered\.com\/(.*) http://steamupdates.squid.internal/$1
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*) http://appupdates.apple.squid.internal/$1
- updatesites.txt
([^.]+.|)adobe.com
([^.]+.|)java.com
([^.]+.|)sun.com
([^.]+.|)oracle.com
([^.]+.|)apple.com
([^.]+.|)apple.com
([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com
([^.]+.|)ubuntu.com
([^.]+.|)steampowered.com
([^.]+.|)google.com
([^.]+.|)gstatic.com
- rc.local
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 10.1.0.21:3129
exit 0
-
Detailed Caching Install Notes
misc/2015brainstormpress.txt · Last modified: 2015/02/20 15:41 by tschulz