ISD 820 Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Update Caching using Squid ======

===== What Stops updates from caching =====
  * fetching updates using HTTP POST and not HTTP GET
  * various HTTP headers
  * file size limits
  * URLs with "?" and parameters also cause problems

===== What updates can be cached =====
  * Windows Updates
  * iOS system updates and iOS App updates
  * Debian/Ubuntu updates (apt-get)
  * Steam game updates (storeid will be need to handle CDNS)
  * adobe (flash/shockwave/reader)
  * Oracle (java)

===== What updates can't be cached =====
  * Chrome updates
  * Android updates

===== Cache Setup =====
  * Ubuntu server 14.04 (VM or Bare Metal)
  * Squid 3.4 or higher (3.5.1 compiled/sorry no package:))
  * recommend setting up cache server as transparent proxy (enable IPv4 packet forwarding, forward port 80 to squid)
  * ZFS raidz works really well for cache store

===== Squid notable config changes =====
  * Enable Transparent Proxying<file>http_port 3129 intercept</file>
  * Set Max Object Size to something large<file>maximum_object_size 5000 MB
range_offset_limit -1
quick_abort_min -1 KB
</file>
  * Set a large cache store<file>cache_dir ufs /var/spool/squid 50000 16 256</file>
  * Only allow caching from update sites<file>acl updatesites dstdom_regex "/etc/squid/updatesites.txt"

cache allow updatesites
cache deny all</file>
  * Setup store_id for CDN Matching<file>store_id_access allow updatesites
#store_id_access allow steamcdn
store_id_access deny all
store_id_program /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
store_id_children 10 startup=3 idle=1 concurrency=0</file>
  * Override Cache timeouts and other cache HTTP headers<file>refresh_pattern ([^.]+.|)adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overr
refresh_pattern ([^.]+.|)java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overri
refresh_pattern ([^.]+.|)sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid
refresh_pattern ([^.]+.|)oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expi

refresh_pattern ([^.]+.|)cs\.steampowered\.com 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire overrid

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+.|)apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod

refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims  i

refresh_pattern -i ([^.]+.|)google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload r
refresh_pattern -i ([^.]+.|)gstatic.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload

refresh_pattern -i ([^.]+.|)ubuntu.com/.*\.(deb) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reloa

refresh_pattern ^ftp: 144000 20% 1008000
refresh_pattern -i \.(deb|rpm)$ 260000 100% 260009 override-expire
</file>

===== Other Squid configs =====
  * <file config storeid_rewrite.conf>^http:\/\/.+?\.ws\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/[^\.]+\.cs\.steampowered\.com\/(.*)   http://steamupdates.squid.internal/$1
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)      http://appupdates.apple.squid.internal/$1
</file>
  * <file config updatesites.txt>([^.]+.|)adobe.com
([^.]+.|)java.com
([^.]+.|)sun.com
([^.]+.|)oracle.com
([^.]+.|)apple.com
([^.]+.|)apple.com
([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com
([^.]+.|)ubuntu.com
([^.]+.|)steampowered.com
([^.]+.|)google.com
([^.]+.|)gstatic.com
</file>
  * <file bash rc.local>iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 10.1.0.21:3129
exit 0
</file>
  * <file bash  sysctl.conf>...
net.ipv4.ip_forward=1
...</file>


===== Detailed Caching Install Notes =====
  * [[web_services:squid_update_cache|Squid Based Update Cache (Ubuntu 14.04)]]
  * [[fileservices:zfs_ubuntu|ZFS on Ubuntu]]
ISD 820 Knowledge Base

User Tools

Site Tools

Page Tools
