User Tools

Site Tools


web_services:squid_update_cache

This is an old revision of the document!


Squid Based Update Cache (Ubuntu 14.04)

Setup

Compile Squid

mkdir /usr/src/arch
cd /usr/src/arch
wget http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.1.tar.gz
cd ..
tar xvzf arch/squid-3.5.1.tar.gz
cd squid-3.5.1/
./configure --prefix=/usr   --localstatedir=/var   --libexecdir=/usr/lib/squid    --srcdir=.   --datadir=/usr/share/squid   --sysconfdir=/etc/squid   --with-default-user=proxy   --with-logdir=/var/log   --with-pidfile=/var/run/squid.pid
make
make install
mkdir /var/log/squid
chown -R proxy:proxy /var/log/squid
mkdir -p /var/spool/squid/
chown -R proxy:proxy /var/spool/squid/

Install missing scripts and configs from Debian/Ubuntu Package

Script from Ubuntu squid package (extract to /etc)

cd
wget http://wiki.sebeka.k12.mn.us/_media/web_services:squid_pkg.tar
tar xvpf squid_pkg.tar.tar -C /

Edit /etc/sysctl.conf

  1. add the following to /etc/sysctl.conf. This will enable packet forwarding
net.ipv6.conf.all.forwarding=1

Edit /etc/rc.local

  • Change 999.999.999.999 to the local ip of squid server. This will redirect every port 80 request through squid.
  • In this example we are using port 3129 as a transparent proxy port
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to 999.999.999.999:3129

Edit /etc/squid/squid.conf

/etc/squid/squid.conf
acl localnet src 10.0.0.0/8     # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines
 
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
 
http_access allow localnet
http_access allow localhost
 
# And finally deny all other access to this proxy
http_access  deny all
 
 
http_port 3128
http_port 3129 intercept
 
cache_dir ufs /var/spool/squid 50000 16 256
 
acl updatesites dstdom_regex "/etc/squid/updatesites.txt"
 
cache allow updatesites
cache deny all
 
maximum_object_size 5000 MB
range_offset_limit -1
quick_abort_min -1 KB
 
cache_store_log /var/log/squid/store.log
access_log daemon:/var/log/squid/access.log squid
cache_log /var/log/squid/cache.log
 
coredump_dir /var/spool/squid
 
store_id_access allow updatesites
store_id_access deny all
store_id_program /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
store_id_children 10 startup=3 idle=1 concurrency=0
 
refresh_pattern ([^.]+.|)adobe.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+.|)java.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+.|)sun.com/.*\.(zip|exe) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern ([^.]+.|)oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern ([^.]+.|)cs\.steampowered\.com 43200 100% 43200 reload-into-ims ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i appldnld\.apple\.com 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
refresh_pattern -i ([^.]+.|)apple.com/.*\.(ipa) 43200 100% 43200 ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200 reload-into-ims  ignore-reload ignore-no-store override-expire override-lastmod
 
refresh_pattern -i ([^.]+.|)google.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
refresh_pattern -i ([^.]+.|)gstatic.com/.*\.(exe|crx) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
 
refresh_pattern -i ([^.]+.|)ubuntu.com/.*\.(deb) 10080 80% 43200 override-expire override-lastmod ignore-no-cache ignore-reload reload-into-ims ignore-private
 
refresh_pattern ^ftp: 144000 20% 1008000
refresh_pattern -i \.(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|tiff|pdf|uxx)$ 260000 100% 260009 override-expire
refresh_pattern    \?                0 0% 4320
refresh_pattern    .                    0 20% 4320

Edit /etc/squid/updatesites.txt

([^.]+.|)adobe.com
([^.]+.|)java.com
([^.]+.|)sun.com
([^.]+.|)oracle.com
([^.]+.|)apple.com
([^.]+.|)apple.com
([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com
([^.]+.|)ubuntu.com
([^.]+.|)steampowered.com
([^.]+.|)google.com
([^.]+.|)gstatic.com

Edit **/etc/squid/storeid_rewrite.conf

(there must be a tab “\t” between the two columns)

^http:\/\/.+?\.ws\.microsoft\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/.+?\.windowsupdate\.com\/.+?_([0-9a-z]{40})\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip|psf|appx) http://wupdate.squid.local/$1
^http:\/\/[^\.]+\.cs\.steampowered\.com\/(.*)   http://steamupdates.squid.internal/$1
^http:\/\/[^\.]+\.phobos\.apple\.com\/(.*)      http://appupdates.apple.squid.internal/$1
^http:\/\/[^\.]+\.c\.android\.clients\.google\.com\/(.*)        http://androidupdates.google.squid.internal/$1

Misc Notes

Test storeid_rewrite

echo "<URL>" | /usr/lib/squid/storeid_file_rewrite /etc/squid/storeid_rewrite.conf
  • You will get a storeid= if storeid_file_rewrite found a match. Otherwise you get err

Setting Proxy Server for Updates

If you use transparent filter you don't need to worry about proxy server settings

Linux

  1. http_proxy or https_proxy set to another proxy server (this can also be used set a proxy server for apt updates)

Windows Update

  1. Use proxycfg to set proxy server for windows update
    proxycfg -p proxyservername:portnumber

iOS (iPad/iPod/iPhone)

  1. iOS uses what ever proxy setting that is set in the Settings App

Android

  1. App updates/downloads can't be cached.

Steam Updates

  1. Steam will not use a proxy

ChromeOS

  1. ChromeOS's proxy can be set in the settings of each ChromeOS device or in the administrative control panel under:
  • Device ManagementChromeNetwork
web_services/squid_update_cache.1424540026.txt.gz · Last modified: 2015/02/21 11:33 by tschulz