Squid Cache Transparency and WCCP
Version 1.1: 12 September 2001

Introduction

In the absence of a Layer-4 switch (eg. Foundry, Alteon) Squid transparency is probably best achieved using Cisco's Web Cache Coordination Protocol (WCCP). Policy Based Routing (PBR) can be used but suffers from one significant limitation: failover. If you choose to use PBR and your Squid cache dies or needs to be rebooted then all your web surfing stops. Further, WCCP allows multiple caches to participate (for further fault tolerance) while PBR only allows one. Squid only supports WCCPv1 at this time. WCCP does place a load on your router in excess of normal HTTP traffic. Hence, ensure that your router has adequate RAM and CPU horsepower.

So how does it work? In a nutshell:

- Your Linux box runs Squid which listens on port 3128.
- WCCP is enabled on your router to redirect encapsulated HTTP requests to your WCCP enabled interception cache (ie. Squid).
- When Squid starts it "registers" itself via WCCP with your router.
- WCCP heatbeats are continually transmitted between router and cache.
- A Generic Routing Encapsulation (GRE) tunnel is established between your Linux box and router.
- All redirected requests from the router are encapsulated down the GRE tunnel to your interception cache.
- The Linux box decapsulates the GRE traffic and redirects the WCCP packets onto Squid.
- This redirection is achieved transparently using IP forwarding and NetFilter (in 2.4.x series kernels).
- Squid pulls apart the request then attempts to deliver the content either from the local cache or via direct request from target.
- The content is then delivered back to the router for delivery to the originator (ie. your browser).

References

We strongly recommend that you read the excellent Squid Cache FAQ. Especially the section on WCCP and related links available from here. Further, we recommend your read Cisco's WCCP documentation to get a better understanding of the protocol.

Test Rig

In the example below we are using the following equipment:

- [Server] Dell PowerEdge 1400SC (1 x PIII 933, 256Mb RAM, 2 x 18Gb 10K RPM SCSI HDD). RedHat Linux v7.1 is installed.
- [Router] Aging Cisco 2511 with 1 x AUI, 2 x Serial, 2 x Async, 18Mb RAM, 4Mb Flash and IOS 12.2(3). IOS is loaded via TFTP from the 1400SC.
- [Client] IBM T21 (1 x PIII 800, 512Mb, 1 x 20Gb 4200 RPM EIDE HDD). SuSE Linux 7.2 is installed.

Architecture

Extremely simple setup:

                            ^
                            | 1Mb Serial to upstream provider 
                            |
                 +---------------------+
                 |     Cisco 2511      |
                 +---------------------+
                            |
                            | 10/100BT Layer-2 Segment
                            |
                 +----------+----------+
                 |                     |
                 |                     |
     +---------------------+ +---------------------+
     |    RH 7.1 + Squid   | | IBM Laptop (Client) |
     +---------------------+ +---------------------+

Software

You are going to need a bunch of software prior to doing anything:

- RedHat Linux v7.1 (or your preferred favourite distro with a 2.4 series kernel)
- Latest Linux kernel from http://www.kernel.org
- Latest Squid release from http://www.squid-cache.org
- WCCP module for Linux

We also installed a bunch of addition software on the machine that will not be covered in this document. Namely:

- DJB's daemontools and djbdns (for service management and local DNS name resolution and caching)
- NTP (for time synchronisation)
- Bastile Linux (for server hardening and optional NetFilter firewall)
- SNORT (for packet filtering and intrusion detection)
- Tripwire (for instrusion detection)

RedHat Linux

RedHat Linux (RHL) is a good general purpose Linux distro. It was chosen because lots of people know it and it's relatively easy to avoid installing the vast amounts of fluff that comes with other distros. We configured RHL on the Dell server using software RAID-1 using the partition map below:

Device Boot Start End Blocks Id System
/dev/sda1 * 1 6 48163+ fd Linux raid autodetect
/dev/sda2 7 2213 177227727+ 5 Extended
/dev/sda5 7 72 530113+ fd Linux raid autodetect
/dev/sda6 73 2213 17197551 fd Linux raid autodetect

Checking mount:

/dev/md2 on / type ext2 (rw)
none on /proc type proc (rw)
/dev/md0 on /boot type ext2 (rw)
none on /dev/pts type devpts

And finally df -k:

Filesystem 1k-blocks Used Available Used% Mounted on
/dev/md2 16927500 725288 15342340 5% /
/dev/md0 46537 8860 35274 21% /boot

We installed RHL using the custom installation option. It is usually a good idea to avoid installing:

- Any development environments (eg. gcc)
- X-Windows (ie. XFree and Gnome)

We then registered on the Red Hat Network using rhn_register and updated the RedHat installation using up2date.

Fun Stuff (Kernel Rebuilds etc.)

We ran a copy of RedHat Linux v7.1 on the SuSE 7.2 laptop using VMware. This was so we could hand build the necessary executables (Squid etc.) and them move them over to the production server via FTP.

Unfortunately, after running up2date and downloading the revised kernel sources (2.4.3-12) we noted that you could not compile modules (which you will need to do later on). The compiler fell over in asm/processor.h. After many hours of mucking around we figured out what had gone on: up2date via diff or rpm had clobbered a bunch of headers. So we immediately gave up and pulled the latest kernel from kernel.org (at the time 2.4.9). It is probably a good idea to rebuild the kernel on RHL since the default kernel is getting a little long in the tooth plus comes with a whole bunch of stuff you probably don't need. Here is our /usr/src/linux/.config file from our kernel build. Please pay special attention to the options that were selected in the Networking options section of that file.

Building Squid

Building squid is a snap. We used the most recent stable release: 2.4.STABLE2. Unpack your sources then configure as follows:

./configure --prefix=/usr/local --enable-wccp --enable-linux-netfilter --enable-async-io
make
strip src/squid src/client

We then tar up the entire squid directory and ftp it over to the production server. There we unbundle the tar and:

make install

You will need to setup squid to run at boot time (via /etc/rc.d/rc.local or /etc/init.d and chkconfig). You will also need to chown nobody.nobody on /usr/local/squid/cache and /usr/local/squid/logs. Squid runs as the user nobody. Consequently it needs read/write permissions on the aforementioned directories.

Configuring Squid for NetFilter and WCCP

We have included our squid.conf for your perusal. The options to pay special attention to in this file are:

wccp_router
httpd_accel_host
httpd_accel_port
httpd_accel_with_proxy
httpd_accel_uses_host_header

Please refer to the Transparent Squid Mini-HowTo for more info.

Compile the WCCP Module

Compile ip_wccp.c as follows for an Intel PIII processor:

gcc -D__KERNEL__ -I/usr/src/linux/include -Wall -Wstrict-prototypes -Wno-trigraphs -O2 -fomit-frame-pointer -fno-strict-aliasing -fno-common -pipe -mpreferred-stack-boundary=2 -march=i686 -DMODULE -DMODVERSIONS -include /usr/src/linux/include/linux/modversions.h -c -o ip_wccp.o ip_wccp.c

Copy ip_wccp.o to /lib/modules/<kernel-version>/kernel/net/ipv4.

Setup GRE, WCCP and Squid

There is a bundle of different ways to do this but we modified /etc/rc.d/rc.local and appended the following lines:

insmod ip_wccp
insmod ip_gre
iptunnel add gre1 mode gre remote <ip-address-of-router> local <ip-address-of-squid-cache> dev eth0
ifconfig gre1 127.0.0.2 up
/usr/local/squid/bin/RunCache &

The first two lines explicitly insert the WCCP and GRE modules. The third line creates a GRE tunnel between router and cache via eth0. The fourth line creates a GRE loopback adapter (ie. the local end-point of the GRE tunnel). The fifth line starts Squid.

Enable IP Forwarding

You need to edit /etc/sysctl.conf and modify the following lines to read:

net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 0

/etc/sysctl.conf is passed at boot time and modifies writeable kernel "variables" located in /proc/sys. ip_forward = 1 enables IP forwarding across interfaces (specifically gre1 and eth0 in this instance). rp_filter enables source route verification. Sysrq = 0 disables the magic-sysrq key.

Enable Transparent HTTP Redirection

You will undoubtably have a more sophisticated NetFilter setup. However, there is one line that you need to redirect inbound traffic on port 80 to Squid on port 3128:

iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128

Checking Out Your Linux Configuration

We recommend that you reboot your machine to ensure that all the various components are initialised. Check out the following:

[[email protected] /root]# lsmod
Module Size Used by
ip_gre 8160 1
ip_wccp 1120 0 (unused)
ipt_REDIRECT 1056 1 (autoclean)
iptable_nat 17136 0 (autoclean) [ipt_REDIRECT]
ip_conntrack 16912 1 (autoclean) [ipt_REDIRECT iptable_nat]
iptable_mangle 2048 0 (autoclean) (unused)
iptable_filter 2048 0 (autoclean) (unused)
ip_tables 11712 6 [ipt_REDIRECT iptable_nat iptable_mangle iptable_filter]

[[email protected] /root]# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
REDIRECT tcp -- anywhere anywhere tcp dpt:http redir ports 3128

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

[[email protected] /root]# iptunnel
gre0: gre/ip remote any local any ttl inherit nopmtudisc
gre1: gre/ip remote "router-ip" local "cache-ip" dev eth0 ttl inherit

[[email protected] /root]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:B0:D0:F9:1A:4E
inet addr:<your-ip> Bcast:<your-bcast> Mask:<your-netmask>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:201608 errors:0 dropped:0 overruns:0 frame:0
TX packets:149111 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
Interrupt:16

gre1 Link encap:UNSPEC HWaddr CB-19-6E-92-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:127.0.0.2 P-t-P:127.0.0.2 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MTU:1476 Metric:1
RX packets:71468 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0

[[email protected] /root]# cat /proc/sys/net/ipv4/ip_forward
1

Configuring IOS

The last (first?) step in all of this is configuring your router. We have included a sample router configuration for completeness. In a nutshell:

wccp>enable
wccp#config t
wccp(config)#ip wccp version 1
wccp(config)#ip wccp web-cache
wccp(config)#int <your-outgoing-interface>
wccp(config-if)#ip wccp web-cache redirect out
wccp(config-if)#end
wccp#write mem

Now check that your Squid cache has registered with the router:

wccp#show ip wccp
Global WCCP information:
Router information:
Router Identifier: <router-ip>
Protocol Version: 1.0

Service Identifier: web-cache
Number of Cache Engines: 1
Number of routers: 1
Total Packets Redirected: 91489
Redirect access-list: -none-
Total Packets Denied Redirect: 0
Total Packets Unassigned: 0
Group access-list: -none-
Total Messages Denied to Group: 0
Total Authentication failures: 0

If you need to debug WCCP try: debug ip wccp

Configuring Your Client

So here goes ... make sure your client box with the browser is setup and configured (ensure that the default gateway is the router you have setup). Point your browser at your favourite site and bingo! It should all be happening! Check out /usr/local/squid/logs/access.log!

Copyright © 2001 Sublime Solutions Pty Ltd