Network redundancy suggestions

[bunyipoz]

Hey everyone,

I'm trying to implement a poor man's network/hardware redundancy setup with 4 fedora10 servers ( 2 nics in each) and two pix 501 firewalls. I've inherited the infrastructure and don't have budget for anymore.

I'm running a public webservice that talks to a mysql database. The client app can connect to the webservice using primary.company.com but if that fails (pix dies, isp down) then they are configured to use secondary.company.com.

I was looking at implementing the following topology:

internet
|
primary.company.com
pix1
192.168.40.1
|
192.168.40.10 server1 (web) 192.168.20.11
192.168.40.11 server2 (mysql) 192.168.20.10
|
192.168.20.1
pix2
secondary.company.com
|
internet

What my terrible ascii art may obscure is that each server has two nics and that one nic is connected to pix1 and the other to pix 2. The pix's are not cabled together.

On server1 and 2 I implemented policy based routing (ala lartc.org-howto-lartc.rpdb.multiple-links) so that web service connections coming in on pix1 go back out through pix1 and connections in on pix2 go back out on pix 2. I tested this and managed to hit the webservices from both urls.

The problem I'm having is that i can't seem to ssh (or anything else for that matter) between server1 and server2 once the policy routing is applied. Both servers can still talk to the pix's (telnet console) and the outside network just not to eachother.

When I replace pix2 with a home router/switch I can communicate between the servers again. A traceroute from server1 to server2 (192.168.20.10->192.168.20.11) shows that the first hop is to the router and then server 2. Being on the same subnet i would not have expected to see the hop through the home router/switch.

There aren't any acl's applied to the inside interface on the pixs.

So any ideas as to why the servers won't talk to eachother anymore? Are there other ways to achieve this "failover".

Once again I don't really have budget to purchase more hardware and I'm trying hard not to introduce single point of failures. I'll be looking at clustering the web server and setting up db replication for the mysql box as another exercise.

Thanks in advance!

[bunyipoz]

I think I've sussed it!
I bound a secondary ip address to each of the network adapters in server1 and server2 using on a different subnet with no gateway specified (as there is no reason for internal host communication to route outside of the subnet). I was then able to ping and ssh between server 1 and 2 using 172.16.10.x host addresses.
so now the config looks like:
server1 eth1 remains the same (host,netmask,gw): 192.168.40.10 255.255.255.0 192.168.40.1
server1 eth0 remains the same: 192.168.20.10 255.255.255.0 192.168.20.1
server 2 eth0 remains the same: 192.168.40.11 255.255.255.0 192.168.40.1
server 2 eth1 remains the same: 192.168.20.11 255.255.255.0 192.168.20.1
server1 created new adapter eth1:0: 172.16.10.10 255.255.255.0 no gateway
server1 created new adapter eth0:0: 172.16.10.20 255.255.255.0 no gateway
server2 created new adapter eth1:0: 172.16.10.11 255.255.255.0 no gateway
server2 created new adapter eth0:0: 172.16.10.21 255.255.255.0 no gateway
So now:

1. from the outside network I can talk to my webservice from primary.company.com and secondary.company.com. PBR allows me send packets back out on the interfaces they came in on.
2. inside the network, server 1 can now talk to server 2 or any other host 172.16.10.0 network. No gateway is needed for the adapters eth1:0 or eth0:0.
3. if either pix goes down the hosts in the internal network should still be able to talk to each other – will need to test

[peter]

Good to hear you got it working!

[bunyipoz]

A suggestion from someone on another forum noticed that I didn't include subnets in my routes. Using the configuration below worked without requiring the use of secondary ip addresses

ip route add 192.168.40.0/24 dev eth1 src 192.168.40.10 table forty
ip route add default via 192.168.40.1 table forty
ip route add 192.168.20.0/24 dev eth0 src 192.168.20.10 table twenty
ip route add default via 192.168.20.1 table twenty
ip route add 192.168.40.0/24 dev eth1 src 192.168.40.10
ip route add 192.168.20.0/24 dev eth0 src 192.168.20.10
ip route add default via 192.168.40.1
ip rule add from 192.168.40.10 table forty
ip rule add from 192.168.20.10 table twenty