problem with Routing for multiple uplinks/providers on RH9 2.4.36.2

[ciuly]

Hello,

This is driving me nuts for a few days now. I just can't seem to make it work. This is the document I am using as a reference the lartc rpdb multiple-links howto
Also note that I am on a redhat 9 with an updated kernel 2.4.36.2 compiled with everything needed for advanced routing and packet filtering.

using the notations from that document, my setup is as follows:
- $IP1 is my main ISP and everything is configured through it and working fine
- $IP2 is my secondary ISP and it is configured and apparently is working (accessing $IP2 from a browser on a remote PC brings up the web page. ping also works. DNS doesn't. I'll get back to it later)
- my internal network is SNAT-ed through $IP1 and it works fine because showmyip-like sites report $IP1

however, if I do a DIG from another external machine, like "dig example.com A @$IP2" I get
reply from unexpected source: $IP1#53, expected $IP2#53

I only need this working for DNS, however it would be nice that ALL packets that come through an interface go out the same interface.

During my testings I found out why this happens:
- the packets comming in on $IP2 are being routed to the default gateway which is $P1
- the SNAT is done on all packets that go out on $IF1 for obvious reasons
- so obviously the response packet got SNATed
How?
iptables -t nat -I POSTROUTING 1 -o $IF1 -p udp --sport 53 -j DROP
and the external dig call timesout with no other error.
Also, doing the same in OUTPUT chain has same effect.

from my RTFM-ing, I learned that the route is decided after the packet is generated and before it hits iptables. I tested it with
iptables -t mangle -A OUTPUT -o $IF1 -p udp --sport 53 -j DROP
it's the red "output routing" here: docum.org/docum.org/kptd

So it is obvious that the packet gets on $IF1 instead of $IF2, based on the routing rules and routes.

I also asked a friend to do that dig and checked the chace:

[root@Impuls root]# ip route show table cache | grep friends_ip
local $IP2 from friends_ip dev lo  src $IP2
friends_ip via $P1 dev $IF1  src $IP1

but I don't know yet how to interpret that information though.

I have tried all sort of tehniques, starting with routing rules and finishing with iptables marking and connection tracking. nothing seems to work. I started from the above mentioend document/how-to which appears to be mentioned lot on the internet, so it must be good but also it is noted that it lacks some scenarios, specifically related to NATing of some kind.
I can mark an incomming packet in PREROUTING and still have it marked in INPUT (again tested with -j DROP by selecting the packed based on mark). but after that, in OUTPUT it's no longer there, which I think makes sense since in OUTPUT I have a new packet which is a response to the incomming one (at least that's how I understand this) so that packet is not marked. I'm not even sure if it should be.
I was marking packets with:
iptables -t mangle -A PREROUTING -i $IF2 -d $IP2 -p udp --dport 53 -j MARK --set-mark 6

All the above being said, I'd like to start out fresh. I cleaned up the routing rules and tables and my iptables rules so all is left is whatever was there before this ordeal started, so I'm back to square 1 as explained initially.

and I am out of ideas. The main question is, how can I get a packet comming in on an interface go out the same interface?

so:

exact routing tables and rules:

[root@Impuls /]# ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup 253
[root@Impuls /]#


[root@Impuls /]# ip route show table local
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast $P0_NET dev $IF0 proto kernel scope link src $IP0
broadcast $P1_NET dev $IF1 proto kernel scope link src $IP1
local $IP2 dev $IF2 proto kernel scope host src $IP2
broadcast $P2_BCAST dev $IF2 proto kernel scope link src $IP2
broadcast $P2_NET dev $IF2 proto kernel scope link src $IP2
local $IP1 dev $IF1 proto kernel scope host src $IP1
broadcast $P0_BCAST dev $IF0 proto kernel scope link src $IP0
broadcast $P1_BCAST dev $IF1 proto kernel scope link src $IP1
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
local $IP0 dev $IF0 proto kernel scope host src $IP0
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
[root@Impuls /]#

[root@Impuls /]# ip route show table main
$P2_NET/30 dev $IF2 scope link
$P1_NET/24 dev $IF1 scope link
$P0_NET/24 dev $IF0 scope link
169.254.0.0/16 dev $IF1 scope link
127.0.0.0/8 dev lo scope link
default via $P1 dev $IF1
[root@Impuls /]#

[root@Impuls /]# ip route show table 253
[root@Impuls /]#

relevant iptables rules

[root@Impuls /]# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- 0.0.0.0/0 0.0.0.0/0 to:$IP1

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain linkup (1 references)
target prot opt source destination
[root@Impuls /]#

[root@Impuls /]# iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
[root@Impuls /]#

Lately I've been reading the routing-selection howto from linux-ip.net but it's not detailed enough for me to actually apply the info on the routes and basically simulate linux routing selection "by hand". I need more something like a MAN with examples for each option.
I also read through the PolicyRoutingBook from policyrouting.org but it's kind of missing on examples.

bottom line: in order to successfully test, I am thinking that the best solution is to "block" the outgoing (response) packets by IP (not iptables) directly with policy routing.
the following did not work for me

[root@Impuls root]# iptables -t mangle -A PREROUTING -i eth0 -d $IP2 -p udp --dport 53 -j MARK --set-mark 2
[root@Impuls root]# ip rule
0:      from all lookup local
32761:  from all to $P2_NET lookup 2
32762:  from all to $IP2 lookup 2
32763:  from $IP2 lookup 2
32764:  from $P2_net lookup 2
32765:  from all fwmark 0x2 lookup 2
32766:  from all lookup main
32767:  from all lookup 253
[root@Impuls root]# ip route show table 2
blackhole 0.0.0.0
[root@Impuls root]#

but since the actual routing routes didn't work either, it was expected.

[ciuly]

as usually, I end up answering my own (painfull, not google-able) problems.
so, I was too concentrated on DNS (that being the only service I actually needed to work right) that I did not bother testing any other services. After doing a lot of rtfm-ing on the linux routing and packet filtering and trying out all sorts of things I decided that until I find a solution, to make the DNS work. long stroy short, I configured my DNS to bind to each interface and it was than when I saw that power dsn can actually change the source address in the packet. So I got DNS working and then I figured I should try another service, http. I did the test without lartc implementation, it behaved as described, packet went out the wrong interface. however, when I implemented the lartc split, it went on the right interface.
So the actual problem was not the lartc but the powerdns. maybe I should log a bug with them.

so for whoever searches for a similar problem: make sure your service is not messing up the packets to begin with. test your setup with a nice playing apache for example and if that works, it's definitelly your service to blaim.