XNUXER.OR.ID

nat on $ext_if inet from any to any -> { 192.0.2.5, 192.0.2.10 } 

nat on $ext_if inet from any to any -> 192.0.2.4/31 source-hash 

web_servers = "{ 10.0.0.10, 10.0.0.11, 10.0.0.13 }"

rdr on $ext_if proto tcp from any to any port 80 -> $web_servers \
 round-robin sticky-address 

lan_net = "192.168.0.0/24"
int_if  = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "68.146.224.1"
ext_gw2 = "142.59.76.1"

pass in on $int_if route-to \
 { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
 from $lan_net to any keep state 

pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 \
 to any
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 \
 to any 

nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2) 

lan_net = "192.168.0.0/24"
int_if  = "dc0"
ext_if1 = "fxp0"
ext_if2 = "fxp1"
ext_gw1 = "68.146.224.1"
ext_gw2 = "142.59.76.1"

#  nat outgoing connections on each internet interface
nat on $ext_if1 from $lan_net to any -> ($ext_if1)
nat on $ext_if2 from $lan_net to any -> ($ext_if2)

#  default deny
block in  from any to any
block out from any to any

#  pass all outgoing packets on internal interface
pass out on $int_if from any to $lan_net
#  pass in quick any packets destined for the gateway itself
pass in quick on $int_if from $lan_net to $int_if
#  load balance outgoing tcp traffic from internal network. 
pass in on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto tcp from $lan_net to any flags S/SA modulate state
#  load balance outgoing udp and icmp traffic from internal network
pass in on $int_if route-to \
    { ($ext_if1 $ext_gw1), ($ext_if2 $ext_gw2) } round-robin \
    proto { udp, icmp } from $lan_net to any keep state

#  general "pass out" rules for external interfaces
pass out on $ext_if1 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if1 proto { udp, icmp } from any to any keep state
pass out on $ext_if2 proto tcp from any to any flags S/SA modulate state
pass out on $ext_if2 proto { udp, icmp } from any to any keep state

#  route packets from any IPs on $ext_if1 to $ext_gw1 and the same for
#  $ext_if2 and $ext_gw2
pass out on $ext_if1 route-to ($ext_if2 $ext_gw2) from $ext_if2 to any 
pass out on $ext_if2 route-to ($ext_if1 $ext_gw1) from $ext_if1 to any 

How can I do equal-cost multipath routing?
Equal-cost multipath routing refers to having multiple routes in the routing table for the same network, such as the default route, 0.0.0.0/0. When the kernel is doing a route lookup to determine where to send packets destined to that network, it can choose from any of the equal-cost routes. In most scenarios, multipath routing is used to provide redundant uplink connections, e.g., redundant connections to the Internet.

The route command is used to add/change/delete routes in the routing table. The -mpath argument is used when adding multipath routes.

# route add -mpath default 10.130.128.1
# route add -mpath default 10.132.0.1

Verify the routes:

# netstat -rnf inet | grep default
default     10.130.128.1      UGS       2      134      -     fxp1
default     10.132.0.1        UGS       0      172      -     fxp2

In this example we can see that one default route points to 10.130.128.1 which is accessible via the fxp1 interface, and the other points to 10.132.0.1 which is accessible via fxp2.

Since the mygate(5) file does not yet support multipath default routes, the above commands should be added to the bottom of the hostname.if(5) files for the fxp1 and fxp2 interfaces. The /etc/mygate file should then be deleted.

/etc/hostname.fxp1
    !route add -mpath default 10.130.128.1
/etc/hostname.fxp2
    !route add -mpath default 10.132.0.1

Lastly, don't forget to activate the use of multipath routes by enabling the proper sysctl(3) variable.

# sysctl net.inet.ip.multipath=1
# sysctl net.inet6.ip6.multipath=1

Be sure to edit sysctl.conf to make the changes permanent.

Now try a traceroute to different destinations. The kernel will load balance the traffic over each multipath route.

# traceroute -n 154.11.0.4
traceroute to 154.11.0.4 (154.11.0.4), 64 hops max, 60 byte packets
1  10.130.128.1  19.337 ms  18.194 ms  18.849 ms
2  154.11.95.170  17.642 ms  18.176 ms  17.731 ms
3  154.11.5.33  110.486 ms  19.478 ms  100.949 ms
4  154.11.0.4  32.772 ms  33.534 ms  32.835 ms

# traceroute -n 154.11.0.5
traceroute to 154.11.0.5 (154.11.0.5), 64 hops max, 60 byte packets
1  10.132.0.1  14.175 ms  14.503 ms  14.58 ms
2  154.11.95.38  13.664 ms  13.962 ms  13.445 ms
3  208.38.16.151  13.964 ms  13.347 ms  13.788 ms
4  154.11.0.5  30.177 ms  30.95 ms  30.593 ms

For more information about how the route is chosen, please refer to RFC2992, "Analysis of an Equal-Cost Multi-Path Algorithm".