Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides

[b]BGP: Multihomed AS with more specific prefixes geographically split[/b]
[attachment=0]BGP-AS-multihomed-dual-ISP1.png[/attachment]

In the following lab, I tried to confirm if what some case studies available on the internet state is true regarding a dual-ISP multihomed

organization.

Let's say AS100 is geographically split. It owns 10.1.20/23 prefix. The requirement is to have incoming traffic for 10.1.20/24 via

location A and incoming traffic for 10.1.21/24 via location B.

Until recently, I was thinking this should be straight forward. R1 exports 10.1.20/24 and 10.1.20/23 to ISP1 (AS101) via location A and R2

exports 10.1.21/24 and 10.1.20/23 to ISP2 (AS102) via location B and the requirement is achieved.

I will focus on 10.1.21/24 prefix (or 10.1.21.0/24). Let's see Internet's visibility for this prefix.



[code]
ISP1#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 174
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1 2
102 100
101.101.0.2 from 101.101.0.2 (102.102.102.102)
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp neighbors 1.1.1.2 rou
BGP table version is 178, local router ID is 101.101.101.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 10.1.20.0/24 1.1.1.2 0 0 100 i
*> 10.1.20.0/23 1.1.1.2 0 0 100 i
*> 10.255.255.0/24 1.1.1.2 0 0 100 i
* 102.102.102.102/32
1.1.1.2 0 100 102 i
* 103.103.0.0/16 1.1.1.2 0 100 102 103 i
* 104.104.0.0/16 1.1.1.2 0 100 102 103 104 i
* 105.105.0.0/16 1.1.1.2 0 100 102 103 104 105 i

[/code]
[code]
ISP103#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 313
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1
102 100
102.102.0.1 from 102.102.0.1 (102.102.102.102)
Origin IGP, localpref 100, valid, external, best
[/code]

[code]
ISP108#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 235
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1
101 102 100
108.108.0.2 from 108.108.0.2 (101.101.101.101)
Origin IGP, localpref 100, valid, external, best
[/code]

[code]
ISP106#sh ip bgp 10.1.21.4
BGP routing table entry for 10.1.21.0/24, version 383
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to update-groups:
1
105 104 103 102 100
105.105.0.1 from 105.105.0.1 (105.105.105.105)
Origin IGP, localpref 100, valid, external
107 108 101 102 100
106.106.0.2 from 106.106.0.2 (107.107.107.107)
Origin IGP, localpref 100, valid, external, best
ISP106#sh ip bgp
BGP table version is 391, local router ID is 106.106.106.106
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 1.1.1.0/24 106.106.0.2 0 107 108 101 ?
*> 9.0.0.0 106.106.0.2 0 107 108 101 i
*> 10.1.20.0/24 106.106.0.2 0 107 108 101 100 i
*> 10.1.20.0/23 106.106.0.2 0 107 108 101 100 i
* 105.105.0.1 0 105 104 103 102 100 i
* 10.1.21.0/24 106.106.0.2 0 107 108 101 102 100 i
*> 105.105.0.1 0 105 104 103 102 100 i
*> 10.255.255.0/24 106.106.0.2 0 107 108 101 100 i
* 105.105.0.1 0 105 104 103 102 100 i
*> 101.101.0.0/24 106.106.0.2 0 107 108 101 ?
*> 101.101.101.101/32
106.106.0.2 0 107 108 101 i
* 102.102.102.102/32
Network Next Hop Metric LocPrf Weight Path
106.106.0.2 0 107 108 101 102 i
*> 105.105.0.1 0 105 104 103 102 i
*> 103.103.0.0/16 105.105.0.1 0 105 104 103 i
*> 104.104.0.0/16 105.105.0.1 0 105 104 i
*> 105.105.0.0/16 105.105.0.1 0 0 105 i
*> 106.106.0.0/16 0.0.0.0 0 32768 i
*> 107.107.0.0/16 106.106.0.2 0 0 107 i
*> 108.108.0.0/24 106.106.0.2 0 107 108 101 ?

[/code]

See above, ISP106 prefers the route via ISP107 (both prefixes for 10.1.21.0.0/24 coming via ISP1-ISP107 and ISP2-ISP105 are external, same

local pref, same as path length and also the older prefix is at the bottom, so, according to BGP path selection algorithm, the first one

installed and second one does not inactivate first one - older route is more preferred because it is more stable).

So as expected, ISP2 has the most specific prefix from R2: 10.1.21.0/24. ISP1 then exports it to it's update-groups: to ISP1 and ISP3, so

it looks like the goal is achieved. Let's see:

[code]
ISP1#traceroute 10.1.21.2 source 101.101.101.101

Type escape sequence to abort.
Tracing the route to 10.1.21.2

1 101.101.0.2 160 msec 100 msec 64 msec
2 2.2.2.2 260 msec 392 msec 280 msec
[/code]

ISP1 sends traffic to location B prefix via ISP2. In this case, organization internal link between siteA and siteB is avoided.

With a hold-timer of 30 seconds between R2 and ISP2, how long would the outage be if L3 link fails between these two routers (assuming a

repeater in the middle that keeps link up on ISP2) ?

Let's see:
[code]
R2(config)#int f0/0
R2(config-if)#sh
R2(config-if)#
[/code]

[code]
ISP1#ping 10.1.21.2 source 101.101.101.101 repeat 9999999

Type escape sequence to abort.
Sending 9999999, 100-byte ICMP Echos to 10.1.21.2, timeout is 2 seconds:
Packet sent with a source address of 101.101.101.101
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!...........!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!...............!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
Success rate is 94 percent (507/534), round-trip min/avg/max = 32/182/1044 ms
[/code]

That doesn't look too good from the perspective of ISP1 (AS101). When link between ISP2 and R2 goes down, ISP2 sends a BGP UPDATE message

to withdraw 10.1.21.0/24 learned from R2 to ISP1 and ISP3. It takes ISP1 more than 10 seconds to remove this route from it's routing table

and then another approx 20 seconds traffic still passes because ISP1 will send to R1 (remember both R1 and R2 send the aggregated route

10.1.20.0/23 to ISp1 and ISP2 respectively).

There is another outage some 10-20 seconds later. Why is that ?

This is ISP1 BGP RIB immediatelly after R2-ISP2 neighbourship is down:
[code]ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.21.0/24, version 182
108 107 106 105 104 103 102 100
Origin IGP, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2 | i entry|100
ISP1#sh ip bgp 10.1.21.2 | i entry|100
BGP routing table entry for 10.1.20.0/23, version 32
100
Origin IGP, metric 0, localpref 100, valid, external, best
[/code]


So for a good number of seconds, ISP1 prefers the aggregated /23 via R1, then switches to the more specific /24 via ISP108. Why is that ?

Remember ISP2 sends a BGP update message withdrawing 10.1.20.0/24 route to ISP1 and ISP103. On the internet, this withdrawl will not

propagate instantly and it will create residue specific routes to be activated because the previous best path was withdrawn. This is the

case for ISP107. Here is what happens on ISP106 and ISP107 in another scenario when ISP106 prefers the path via ISP2-ISP105 because this

path is older than the one via ISP1-ISP107:
[code]
ISP106#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 394
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to update-groups:
1
107 108 101 102 100
106.106.0.2 from 106.106.0.2 (107.107.107.107)
Origin IGP, localpref 100, valid, external
105 104 103 102 100
105.105.0.1 from 105.105.0.1 (105.105.105.105)
Origin IGP, localpref 100, valid, external, best
[/code]
ISP107 has two copies of this prefix: one from ISP108 (shorter) and one from ISP106 via ISP105-ISP102:
[code]
ISP107#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 419
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to update-groups:
1
106 105 104 103 102 100
106.106.0.1 from 106.106.0.1 (106.106.106.106)
Origin IGP, localpref 100, valid, external
108 101 102 100
107.107.0.2 from 107.107.0.2 (108.108.108.108)
Origin IGP, localpref 100, valid, external, best
[/code]


BGP Path selection algorithm on ISP107 decides the path via ISP108 is best one as it has a shorter AS path (it is also older - at the

bottom).

How does ISP107 router react when it receives the first BGP update message to withdraw 10.1.20.0/24 from ISP108 ? It will remove the

prefix from ISP108 and will chose prefix via ISP106-ISP2 and start advertising to all it's peers the path via "106 105 104 103 102 100".

Let's take a look at ISP7:

[code]
ISP107#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 419
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Advertised to update-groups:
1
106 105 104 103 102 100
106.106.0.1 from 106.106.0.1 (106.106.106.106)
Origin IGP, localpref 100, valid, external
108 101 102 100
107.107.0.2 from 107.107.0.2 (108.108.108.108)
Origin IGP, localpref 100, valid, external, best
ISP107#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 420
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Advertised to update-groups:
1
106 105 104 103 102 100
106.106.0.1 from 106.106.0.1 (106.106.106.106)
Origin IGP, localpref 100, valid, external, best
[/code]

This moment, I would say, is the start of blackhole. THis update will propagate down to ISP108 and ISP1 (ISP2 will not use it because 102

is already in AS PATH). Let's see ISP108:

[code]
ISP108#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 268
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1
101 102 100
108.108.0.2 from 108.108.0.2 (101.101.101.101)
Origin IGP, localpref 100, valid, external, best
ISP108#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 270
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Advertised to update-groups:
1
107 106 105 104 103 102 100
107.107.0.1 from 107.107.0.1 (107.107.107.107)
Origin IGP, localpref 100, valid, external, best
[/code]
[code]
ISP1#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.20.0/23, version 32
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
1
100
1.1.1.2 from 1.1.1.2 (10.255.255.1)
Origin IGP, metric 0, localpref 100, valid, external, best
ISP1#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 223
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Flag: 0x820
Advertised to update-groups:
1 2
108 107 106 105 104 103 102 100
108.108.0.1 from 108.108.0.1 (108.108.108.108)
Origin IGP, localpref 100, valid, external, best
[/code]


The first part of the problem is when AS 106 prefers the prefix via ISP2-ISP105 because it is older. ISP106 will advertise this prefix to

ISP107:
[code]
ISP106#sh ip bgp 10.1.21.2
BGP routing table entry for 10.1.21.0/24, version 438
Paths: (2 available, best #2, table Default-IP-Routing-Table)
Flag: 0x820
Advertised to update-groups:
1
107 108 101 102 100
106.106.0.2 from 106.106.0.2 (107.107.107.107)
Origin IGP, localpref 100, valid, external
105 104 103 102 100
105.105.0.1 from 105.105.0.1 (105.105.105.105)
Origin IGP, localpref 100, valid, external, best
ISP106#sh ip bgp nei 106.106.0.2 adv
BGP table version is 452, local router ID is 106.106.106.106
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network Next Hop Metric LocPrf Weight Path
*> 1.1.1.0/24 106.106.0.2 0 107 108 101 ?
*> 9.0.0.0 106.106.0.2 0 107 108 101 i
*> 10.1.20.0/24 106.106.0.2 0 107 108 101 100 i
*> 10.1.20.0/23 106.106.0.2 0 107 108 101 100 i
*> 10.1.21.0/24 105.105.0.1 0 105 104 103 102 100 i
*> 10.255.255.0/24 106.106.0.2 0 107 108 101 100 i
*> 101.101.0.0/24 106.106.0.2 0 107 108 101 ?
*> 101.101.101.101/32
106.106.0.2 0 107 108 101 i
*> 102.102.102.102/32
105.105.0.1 0 105 104 103 102 i
*> 103.103.0.0/16 105.105.0.1 0 105 104 103 i
*> 104.104.0.0/16 105.105.0.1 0 105 104 i
*> 105.105.0.0/16 105.105.0.1 0 0 105 i
*> 106.106.0.0/16 0.0.0.0 0 32768 i
*> 107.107.0.0/16 106.106.0.2 0 0 107 i
Network Next Hop Metric LocPrf Weight Path
*> 108.108.0.0/24 106.106.0.2 0 107 108 101 ?

Total number of prefixes 15
[/code]

The second part of the problem, however, is the fact that ISP107 will not chose it as best because it will prefer the one via ISP108

(shorter AS-PATH) and when it receives the withdraw BGP update message from ISP108, it will install the this prefix as best one and will

start notifying other neighbors.

Until the withdraw from ISP106 arrives on ISP107, it will attract traffic for the more specific prefix. Third part of the problem would be

the unequal speed of the withdraw messages for the specific prefix propagating throughout the internet.

There are ways to prevent this from happening. One would be advertising both specifics + the aggregate to both ISP1 and ISP2 and using the

"as path prepend" feature available in both Cisco and Juniper in combination with the "no-export" bgp community for site B specific prefix

on ISP1 with longer AS path so that ISP1 prefers path via ISP2 and possibly no-export so that ISP1 and ISP2 only send out to the internet

the aggregated /23 prefix.

Comments are welcome.