ISP aggregation

HI Bruce,

I don't think anyone has yet mentioned that BGP multihoming is a very inexact science. Multiple upstreams are very good and stable for resiliency if one goes down. Load-sharing is another kettle of fish. I will assume that you (like us) have far greater amount of inbound traffic than outbound traffic. (Large research Unis may reverse this pattern.) One can kind of distribute the inbound traffic with AS-path prepends, however conditions change. Upstreams form different peering relationships, an Akamai cluster gets put in one place or taken out another and the traffic pattern changes. To quote from Jeff Doyle and Jennifer Carroll's Routing TCP/IP volume 2: "Do not be too concerned if 75 percent of your traffic uses one link while only 25 percent of your traffic uses the other link. Multihoming is for redundancy and increased routing efficiency, not load balancing." We tend to only try to rebalance our inbound traffic when a link is approaching 100% and the other link has a great deal of unused bandwidth. We take directly connected and default routes from our upstreams, and do a little of source routing to share the outbound, though it really isn't necessary for us, since the outbound is not large compared to inbound. For us there is no value in taking a full internet routing table. YMMV.

best,
Dennis Bohn
Manager of Network and Systems
Adelphi University
bohn@adelphi.edu
5168773327

Posted by: dogbohn on November 6, 2012

One more thing... I sleep much better at night having a separate router for each upstream with the internal addresses in HSRP configuration. They are actually in two separate buildings, with the fiber coming in on two separate conduit paths and crossing the (East) river with two separate tunnels and ultimately going to separate carrier hotels. I think it was the bridge collapse in Minneapolis when folks learned that though they had different upstreams, most fiber went over the same bridge. Oops.
Dennis Bohn
Manager of Network and Systems
Adelphi University
bohn@adelphi.edu
5168773327

Posted by: dogbohn on November 6, 2012

Message from mark.duling@biola.edu

I just looked at that quote in google books and it is referring to "load balancing." If multihoming simply means having multiple paths and/or ISPs, I think the "multihoming is for redundancy" statement is an over generalization.

Load sharing can quite effectively be done over multiple links in a determinate fashion by policy routing outbound (PBR in Cisco-speak), and using AS-PATH prepends to make it so that whatever pipe a packet is routed out, then the return transaction will also use the same pipe inbound unless that pipe is down. You select which internal networks will use which pipes (I'm assuming NAT is in use) and set your dynamic NAT statements accordingly. This method is highly determinate and very simple to setup, but still very flexible since it is easy to adjust the dynamic NAT statements to fine tune the traffic loads if adjustments are needed over time.

So if the costs or your budget circumstances don't permit two similarly sized pipes you can still get the benefits of a total bandwidth increase for a reasonable cost by adding a pipe using this method, and using very unequal sized pipes isn't a problem because of the PBR out and AS-PATH prepend in combination of route determinacy I've described. Now obviously, in this case you wouldn't get redundancy in any effective way if you have pipes of sizes 1X and 3X and are using 80% of the total bandwidth, since you can't lose the larger pipe without a devastating degradation of service. But after setting up the multihomed infrastructure, and assuming you want at some point to be able to survive an ISP link failure, you can upgrade the smaller pipe quite easily when the money is available or the costs come down.

All to say if getting more bandwidth for a decent price is more critical for you or more attainable for some reason at a given point in time than path redundancy there is nothing wrong with that. We are multihomed exactly this way and are quite happy with the results. Dennis, we also have ASA's and a Cisco border router so I could share the relevant configuration components with you if you want to contact me offline for details.

Posted by: listserv-anon on November 6, 2012

Message from peter.charbonneau@williams.edu

I'll "second" this architecture. We have used this for years and years.

Cisco's Policy Based Routing (pushing the outbound data) is predicated on link up and link down. Unfortunately, for us, we have intermediate switches between us and our ISPs. There have been many times that an ISP's link has gone down (mostly for maintenance), the BGP routes for that ISP get dropped, but the router is still trying to send/route data out that interface, because it sees the link as "up". Therefore, this becomes a maintainability issue ... someone here has to shut the interface down at the router, so outbound data will flow correctly.

The issue, after, is ... "How do you know when that interface is really back up and ready for bidirectional traffic?"

**sigh**

P

On Nov 6, 2012, at 2:24 PM, Mark Duling wrote:

I just looked at that quote in google books and it is referring to "load balancing." If multihoming simply means having multiple paths and/or ISPs, I think the "multihoming is for redundancy" statement is an over generalization.

Load sharing can quite effectively be done over multiple links in a determinate fashion by policy routing outbound (PBR in Cisco-speak), and using AS-PATH prepends to make it so that whatever pipe a packet is routed out, then the return transaction will also use the same pipe inbound unless that pipe is down. You select which internal networks will use which pipes (I'm assuming NAT is in use) and set your dynamic NAT statements accordingly. This method is highly determinate and very simple to setup, but still very flexible since it is easy to adjust the dynamic NAT statements to fine tune the traffic loads if adjustments are needed over time.

So if the costs or your budget circumstances don't permit two similarly sized pipes you can still get the benefits of a total bandwidth increase for a reasonable cost by adding a pipe using this method, and using very unequal sized pipes isn't a problem because of the PBR out and AS-PATH prepend in combination of route determinacy I've described. Now obviously, in this case you wouldn't get redundancy in any effective way if you have pipes of sizes 1X and 3X and are using 80% of the total bandwidth, since you can't lose the larger pipe without a devastating degradation of service. But after setting up the multihomed infrastructure, and assuming you want at some point to be able to survive an ISP link failure, you can upgrade the smaller pipe quite easily when the money is available or the costs come down.

All to say if getting more bandwidth for a decent price is more critical for you or more attainable for some reason at a given point in time than path redundancy there is nothing wrong with that. We are multihomed exactly this way and are quite happy with the results. Dennis, we also have ASA's and a Cisco border router so I could share the relevant configuration components with you if you want to contact me offline for details.

Posted by: listserv-anon on November 6, 2012

As long as you have a more up-to-date IOS version, IP SLA / object tracking should address that issue.

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Peter Charbonneau
Sent: Tuesday, November 06, 2012 2:34 PM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] ISP aggregation

I'll "second" this architecture. We have used this for years and years.

Cisco's Policy Based Routing (pushing the outbound data) is predicated on link up and link down. Unfortunately, for us, we have intermediate switches between us and our ISPs. There have been many times that an ISP's link has gone down (mostly for maintenance), the BGP routes for that ISP get dropped, but the router is still trying to send/route data out that interface, because it sees the link as "up". Therefore, this becomes a maintainability issue ... someone here has to shut the interface down at the router, so outbound data will flow correctly.

The issue, after, is ... "How do you know when that interface is really back up and ready for bidirectional traffic?"

**sigh**

P

On Nov 6, 2012, at 2:24 PM, Mark Duling wrote:

I just looked at that quote in google books and it is referring to "load balancing." If multihoming simply means having multiple paths and/or ISPs, I think the "multihoming is for redundancy" statement is an over generalization.

Load sharing can quite effectively be done over multiple links in a determinate fashion by policy routing outbound (PBR in Cisco-speak), and using AS-PATH prepends to make it so that whatever pipe a packet is routed out, then the return transaction will also use the same pipe inbound unless that pipe is down. You select which internal networks will use which pipes (I'm assuming NAT is in use) and set your dynamic NAT statements accordingly. This method is highly determinate and very simple to setup, but still very flexible since it is easy to adjust the dynamic NAT statements to fine tune the traffic loads if adjustments are needed over time.

So if the costs or your budget circumstances don't permit two similarly sized pipes you can still get the benefits of a total bandwidth increase for a reasonable cost by adding a pipe using this method, and using very unequal sized pipes isn't a problem because of the PBR out and AS-PATH prepend in combination of route determinacy I've described. Now obviously, in this case you wouldn't get redundancy in any effective way if you have pipes of sizes 1X and 3X and are using 80% of the total bandwidth, since you can't lose the larger pipe without a devastating degradation of service. But after setting up the multihomed infrastructure, and assuming you want at some point to be able to survive an ISP link failure, you can upgrade the smaller pipe quite easily when the money is available or the costs come down.

All to say if getting more bandwidth for a decent price is more critical for you or more attainable for some reason at a given point in time than path redundancy there is nothing wrong with that. We are multihomed exactly this way and are quite happy with the results. Dennis, we also have ASA's and a Cisco border router so I could share the relevant configuration components with you if you want to contact me offline for details.

Posted by: triegert@towson.edu on November 6, 2012

Tim is correct. Look up IP SLA, and the TRACK feature in Cisco IOS. That's what I use to manage this issue.

-
Pete Hoffswell - Network Manager
pete.hoffswell@davenport.edu
http://www.davenport.edu
616-732-1101

Posted by: hoffswell on November 6, 2012

Message from peter.charbonneau@williams.edu

Many thanks!

p

On Nov 6, 2012, at 2:51 PM, Pete Hoffswell wrote:

Tim is correct. Look up IP SLA, and the TRACK feature in Cisco IOS. That's what I use to manage this issue.

-
Pete Hoffswell - Network Manager
pete.hoffswell@davenport.edu
http://www.davenport.edu
616-732-1101

Posted by: listserv-anon on November 6, 2012

We have not found AS path prepending to be very deterministic. You can be very deterministic for outgoing using “local pref” but incoming often does not always respond very well to prepending, and at least in our case, that the vast majority of the traffic is incoming. We have also found that once you get it all set up, something does indeed change, and your well crafted schemes get messed up. We do not do NAT and are balancing chunks of a Class B and more, so maybe that is part of the issue for us.

Thank you for pointing out the fallacy of thinking that load balancing between two links can achieve both more available bandwidth and redundancy. The two are mutually exclusive goals unless you can implement per IP address rate limiting very quickly when you fail the oversubscribed traffic on to the one link.

If you simply need more bandwidth it is usually much cheaper to just increase the volume from your current provider as the cost per megabit per month is always going down, and you can get quantity discounts for it just like anything else.

What we do now is route all our traffic in and out of one provider, and fail it over to a backup provider who gives us a burstable service. The burst will support our current traffic levels, and if the failure lasts long enough we will have to pay extra which we would be glad to do at that point to keep our link up given the increasing importance.

Pete Morrissye

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Mark Duling
Sent: Tuesday, November 06, 2012 2:25 PM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] ISP aggregation

I just looked at that quote in google books and it is referring to "load balancing." If multihoming simply means having multiple paths and/or ISPs, I think the "multihoming is for redundancy" statement is an over generalization.

Load sharing can quite effectively be done over multiple links in a determinate fashion by policy routing outbound (PBR in Cisco-speak), and using AS-PATH prepends to make it so that whatever pipe a packet is routed out, then the return transaction will also use the same pipe inbound unless that pipe is down. You select which internal networks will use which pipes (I'm assuming NAT is in use) and set your dynamic NAT statements accordingly. This method is highly determinate and very simple to setup, but still very flexible since it is easy to adjust the dynamic NAT statements to fine tune the traffic loads if adjustments are needed over time.

So if the costs or your budget circumstances don't permit two similarly sized pipes you can still get the benefits of a total bandwidth increase for a reasonable cost by adding a pipe using this method, and using very unequal sized pipes isn't a problem because of the PBR out and AS-PATH prepend in combination of route determinacy I've described. Now obviously, in this case you wouldn't get redundancy in any effective way if you have pipes of sizes 1X and 3X and are using 80% of the total bandwidth, since you can't lose the larger pipe without a devastating degradation of service. But after setting up the multihomed infrastructure, and assuming you want at some point to be able to survive an ISP link failure, you can upgrade the smaller pipe quite easily when the money is available or the costs come down.

All to say if getting more bandwidth for a decent price is more critical for you or more attainable for some reason at a given point in time than path redundancy there is nothing wrong with that. We are multihomed exactly this way and are quite happy with the results. Dennis, we also have ASA's and a Cisco border router so I could share the relevant configuration components with you if you want to contact me offline for details.

Posted by: ppmorris on November 6, 2012

I haven't jumped in yet, but I feel compelled to add my experiences.

I am using BGP to provide both redundancy and load sharing.

I have two ISPs, both connected by Metro-Ethernet like services over fiberoptics. ISP #1 connects to us by two geographically diverse fiber paths. ISP #2 uses a single path which is geographically diverse from the two paths of ISP #1. This gives us protection from what I call "Dump-Truck attacks" where telephone poles are lost due to automobile accidents, and "Backhoe Attacks" where the underground fiber is damaged. Yes, we've experienced both...

Our Internet router peers with each of the ISPs by BGP. Our network uses a single public class B for networking. Inside our network we give subnets to each building and each WiFi vlan. (WiFi VLANs are assigned to clients by student class or by employment / guest status).

With each ISP I advertise our full class-b network. I also advertise half of the largest bandwidth consumers' subnets to each provider. For example with ISP #1 I advertise the dorms # 1, 3, 5, and the Freshman and Sophomore WiFI VLANs. Then with ISP #2 I advertise dorms #2, 4, 6, and the Junior and Senior WiFi VLANs. Based on occasional statistics I gather, this covers 80% or more of our bandwidth usage. Any other bandwidth is routed to us however the remote ISPs see fit.

The result is our inbound bandwidth is very well balanced, they share the same daily peaks and troughs, and I've had very few complaints.

Our outbound bandwidth is small compared to inbound. I send it our either one link or the other. If it became a problem I could institute policy-based routing or something similar.

My experience with AS-Path prepends is that the remote ISPs are going to make their own decisions on routing, and there's not much I can do about it. AS-Path prepends are just a suggestion to the ISP and aren't often taken seriously.

There are two philosophical schools of thought about tools. As my philosopher professor once told our class, a tool has an intended use, i.e. a screwdriver is "for" driving screws. Any other use of that tool is a misuse of it. However I subscribe to the other school that declares a good tool is one that can be used elegantly in ways never imagined by its designer.

Best,
Matt

-- Matt Richard '08 Access and Security Coordinator Information Technology Services Franklin & Marshall College matt.richard@fandm.edu

On 11/6/12 3:39 PM, Peter P Morrissey wrote:

We have not found AS path prepending to be very deterministic. You can be very deterministic for outgoing using “local pref” but incoming often does not always respond very well to prepending, and at least in our case, that the vast majority of the traffic is incoming. We have also found that once you get it all set up, something does indeed change, and your well crafted schemes get messed up. We do not do NAT and are balancing chunks of a Class B and more, so maybe that is part of the issue for us.

Thank you for pointing out the fallacy of thinking that load balancing between two links can achieve both more available bandwidth and redundancy. The two are mutually exclusive goals unless you can implement per IP address rate limiting very quickly when you fail the oversubscribed traffic on to the one link.

If you simply need more bandwidth it is usually much cheaper to just increase the volume from your current provider as the cost per megabit per month is always going down, and you can get quantity discounts for it just like anything else.

What we do now is route all our traffic in and out of one provider, and fail it over to a backup provider who gives us a burstable service. The burst will support our current traffic levels, and if the failure lasts long enough we will have to pay extra which we would be glad to do at that point to keep our link up given the increasing importance.

Pete Morrissye

From: The EDUCAUSE Network Management Constituent Group Listserv [mailto:NETMAN@LISTSERV.EDUCAUSE.EDU] On Behalf Of Mark Duling
Sent: Tuesday, November 06, 2012 2:25 PM
To: NETMAN@LISTSERV.EDUCAUSE.EDU
Subject: Re: [NETMAN] ISP aggregation

I just looked at that quote in google books and it is referring to "load balancing." If multihoming simply means having multiple paths and/or ISPs, I think the "multihoming is for redundancy" statement is an over generalization.

Load sharing can quite effectively be done over multiple links in a determinate fashion by policy routing outbound (PBR in Cisco-speak), and using AS-PATH prepends to make it so that whatever pipe a packet is routed out, then the return transaction will also use the same pipe inbound unless that pipe is down. You select which internal networks will use which pipes (I'm assuming NAT is in use) and set your dynamic NAT statements accordingly. This method is highly determinate and very simple to setup, but still very flexible since it is easy to adjust the dynamic NAT statements to fine tune the traffic loads if adjustments are needed over time.

So if the costs or your budget circumstances don't permit two similarly sized pipes you can still get the benefits of a total bandwidth increase for a reasonable cost by adding a pipe using this method, and using very unequal sized pipes isn't a problem because of the PBR out and AS-PATH prepend in combination of route determinacy I've described. Now obviously, in this case you wouldn't get redundancy in any effective way if you have pipes of sizes 1X and 3X and are using 80% of the total bandwidth, since you can't lose the larger pipe without a devastating degradation of service. But after setting up the multihomed infrastructure, and assuming you want at some point to be able to survive an ISP link failure, you can upgrade the smaller pipe quite easily when the money is available or the costs come down.

All to say if getting more bandwidth for a decent price is more critical for you or more attainable for some reason at a given point in time than path redundancy there is nothing wrong with that. We are multihomed exactly this way and are quite happy with the results. Dennis, we also have ASA's and a Cisco border router so I could share the relevant configuration components with you if you want to contact me offline for details.

Posted by: matt.richard on November 6, 2012

Message from mark.duling@biola.edu

I'm not a routing expert, but it seems to me all routing metrics are merely suggestions strictly speaking. It all depends on how they are used. But in the context of the scheme I described with single links to multiple ISPs, it seems to me that appending multiple AS-PATHs onto route advertisements combined with outbound policy routing is very highly deterministic. We have two pipes of wildly different bandwidths right now (though I hope we'll be able to equalize this soon) and it would be painfully obvious if AS-PATH prepending in this scenario were anything less than highly deterministic.

If I append my own ASN 4 times into a route advertisement to the immediate upstream router of one of my ISPs, I think that effectively tells it that the other link to me is actually four hops away. It isn't, but I want him to think that so he won't try a supposed 4-hop route unless the other is down. Ditto for the other ISP with the link preference reversed. So maybe I'm missing something, but I don't see what kind of event there could be in the Internet that would make my immediate upstream peer decide it prefers a 4-hop route to a 1-hop route unless it goes down, which is the desired behavior in our case. I've also never seen anything other than this behavior in our deployment.

Posted by: listserv-anon on November 6, 2012

One useful thing to do in general would be to read the BGP path selection algorithm document from your favorite router vendor. Here's the one from Juniper: https://www.juniper.net/techpubs/en_US/junos11.4/topics/reference/genera... As you can see, localpref trumps as-path length (this is also generally true of other vendors). So if I am a big content provider who happens to peer with both ISPs that you use, it doesn't matter how many times you prepend if I have localpref set to prefer sending my traffic to you via one of your ISPs. Let's say you have ISP 1 and ISP 2, and you want most of your traffic going over ISP 1. You therefore prepend in your announcements to ISP 2 and don't prepend to ISP 1. But I (the Big Content Provider) have a much bigger pipe to ISP 2, so I localpref your routes to ISP 2, and thence shall the traffic flow, despite your AS prepending. Even if ISP 1 and ISP 2 peer with each other, ISPs rarely localpref routes learned from peers higher--or the same--as routes learned from direct customers. So the traffic will get to you directly via ISP 2. So BGP really _is_ deterministic, it is just not deterministic from the standpoint of any one AS that is trying to manage inbound traffic. :) This also affects *which* ISPs to choose as your upstream providers, since they will have different customer and peering relationships with other ISPs and content providers, and these relationships can make traffic engineering easier (or harder). michael On 11/6/12 2:39 PM, Mark Duling wrote: > I'm not a routing expert, but it seems to me all routing metrics are > merely suggestions strictly speaking. It all depends on how they are > used. But in the context of the scheme I described with single links to > multiple ISPs, it seems to me that appending multiple AS-PATHs onto > route advertisements combined with outbound policy routing is very > highly deterministic. We have two pipes of wildly different bandwidths > right now (though I hope we'll be able to equalize this soon) and it > would be painfully obvious if AS-PATH prepending in this scenario were > anything less than highly deterministic. > > If I append my own ASN 4 times into a route advertisement to the > immediate upstream router of one of my ISPs, I think that effectively > tells it that the other link to me is actually four hops away. It > isn't, but I want him to think that so he won't try a supposed 4-hop > route unless the other is down. Ditto for the other ISP with the link > preference reversed. So maybe I'm missing something, but I don't see > what kind of event there could be in the Internet that would make my > immediate upstream peer decide it prefers a 4-hop route to a 1-hop route > unless it goes down, which is the desired behavior in our case. I've > also never seen anything other than this behavior in our deployment. > > > >

Posted by: msinatra on November 6, 2012

On 11/6/2012 5:56 PM, Michael Sinatra wrote: > As you can see, localpref trumps as-path length (this is also generally > true of other vendors). So if I am a big content provider who happens > to peer with both ISPs that you use, it doesn't matter how many times > you prepend if I have localpref set to prefer sending my traffic to you > via one of your ISPs. Yes, and localpref only works (for you) for outbound traffic. Inbound is quite different, you must "effect policy" on your upstreams. Some providers will accept communities on prefixes that they will in turn tag with *their* localpreferences. For others, you can send "more specific" prefixes for traffic than you announce to other providers. Some providers will accept communities that influence to which of their peers they will announce a particular prefix, so you can extend the "more-specific" override beyond the immediate upstream. "More specific" prefixes trump everything :) For all cases, you can often tag with the "no-export" community to keep those advertisements local to your upstream only (will affect their local customers only, if you don't want to fight the whole internet over traffic engineering your prefixes). Prepending is essentially network-wide (and for providers using localpref, can be totally ignored). If you have multiple links to a single provider, these can be reasonably used to load-balance the links. Beyond that, with multiple providers, "it depends". > This also affects *which* ISPs to choose as your upstream providers, > since they will have different customer and peering relationships with > other ISPs and content providers, and these relationships can make > traffic engineering easier (or harder). Yes. If your upstreams support these functions (localpref communities, and honor no-export, and peering communities) it helps greatly. Jeff ********** Participation and subscription information for this EDUCAUSE Constituent Group discussion list can be found at http://www.educause.edu/groups/.

Posted by: jeffkell on November 6, 2012

Message from mark.duling@biola.edu

Ok, so I now get that local preference trumps. I guess the question is how likely is this to actually occur. It hasn't been a problem for us, but it would be a big problem at the moment if it happened because of the unequal sized pipes we have at the moment. It sounds like from what I'm hearing that some large service such as Akamai wired in the different way to enhance regional performance could change our inbound routing behavior.

I see in this NANOG presentation it mentions a method (different than what we're discussing) that "works best in a network that can be divided into (usually regional) ISP transit islands that are usually limited to a metro area". http://www.nanog.org/meetings/nanog45/presentations/Monday/Roisman_bgp_metric_N45.pdf

Posted by: listserv-anon on November 6, 2012

We are in the process of building this out now. I don't know your situation but in our neighborhood one vendor was half the price of the rest when we were upgrading last. So we went with a model of having one ISP as a primary and the second as a hot standby. We have 300 Mb for our primary connection, 100 for Admin and 200 for the students. Our backup connection is just 100 Mb for the Admin side. We figure that we have to keep our vitals up, i.e. the website, DNS, remote access, email.

We are in the process of setting this up with BGP now. Keep in mind that crossing the 100 Mb barrier for routing can be a slap budget-wise. We shopped around and settled on a pair of used Cisco 3825s from Network Hardware Resale = VERY affordable. They come with 2 copper Gb ports and a SFP. The SFP is handy b/c ISPs are just handing off a layer 2 ethernet fiber connection these days.

As for outbound routing we will just be learning our default route via BGP from our pirmary ISP. We will have a static route to our backup ISP with a higher cost than the learned BGP route. If our primary goes away so will the learned default route and we will failover to our secondary for outbound traffic. Between the edge routers and the FWs we will be running HRSP.

As for inbound we will be prepending the advertised routes 4-5 times to the backup ISP per our agreement with them. We actually negociated a 0 Cir circuit. We can burst up to 100 Mb for 3 days after which we have to pay.

John Kaftan

Infrastructure Manager

Utica College

Posted by: jkaftan on November 6, 2012

You can trump localpref/weight with BGP conditional advertisement; however, ISP2 can be used only as backup in this case Regards, Adrian -----Original Message----- From: Michael Sinatra [mailto:michael@RANCID.BERKELEY.EDU] Sent: Tuesday, November 06, 2012 4:56 PM Subject: Re: ISP aggregation One useful thing to do in general would be to read the BGP path selection algorithm document from your favorite router vendor. Here's the one from Juniper: https://www.juniper.net/techpubs/en_US/junos11.4/topics/reference/general/ routing-ptotocols-address-representation.html As you can see, localpref trumps as-path length (this is also generally true of other vendors). So if I am a big content provider who happens to peer with both ISPs that you use, it doesn't matter how many times you prepend if I have localpref set to prefer sending my traffic to you via one of your ISPs. Let's say you have ISP 1 and ISP 2, and you want most of your traffic going over ISP 1. You therefore prepend in your announcements to ISP 2 and don't prepend to ISP 1. But I (the Big Content Provider) have a much bigger pipe to ISP 2, so I localpref your routes to ISP 2, and thence shall the traffic flow, despite your AS prepending. Even if ISP 1 and ISP 2 peer with each other, ISPs rarely localpref routes learned from peers higher--or the same--as routes learned from direct customers. So the traffic will get to you directly via ISP 2. So BGP really _is_ deterministic, it is just not deterministic from the standpoint of any one AS that is trying to manage inbound traffic. :) This also affects *which* ISPs to choose as your upstream providers, since they will have different customer and peering relationships with other ISPs and content providers, and these relationships can make traffic engineering easier (or harder). michael On 11/6/12 2:39 PM, Mark Duling wrote: > I'm not a routing expert, but it seems to me all routing metrics are > merely suggestions strictly speaking. It all depends on how they are > used. But in the context of the scheme I described with single links > to multiple ISPs, it seems to me that appending multiple AS-PATHs onto > route advertisements combined with outbound policy routing is very > highly deterministic. We have two pipes of wildly different > bandwidths right now (though I hope we'll be able to equalize this > soon) and it would be painfully obvious if AS-PATH prepending in this > scenario were anything less than highly deterministic. > > If I append my own ASN 4 times into a route advertisement to the > immediate upstream router of one of my ISPs, I think that effectively > tells it that the other link to me is actually four hops away. It > isn't, but I want him to think that so he won't try a supposed 4-hop > route unless the other is down. Ditto for the other ISP with the link > preference reversed. So maybe I'm missing something, but I don't see > what kind of event there could be in the Internet that would make my > immediate upstream peer decide it prefers a 4-hop route to a 1-hop > route unless it goes down, which is the desired behavior in our case. > I've also never seen anything other than this behavior in our deployment. > > > >

Posted by: bucurica@iit.edu on November 7, 2012

Hi All,

A really interesting talk. I noted that presentation is vectored for content providers balancing traffic out. From slide 3:

• Techniques are different for inbound versus

outbound BGP load balancing at your network’s

border

• Today’s talk focuses on outbound

• Typically for content networks, outbound traffic is

the billable value, the benefit is that there is

more control on outbound load balancing

• Technique used mostly for ISP uplinks, peering

uplinks are usually considered differently

• Inbound load balancing is less precise, and

often relies on behavior of upstream and further

remote networksTechniques are different for inbound versus

outbound BGP load balancing at your network’s

border

• Today’s talk focuses on outbound

I think where the inbound load balancing gets a little difficult, is when there are two (roughly) equal-sized pipes running for a portion of the day at somewhat > 50%, maybe up to 80%. This as compared with a large circuit and a small one. With two largish circuits, one of them can go down without too much problem. During the storm last week, we lost two of our upstreams (we are fortunate enough to have three), one for a day and one for several days without any trouble for the campus (partially because traffic was low.) OTOH, in 2001, we had a large pipe and a small pipe. In the WTC event, we lost the large pipe for several months, and it was really a problem. I would not want to risk that kind of situation ten years later, when everyone is so much more dependent on the internet, when university business really grinds to a halt without a robust internet connection.

My $.02.

Dennis Bohn
Manager of Network and Systems
Adelphi University
bohn@adelphi.edu
5168773327

Posted by: dogbohn on November 7, 2012

Message from mark.duling@biola.edu

Dennis,

Yes, I noted the scenario for the presentation was different from the ones we're discussing. I was just wondering if the part I quoted from it could be relevant to the "it depends" part of the possible inbound unpredictability others have referred to. We've got a routing trump card in play here but no way yet for me to tell how likely it is I'll encounter it. It's hard for me to tell if people are talking about possibilities more theoretical than actual or not.

Say, what type of event was it that caused one of your pipes to be down for several months?

Mark

Posted by: listserv-anon on November 7, 2012

Message from mark.duling@biola.edu

Oh I didn't get that WTC must mean World Trade Center.

Posted by: listserv-anon on November 7, 2012

Hi Mark,

I think it depends on who that smaller upstream is, and what else is going on at the POP or carrier hotel which hosts it. Four prepends is quite a good bit, clearly enough to keep most traffic away from that link. Say that the small circuit is a customer of Level3. And Level3 puts in an Akamai server cluster in that pop. Traffic would likely flow through that smaller pipe, even if it originated from the larger pipe. I have noticed that without any changes on our part, our microsoft updates shift from link to link, based on where the Akamai presence is. A few months ago, there was a google server very close by one of our upstreams: <5 msec away. Then something happened and it was 20 msec away. Now there is some movement and it is 10 msec away. Changing relationships between isps? New servers? Who knows?

I wouldn't mind seeing your configs, see if there is something else I should be doing. Oh, and yes it was the World Trade Center.... there was a large connection point right across west street from the WTC that was taken out for months.
Dennis Bohn
Manager of Network and Systems
Adelphi University
bohn@adelphi.edu
5168773327

Posted by: dogbohn on November 7, 2012

Main Nav

Publications

Research

Major Conferences

Online Events

Special Topic Events

Events for all Levels and Interests

Career Center

Leadership and Management Programs

Fellowships and Awards

Getting Involved

Jump Start Your Career Growth

Latest Topics

EDUCAUSE organizes its efforts around three IT Focus Areas

Join These Programs If Your Focus Is

From the Blogs

Find Others

Share Ideas

Get on the Higher Ed IT Map

2013 Strategic Priorities

Membership

Who We Are

Stay Informed

Uncommon Thinking for the Common Good™

Comments

Recommend

Research and Publications

Conferences and Events

Career Development

Focus Areas and Initiatives

Connect and Contribute

About EDUCAUSE

Publications

Research

Major Conferences

Online Events

Special Topic Events

Events for all Levels and Interests

Career Center

Leadership and Management Programs

Fellowships and Awards

Getting Involved

Jump Start Your Career Growth

Latest Topics

EDUCAUSE organizes its efforts around three IT Focus Areas

Join These Programs If Your Focus Is

From the Blogs

Find Others

Share Ideas

Get on the Higher Ed IT Map

2013 Strategic Priorities

Membership

Who We Are

Stay Informed

Uncommon Thinking for the Common Good™