Document Links

Contents Introduction Routing Cost MOREnet Resources and Support Conclusion Comparison Endnotes

Adding Internet Bandwidth

Introduction

Adding a second Internet provider (called "multi-homing") for additional bandwidth or as a backup can cause significant technical issues for an organization. Unfortunately, increasing bandwidth to the Internet is not as simple as "plugging in" another circuit, whether it is DSL, cable, satellite or T-1, unless the circuit comes from the same provider as the existing circuit.

If utilization statistics show an organization is experiencing bandwidth congestion on the MOREnet-supplied circuit (circuit 1 in Figure 1), then traffic controls or additional Internet bandwidth is needed.¹ Traffic control can limit the amount of bandwidth used by certain applications, block applications that the organization does not want utilizing its network and prioritize traffic based on organizational policy.² Adding these kinds of control may relieve the congestion. Without traffic control, adding additional bandwidth is the only solution.

If the bandwidth congestion is not on the MOREnet-supplied circuit but on one or more of the organization's WAN circuits (circuit 2 in Figure 1), then the organization's network will need to be evaluated. Is there congestion at a specific point in the network (i.e., an over-utilized server, a hub where a switch should be, etc.) or is there a bandwidth limitation on a WAN circuit (i.e., a 56K circuit when a 384K or T-1 is needed)? Upgrading the organization's network can solve internal congestion problems, which may be the actual cause of the problem.

Figure 1

Routing

The key issue with adding bandwidth from a second provider is routing. The rules and protocols for routing Internet traffic when an organization has a single provider (regardless of the number/type of circuits the provider is supplying) are simple-route all non-local traffic to the provider and let the provider's router forward it. This configuration is called static routing and is how MOREnet's customer connections are routed.

When a second provider is added, the organization must take on the routing decisions, as it now has multiple paths for forwarding data.³ The organization must add at least one router to the network to "learn" about the networks available from each provider and make the appropriate routing decisions.⁴ Although each provider may be able to connect to the entire Internet, the router will determine preferred paths for forwarding traffic. The most common protocol used for multi-homed networks is BGP, or Border Gateway Protocol.⁵

MOREnet cannot route IP addresses from other providers, and other providers cannot route MOREnet-supplied customer IP addresses. Therefore, the organization must decide which provider to use for the default routes (default routes are everything except what is going to the other provider's directly-connected customers). For instance, if XYZnet was chosen as the default route, the organization would renumber all the devices in the network to use XYZnet IP addresses. All Internet traffic would be routed to XYZnet, and only traffic for MOREnet networks (i.e., other school districts, libraries, higher education campuses and the State of Missouri) would be routed to MOREnet.

In addition to the routing decisions that need to be made, an organization needs to understand that it only controls it's outbound traffic; that is, the traffic it is sending out to the Internet. Traffic coming back into the organization's network will take the best path that the other end knows about, and it may not be the same path the traffic was sent out on. This problem is known as asymmetric routing and can cause serious problems if traffic utilization is not carefully watched and BGP metrics, or routing information, is not carefully tuned to minimize (but never eliminate) asymmetric routing.⁶

Since routing is not easily controlled, issues such as Quality of Service (QoS) become difficult to implement. Because an organization cannot specify the return path, it depends on the other networks to appropriately pass the traffic and hope that it meets the QoS parameters demanded by the application.

In the example network (Figure 2), the customer network is multi-homed to MOREnet and to a second provider using BGP with an ASN and IP addresses assigned by ARIN, the American Registry for Internet Numbers (see footnote 5). When a PC requests information from a server (e.g., www.web.com), the customer's router will look at its BGP tables to determine the best route to that server. If the MOREnet link is chosen, then the data will be forwarded via links A, B and C.

Figure 2

When the server (www.web.com) returns data to the PC, its local router will look at its BGP tables and determine the best path to the PC, which may be via links D and E. The remote server has no knowledge of the path the original data took (A-B-C). Thus, there will be traffic leaving via connection A that returns via E.

In the second example network (Figure 3), the customer network is multi-homed to MOREnet and to a second provider, but without using BGP. For machines in the MOREnet-addresses space, all traffic will go in and out via the MOREnet connection (A) except for traffic destined for the other portion of the customer's network, which will be routed over link F. No traffic from the MOREnet portion of the network would ever travel over link E, and no traffic from the other IP space would travel over link A.

Figure 3

So, repeating our example of accessing the server (www.web.com), a PC in the MOREnet portion of the customer network would have its data send via A-B-C, and the server would return the data over the same links (C-B-A). A PC in the second provider portion of the customer network trying to access the same server would see data forwarded from E to D and back via E.

Local customer traffic can be routed internally, allowing machines in both clouds to share file servers and printers regardless of which portion of the network a user or a resource is located in. However, resources that are controlled via source IP address (such as MOREnet online databases) will not be accessible from the whole customer network-only from the machines that are addressed within the allowed IP address space.

While this configuration does solve some of the issues identified with dual-homing via BGP, the problem of adding bandwidth to the entire network has not been solved, as each piece of the network only has it's connection (A or E) as the total available bandwidth. If an application (or user) needs bandwidth in excess of what one link has available, the scenario shown will not meet the need, as traffic load cannot be shared across the two providers.

The best solution for adding additional bandwidth to the entire organization (provided that the organization WAN is not the bottleneck) is to purchase additional bandwidth from the current provider (Figure 4). Adding bandwidth allows for load balancing across the connections, providing higher bandwidth to users, as well as ensuring consistent QoS and network response time.

Figure 4

Cost

As complexity of the network increases, so does the level of training and knowledge required on the part of the organization's technical staff. Providing training for existing staff or hiring new staff with the necessary additional knowledge can be quite costly.⁷ Network costs also increase; the router(s) needed for a more complex network will be more expensive than the simple gateway routers in use today. Newer routers will of necessity have larger CPUs, more memory, more physical interfaces and additional licensed features.⁸ The purchase cost as well as the ongoing maintenance costs can add up quickly. In addition, organizations should also consider the administrative overhead of handling a second provider. Bidding, billing and vendor interaction, especially when troubleshooting problems, can eat up time (and labor dollars) quickly.

School districts, libraries and other eligible organizations must also consider E-rate issues. A second provider may not have a Service Provider Identifier Number (SPIN) and, thus be unable to offer E-rate-eligible services.⁹ This increases the cost to the district in real dollars in the form of additional costs in dealing directly with E-rate filing.

MOREnet Resources and Support

Access to MOREnet online resources will not be possible to that portion of the organization's network that is served by a second ISP, regardless of routing and addressing. Organizations that desire access to the online databases and resources must use a MOREnet-supplied connection and IP addresses.

MOREnet support is limited to the router and networks connected to the MOREnet network. Devices, including routers, switches and workstations, connected to non-MOREnet networks are not supported by MOREnet. If an organization has installed device(s) to route between MOREnet and a second connection, that device and any support issues are the sole responsibility of the organization. Any second connection into an organization must be made with non-MOREnet equipment. In other words, the connection cannot terminate in the MOREnet-owned and managed edge router.

Conclusion

Adding additional Internet bandwidth is not an easy task when more than one provider is involved. Although the circuit, router and final destinations may be the same, underlying technologies such as routing can cause a great deal of work and frustration on the part of an organization that may not have the skill set to manage the complex technologies. BGP routing, asymmetric paths and router updates can easily overwhelm an organization's technical staff, as the nature of multi-path networks is complex.

If the congestion is solely on the Internet access portion of the network, the most efficient method of adding bandwidth for an organization is to purchase additional bandwidth from the current provider. Staying with a single provider allows routing to remain simple and predictable (using static routing), and asymmetric routing, QoS and access to support and resources are no longer an issue. End users of the network have more true bandwidth available, experience consistent response time and are more insulated from outages.

If the congestion is within the organizational LAN or WAN, then additional bandwidth, bandwidth controls, a network redesign or a combination of all three may be needed to supply the connectivity required. The organization must take a careful look at the network needs, including the applications, protocols and bandwidth needed to support the users.

To discuss this issue further with MOREnet, contact techsupp@more.net.

Comparison

What does MOREnet supply for additional bandwidth to a customer organization as compared to another network provider offering additional bandwidth?

Issue	Second MOREnet Circuit	Second Network Provider
Router and CSU/DSU provided MOREnet supplies the appropriate router and CSU/DSU for the connection. Additional charges will depend on the customer type (DESE, REAL, CBHE, MERC) and bandwidth ordered-check with your MOREnet program manager for details.	Yes	Generally no, but it depends.a
Need ASN and BGP routing As discussed, routing becomes difficult with two network providers.	Not applicable	Yes: multi-homing typically requires BGP.
Need new IP addresses IP addresses are generally not portable between ISPs. Portable IPs are assigned by ARIN, the American Registry of Internet Numbers, and are difficult to acquire.	No	Yes, for devices served by the second provider.
Internal routing protocol With 2 providers, a method to distribute routes is needed so internal routers know where to send traffic.	No additional routing protocols required.	Yesb
Load balancing Not possible with two different providers, as the return path is controlled by the far end. Load balancing allows for the use of two or more connections as if they were one larger connection.	Yes	Not available
DNS Updates Updates will have to be made for the organizations web, email, and other servers and services if IP addresses are reassigned.	No	Yes, for devices to be served by the second provider.
Access to MOREnet online resourcesc	Yes	Not available
QoS available to MOREnet QoS is critical for high-quality H.323 video conferencing, and is configured in the MOREnet network for priority access to the video MCUs.	Yes	Nod
Technical Support If there is a problem with the connection, who gets the call? If the problem is not with the connection, but an organization needs additional assistance with troubleshooting internal network issues, will the provider offer assistance?	Yes	Variese

Table Notes:

1. Depends on the offering-a managed service including providing, configuring, and maintaining the router is more expensive than basic service where the customer provides the equipment and configures it.
2. If an internal routing protocol already exists, it will need to be modified and may need to be changed if it is incapable of performing the functions needed.
3. MOREnet-provided resources include EBSCOhost, the Gale Business and Company Resource Center and the DISCovering® Collection, NewsBank.
4. Some providers offer QoS within their network between their customers, but traffic leaving their network for other networks, including MOREnet, has no QoS mechanisms.
5. The level of technical support varies from provider to provider. Some offer a fully managed service, including router configuration and support, some offer no support beyond the connection. Few, if any, providers provide assistance in network design, troubleshooting, and security as part of the basic price-these are typically fee-for-service offerings.

Endnotes

1. See http://radar.more.net/cgi-bin/stats.rrd.f.cgi for MOREnet network utilization information.
2. See http://www.more.net/services/bandwidth/index.html for more information.
3. If the second provider is used solely for an isolated network, then routing issues do not apply. However, access to organizational resources and the rest of the organization's network will be via the Internet, which can cause performance and security problems.
4. If the second provider is used to connect a lab or classroom to the Internet and no connection to the rest of the district network is needed or desired, then the routing protocol issues do not exist; however, all the rest of the issues remain.
5. BGP is usually an extra-cost option on a router, and requires additional memory and CPU power to run. In addition, BGP requires an Autonomous System Number (ASN) and new IP address ranges from ARIN, the American Registry for Internet Numbers, which will be different than the IP addresses MOREnet has provided. The routing protocol uses ASN to determine how to route traffic to a specific network. If more than one path is possible (by having 2 or more different ISPs), the path chosen is based on the number of ASNs the traffic will cross and the bandwidth speed available. "Most end-users receive IP address space from an Internet Service Provider (ISP), not directly from ARIN. Assigned addresses obtained directly from ARIN are the least likely to be globally routable. Organizations that do not meet the requirements for a /20 must secure address space from an ISP." (From http://www.arin.net/library/templates/net-end-user.txt, emphasis added.) A /20 network is equivalent to over 32,000 IP addresses.
6. If multiple connections are from a single provider, regardless of technology, asymmetric routing is not an issue as the provider has one "path" to send data to the organization, and the organization has one "path" to the provider. Load balancing is done between the paths by the equipment at each end and eliminates the asymmetric routing issue.
7. For example, a five-day Cisco course on BGP is $3,495, and with travel expenses added in, can exceed $5,000 per person. Depending on the skill level of the individual, other classes may be required prior to taking the BGP routing class.
8. As of 3/15/2003, a Cisco 2621 router with two WAN connections, two LAN connections, and IP routing with BGP has a list price just over $10,000.
9. See http://www.universalservice.org.