|
Conferences and Events | Online Resources | Programs | Security | Services | Shared Network | Technical Support | Training Home » Technical Support » Research and Innovation » MOREnet Network Architecture |
|||||||||||
MOREnet Network ArchitectureBackgroundSince its inception in 1991, MOREnet has evolved through four different core network designs, each put in place to meet the demands of a rapidly growing and changing Internet. The most recent version of the network, MOREnet3 (M3), was designed in 1998 and went into production service in August of 1999. The network has undergone some incremental, and one major, upgrade since its installation. Over the last five years, however, technologies and customer demands have changed. When M3 was put into production, the industry-wide video connection technology was H.320, which MOREnet implemented using Circuit Emulation Services (CES) over an ATM core. Today, all video is H.323, an IP-based protocol that does not require CES services. Quality of Service (QoS) was just beginning to be understood five years ago, but in 2002, new routers were added to the core to implement some QoS functionality for our customers. In late 2002 and early 2003, MOREnet conducted a needs assessment to determine what capabilities the MOREnet network should have. This needs assessment was conducted by interviewing MOREnet staff and management for the following information:
The results are documented in the "MOREnet4: Network Requirements" document. Next-Generation Network NeedsAs documented in "MOREnet4: Network Needs," the next generation MOREnet network should support the following principles (new items): General
Core Network
Internet Access
Customer Connections
Network Monitoring
These principles will be used to ensure that the next-generation design meets all of the goals set in the "MOREnet4: Needs Analysis" document. One of the key issues in any next-generation design is the suitability of existing equipment in the network. Can existing equipment continue to be used, increasing its lifetime and lowering the cost of the network upgrade? Can the existing equipment be upgraded or modified to work in the new architecture? Issues such as port density, port speeds, features and capabilities must be balanced against the cost of replacement equipment. Similarly, does the network need to be completely replaced (forklifted), or can it be incrementally upgraded? A forklift upgrade offers the opportunity to build a new infrastructure parallel to the existing network, test and move connections when it's ready. However, the price for a "clean" network is usually substantial, as new equipment and circuits are operated in parallel to the existing production network. The incremental upgrade allows for portions of the network to be replaced or upgraded as funding, time and people resources allow and to be done in a just-in-time fashion (i.e., upgrade when the need exists vs. upgrade and wait for the need to exist). In determining the feasibility of using existing equipment or forklifting an upgrade, the current capabilities of the equipment, its projected lifespan and upgradeability must be evaluated. If the capacity or features are not available, then a replacement piece must be selected for the network implementation. Building the Next Generation NetworkATMThe current network (M3) is an ATM-based network, which was the most effective way to implement support for circuit-switched (H.320) video. The core network is based on Cisco StrataCom BPX 8600 ATM switches, which, coupled with Cisco 7200 routers equipped with PA-A2 ATM blades and a Cisco StrataCom MGX 8220, provided the ability to transport T-1s (or fractional T-1s) across the network for video via Circuit Emulation Services (CES). The IP (packet) data was also transmitted via a separate ATM PVC and was switched to Cisco 12000 IP routers (See Figure A).
ATM is also used in the M3 network to attach to carrier networks for circuit aggregation. Fractional T-1 (fT-1) and T-1 Frame Relay circuits are brought into the MOREnet core after being converted to ATM in the carrier network using service internetworking based on FRF.8. OC-3c ATM connections to carrier networks provide a cost-effective method to aggregate 80 or more circuits to a single physical interface in the MOREnet core. For circuits that are IP-only, these ATM aggregation circuits are terminated directly into the IP router. Circuits that carry both IP and CES traffic terminate in the ATM switch to allow video traffic to be switched to the MGX for delivery to the MCU, while switching IP traffic to the IP router (see Figure B).
Changes in videoconferencing technology, however, have shifted video from a circuit-switched environment using H.320 to a packet-based system using H.323. This shift has caused MOREnet to discontinue the support of H.320 services in the network for end-station codecs as of June 30, 2002, and for MCU to MCU cascades as of June 30, 2003. Without a requirement for H.320 support, ATM Circuit Emulation Services are no longer needed in the network. When the University of Missouri desired to obtain access to the Abilene network for Internet2 (I2) activities, a new edge router was placed in the core to manage the Abilene traffic, and separate PVCs were created to carry the I2 traffic from the four University campuses. These ATM PVCs define an I2 VPN, controlling traffic and QoS for the I2 member institutions. Additionally, the ability to create independent and isolated paths allows the MOREnet Network Research Lab (NRL) to have direct connectivity to advanced networks in an isolated fashion, insulating the production network from potentially disruptive research efforts. The NRL connects to the Great Plains Network in Kansas City, which is the Midwest I2 GigaPOP and Abilene network provider. This model is based on the MORPHnet model, first described in 1997 by researchers at the Argonne National Laboratories.1 The University plans to continue membership in Internet2 and will continue to need managed access to the Abilene network, and the NRL will continue to need isolated traffic paths for research, so this ATM VPN structure cannot be abolished without a suitable replacement. As of fiscal year 2003, some thought has been made to consolidate several networks onto a single state-wide infrastructure. The potential need for multiple VPNs in the future requires that whatever architecture is developed, the need for VPN support is an immediate, as well as a future, requirement. Without the need to support CES services, however, the implementation of a VPN capability does not have to be ATM-based; IP-based VPN solutions are available and may be capable of providing the services needed. ATM will not likely be removed from the network as no longer necessary anytime in the near future, as aggregation circuits are still delivered via ATM, and the need to support a VPN for I2 connectivity continues. However, the need for an ATM switching infrastructure can be evaluated and eliminated if there is no longer a need to switch PVCs at a pure ATM level. The need for ATM services will need to be evaluated on both a customer aggregation circuit level and a backbone level, as the needs and demands differ between circuit types. OpticalOne of the key drivers for a next-generation network is the ability to take advantage of new opportunities and technologies. A rapidly growing opportunity for more bandwidth and lower costs is the availability of dark or dim fiber services, allowing MOREnet to connect any type of equipment to the fiber and carry traffic over it. This technology allows MOREnet, with a the expenditure of one-time capital, to increase the capacity of the network, rather than the expenditure of the capital as well as an increase in the annual circuit costs for a larger-capacity circuit. The drop in prices of Dense Wave Division Multiplexing (DWDM) equipment offers MOREnet the opportunity to build a core network from dim/dark fiber, lighting wavelengths as necessary to carry traffic. Several options for the increase in bandwidth are available via DWDM:
Other advantages of optical carrier technology include:
With the flexibility that optical carrier services are offering now and into the foreseeable future, an optical core is the most flexible and growth-capable platform to build the next generation MOREnet network. As MOREnet's services evolve, a DWDM-based transport facility offers growth and flexibility in an economical manner, both in initial (capital) costs as well as recurring operational costs. In evaluating the deployment of dark/dim fiber, there are two factors that must be recognized and addressed. First, that the availability of dark/dim fiber between MOREnet locations may be limited or unavailable. Second, that the current decline in costs for OC-12 vs. the existing OC-3 backbone makes dark/dim fiber a much tougher sell up front for capacity not needed today (or tomorrow), unless a major change in the network was substantiated (new service offering, new technology, new customer(s)). IP Services"MOREnet4: Needs Analysis" outlines a clear rationale for enhancing IP support in the network. Quality of Service, VPNs and IPv6 are all clear needs for the next generation of the MOREnet network and must be clearly addressed in the architecture and design. Additional Quality of Service mechanisms are needed to control the growing volume of traffic as circuit capacities cannot be increased due to funding restrictions. Mission-critical services must be handled above entertainment traffic, and each end organization should have some capability to determine what it deems "mission-critical" vs. "best-effort". The rising use of H.323 packet-based videoconferencing has highlighted the problems of QoS mechanisms in the network, as other traffic, from web to FTP to peer-to-peer networks, has wreaked havoc on video calls because QoS is not end-to-end, but just in the MOREnet core-as most customer networks are not QoS enabled. MOREnet has placed some QoS mechanisms in the M3 network by adding edge service routers (Cisco 10000) and enabling QoS on customer edge routers. However, the ability of these mechanisms is limited and will not suffice in the future. End-to-end QoS for an application should be the goal. Although organizations may set differing policies for various applications, once a policy is set, it should be honored across the MOREnet network. Differences between organizational policies will have to be arbitrated by a policy broker (likely MOREnet). In addition, general policies should be put in place and agreed upon for certain types of traffic, such as H.323 video, VoIP and VPN-managed traffic (such as PeopleSoft, MOBIUS, I2, etc.). Moving VPN services from the ATM layer to the IP layer offers one large advantage: the ability to deliver VPN services to any customer, regardless of connection type. Currently, VPN services are delivered via ATM, so only ATM-connected customers can take advantage of VPN services. This is not a scalable solution, and with the increased need for VPN services (such as PeopleSoft, MOBIUS and HIPPA-related services), moving the VPN control from the ATM layer to the IP layer makes it a more deployable and scalable solution. By providing a VPN solution to customers, MOREnet can build private, secure, and controlled "virtual" networks with managed QoS that is independent of MOREnet's normal traffic, even though it share the same physical infrastructure. Many organizations looking to implement a VPN solution do not need ATM; for these organizations, there is no benefit other than VPN capabilities, and the additional costs may be prohibitive. By switching to an IP-based VPN solution, any circuit type (fT-1, T-1, Frame Relay, ATM, xDSL, PPP) can carry VPN traffic, extending the service out to all customers equally. For new customers that MOREnet may add to the network, the ability to carry VPN services at the IP layer rather than the ATM layer will offer the ability to reuse existing IP equipment rather than investing in new, more expensive ATM equipment. In addition, the VPN capabilities will provide MOREnet the means to build VPN structures for customers as necessary, rather than being limited only to ATM-based customers. Support for New ServicesOne of the limiting constraints of the MOREnet3 network is the use of ATM as a circuit technology between core devices, as well as the collapsed backbone structure of the original M3 network (the current M3 network has a distribution layer added between the core and the customer edge). When evaluating new services and products, Ethernet (generally Gigabit Ethernet) is currently the supported technology by most manufacturers for interconnecting equipment. A key criterion in the architecture design, therefore, is to separate the distribution network from the core network in such a way that Gigabit Ethernet can be utilized as the interconnectivity method. This separation allows for the inclusion of application-aware (sometimes called "layer 4" or "layer 7" switches) to manipulate traffic to service devices. These devices could be DNS servers, web hosts, content filters, file caches, storage, videoconferencing servers or any other core service available to customers. By utilizing Gigabit Ethernet, including these services becomes much easier. In addition, new services can be more easily introduced when they can be plugged into a Gigabit Ethernet switch, turned on and then introduced in a controlled fashion. Currently, introducing a feature such as filtering or caching would involve a disruptive upgrade to the network, as well as changes to the network's routing structure (and an upgrade of the routers if they were not capable of handing the additional routing load). Customer ConnectivityMOREnet has been using 56/64K, fT-1 and T-1 circuits to connect the majority of customers for many years. While this technology is mature and generally available, it tends to be expensive. Due to the patchwork nature of telephone companies within Missouri, there is no dominant company, and many small companies serve isolated areas. MOREnet has (operationally) chosen to work with a limited number of companies, primarily Southwestern Bell, CenturyTel (formerly GTE/Verizon), AT&T and Sprint for customer access. Although the customer location may fall within an independent telephone company operating area, MOREnet orders the circuit to be connected to one of the primary aggregators network for delivery back to a MOREnet POP on an aggregation circuit. With MOREnet2, aggregation circuits went from T-1 to DS-3 and OC-3c, dropping the number of aggregation circuits needed while allowing for an increase in available bandwidth. While this has lowered costs, it has also limited the number of companies MOREnet is connected directly to, and its effect on total pricing is unknown. However, cost issues such as floor space in the POPs, physical ports on the router/switch, and aggregation circuit costs tend to favor continued limitation on providers for the near term. In looking at customer connections, it becomes apparent that many customers are not making use of the bandwidth supplied. The original premise was that customers' use of the Internet would grow. While some customers' use has increased, many others have been stable or shown slow growth in their bandwidth demands. With the availability of new connection methods from carriers, such as xDSL, cable modems and EtherLoop, opportunities to provide the necessary bandwidth at lower costs have become available. In a region (generally a telephone company operating territory), customers with access to xDSL services should be evaluated to determine if switching to DSL will provide the bandwidth necessary for the organization. If enough customers in an area are xDSL candidates, then a single connection to the telephone company for backhaul to a MOREnet POP is needed to support all of the locally connected customers. This process can be done for cable modems and EtherLoop where available. Another alternative is radio, both line-of-sight (LOS) and non-line-of-sight (NLOS). Depending on terrain, a single multipoint base tower can serve many sites (up to 200, depending on technology), with a single backhaul circuit to deliver the aggregated bandwidth back to a MOREnet POP. This solution can also be used to provide customer site-site connectivity as a fee-based service, where appropriate. Continued aggregation at 1:1 is likely not sustainable in the foreseeable future due to budget constraints. A managed aggregation, where MOREnet engineers monitor the total bandwidth consumed and ensure that it remains below a trigger value (such as 80%) that would cause a new circuit to be added to carry traffic, would be a simple, cost-effective alternative. Customer connections will still be driven by bandwidth; however, the network will revert to a packet-only transport, using an edge router for each customer site. Each edge router must terminate the WAN circuit and provide the following functionality at a minimum:
MOREnet should continue the practice of operating a managed edge device to demarcate the MOREnet network from the customer network. A MOREnet-owned and managed router will be at the customer premises at the end of each tail circuit, providing layer 3 routing and services to the customer network (see Figure C).
Network ManagementThe MOREnet network should have the following network management tools:
In addition, tools to monitor flows and applications, such as H.323 video or a layer 3 VPN, should be implemented to ensure end-to-end traffic flow control in the network. Ongoing analysis of the network traffic statistics collected will be used by the traffic engineering group to determine when a circuit needs to be upgraded, when customer connections need to be rearranged for more effective use of bandwidth and to ensure that traffic is being handled in the core of the network with the appropriate QoS parameters and flow controls. A separate out-of-band network management system should be put into place for two reasons: first, it will provide an independent, secure path to the MOREnet core nodes for management; second, it will provide access to the control functions of the POP/RAC equipment during backbone outages. A fT-1 Frame Relay network between the various locations, all terminating in Columbia, would provide the network access needed (see Figure D). The out-of-band access provides several benefits:
Network ServicesNetwork services will see an expanded role in next-generation networks. M3 saw the introduction of Akamai clusters in Columbia and Jefferson City; distributed DNS at all hub sites; distributed polling and network monitoring. Services that will likely be offered in next-generation networks include:
Other network services that may be added to the network will likely center around enabling technologies, providing the basic services necessary for applications to function across the network. For example, ENUM is a proposed system to tie DNS, telephone numbers, email, SIP URLs, and other multimedia addressing together into a single database. MOREnet may provide ENUM servers in the network for customer use, similar to how DNS services are deployed today. Reliability and RedundancyThe continuing goal of the MOREnet network is to provide 99.999% availability at the core of the network. As with M3, the overall architecture of the network contributes to this goal, as each POP/RAC has at least two connections to diverse locations, minimizing the possibility of an outage due to circuit failure. In addition, equipment chosen for deployment should have, where technically possible and financially feasible:
As additional services and equipment are deployed, the need for reliability and redundancy (R&R) of the network as a whole is as important as the R&R of any specific piece of equipment. Key services in the network must have failover capabilities, whether to another system in the same hub or to a backup system in another hub. Failure to provide a 99.99% or greater level of reliability in the network will fail to meet customer expectations of "always on and working." MOREnet should also continue look to look at ways to increase the reliability of customer connections, as these are the weakest point for an individual customer. Alternate technologies, such as wireless, may offer a quick response to an outage by providing reduced services within hours. Vendors should be pressed to improve the reliability of their networks, and to find ways to reduce the Mean Time To Repair (MTTR) for a circuit outage. Regional Aggregation CentersDue to the fractured nature of telephone company territories in Missouri, one option for lowering costs is to build regional aggregation centers (RAC). These aggregation centers would be layer 2 only, bringing in multiple customer tail circuits and aggregating them onto a high-speed circuit for delivery to a POP. By locating a RAC in various territories, the amount of aggregate bandwidth used to deliver services to POPs can be reduced, lowering operational costs. In addition, the RAC allows other connectivity methods to be employed in the local area, such as cable modems, xDSL, EtherLoop and wireless, presenting further opportunities for cost reductions (see Figure E).
Metro FiberIn the four major metro areas that MOREnet operates a POP in (Kansas City, St. Louis, Columbia, Springfield), local fiber rings offer an opportunity to provide services for both interconnecting customers as well as building a service ring to deliver connectivity to carriers (see Figure F). By utilizing DWDM on a fiber ring, multiple customers or multiple circuits from carriers at various locations could be delivered to the MOREnet POP at a fixed cost. For example, in the KC metro area, a ring from UMKC to two or three carrier hotels or carrier POPs would provide MOREnet the ability to connect to services without paying for local circuits. A recent quote for a circuit from UMKC to UMSL highlights why this is an issue: the long-haul portion was one third of the cost; the other two thirds of the cost were the local loops from the carrier POP to the MOREnet POP.
Overall ArchitectureIn the preceding sections, the issues of customer access, regional aggregation and the backbone have all been discussed separately. In this section, the overall architecture and options for building the next-generation network will be covered. CoreMOREnet has been operating hubs in five cities for MOREnet2 and MOREnet3. However, in analyzing the traffic loads placed on the backbone, there is no compelling reason to maintain Jefferson City as a fully-interconnected node (Figure G) on the backbone, as the traffic originating and destined for Jefferson City locations is less than 1G during a peak hour (approx. 2.8Mbps x 1 hour = 1G). Operating it as a distributed remote node with Columbia will save significant circuit costs, as only one circuit will have to be connected to Jefferson City, rather than the three that exist today, one of which is less than 5% utilized.
As a phase one approach, changing the backbone to be as shown will reduce costs while continuing to provide ample capacity in the Jefferson City hub area. This change can be made with existing equipment. See Figure H.
If dark/dim fiber (or one or more lambdas (?)) can be obtained, an xWDM2 solution can be built as a transport platform for MOREnet services. If fiber is available as shown, both a core (St. Louis, Springfield, Kansas City, Columbia) and regional rings (Kansas City, St. Joseph, Kirksville; St. Louis, Hannibal, Kirksville; St. Louis, Cape Girardeau, Rolla, Jefferson City, Columbia; Springfield, Joplin, Rolla) can be built to provide redundancy for the core as well as redundant regional aggregation centers (see Figure I). In this scenario, actual bandwidth delivered to a POP/RAC is no longer a cost per bit, but a capital cost for the endpoint hardware, as the fiber (or lambda) is a fixed cost.
Another advantage to using an optical transport as the core of the network is the ability to add connections for specific customers on diverse wavelengths. For example, if the University of Missouri wished to deploy a Gigabit Ethernet connection between the four campuses, a wavelength could be dedicated to this connection independent of the MOREnet backbone or other customers. Another potential use of independent wavelengths is research networks, either isolated from production networks or connectivity to Abilene/Great Plains Network. If optical transports are available, then existing VPNs can be re-implemented as separate wavelengths. If not, deployment of MPLS (or similar layer 3 technology) to handle VPNs should be done early in the deployment of the NGN. This would allow the backbone circuits to terminate on the core routers, and the ATM switches would then be aggregation feeders to the core routers as necessary. If a customer needed ATM, Frame Relay, or other layer 2 VPN service, the ATM switches could be interconnected on a separate wavelength to provide services. By directly connecting the core routers, rerouting can happen at the optical layer (if DWDM with BLSR or UPSR) and network convergence at layer 3 can be minimized. With MOREnet3, transit Internet access is supplied out of Kansas City and St. Louis only. As the bandwidth demand continues to grow, it is unlikely that cost-effective facilities will be built to Columbia or Springfield, as current services are not easily obtained in these locations, and even after a facility build, would be costly to maintain. One option that would provide a great deal of consolidation in the network is to use the layer 1 optical transport to deliver connections to Kansas City, Columbia, or St. Louis for layer 3 routing services. Jefferson City and Springfield would become large aggregation centers at layers 1 and 2 only, with all connections terminating in routing engines at UMKC, UMC or UMSL (Figure J). Several advantages are gained from this:
In addition, the grooming on/off ring of customer or carrier traffic at a hub site provides a great deal of control for delivery of services via optical switching. This control provides an easy mechanism for adding new services, moving bulk traffic and providing redundancy via spare wavelengths in alternate paths. Routing traffic that needs to be routed to content devices (cache engines, content filters, etc) would best be accomplished by performing the redirections at the customer site. This requires that the customer routers have sufficient CPU, memory and feature sets to enable policy-based routing, WCCP or other redirection means. By offloading the redirections to the edge, the core routers maintain a "big-fast-dumb" approach to routing, as well as keeping policy implementation issues isolated to a single customer site.
ConclusionOverall, the need for ATM as a core transport no longer exists. ATM is still a viable solution for VPNs and aggregation circuit transport; however, the nature of the traffic (other than VPN support) on the network is strictly layer 3 packet data. As opportunities present to move away from ATM as a bearer service, they should be taken if possible. In the backbone, POS or Long Haul Ethernet are viable choices (via xWDM as necessary) to connect the MOREnet POPs. The general core architecture should be a layer 3 service, providing:
Continued evaluation of aggregation methods will provide MOREnet with opportunities to increase bandwidth and lower costs, as carriers build out networks into the non-metropolitan areas of the state. A restructuring of the current network to an I-70 core with aggregation centers around the state will ultimately provide the best mix of aggregation, cost and control, as floor space, equipment costs, administration and access to upstream Internet services will continue to be in the Kansas City and St. Louis metropolitan areas. The migration of the backbone circuits from OC-3 to OC-12 in the 2004 calendar year will provide bandwidth relief for some time. However, the continuing growth of traffic and the persistent need for flexibility coupled with shrinking budgets and rising costs will support a change in the next 12-18 months from an ATM core supporting VPNs to an IP core, using MPLS for VPNs. As more multi-media applications, collaborative tools and interactive services become pervasive, the resulting demand for bandwidth from customer sites will rise. This bandwidth growth can be met in several ways:
However, funding issues will prevent MOREnet from just increasing the bandwidth. A more conservative approach would be for MOREnet to encourage customers to begin developing an appropriate traffic policy and assist them in implementing it on their networks, along with a revision of the 1:1 aggregation ratio on customer aggregation circuits. A managed aggregation policy can provide an effective non-blocking solution, although there will be a small amount of additional overhead in monitoring and issuing orders to move circuits from congested circuits to uncongested circuits. MOREnet will also continue to struggle with the issues of cost and availability for tail circuits to customers. With Missouri being a rural state, the MOREnet philosophy of "equal access everywhere" is getting harder to accomplish. Network capacity is being built in the metropolitan areas but not in rural areas, hampering the delivery of high-speed circuits to customers who need the bandwidth. New opportunities with non-traditional providers, supporting a broader range of technologies, may alleviate these issues in the future. Endnotes
|
|||||||||||
|
Copyright © 2002 MOREnet. All rights reserved. Reviewed December 5, 2003.
Contact strategic-tech@more.net. DMCA and other copyright information. Site Information: Copyright, accessibility, privacy and other information about this site. PageMinder: Receive an e-mail notice when this page updates.
|