Quick HOWTO : App02 : Data Center Relocation - Selecting an ISP

From Linux Home Networking
Jump to: navigation, search


In Chapter 2, "Preparation"I outlined the importance of ISP preparation. This chapter will discuss the many technical factors that govern the selection process in detail. These factors include:

  1. Pricing
  2. Determining the type of data circuit to use.
  3. Deciding on whether to use IP addresses issued by your ISP or addresses owned by your company.
  4. Configuring a routing protocol to use with your ISP.

Check lists are also included in Appendix I, "Relocation Check Sheets" to help you make a better decision and facilitate your monitoring of the status of all the required tasks. First let's discuss these factors in more detail.

Data Circuit Pricing

Pricing varies depending on the type of service you purchase. Internet circuits typically require you to commit to a minimum data rate and charge a variable fee for usage above that rate to a defined maximum. Non-Internet dedicated point to point services from data carriers usually charge a fixed fee that allows transfers up to the maximum data rate. There is no variable component. This will be discussed in more detail next.

Internet Services

A very common ISP billing technique used is called the 95th percentile method. Here the internet service provider provides an absolute maximum data rate, also known as a committed information rate (CIR), but you are billed based on actual usage. The ISP samples your data rate every five minutes and sorts all the sample readings for the month from high to low. They then discard the top 5% of the samples with the highest utilization. You are then billed at the rate of the highest sample that remains, not the average of those remaining. One of the advantages of this billing method is that it allows you to download files, a usually bandwidth intensive process, for up to about an hour a day without it affecting your bill.

In addition to bandwidth, you may also be charged a local loop rate which amounts to a monthly fee that covers your connection from your facility to the nearest telephone / Internet exchange. This is frequently proportional to the distance between the exchange and your facility. Sometimes this fee is also related to your CIR and you may find that you can reduce this monthly fixed cost my negotiating a lower CIR. You may also be able to reduce your 95th percentile rate by committing to a longer contract or by convincing your ISP that you will be generating sufficient traffic to justify a bulk discount.

Non-Internet (Carrier) Services

Data carriers typically will charge a flat fee for circuits with a pre-defined maximum data rate. You will also be charged a local loop rate. The complexities of a CIR and 95th percentile are usually absent.

Data Circuit Types

The selection of the type of data circuit to be used will depend upon the amount of bandwidth you expect to use, the equipment available to your ISP in the area and the capabilities of your networking equipment. The most commonly used data circuit technologies include those listed in Table A3.1.

Table A3.1 - Common Data Circuit Terminologies

Term Description
T1 1.544 Mbps link that can be split into up to 24 x 64 Kbps channels. Channels can be aggregated together to increase throughput up to the T1 maximum in a configuration called a "fractional T1". Channels may be used for voice or data. Typically runs over copper
E1 2.048 Mbps link that can be split into up to 32 x 64 Kbps channels. Two channels are used for signaling. Channels can be aggregated together to increase throughput up to the E1 maximum in a configuration called a "fractional E1". Channels may be used for voice or data. Typically runs over copper.
T3 Circuit configured to carry DS3 formatted data at up to 44.736 Mbps. A DS3 can be fractionalized with up to 30 T1 circuits. Two circuits are used for signaling. This means a T3 can up to 672 x 64 Kbps voice/data channels. Typically runs over copper.
DS3 See T3
HSSI High Speed Serial Interface capable of supporting up to 52 Mbps.
Packet Over SONET (POS) Methodology carrying TCP/IP traffic over SONET networks. The TCP/IP packets are inserted into ATM packets which are then placed on the SONET circuit. You will most likely need a POS interface on your router if you intend to transmit TCP/IP, Voice over IP (VoIP) or Internet packets on an OC type circuit. ATM interfaces may look the same, but are designed to strictly carry traditional voice traffic. Typically runs over fiber optics.
SONET Synchronous Optical Network. International standard for transmitting digitized voice over optical fiber circuits. There are a number of SONET circuit types, the most commonly used ones being optical carrier levels 3 and 12 mentioned in this table. (OC-3, OC-12). Data traveling on SONET networks use asynchronous transfer mode (ATM) formatted packets which were originally designed to carry voice traffic. Typically runs over fiber optics.
Fast Ethernet Primarily copper based version of Ethernet that operates at 100 Mbps.
Gigabit Ethernet (Fiber) Optical version of Ethernet that operates at 1 Gbps.
Gigabit Ethernet (Copper) Copper based version of Ethernet that operates at 1 Gbps.
Wireless Circuits Uses a variety of methods to transmit data through the air. In most cases the ISP manages the antenna equipment and hands off a physical cable link to their customer. This link may be of any of the circuit types mentioned in this table. Wireless links can be quickly installed but they tend to be subject to interference that can reduce their reliability.
DSL An adequate solution for low volume web sites. Usually offers a maximum bandwidth of 2 Mbps.
Cable Modem Unsuitable for high bandwidth websites as your circuit is shared by many other subscribers who could affect your performance. Your bandwidth usage could also affect video quality of other subscribers. Some ISPs purposely restrict traffic to web servers on their cable links for this reason

It is good to note that data services are sometimes asymmetrical in nature, especially with DSL and cable modem circuits. This means that the incoming downstream data rate from the Internet is different from the reverse outgoing upstream speed. You should be most concerned about the upstream speed for your Web site to the Internet. Inbound Web browser queries don't use a lot of data bandwidth, but the Web pages that contain the outbound replies do. Internet service providers provide asymmetric services for residential users and the downstream rate is almost always higher than the upstream. They reserve symmetrical data circuits for businesses which usually need high bandwidth to both surf the web and serve Web pages and more reliable guaranteed service. The ISP will usually provide the business with a fixed range of Internet addresses as part of the service; residential customers usually get a dynamic address allocation which is unsuitable for most businesses. Whenever possible always opt for symmetrical services for your business. Remember that there are many ways to plan the expansion of your data circuit bandwidth. You can:

  1. Add more circuits.
  2. Order a high speed circuit and throttle it with a lower valued CIR. Increasing the CIR increases your bandwidth.
  3. Use a factional or channelized service and expand your usage one channel at a time till the maximum capacity of the circuit is reached.

Select your data circuit with care. A wrong decision could inhibit the growth of your business.

Data Circuit Provisioning

You should always be aware of the environment in which data circuit providers work. In most cases neighborhoods are grouped into geographic zones which receive data communication services from a central office (CO). COs can also be called telephone or Internet exchanges.

Usually the CO is owned and operated by a single incumbent carrier (eg. AT&T) that owns the wiring infrastructure all the way to the neighborhoods' homes and businesses. Competing carrier can sometimes arrange with the incumbent to provide competing services over the wired infrastructure for a fee. The connection between a CO and your business or home is often called the local loop.

Ideally, a dedicated point to point data circuit between two neighborhoods should have a local loop in neighborhood "A", which then connects to the carrier's backbone network. The backbone should then provide services to the CO in neighborhood "B", which connects to the remote business via another local loop. For Internet services, there need only be a single local loop to your ISPs Internet infrastructure.

Not all ISPs are present in all COs. In order to provide services to all neighborhoods in a city, ISPs may have to negotiate interconnections between COs. Therefore it is possible to purchase services from an ISP who then has to negotiate multiple local loops for the circuit to finally reach its backbone infrastructure.

It is best to minimize the number of local loops in your circuit design. Coordinating the installation and troubleshooting activities of one ISP can be difficult. Extending this to multiple ISPs can be tricky.

You should also realize that not all data centers allow access to all carriers and in some cases there may be only a limited number of circuit types available. Make sure you understand how your desired types of circuits and carriers will gain access to the facility before making a final data center decision.

The relationship between carriers and ISPs in a CO leads to a variety of additional terminologies you'll need to know:

LOA-CFA (Letter of Authority and Customer Facility Assignment): This document does two things. Firstly, it allows a carrier to have access to another carrier's facility to do work (LOA). Secondly the carrier that issues the document also provides a facility assignment (CFA) which indicates the specific interconnection point within the CO for the other carrier to use. Work cannot proceed without a LOA-CFA for the local loops. The more local loops you have, the more LOA-CFAs are required. It is important to keep a very close eye on this process.

DLR (Design Layout Record): This document describes all the details of the circuit path from one end to the other. It can include physical information such as location, floor, row, rack, panel and port. It can also refer to virtual circuits, in other words, circuits that are securely shared with other customers, such as a channelized DS3. A DLR can also mention interconnections with other known circuits, which can help reduce the complexity of the document. You should always verify that a DLR has been created on time in order for it not to hold up the rest of your operation.

FOC (Firm Order Commitment): It may sound rude, but FOC is a common term used in the industry. It is the date your carrier will commit to having a fully functional circuit delivered to you. Always ask what additional tasks will be required after the FOC date. You will almost certainly have to coordinate your engineers and those of the carrier to harmonize and test their configurations before data flows correctly. It is very possible for carriers to test their local loops correctly but make a mistake on the CFA with an incorrect cross-connection.

MPOE (Main Point of Entry): Carriers and ISPs need to deliver data circuits to a specific room at a business address. It is typically the same room in which all telephone lines enter the building.

MDF (Main Distribution Frame): Is usually a rack in the MPOE in which carriers will install the equipment required to terminate the circuit's local loop coming from the CO. This rack and equipment is usually the property of the carrier / ISP. Your equipment will usually connect to the MDF gear through a patch panel provided by you carrier / ISP.

IDF (Intermediate distribution frame): In buildings with multiple tenants it is common to extend connectivity from the MPOE to each tenant's premises. Each tenant location, (eg. a server room or the location of their PBX) will have their own IDF for their own equipment. Connectivity between gear in the MDF and the IDF is usually achieved by using patch panels.

Data center Cross-connects: A carrier or ISP will deliver your circuit to the MPOE, but you'll need to have a cross-connect created to link your server room's IDF to the MPOE's MDF. Remarkably, data centers often charge for this accessibility on a per circuit basis. It can be an unexpected hidden cost.

With the knowledge of these terminologies you should be in a much stronger position when talking to your ISP and carriers.

IP Address Ownership

In a data center environment you will normally request a block of IP addresses (the data equivalent of a telephone number) from your ISP for use by your servers. The ISP will assign a range of addresses to you and will configure their equipment to route traffic to this range via the data circuit they provide. There is a disadvantage to this. If you cancel your ISP data circuit, you will lose the IP addresses they assigned to you. This could force you to reassign brand new IP addresses to your servers.

Always consider applying for your own IP addresses from your Regional Internet Registry (RIR). Here is a useful list of RIRs you can use for your area:

  1. AfriNIC (African Network Information Centre) - Africa region
  2. APNIC (Asia Pacific Network Information Centre) - Asia Pacific region
  3. ARIN (American Registry for Internet Numbers) - Americas and Southern Africa
  4. LACNIC (Regional Latin-American and Caribbean IP Address Registry) - Latin America and some Caribbean
  5. RIPE NCC (Réseaux IP Européens Network Coordination Centre) - Europe and surrounding areas will recognize your operation as being similar to that of an ISP and will assign you your own AS and IP addresses. The circumstances for doing so are slightly different for each affiliate but the main factors are that you can prove that your routing policy is different from that of your ISP and/or that your connectivity requires links to multiple ISPs.

If you cannot obtain your own IP block then you will have to ensure that all your applications use DNS names to refer to other servers in your environment and not their actual IP addresses. When new IP addresses are required, you can just modify the DNS name to map to the new address. This minimizes the impact of forced IP address changes on your operation.

Routing Protocols

Internet routing can be quite complicated and you will often need a network engineer to configure your equipment to get access. This section will provide an overview for project managers of the most common Internet routing challenges data center based web sites face. It will provide insights into what can be done if things go wrong during your data center relocation. ISPs usually use two methods to provide internet access to their clients. The first is by providing a simple default gateway through which all network traffic should pass. This is the usual option when only a single link is provided. The second method relies on the border gateway protocol (BGP) and is used primarily when Internet connectivity is provided via multiple ISP links.

Border Gateway Protocol

BGP is a dynamic protocol that can be adjusted relatively easily to influence traffic to and from your site in order to reduce bandwidth costs when your ISPs charge different rates, or to divert excess traffic from an overloaded circuit to a lesser utilized one. Unlike the configuration of a static route that can never change even if a link fails, BGP routes adjust themselves automatically depending on the availability of network links to reach target destinations. This section will cover BGP for use by project managers in some detail and Table A3.2 summarizes many of the terms that will be used later.

Table A3.2 - Common BGP Routing Terms

Term Description
BGP Autonomous System (AS) A BGP routing management domain usually owned by ISPs. Autonomous systems are assigned blocks of IP addresses which the ISP advertises to neighboring autonomous systems. The ISP is also responsible for the routing of traffic destined to their AS or passing through it to another AS.
Autonomous System Number (ASN) A unique number assigned to the ISP for their autonomous system.
AS Path A sequential list of ASs traffic must pass through in order to reach its destination network.
AS Path Prepending A method of repeatedly adding on your AS number to the beginning of the AS path to your network to bias traffic away from the link that is advertising the prepended list.
Local Preference A method of biasing the desirability of routes to the internet via a particular link in favor of an alternative one within your BGP AS.
Multi-Exit Discriminator (MED) A method in which an ISP, with multiple links to your AS, can bias your routers to select one of its paths to the Internet over another.

With BGP, each ISP is provided with a BGP autonomous system (AS) number from the Internet Assigned Numbers Authority (IANA). The ISP then associates the IP addresses it owns to this AS. Routes are exchanged with other ASs in the form of "I am AS number X and in my AS I have the following networks". This results in BGP routers having long lists of all the available networks on the Internet tied to a sequence of ASs that need to crossed in order to get to each one. This sequence is called an AS path list.

BGP routers update their neighbors of changes they detect in the Internet. If a BGP router loses connectivity with a peer responsible for advertising networks for a particular AS, it then notifies its remaining neighbors of the failure and instructs them to remove their routes to the failed AS from their routing tables. ISPs can advertise default routes to your routers via BGP. This can be useful when your equipment doesn't have sufficient memory to store the entire Internet routing table. Some manufacturers recommend a minimum of 512 KB of RAM to support full routes.

No matter what types of routes you receive you can influence how traffic leaves your site with a number of commonly used techniques. If the links terminate on the same router you can use a system of weighting to route traffic completely over one link versus the other. Weights aren't exchanged between routers, so when the links terminate on different routers within your control you'll need to use BGP's local preference feature help them negotiate the preferred link. When both your links are provided by the same ISP, you can also have them advertise a unique multi-exit discriminator (MED) metric value in the advertisements on each link which will bias BGP on your router to route its traffic on one link versus the other.

The methods previously discussed only refer to outbound traffic. Inbound traffic can be influenced too. One method uses AS path prepending in which you repeatedly add on your own AS number in your BGP advertisements. This lengthens your AS path list without making the traffic pass through any additional ASs. Prepending can be applied on a per link basis so that internet routers will feel that the AS path to your network on your preferred link is much shorter than the one to your network via the less favorable link.

These modifications don't have to apply to all Internet routes. You can bias traffic on a per-network and per-AS basis. This can be very useful. Say for example you have to email a weekly newsletter to thousands of customers but the additional traffic saturates one of your ISP links, you can use local preferences to make traffic to Hotmail, MSN and Yahoo! go through the original link, but traffic to AOL and Gmail pass through the other. Another common example would be a situation in which BGP automatically passes most of your outbound traffic over your most expensive link. You can use some of the techniques mentioned to make BGP favor the cheaper link for your traffic until your safe link bandwidth threshold is reached. You can usually guess the most popular ISPs from which web surfers would be coming, if not there are automated tools such as netflow on routers, and webalizer on web servers, that can provide more accurate insights. You can also figure out an AS number manually using the method in the following section.

Determining a BGP Autonomous System Number

Determining the AS number of an ISP or mail service manually is usually straight forward. In this case I'll attempt to determine the AS to which the IP address of mail.aol.com belongs.

  1. Use the nslookup or host command on a Windows or Linux server to determine the IP address of mail.aol.com
[root@bigboy tmp]# host mail.aol.com mail.aol.com has address
mail.aol.com has address mail.aol.com has address
[root@bigboy tmp]#
  1. Use a search engine to find a site that will provide access to "BGP looking glass" routers. Log on to one of the looking glass routers listed on the site.
  2. Enter one of the AOL IP addresses, in this case, and select "BGP", not "BGP summary". Click on the submit button and you will get output looking like this.
TELIA (1299) AOL (1668) 8176, (aggregated by 8176 172.21.44.GENUITY/BBN (1)) (metric 7) from (
    Origin IGP, metric 46, localpref 100, valid, internal, atomic-aggregate
    Community: 6939:2000

The output shows that the AS path list is 1299, 1688, 8176. This means to get to mail.aol.com the looking glass router had to pass through AS 1299, then 1688 and finally AS 8176. The final AS, 8176, is the AS for mail.aol.com. You will now have to do this again for all the other IP addresses returned by the hosts command.

Administrative Tasks Needed to Advertise BGP Routes

Network engineers will have to configure BGP on their routers, but the project manager will have to contact the ISP to make sure they are prepared to receive your routes. You will have to inform them ahead of time of your AS number, the networks you wish to advertise and the possibility of using AS path prepending. Once they receive this they will configure their equipment and then provide you with:

  1. A data circuit.
  2. IP address assignments for both your and their equipment connected to the circuit.
  3. Their BGP AS number.

Your network engineer will then be able to configure your equipment to provide correct BGP connectivity to the Internet.

Note: When you use multiple ISPs, make sure your network engineer's BGP configuration guarantees that data traveling from ISP #1 to ISP #2 doesn't pass through your AS. If this happens, you will find yourself paying for traffic that wasn't destined to your site. The volume of traffic passing through your AS could cripple your data circuits too.


Data center ISP selection is a very important part of any relocation activity. This Appendix has provided a summary of the issues that need to be addressed in the process and will make the overall task much simpler to complete. Activity checklists are provided in Appendix I to help facilitate this further.