Quick HOWTO : App02 : Data Center Relocation - Selecting an ISP
From Linux Home Networking
In Chapter 2, "Preparation"I outlined the importance of ISP preparation. This chapter will discuss the many technical factors that govern the selection process in detail. These factors include:
Check lists are also included in Appendix I, "Relocation Check Sheets" to help you make a better decision and facilitate your monitoring of the status of all the required tasks. First let's discuss these factors in more detail.
Data Circuit Pricing
Pricing varies depending on the type of service you purchase. Internet circuits typically require you to commit to a minimum data rate and charge a variable fee for usage above that rate to a defined maximum. Non-Internet dedicated point to point services from data carriers usually charge a fixed fee that allows transfers up to the maximum data rate. There is no variable component. This will be discussed in more detail next.
A very common ISP billing technique used is called the 95th percentile method. Here the internet service provider provides an absolute maximum data rate, also known as a committed information rate (CIR), but you are billed based on actual usage. The ISP samples your data rate every five minutes and sorts all the sample readings for the month from high to low. They then discard the top 5% of the samples with the highest utilization. You are then billed at the rate of the highest sample that remains, not the average of those remaining. One of the advantages of this billing method is that it allows you to download files, a usually bandwidth intensive process, for up to about an hour a day without it affecting your bill.
In addition to bandwidth, you may also be charged a local loop rate which amounts to a monthly fee that covers your connection from your facility to the nearest telephone / Internet exchange. This is frequently proportional to the distance between the exchange and your facility. Sometimes this fee is also related to your CIR and you may find that you can reduce this monthly fixed cost my negotiating a lower CIR. You may also be able to reduce your 95th percentile rate by committing to a longer contract or by convincing your ISP that you will be generating sufficient traffic to justify a bulk discount.
Non-Internet (Carrier) Services
Data carriers typically will charge a flat fee for circuits with a pre-defined maximum data rate. You will also be charged a local loop rate. The complexities of a CIR and 95th percentile are usually absent.
Data Circuit Types
The selection of the type of data circuit to be used will depend upon the amount of bandwidth you expect to use, the equipment available to your ISP in the area and the capabilities of your networking equipment. The most commonly used data circuit technologies include those listed in Table A3.1.
Table A3.1 - Common Data Circuit Terminologies
Select your data circuit with care. A wrong decision could inhibit the growth of your business.
Data Circuit Provisioning
You should always be aware of the environment in which data circuit providers work. In most cases neighborhoods are grouped into geographic zones which receive data communication services from a central office (CO). COs can also be called telephone or Internet exchanges.
Usually the CO is owned and operated by a single incumbent carrier (eg. AT&T) that owns the wiring infrastructure all the way to the neighborhoods' homes and businesses. Competing carrier can sometimes arrange with the incumbent to provide competing services over the wired infrastructure for a fee. The connection between a CO and your business or home is often called the local loop.
Ideally, a dedicated point to point data circuit between two neighborhoods should have a local loop in neighborhood "A", which then connects to the carrier's backbone network. The backbone should then provide services to the CO in neighborhood "B", which connects to the remote business via another local loop. For Internet services, there need only be a single local loop to your ISPs Internet infrastructure.
Not all ISPs are present in all COs. In order to provide services to all neighborhoods in a city, ISPs may have to negotiate interconnections between COs. Therefore it is possible to purchase services from an ISP who then has to negotiate multiple local loops for the circuit to finally reach its backbone infrastructure.
It is best to minimize the number of local loops in your circuit design. Coordinating the installation and troubleshooting activities of one ISP can be difficult. Extending this to multiple ISPs can be tricky.
You should also realize that not all data centers allow access to all carriers and in some cases there may be only a limited number of circuit types available. Make sure you understand how your desired types of circuits and carriers will gain access to the facility before making a final data center decision.
The relationship between carriers and ISPs in a CO leads to a variety of additional terminologies you'll need to know:
LOA-CFA (Letter of Authority and Customer Facility Assignment): This document does two things. Firstly, it allows a carrier to have access to another carrier's facility to do work (LOA). Secondly the carrier that issues the document also provides a facility assignment (CFA) which indicates the specific interconnection point within the CO for the other carrier to use. Work cannot proceed without a LOA-CFA for the local loops. The more local loops you have, the more LOA-CFAs are required. It is important to keep a very close eye on this process.
DLR (Design Layout Record): This document describes all the details of the circuit path from one end to the other. It can include physical information such as location, floor, row, rack, panel and port. It can also refer to virtual circuits, in other words, circuits that are securely shared with other customers, such as a channelized DS3. A DLR can also mention interconnections with other known circuits, which can help reduce the complexity of the document. You should always verify that a DLR has been created on time in order for it not to hold up the rest of your operation.
FOC (Firm Order Commitment): It may sound rude, but FOC is a common term used in the industry. It is the date your carrier will commit to having a fully functional circuit delivered to you. Always ask what additional tasks will be required after the FOC date. You will almost certainly have to coordinate your engineers and those of the carrier to harmonize and test their configurations before data flows correctly. It is very possible for carriers to test their local loops correctly but make a mistake on the CFA with an incorrect cross-connection.
MPOE (Main Point of Entry): Carriers and ISPs need to deliver data circuits to a specific room at a business address. It is typically the same room in which all telephone lines enter the building.
MDF (Main Distribution Frame): Is usually a rack in the MPOE in which carriers will install the equipment required to terminate the circuit's local loop coming from the CO. This rack and equipment is usually the property of the carrier / ISP. Your equipment will usually connect to the MDF gear through a patch panel provided by you carrier / ISP.
IDF (Intermediate distribution frame): In buildings with multiple tenants it is common to extend connectivity from the MPOE to each tenant's premises. Each tenant location, (eg. a server room or the location of their PBX) will have their own IDF for their own equipment. Connectivity between gear in the MDF and the IDF is usually achieved by using patch panels.
Data center Cross-connects: A carrier or ISP will deliver your circuit to the MPOE, but you'll need to have a cross-connect created to link your server room's IDF to the MPOE's MDF. Remarkably, data centers often charge for this accessibility on a per circuit basis. It can be an unexpected hidden cost.
With the knowledge of these terminologies you should be in a much stronger position when talking to your ISP and carriers.
IP Address Ownership
In a data center environment you will normally request a block of IP addresses (the data equivalent of a telephone number) from your ISP for use by your servers. The ISP will assign a range of addresses to you and will configure their equipment to route traffic to this range via the data circuit they provide. There is a disadvantage to this. If you cancel your ISP data circuit, you will lose the IP addresses they assigned to you. This could force you to reassign brand new IP addresses to your servers.
Always consider applying for your own IP addresses from your Regional Internet Registry (RIR). Here is a useful list of RIRs you can use for your area:
If you cannot obtain your own IP block then you will have to ensure that all your applications use DNS names to refer to other servers in your environment and not their actual IP addresses. When new IP addresses are required, you can just modify the DNS name to map to the new address. This minimizes the impact of forced IP address changes on your operation.
Internet routing can be quite complicated and you will often need a network engineer to configure your equipment to get access. This section will provide an overview for project managers of the most common Internet routing challenges data center based web sites face. It will provide insights into what can be done if things go wrong during your data center relocation. ISPs usually use two methods to provide internet access to their clients. The first is by providing a simple default gateway through which all network traffic should pass. This is the usual option when only a single link is provided. The second method relies on the border gateway protocol (BGP) and is used primarily when Internet connectivity is provided via multiple ISP links.
Border Gateway Protocol
BGP is a dynamic protocol that can be adjusted relatively easily to influence traffic to and from your site in order to reduce bandwidth costs when your ISPs charge different rates, or to divert excess traffic from an overloaded circuit to a lesser utilized one. Unlike the configuration of a static route that can never change even if a link fails, BGP routes adjust themselves automatically depending on the availability of network links to reach target destinations. This section will cover BGP for use by project managers in some detail and Table A3.2 summarizes many of the terms that will be used later.
Table A3.2 - Common BGP Routing Terms
BGP routers update their neighbors of changes they detect in the Internet. If a BGP router loses connectivity with a peer responsible for advertising networks for a particular AS, it then notifies its remaining neighbors of the failure and instructs them to remove their routes to the failed AS from their routing tables. ISPs can advertise default routes to your routers via BGP. This can be useful when your equipment doesn't have sufficient memory to store the entire Internet routing table. Some manufacturers recommend a minimum of 512 KB of RAM to support full routes.
No matter what types of routes you receive you can influence how traffic leaves your site with a number of commonly used techniques. If the links terminate on the same router you can use a system of weighting to route traffic completely over one link versus the other. Weights aren't exchanged between routers, so when the links terminate on different routers within your control you'll need to use BGP's local preference feature help them negotiate the preferred link. When both your links are provided by the same ISP, you can also have them advertise a unique multi-exit discriminator (MED) metric value in the advertisements on each link which will bias BGP on your router to route its traffic on one link versus the other.
The methods previously discussed only refer to outbound traffic. Inbound traffic can be influenced too. One method uses AS path prepending in which you repeatedly add on your own AS number in your BGP advertisements. This lengthens your AS path list without making the traffic pass through any additional ASs. Prepending can be applied on a per link basis so that internet routers will feel that the AS path to your network on your preferred link is much shorter than the one to your network via the less favorable link.
These modifications don't have to apply to all Internet routes. You can bias traffic on a per-network and per-AS basis. This can be very useful. Say for example you have to email a weekly newsletter to thousands of customers but the additional traffic saturates one of your ISP links, you can use local preferences to make traffic to Hotmail, MSN and Yahoo! go through the original link, but traffic to AOL and Gmail pass through the other. Another common example would be a situation in which BGP automatically passes most of your outbound traffic over your most expensive link. You can use some of the techniques mentioned to make BGP favor the cheaper link for your traffic until your safe link bandwidth threshold is reached. You can usually guess the most popular ISPs from which web surfers would be coming, if not there are automated tools such as netflow on routers, and webalizer on web servers, that can provide more accurate insights. You can also figure out an AS number manually using the method in the following section.
Determining a BGP Autonomous System Number
Determining the AS number of an ISP or mail service manually is usually straight forward. In this case I'll attempt to determine the AS to which the IP address of mail.aol.com belongs.
[root@bigboy tmp]# host mail.aol.com mail.aol.com has address 188.8.131.52 mail.aol.com has address 184.108.40.206 mail.aol.com has address 220.127.116.11 [root@bigboy tmp]#
TELIA (1299) AOL (1668) 8176, (aggregated by 8176 172.21.44.GENUITY/BBN (1)) 18.104.22.168 (metric 7) from 22.214.171.124 (126.96.36.199) Origin IGP, metric 46, localpref 100, valid, internal, atomic-aggregate Community: 6939:2000
The output shows that the AS path list is 1299, 1688, 8176. This means to get to
Administrative Tasks Needed to Advertise BGP Routes
Network engineers will have to configure BGP on their routers, but the project manager will have to contact the ISP to make sure they are prepared to receive your routes. You will have to inform them ahead of time of your AS number, the networks you wish to advertise and the possibility of using AS path prepending. Once they receive this they will configure their equipment and then provide you with:
Your network engineer will then be able to configure your equipment to provide correct BGP connectivity to the Internet.
Note: When you use multiple ISPs, make sure your network engineer's BGP configuration guarantees that data traveling from ISP #1 to ISP #2 doesn't pass through your AS. If this happens, you will find yourself paying for traffic that wasn't destined to your site. The volume of traffic passing through your AS could cripple your data circuits too.
Data center ISP selection is a very important part of any relocation activity. This Appendix has provided a summary of the issues that need to be addressed in the process and will make the overall task much simpler to complete. Activity checklists are provided in Appendix I to help facilitate this further.