This was the first real world performance of our 9 years ago designed unique system of numerous checks and balances and advanced, nearly never used by other networks, configuration directives, designed to automatically make one network site that is cut off from the rest communicate to the rest of our network over the public Internet – that is, accept our own AS routes in our own AS, but, only so when they are not reachable via the hopone.net backbone itself.

While this advanced configuration – that no other network has been able to architect and implement safely and transparently – was in place for nearly a decade, we had never had any network outage or issues that would have invoked it. Until, that is, this night’s scheduled replacement (upgrade) of the core1.iad1 router. And, much to our delight, it worked flawlessly just as designed. And, unlike any other backbone that would have had massive outages and customer impact in such a case, there was just a few second blip as various IP networks recalculated their routes to us, and all continued running normally.

This is a truly amazing achievement (and I am never one to use such words lightly) worth noting, showing how the hopone.net backbone is truly far ahead of the rest and is more fault tolerant by its numerous redundancy and failover layers than any other backbone.

Have you stopped by our booth at Search Engine Strategies Conference New York yet? We are exhibiting in Hilton New York today 3/23 and tomorrow 3/24 at booth 223. Stop by and learn about our Search Engine Optimization-Friendly web hosting and how our web hosting service can help enhance your SEO efforts.

At Superb Internet, we understand that choosing the right web hosting provider can have a major impact on search engine rankings. From Dedicated IP addresses to 100% Uptime guarantee SLA, we have just the right quality you are looking for in a SEO-friendly web hosting provider. For more information about our products and services, please visit www.superb.net or speak with one of our sales representatives at 1-888-354-6128.

Internet2 is a non-profit consortium made up of 212 universities and 60 partner companies from Industries like networking (including heavyweights like Cisco Systems and Nortel), publishing, and technology (including Intel, Comcast, and Sun Microsystems), which develops and deploys advanced network applications and technologies. Although there was a concern the network was being used by students for P2P transfers, a couple hundred lawsuits seems to have fixed the problem – at least in the public’s eye.

While the average website may not require these types of transfer rates, with increasing demands on streaming media and shared data, these types of advances are important in maintaining and evolving today’s Internet experience. For more information, see this article on ars technica.

If one does see some issues in a traceroute – that start at one hop and carry on forward (e.g. packet loss, or latency past the expected given the distance traveled between the two hops³), then it is imperative to run it ­bi-directionally. What I mean by this is, run, if possible at the same time or as close together time-wise as possible, a route from the user host to the server, and from the server to the user host.

It is important to do this, as often routes are asymmetric (take one path in one direction, yet another path in the return direction), and one needs to see the route both ways to be able to more accurately pinpoint where a problem may lie.

For example, trace from a customer to their server (this example now also brings together all the matters discussed in this article):

Tracing route to cyberfret.com [209.103.172.27] over a maximum of 30 hops:

1 7 ms 7 ms 9 ms 10.33.96.1
2 7 ms 9 ms 17 ms gig2-1.ghnaoh1-ybr1.columbus.rr.com [24.95.81.225]
3 15 ms 9 ms 9 ms srp9-0.clmboh1-rtr2.columbus.rr.com [65.25.129.94]
4 22 ms 20 ms 20 ms pos14-0.clmboh1-rtr3.columbus.rr.com [65.25.128.138]
5 22 ms 20 ms 19 ms son0-1-2.mtgmoh1-rtr0.columbus.rr.com [65.25.128.229]
6 25 ms 24 ms 22 ms te-3-2.car1.Cincinnati1.Level3.net [4.78.216.13]
7 23 ms 31 ms 34 ms ae-5-5.ebr2.Chicago1.Level3.net [4.69.132.206]
8 22 ms 35 ms 35 ms ae-78.ebr3.Chicago1.Level3.net [4.69.134.62]
9 31 ms 33 ms 35 ms ae-2.ebr2.Washington1.Level3.net [4.69.132.70]
10 30 ms 31 ms 54 ms ae-82-82.csw3.Washington1.Level3.net [4.69.134.154]
11 * * * Request timed out.
12 38 ms 38 ms 40 ms ge6-2.core1.dca2.hopone.net [66.36.224.190]
13 * * * Request timed out.
14 34 ms 34 ms 32 ms client.covesoft.net [209.103.172.27]

And a trace from their server back to their user host:

traceroute to 24.95.81.225 (24.95.81.225), 64 hops max, 40 byte packets
1 c-vl101-d1.acc.dca2.hopone.net (66.36.230.2) 0.381 ms 0.322 ms 0.241 ms
2 gec2.core2.dca2.hopone.net (66.36.224.233) 0.213 ms 0.199 ms 0.167 ms
3 gi1-0-0.ar2.wdc2.gblx.net (66.36.224.169) 1.339 ms 1.406 ms 1.368 ms
4 te1-1-10g.ar3.DCA3.gblx.net (67.17.108.134) 2.887 ms 2.585 ms 2.523 ms
5 pop2-ash-S2-1-0.atdn.net (66.185.136.9) 2.488 ms 2.608 ms 2.444 ms
6 bb2-ash-P3-0.atdn.net (66.185.151.150) 1.878 ms 1.941 ms 1.894 ms
7 bb2-cin-P3-0.atdn.net (66.185.153.61) 22.294 ms 22.182 ms 22.302 ms
8 pop1-cin-P1-0.atdn.net (66.185.133.3) 22.655 ms 22.407 ms 22.752 ms
9 RR-Cincinnati.atdn.net (66.185.133.6) 23.632 ms 23.698 ms 23.783 ms
10 pos13-1.clmboh1-rtr2.columbus.rr.com (24.95.81.194) 27.110 ms 26.940 ms 26.829 ms
11 srp1-1.ghnaoh1-ybr1.columbus.rr.com (65.25.129.68) 27.991 ms * 28.186 ms

As clearly evident, the route here is asymmetrical (as is often the case, given the various different routing preferences among various providers (and multiple, often countless, routes from point A to point B), each selecting the “best” routes differently from others – many often do that based on cost vs. performance, though the hopone.net network we use always does everything 100% based on optimal performance: shortest path, lowest latency, and always no packet loss). The route in (from customer to their server) goes from Road Runner in Columbus, OH to Level 3 in Cincinnati, OH and from there to HopOne in McLean, VA (the Level 3 [Old Meadow Road] McLean, VA site is misleadingly called Washington.Level3.net). The route out (from server to customer host), however, goes from HopOne in McLean, VA to GlobalCrossing in Washington, DC, then to AOL in Ashburn, VA and then to Road Runner in Cincinnati, OH, from where it travels to the end host in Columbus, OH.

The latency on the out route is lower, as it is tracing to the first real (not 10.x.x.x or 192.x.x.x non-routable IPs, that are not routable, private IP space used by various networks) hop in the customer’s in route, since the customer did not provide us with their actual host IP. As is evident in the in route, the customer has 7ms latency from their cable modem to their first Internet hop. As the out route traces to their gateway, it is, as expected, 7ms lower, as it does not trace all the way to the customer’s end host that is 7ms further away.

The in route shows two hops as “* * * Request timed out.” that, as earlier explained in this article, are no reason for concern, as long as the end host is reached with no problems. One of the hops, hop 11, is a Level 3 router that has been configured not to respond to traceroutes or pings (most likely for security and/or performance reasons), while the other hop, hop 13, is a HopOne distribution switch that is in non-routable IP space – which typically will show up in normal traceroutes (as hop 12 does), but in this case does not. However, as the end host is reached and shows no packet loss (3/3 pings to it have arrived perfectly fine, and all with consistent latency), it is evident that no problems are present. To ensure, however, the customer in this case would be suggested to run a recursive ping (ping –t) a 100+ times from their end host to the server, to verify that there are indeed no issues (there should not be, as the routes both ways are as close to perfect as possible, with stable latencies and no real packet loss anywhere). If there are any issues, such would be most likely present on the Road Runner end in Columbus, OH, as in the outgoing trace there are some packets lost there at the customer’s end gateway hop (however 1/3 packet loss does not have enough samples to go by to be objective, and it most likely is not reflective of 33% packet loss on that end host – as it is a router, it likely treats ping with the lowest priority and has a maximum of ping requests responses per second configured, so while it may be an indication of an issue there, it may be also simply the router doing what it has been configured to do and reject frequent pings).

/mtr –report-cycles=30 –report extby.neolocation.net
HOST: salviori.us.ded.neolocation Loss% Snt Last Avg Best Wrst StDev
1. h-vl202-d3.acc.dca2.hopone.n 0.0% 30 0.4 0.6 0.4 2.6 0.6
2. gec3.core2.dca2.hopone.net 0.0% 30 0.4 0.5 0.4 2.2 0.4
3. ge6-0.core1.dca2.hopone.net 0.0% 30 0.4 0.5 0.3 3.5 0.6
4. cw.peer.nyc2.hopone.net 0.0% 30 5.9 11.2 5.6 117.3 21.3
5. so-3-0-0-dcr1.nyk.cw.net 0.0% 30 5.8 13.0 5.7 121.9 27.2
6. so-7-0-0-dcr1.was.cw.net 0.0% 30 6.2 11.7 6.1 98.9 19.0
7. so-0-0-0-dcr1.par.cw.net 0.0% 30 86.4 91.2 86.3 136.4 11.7
8. so-7-0-0-dcr1.fra.cw.net 0.0% 30 103.4 97.0 94.9 124.1 5.5
9. so-4-0-0-zar1.fri.cw.net 0.0% 30 95.1 96.1 94.9 120.0 4.5
10. romteleco-gw.fri.cw.net 50.0% 30 455.1 462.5 450.4 532.3 23.3
11. spb-dsr2-ge0-2-0-0.rt-comm.r 40.0% 30 488.0 492.3 484.1 568.6 19.1
12. 195.161.4.34 26.7% 30 486.7 487.0 484.2 491.4 1.7
13. J10-1-B57.ge-1-0-0-7.peterst 56.7% 30 488.1 487.4 485.3 488.8 1.2
14. BELPAK.peterstar.net 60.0% 30 498.7 498.3 494.5 501.6 1.9
15. 193.232.248.110 56.7% 30 497.8 519.5 494.1 778.8 77.9
16. 193.232.249.185 53.3% 30 499.9 499.2 494.5 503.3 2.7
17. 86.57.250.98 63.3% 30 498.0 500.6 495.3 521.4 7.4

In this route, it is evident in the highlighted section, that there is an issue with “romteleco” (rt-comm.ru) gateway to their IP transit provider, Cable & Wireless (cw.net) – it would seem that this their gateway to C&W is seriously overloaded/congested, as the high packet loss carries all the way through past it on every hop. At such high packet loss (27-60%), the connectivity is likely to be severely limited.

In this particular case, an incoming route (from the customer to their server) would likely show the opposite: packet loss always after the “romteleco” gateway with C&W and onwards from there, as that is where the issue is caused. Seeing such an incoming route to compare it with the outgoing route would then confirm where the problem lies, if in both direction of routes it starts happening after one particular hop (in this case, the “romteleco” link to C&W).

Understanding Traceroute: A Four-part Series

Part 1: Traceroute can be used to measure performance

Part 2: Traceroute/tracert-like applications or commands will provide accurate results

Part 3: * * * s (apparent packet loss) somewhere in the route is always bad and a sign of problems

Part 4: A uni-directional traceroute (run only in one direction) can be used to accurately spot a problem area in a route

Also Check out our VPS Hosting Page

Certain hosts in the path will simply show a misleading 100% packet loss using applications such as Ping Plotter, which run a trace and then do multiple pings of each hop directly afterwards, despite the fact that often there is no packet loss; instead, the destination host may not be publicly routable. For example, the HopOne network that Superb uses exclusively is entirely secure, and its whole network device IP address space is not routable and not accessible from anywhere on the Internet (for security, performance, and DoS and intrusion mitigation measure); when pinging any of its routers using a traceroute imitation application, one will get apparent 100% packet loss. However, it is simply not reachable – e.g. result of a ping of a non-routable IP:

Reply from 64.251.86.49: Destination host unreachable.

This does not indicate a problem – as the hops in-between the origin and destination host do not need to be accessible to either host – they only need to be accessible to each other, as they, of course, are, if the traceroute completes at the end host (if they were not, then the trace would not complete and would not reach the end host).

Using the real tracert/traceroute commands will overcome most (but not all) of these issues, and will usually show even those unreachable to plain ping hosts and the latency to them – something the traceroute imitation applications using plain ICMP ping are not able to do.

In short, the most important measure (unless the end host rejects ICMP ping and also real traceroutes due to its firewalling, as many major sites do – e.g. EBay²) is to see if the end host was reached without excessive latency or packet loss. If it is reached without problems appearing at the end host itself, then any apparent issues in the route in between are irrelevant, and may be due to low priority or rejection of ICMP ping requests in the route. Any issue seen in a traceroute that does not carry through to the end host (e.g. packet loss or increased latency) is not a real issue. Any issue that does carry through, however (that is, starts at one hop and happens at all the subsequent hops), may be an issue with the route.

If one prefers to use a traceroute imitation application, then it is suggested to still use the real tracert/traceroute command if the application shows some packet loss or latency issues somewhere in the route in between but not at the end host to verify if there really is an issue or if it is due to the limitations of the application being used.

Understanding Traceroute: A Four-part Series

Part 1: Traceroute can be used to measure performance

Part 2: Traceroute/tracert-like applications or commands will provide accurate results

Part 3: * * * s (apparent packet loss) somewhere in the route is always bad and a sign of problems

Part 4: A uni-directional traceroute (run only in one direction) can be used to accurately spot a problem area in a route