Wednesday, July 11, 2001
A routing mess
Theoretically, TCP/IP packets don't have to take the same route but
in real life all the packets travel the same path. Do two
traceroutes and you'll see the same path.
But today, something seriously odd is going on with the Telocity network. I noticed some annoying network lags between my home system and my colocated server and did some traceroutes. And I got something I've never seen before: no two traceroutes (done seconds after each other) where the same!
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 1.814 ms 1.737 ms 1.755 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.196 ms 3.244 ms 3.321 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 14.046 ms 14.042 ms 14.335 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 14.353 ms 15.842 ms 15.055 ms
5 63.209.144.5 (63.209.144.5) 28.034 ms 15.807 ms 15.295 ms
6 63.212.176.30 (63.212.176.30) 64.815 ms 64.130 ms 64.096 ms
7 tower.conman.org (66.33.1.143) 63.686 ms 64.580 ms 65.116 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 1.811 ms 1.739 ms 1.814 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.203 ms 3.250 ms 3.166 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 14.206 ms 16.701 ms 14.037 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 55.927 ms 14.870 ms 15.310 ms
5 pos8-1-hfr2.atl.tlct.net (216.227.126.17) 29.778 ms 27.010 ms 28.602 ms
6 ge-v5-hfr1.atl.tlct.net (216.227.126.57) 26.929 ms 27.705 ms 26.858 ms
7 pos8-1-hfr2.dfw.tlct.net (216.227.126.22) 72.251 ms 75.729 ms 73.677 ms
8 ge-v5-hfr1.dfw.tlct.net (216.227.126.53) 74.153 ms 73.005 ms 75.141 ms
9 pos8-1-hfr2.lax.tlct.net (216.227.126.161) 105.344 ms 107.666 ms 106.208 ms
10 ge-v5-hfr1.lax.tlct.net (216.227.126.49) 105.677 ms 105.910 ms 106.452 ms
11 pos8-1-hfr2.sjc.tlct.net (216.227.126.30) 132.903 ms 130.575 ms 125.659 ms
12 fe1-0-pbnap.sjc.tlct.net (216.227.126.169) 130.631 ms 124.767 ms 125.355 ms
13 ge-v5-hfr1.sjc.tlct.net (216.227.126.45) 119.255 ms 117.430 ms 125.996 ms
14 fe0-0-pbnap.sjc.tlct.net (216.227.126.165) 128.272 ms 130.919 ms 307.509 ms
15 * tower.conman.org (66.33.1.143) 64.129 ms 65.449 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 1.800 ms 1.755 ms 1.731 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.212 ms 3.170 ms 3.230 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 13.544 ms 14.038 ms 12.598 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 15.749 ms 14.805 ms 15.291 ms
5 pos8-1-hfr2.atl.tlct.net (216.227.126.17) 31.635 ms 28.690 ms 28.623 ms
6 ge-v5-hfr1.atl.tlct.net (216.227.126.57) 27.187 ms 28.049 ms 28.120 ms
7 pos8-1-hfr2.dfw.tlct.net (216.227.126.22) 74.507 ms 73.200 ms 72.905 ms
8 ge-v5-hfr1.dfw.tlct.net (216.227.126.53) 72.751 ms 73.105 ms 75.415 ms
9 pos8-1-hfr2.lax.tlct.net (216.227.126.161) 106.234 ms 106.130 ms 106.299 ms
10 ge-v5-hfr1.lax.tlct.net (216.227.126.49) 105.595 ms 106.418 ms 106.217 ms
11 pos8-1-hfr2.sjc.tlct.net (216.227.126.30) 135.297 ms 128.321 ms 126.335 ms
12 fe1-0-pbnap.sjc.tlct.net (216.227.126.169) 118.552 ms 130.986 ms 126.928 ms
13 ge-v5-hfr1.sjc.tlct.net (216.227.126.45) 114.866 ms 117.073 ms 124.424 ms
14 fe0-0-pbnap.sjc.tlct.net (216.227.126.165) 127.696 ms 130.745 ms 125.977 ms
15 * pos8-1-hfr2.atl.tlct.net (216.227.126.17) 191.308 ms *
16 * tower.conman.org (66.33.1.143) 101.322 ms 100.327 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 2.492 ms 1.769 ms 1.748 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 4.209 ms 3.150 ms 3.165 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 14.623 ms 13.612 ms 15.036 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 31.694 ms 16.130 ms 15.292 ms
5 vlan20-core2.mia.tlct.net (216.227.80.34) 15.440 ms 15.476 ms 16.583 ms
6 pos8-2-hfr1.was.tlct.net (216.227.126.217) 41.190 ms 62.272 ms 41.618 ms
7 fe0-0-mae-east.was.tlct.net (216.227.126.109) 47.128 ms 40.831 ms 40.425 ms
8 at0-0-0-170.br1.WDC2.gblx.net (208.49.231.89) 51.271 ms 40.997 ms 41.915 ms
9 pos2-0-155M.cr2.WDC2.gblx.net (208.178.174.61) 41.269 ms 41.624 ms 41.947 ms
10 pos1-0-622M.cr1.ATL1.gblx.net (206.132.115.94) 57.903 ms 57.344 ms 58.573 ms
11 pos5-0-0-155M.ar1.TPA1.gblx.net (208.178.116.34) 91.555 ms 93.025 ms 93.167 ms
12 Customer.s4-0-0.ar1.TPA1.gblx.net (64.212.109.138) 99.760 ms 98.111 ms 216.351 ms
13 tower.conman.org (66.33.1.143) 101.450 ms 102.368 ms 101.038 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 1.770 ms 1.852 ms 1.705 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.175 ms 3.220 ms 3.290 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 12.757 ms 13.936 ms 14.244 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 15.600 ms 16.065 ms 13.803 ms
5 pos8-1-hfr2.atl.tlct.net (216.227.126.17) 27.430 ms 28.691 ms 28.694 ms
6 ge-v5-hfr1.atl.tlct.net (216.227.126.57) 27.660 ms 26.413 ms 26.573 ms
7 pos8-1-hfr2.dfw.tlct.net (216.227.126.22) 74.026 ms 82.005 ms 73.880 ms
8 ge-v5-hfr1.dfw.tlct.net (216.227.126.53) 74.488 ms 73.386 ms 73.193 ms
9 pos8-1-hfr2.lax.tlct.net (216.227.126.161) 105.209 ms 106.414 ms 106.708 ms
10 ge-v5-hfr1.lax.tlct.net (216.227.126.49) 107.248 ms 106.171 ms 127.723 ms
11 pos8-1-hfr2.sjc.tlct.net (216.227.126.30) 135.017 ms 127.588 ms 126.325 ms
12 fe1-0-pbnap.sjc.tlct.net (216.227.126.169) 133.066 ms 125.410 ms 128.867 ms
13 ge-v5-hfr1.sjc.tlct.net (216.227.126.45) 115.226 ms 121.554 ms 119.980 ms
14 fe0-0-pbnap.sjc.tlct.net (216.227.126.165) 130.603 ms 134.301 ms 130.845 ms
15 ge-v5-hfr1.sjc.tlct.net (216.227.126.45) 119.819 ms 130.060 ms 142.445 ms
16 fe0-0-pbnap.sjc.tlct.net (216.227.126.165) 139.018 ms 133.283 ms *
17 tower.conman.org (66.33.1.143) 102.361 ms 101.642 ms 100.291 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 1.740 ms 1.757 ms 1.728 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.201 ms 3.244 ms 3.170 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 13.510 ms 13.748 ms 14.303 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 14.592 ms 14.997 ms 14.759 ms
5 vlan20-core2.mia.tlct.net (216.227.80.34) 17.374 ms 16.767 ms 16.539 ms
6 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.462 ms 15.245 ms 14.954 ms
7 vlan20-core2.mia.tlct.net (216.227.80.34) 14.788 ms 16.020 ms 16.619 ms
8 fe1-2-core1.mia.tlct.net (216.227.80.65) 15.897 ms 15.401 ms 16.995 ms
9 vlan20-core2.mia.tlct.net (216.227.80.34) 16.266 ms 16.682 ms 15.344 ms
10 fe1-2-core1.mia.tlct.net (216.227.80.65) 15.385 ms 16.295 ms 15.085 ms
11 vlan20-core2.mia.tlct.net (216.227.80.34) 16.568 ms 17.000 ms 16.552 ms
12 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.599 ms 17.155 ms 14.814 ms
13 vlan20-core2.mia.tlct.net (216.227.80.34) 14.799 ms 40.861 ms 18.034 ms
14 fe1-2-core1.mia.tlct.net (216.227.80.65) 17.572 ms 91.453 ms 15.259 ms
15 vlan20-core2.mia.tlct.net (216.227.80.34) 23.225 ms 16.968 ms 16.886 ms
16 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.060 ms 17.160 ms 15.896 ms
17 vlan20-core2.mia.tlct.net (216.227.80.34) 15.333 ms 16.403 ms 25.721 ms
18 fe1-2-core1.mia.tlct.net (216.227.80.65) 17.314 ms 15.686 ms 16.859 ms
19 vlan20-core2.mia.tlct.net (216.227.80.34) 23.482 ms 15.241 ms 15.339 ms
20 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.103 ms 16.605 ms 17.546 ms
21 vlan20-core2.mia.tlct.net (216.227.80.34) 15.845 ms 16.959 ms 18.241 ms
22 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.839 ms 15.091 ms 15.442 ms
23 vlan20-core2.mia.tlct.net (216.227.80.34) 15.910 ms 16.546 ms 16.501 ms
24 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.934 ms 16.800 ms 17.549 ms
25 vlan20-core2.mia.tlct.net (216.227.80.34) 17.931 ms 16.327 ms 16.082 ms
26 fe1-2-core1.mia.tlct.net (216.227.80.65) 15.973 ms 16.469 ms 16.467 ms
27 vlan20-core2.mia.tlct.net (216.227.80.34) 52.760 ms 17.235 ms 15.509 ms
28 fe1-2-core1.mia.tlct.net (216.227.80.65) 16.039 ms 16.770 ms 16.421 ms
29 vlan20-core2.mia.tlct.net (216.227.80.34) 15.857 ms 16.527 ms 17.723 ms
30 fe1-2-core1.mia.tlct.net (216.227.80.65) 17.253 ms 18.240 ms 16.017 ms
[spc]linus:/usr/src/linux-2.0.39>/usr/sbin/traceroute tower.conman.org
traceroute to tower.conman.org (66.33.1.143), 30 hops max, 40 byte packets
1 janet (10.0.0.1) 43.666 ms 1.827 ms 1.813 ms
2 dsl-64-128-156-186.telocity.com (64.128.156.186) 3.306 ms 3.232 ms 3.102 ms
3 route-64-131-188-129.telocity.com (64.131.188.129) 14.623 ms 14.539 ms 14.481 ms
4 fe1-2-core1.mia.tlct.net (216.227.80.65) 65.600 ms 81.921 ms 58.918 ms
5 vlan20-core2.mia.tlct.net (216.227.80.34) 85.022 ms 14.867 ms 14.560 ms
6 fe1-2-core1.mia.tlct.net (216.227.80.65) 119.265 ms * ge-v5-hfr1.atl.tlct.net (216.227.126.57) 46.996 ms
7 pos8-1-hfr2.dfw.tlct.net (216.227.126.22) 73.053 ms 81.261 ms 94.668 ms
8 ge-v5-hfr1.dfw.tlct.net (216.227.126.53) 105.210 ms 75.482 ms *
9 209.246.152.61 (209.246.152.61) 80.207 ms 71.852 ms 72.419 ms
10 gigabitethernet5-0.core1.Dallas1.Level3.net (209.244.15.21) 71.978 ms 74.689 ms
ge-v5-hfr1.lax.tlct.net (216.227.126.49) 106.497 ms
11 pos8-1-hfr2.sjc.tlct.net (216.227.126.30) 133.156 ms 141.849 ms 134.666 ms
12 fe1-0-pbnap.sjc.tlct.net (216.227.126.169) 125.158 ms 117.098 ms 115.600 ms
13 pos2-1-paix.pao.tlct.net (216.227.126.210) 115.742 ms 121.505 ms 162.642 ms
14 gblx-px.pao.tlct.net (64.192.16.14) 124.839 ms 117.256 ms 124.204 ms
15 so1-0-0-622M.cr1.pao2.gblx.net (208.50.169.141) 122.668 ms 126.308 ms 138.030 ms
16 pos1-0-622M.cr1.HOU1.gblx.net (206.132.116.126) 121.666 ms 132.773 ms 124.458 ms
17 * pos0-0-155M.ar1.TPA1.gblx.net (206.132.116.98) 151.364 ms *
18 Customer.s4-0-0.ar1.TPA1.gblx.net (64.212.109.138) 158.682 ms 246.846 ms 158.229 ms
19 tower.conman.org (66.33.1.143) 161.658 ms 163.065 ms 161.680 ms
[spc]linus:/usr/src/linux-2.0.39>
Got to see major network rerouting at work here. From 7 to 19 hops no less! Sheesh!
