The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, January 04, 2011

It's hard to break a program when the network keeps breaking

One of my jobs at The Corporation has been to load test a phone network service by simulating a ton of calls and see where our service breaks. Which means I get to write code to simulate a phone network initiating calls. It's not easy, and no, it's not entirely related to The Protocol Stack From Hell™ (but don't get me wrong—there's still plenty of blame to go around).

Problem the first: generating a given load, a measured set of packets per second. It's harder than it sounds. Under the operating system we're using, the smallest unit we can reliably pause is 1/100 of a second. If I want to send out 500 messages per second, the best I can do is 5 packets every 0.01 seconds, which isn't the same as one packet every 0.002 seconds (even though it averages out). The end result tends towards bursty traffic (that is, if I attempt to control the rate; if I don't bother with that, I tend to break the phone network connection—more on that below).

Sure, there's some form of congestion control in The Protocol Stack From Hell™, but attempting to integrate the sample code provided into my testing program failed—not only do I not get the proper messages, but what I do get is completely different from the sample program. This is compounded by the documentation (which everybody agrees is completely worthless) and the fact that this is the first time I've ever worked on anything remotely related to telephony. I'm unfamiliar with the protocols, and with the ins and outs of The Protocol Stack From Hell™ (unlike my manager R, who's worked with this stuff for the past fifteen to twenty years, but is swamped with other, manager-type work).

Now the second problem: even though the testing system is in the same cabinet, and hooked to the same network switch, as the target system (in fact, I think they're physcially touching each other) due to the nature of the phone system, communications between phone network components must go through an intermediary system known as an STP; actually, a pair of STPs (for redundancy). Unfortunately for us, the only STP we have access to is out in Washington State (where The Corporate Master Headquarters are stationed) and said traffic between our two testing systems (here in Lower Sheol) goes back and forth across the Inernet over a VPN.

Yeah, what I can say? When I asked about getting an STP a bit closer to us, I was told it wasn't in the budget (and no wonder—the price is far into the “if you have to ask, you can't afford it” territory—deep into that territory).

So we're stuck with a six thousand mile round trip for the phone network traffic. And now we come to the punch line—the Internet is broken. The quick synopsis: excessive buffering of Internet traffic by various routers is causing a breakdown of anti-congestion algorithms used by TCP/IP. Now, the article is talking about buffer bloat in consumer grade equipment, but it is possible that there are commercial grade routers doing the same thing—excessive buffering and that could be the cause of largish spikes in traffic, as well as increased latency in round trips. If there's a spike in traffic, the STP will attempt to assert flow control, but if there's still traffic coming in, it's considered an error. Also, the phone network is very time sensitive and execessive latencies are also an error condition.

Worse, if the STP receives too many errors from an endpoint, it (like every other STP on the phone network) is programmed to take that endpoint out of service. It's hard to say where that point is, but it happens with frightening regularity when I attempt to do load testing. The packets are being pushed when suddenly we start receiving either canceled messages, or delivery failures about five levels down in the protocol stack, which means one (or both) endpoints have been cut loose from the phone network due to excessive errors. It then requires manual intervention to restart the entire stack on both sides.

So, there's bursty traffic due to my attempts at sending a settable amount of traffic. Then there's the (possible) bursty traffic due to excessive buffering across the Internet. Oh, and I forgot to mention the licensing restriction on The Protocol Stack From Hell™ that limits the number of messages we can send and receive. All that makes it quite difficult to find the breaking point of the program I'm testing. I keep breaking the communications channel.

That tends to put a damper on load testing.

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Contact Info

Obligatory Feeds

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2022 by Sean Conner. All Rights Reserved.