The Boston Diaries

The ongoing saga of a programmer who doesn't live in Boston, nor does he even like Boston, but yet named his weblog/journal “The Boston Diaries.”

Go figure.

Tuesday, January 17, 2006

Some initial data from a real-time LaBrea data processing program

While I'm waiting a call back, some more on LaBrea.

Yesterday (from January 16 at 06:28:25 to January 17 08:54:50) LaBrea generated 1.1G of log data, and it took full five minutes to run grep 'Initial Connect' daemon.log.0 | wc -l (255,344 new tarpitting connections by the way).

LaBrea was also running at full speed, maxed out at 64Kbps bandwidth to keep all these connections tarpitted (the maximum I set LaBrea to use, by the way).

[NETWORK TARPIT bandwidth usage]

That first large dip in the graph (the one around 6:30 in the morning) is probably due to the system attempting to rotate a 1.1G log file. The second dip, at the right (around 3:00 pm) is when I restarted LaBrea so its logging information would go through ltpstat. After an hour of running:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 15:55:59 2006 Running time: 1h
Pool-max: 1048576
Pool-num: 24322
Rec-max:  1048576
Rec-num:  24322
UIP-max:  1048576
UIP-num:  1282
Reported-bandwidth: 40 (Kb/sec)

And after two hours:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 16:56:19 2006 Running time: 2h 20s
Pool-max: 1048576
Pool-num: 33326
Rec-max:  1048576
Rec-num:  33326
UIP-max:  1048576
UIP-num:  1632
Reported-bandwidth: 40 (Kb/sec)

And right this second:

Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 18:37:19 2006 Running time: 3h 41m 20s
Pool-max: 1048576
Pool-num: 42931
Rec-max:  1048576
Rec-num:  42931
UIP-max:  1048576
UIP-num:  2148
Reported-bandwidth: 40 (Kb/sec)

Okay, pool-max and rec-max are the maximum sizes for the structure array and pointer array, and both should always be equal at all times (I'm displaying this number more for debugging purposes than anything else), while pool-num and rec-num (which should also be equal at all times) represent the current number of connectioned tarpitted. I also keep track of unique IPs, which just now is currently 2,148 (out of 1,048,576 that I can store). I also just found out that IP address has 4,809 connections currently tarpitted (and in the few seconds it took to do that query, five more connections were tarpitted).

I'll be releasing this code in the next few days, when I can write up some documentation and slap on a license.

Obligatory Picture

[It's the most wonderful time of the year!]

Obligatory Links

Obligatory Miscellaneous

You have my permission to link freely to any entry here. Go ahead, I won't bite. I promise.

The dates are the permanent links to that day's entries (or entry, if there is only one entry). The titles are the permanent links to that entry only. The format for the links are simple: Start with the base link for this site:, then add the date you are interested in, say 2000/08/01, so that would make the final URL:

You can also specify the entire month by leaving off the day portion. You can even select an arbitrary portion of time.

You may also note subtle shading of the links and that's intentional: the “closer” the link is (relative to the page) the “brighter” it appears. It's an experiment in using color shading to denote the distance a link is from here. If you don't notice it, don't worry; it's not all that important.

It is assumed that every brand name, slogan, corporate name, symbol, design element, et cetera mentioned in these pages is a protected and/or trademarked entity, the sole property of its owner(s), and acknowledgement of this status is implied.

Copyright © 1999-2019 by Sean Conner. All Rights Reserved.