Does this site look plain?

This site uses advanced css techniques

On around 23 Sept, 2009, Paychoice's "Online Employer" portal had a security incident where portal users received emails purportedly from Paychoice, informing them that they had to download and install an update to work with the portal.

The English was not particularly well written, but the email included their portal username and part of their real password. This was not a random phishing expedition: somebody had private information.

The links were to websites hosted at Yahoo!, and the badware was a password-stealing Trojan horse that phones home its stolen information to a webserver in Sweden.

It's said that Paychoice took down the portal at ~noon on Wednesday and provided new usernames and passwords. Apparently, a lot of service bureaus had some very uncomfortable phone calls to make, while many of their staff were at the Paychoice annual user conference.

I became involved on Thursday afternoon when a customer casually mentioned that he was having a bad day, and that started the process: I'm a security consultant.

The log below represents activies that I was directly involved in; there have been rumors and claims of other events or actions; I'll not address them.

Update 2009/09/30 — After following this incident for almost a week, I've come to the conclusion that this is far less serious than it appears at first. The bad guy did not get private data, and it was not an inside job. I have written my analysis here, though of course this is all speculation.

Blog posts

24 Sep 2009 — Thursday

(All times are Pacific Daylight Time)

Obtained samples of the malicious emails, including the links.
fake email - Sept 23
Emailed Yahoo! Security with two domains used in the scam, plus introductory/background information:
update - onlineemployer . com online - employer - dnl . com
I'm fortunate to have a good working relationship with the Yahoo! Paranoids, the top-tier security guys there.
Yahoo! disables those two sites.
Yahoo! Security reports that my submission was the first they have heard of this matter. This does not indicate anything about other departments at Yahoo!, including the website support center (about which I have no idea).
I provide Yahoo! Security with the email address of Charles Kershes (, which was given to me by a customer.
Reported four more domains hosted at Yahoo!
onlineemployer - update . net onlineemployer - download . net secure - onlineemployer . com plugin - online - employer . org
Yahoo! Security first submits the badware, plugin_setup.exe (but named e.png) to VirusTotal with detection of 5/11.
Noticing a pattern in the domain names, wrote some perl code that generated all the permutations of the known words of the bad-guy's domains, and it found several more (all reported to Yahoo! Security)
onlineemployer - download . biz plugin - online - employer . com plugin - online - employer . net
Perl program: find-malurls (but a .txt file).
Yahoo! Security tells me that two of the previous set had been registered 2 hours ago and had not been deployed yet.
Paychoice Support emails their customers:

25 Sep — Friday

Reported more domains to Yahoo!
onlineemployer - download . biz plugin - online - employer . com plugin - online-emp . info onlimemp - plugin . com onlimemp - software . org
Virustotal detects 14/40. Symantec does not detected yet.
Obtained a copy of the Friday fake mailes.
fake email - Sept 25
I email support @, Cc: Yahoo! Security attempting to make an introduction.
Yahoo! Security reports they've made contact with Paychoice (20 minutes!)
Paychoice sends this message to its licensees:
I submit the samples of malware to a private submission list that gets it to ~60 antimalware and protection companies (to their priority queue).
Ran the malware in a protected Virtual Manchine to watch behavior of the malware with the WireShark network sniffer: it phones home to http: // iicon - metal . org, and that IP is located in Sweden.
The badware fetches a file, /la/config.bin, which appears to be encrypted, then a few exchanges via POST /forum/showthread.php, also transacting small encrypted packets.
I understand there are other maldomains and other malware that behave differently; I only captured one.

26 Sep — Saturday

Emailed & (cc: Yahoo! Security) reporting the botnet phone home information noted above (hoping for a takedown).
Received auto-response from everydns, ticket #4519
Reran the badware in a virtual machine after localhost-ing iicon - metal . org (cutting off the mothership), found it doesn't appear to have a backup phone-home contact.
Resubmitted plugin_setup to Virustotal: now 18/40. Symantec not detected.

28 Sep — Monday

Virustotal picks up 22/41. Symantec still not detected.
Forwarded a copy of this log to Phil McLaughlin and Jim Costello of Paychoice.
Developed a technique to help partly protect a network from infection: Protecting from the badware impersonating Online Employer.

29 Sep — Tuesday

Virustotal picks up 23/41. Symantec still doesn't have it.
Virustotal picks up 24/41. Symantec defs (2009.09.29) still don't have it.
Link to this analysis posted to the Payroll Group member mailing list.

30 Sep — Wednesday

Virustotal picks up 28/41. Symantec seems to have them now, more than a week later.

1 Oct — Thursday

Virustotal still at 28/41.

2 Oct — Friday

DNS for (one of the botnet command-and-control systems) is pulled by EveryDNS — Thanks David! Thanks
Email EveryDNS requesting termination of the domain.
Blog post: How not to react to a targetted malware attack

4 Oct — Sunday

DNS for (the other domain) is pulled by EveryDNS.
Noticed that moved to EditDNS, emailed their abuse department with a report asking for the domain to be terminated. The webserver at is down.
Found that the webserver ( is back up and commanding the botnet. Ugh.
Observe that does not resolve in DNS; Tyler @ EditDNS reports that mine was the only report he received for this domain.

5 Oct — Monday

Observe that DNS for is now hosted at DNS.GEN.IN, and the C&C is live (ugh).
Spent the last hour researching DNS.GEN.IN, could not find any obvious contact information. Sent abuse report to the three ISPs (HopOne, The Planet, Optical Jungle) in whose address space the nameservers reside.
Got autoresponse ticket [ThePlanetAbuse-C46427121Z] for DNS.GEN.IN.
Got autoresponse ticket [ #11535] for DNS.GEN.IN.
Virustotal shows 31/41

6 Oct — Tuesday

DNS for disappears (!). No root resolution, but "whois" shows registration to NS1/NS2.MEGABANDISA.COM - this name does not show up anywhere in whois or the root servers. Odd.

10 Oct — Friday

Bredband2 (the Swedish ISP) reports that the mothership IP has been killed.

This page will be updated as events unfold.

VirusTotal Results

Other references

Important Disclaimer

Though I've performed security responses like this for years, in this case I have a substantial conflict of interest: I do consulting in the payroll industry, and am closely associated with a competitor of Paychoice.

My efforts on this security response were strictly a private effort to protect customers, have been entirely off the clock (of anybody), and have not been coordinated with any other party.

I have no axe to grind with Paychoice — with whom I have a great deal of sympathy — and believe I'm presenting this information in good faith. But you can reach your own conclusions based on what you read here and elsewhere.