Does this site look plain?

This site uses advanced css techniques

On around 23 Sept, 2009, Paychoice's "Online Employer" portal had a security incident where portal users received emails purportedly from Paychoice, informing them that they had to download and install an update to work with the portal.

The English was not particularly well written, but the email included their portal username and part of their real password. This was not a random phishing expedition: somebody had private information.

The links were to websites hosted at Yahoo!, and the badware was a password-stealing Trojan horse that phones home its stolen information to a webserver in Sweden.

It's said that Paychoice took down the portal at ~noon on Wednesday and provided new usernames and passwords. Apparently, a lot of service bureaus had some very uncomfortable phone calls to make, while many of their staff were at the Paychoice annual user conference.

I became involved on Thursday afternoon when a customer casually mentioned that he was having a bad day, and that started the process: I'm a security consultant.

The log below represents activies that I was directly involved in; there have been rumors and claims of other events or actions; I'll not address them.

Update 2009/09/30 — After following this incident for almost a week, I've come to the conclusion that this is far less serious than it appears at first. The bad guy did not get private data, and it was not an inside job. I have written my analysis here, though of course this is all speculation.

Blog posts

24 Sep 2009 — Thursday

(All times are Pacific Daylight Time)

14:00~
Obtained samples of the malicious emails, including the links.
fake email - Sept 23
14:15
Emailed Yahoo! Security with two domains used in the scam, plus introductory/background information:
update - onlineemployer . com online - employer - dnl . com
I'm fortunate to have a good working relationship with the Yahoo! Paranoids, the top-tier security guys there.
15:10~
Yahoo! disables those two sites.
15:37
Yahoo! Security reports that my submission was the first they have heard of this matter. This does not indicate anything about other departments at Yahoo!, including the website support center (about which I have no idea).
15:42
I provide Yahoo! Security with the email address of Charles Kershes (@paychoice.com), which was given to me by a customer.
16:25
Reported four more domains hosted at Yahoo!
onlineemployer - update . net onlineemployer - download . net secure - onlineemployer . com plugin - online - employer . org
16:29
Yahoo! Security first submits the badware, plugin_setup.exe (but named e.png) to VirusTotal with detection of 5/11.
17:25
Noticing a pattern in the domain names, wrote some perl code that generated all the permutations of the known words of the bad-guy's domains, and it found several more (all reported to Yahoo! Security)
onlineemployer - download . biz plugin - online - employer . com plugin - online - employer . net
Perl program: find-malurls (but a .txt file).
17:38
Yahoo! Security tells me that two of the previous set had been registered 2 hours ago and had not been deployed yet.
(unk)
Paychoice Support emails their customers:

25 Sep — Friday

07:27
Reported more domains to Yahoo!
onlineemployer - download . biz plugin - online - employer . com plugin - online-emp . info onlimemp - plugin . com onlimemp - software . org
11:18
Virustotal detects 14/40. Symantec does not detected yet.
Morning
Obtained a copy of the Friday fake mailes.
fake email - Sept 25
11:47
I email support @ paychoice.com, Cc: Yahoo! Security attempting to make an introduction.
12:09
Yahoo! Security reports they've made contact with Paychoice (20 minutes!)
~13:00
Paychoice sends this message to its licensees:
15:00
I submit the samples of malware to a private submission list that gets it to ~60 antimalware and protection companies (to their priority queue).
17:00
Ran the malware in a protected Virtual Manchine to watch behavior of the malware with the WireShark network sniffer: it phones home to http: // iicon - metal . org, and that IP 83.233.30.157 is located in Sweden.
The badware fetches a file, /la/config.bin, which appears to be encrypted, then a few exchanges via POST /forum/showthread.php, also transacting small encrypted packets.
I understand there are other maldomains and other malware that behave differently; I only captured one.

26 Sep — Saturday

08:08
Emailed support@everydns.net & abuse@bredband2.se (cc: Yahoo! Security) reporting the botnet phone home information noted above (hoping for a takedown).
08:08
Received auto-response from everydns, ticket #4519
09:00
Reran the badware in a virtual machine after localhost-ing iicon - metal . org (cutting off the mothership), found it doesn't appear to have a backup phone-home contact.
09:26
Resubmitted plugin_setup to Virustotal: now 18/40. Symantec not detected.

28 Sep — Monday

08:04
Virustotal picks up 22/41. Symantec still not detected.
10:10
Forwarded a copy of this log to Phil McLaughlin and Jim Costello of Paychoice.
12:00~
Developed a technique to help partly protect a network from infection: Protecting from the badware impersonating Online Employer.

29 Sep — Tuesday

01:59
Virustotal picks up 23/41. Symantec still doesn't have it.
10:04
Virustotal picks up 24/41. Symantec defs 1.4.4.12 (2009.09.29) still don't have it.
13:50
Link to this analysis posted to the Payroll Group member mailing list.

30 Sep — Wednesday

14:18
Virustotal picks up 28/41. Symantec seems to have them now, more than a week later.

1 Oct — Thursday

11:42
Virustotal still at 28/41.

2 Oct — Friday

~15:00
DNS for iicon-metal.org (one of the botnet command-and-control systems) is pulled by EveryDNS — Thanks David! Thanks
15:09
Email EveryDNS requesting termination of the calvinkleinstuffz.com domain.
16:24
Blog post: How not to react to a targetted malware attack

4 Oct — Sunday

~9:00
DNS for calvinkleinstuffz.com (the other domain) is pulled by EveryDNS.
09:40
Noticed that iicon-metal.org moved to EditDNS, emailed their abuse department with a report asking for the domain to be terminated. The webserver at iicon-metal.org is down.
14:49
Found that the webserver iicon-metal.org (83.233.30.157) is back up and commanding the botnet. Ugh.
17:26
Observe that iicon-metal.org does not resolve in DNS; Tyler @ EditDNS reports that mine was the only report he received for this domain.

5 Oct — Monday

04:45
Observe that DNS for iicon-metal.org is now hosted at DNS.GEN.IN, and the C&C is live (ugh).
05:45
Spent the last hour researching DNS.GEN.IN, could not find any obvious contact information. Sent abuse report to the three ISPs (HopOne, The Planet, Optical Jungle) in whose address space the nameservers reside.
05:46
Got autoresponse ticket [ThePlanetAbuse-C46427121Z] for DNS.GEN.IN.
06:02
Got autoresponse ticket [ir.hopone.net #11535] for DNS.GEN.IN.
07:59
Virustotal shows 31/41

6 Oct — Tuesday

13:01
DNS for iicon-metal.org disappears (!). No root resolution, but "whois" shows registration to NS1/NS2.MEGABANDISA.COM - this name does not show up anywhere in whois or the root servers. Odd.

10 Oct — Friday

05:00
Bredband2 (the Swedish ISP) reports that the mothership IP has been killed.

This page will be updated as events unfold.

VirusTotal Results

Other references

Important Disclaimer

Though I've performed security responses like this for years, in this case I have a substantial conflict of interest: I do consulting in the payroll industry, and am closely associated with a competitor of Paychoice.

My efforts on this security response were strictly a private effort to protect customers, have been entirely off the clock (of anybody), and have not been coordinated with any other party.

I have no axe to grind with Paychoice — with whom I have a great deal of sympathy — and believe I'm presenting this information in good faith. But you can reach your own conclusions based on what you read here and elsewhere.