Weblogic Joomla Template Demo

Hi all,

Just got through working through a 17 hour security incident and it was interesting enough I thought I would share the details.

A client contacted me through a referral (hint, hint) and said they had been hacked and had been defaced, and then they were down. They had some backups, but did not know what steps to take next.

As it turns out the defacement was not the original intent of the hacker. That came from another hacker most likely. The original intent was to SPAM from this box. Which the hacker had been successfully doing for 3 months, all the while wiping the log files out to hide his tracks.

This box had been penetrated in or around October 2007 and the box was being used to spam. I think that forensics of this are worth while and hope you find them valuable.

The Box:

The server was located with a hosting company, whose reputation is not a shining one, and for good reason.

They are running:

Linux 2.x.x
Cpanel
Joomla 1.0.13 (or was it 1.0.14) the client could not remember, likely due to panic)

The client said, in his panic to get the box up he upgraded it to 1.0.14, rc1. This did not help and muddied up the water. Or so he thought. Again he could not remember.

The situation

As I said, they were down hard, and had a splash page up. I first went to the log files to see what was what, surprise, surprise! They had been deleted.

Upon digging further, I did find a few log files of interest that had not been deleted.

In the log file, I found that the client had changed a file called c99.php to c9badfile.php. This is the command “shell” to allow hackers into the backend.

Further, I found the database was around 22 megabytes. Not unheard of but seemed rather large – more on this in a minute.

So I proceeded to examine the logs and found out their average bandwidth was about 30 Meg a month. In the first two weeks of this month they had transferred nearly 700 Meg. Yes, point 7 (.7) terabytes of bandwidth. – Gee –that’s a lot!
It had been going up on a regular basis.

Easy enough, he had been attacked, but why and from where? We know that a backdoor was present but from whom?

Forensics:

I took the only known good backup he had, downloaded it to my machine and scanned it with Kapersky (which rocks!) and it revealed the following:

The system had five copies of : Virus.Linux.Osf.8759 and it had one copy of: SpamTool.PHP.Massma.l

Gee is that YOUR viruses on here? The malware SpamToo.Php.Massma.l doesn’t have an exact classification but.

However the Virus.Linux.OsF.8759 was very impressive!

Linux.OSF.8759 is a virus with enhanced backdoor capabilities that replicates on Linux systems and infects ELF executables.

The files infected by the virus have their file size increased by 8759 bytes. 3979 bytes belong to the actual virus code while the other 4662 belong to the code of a backdoor attached by the virus at the end of the file.

Although the backdoor code is copied along with the virus, it seems it appears designed in such way that it can be easily replaced with updated versions - the backdoor is not linked into the ELF structure, but is instead loaded and executed by the virus itself. Therefore improved versions of this virus, especially of the backdoor code can be expected in the future.

The virus infects all the files in the current directory, but avoids infecting files with file names ending with "ps".

If run from a root account the virus will also attempt to infect the files from the "/bin" system directory. In all cases no more than 201 files are infected in one run.

The backdoor found in this version of the virus is listening on the UDP port 3049, or if the respective port is not available, it will try to increase the port number until one which can be used is found. Various internal commands are available to directly execute files on the target system or to launch a sniffer and forward the traffic to the other machine. The backdoor will also attempt to edit the firewall rules list and wipe out any entries that might prevent it from communicating on the hooked port, or, on the port used to communicate with the remote machine in the case of the sniffer.

Besides the above, the virus also attempts to prevent tracing by various debugging utilities by spawning a copy of itself, and attempting to debug itself from the spawned copy. If any debugger is already running, these steps will fail, and the virus will immediately terminate execution.

Another detail is if the system uptime is 5 minutes or less, the virus will also terminate execution, probably in order to prevent simple inspection on "test" machines.

It would seem the virus is the transport for the backdoor payload. It can hide for up to five minutes by terminating but stay resident; it can propagate with ROOT access...

Remember our C99 shell? – Yep – probably root access.

It “listens” on port 3049, and will increment till it finds an open port. And my favorite part? – It can spawn and morph.

Like I said – slick. In any case, we verified all traces of this had been removed.

After we cleaned it up, we reviewed the files and found some “fun facts” about our perp!

~Tue Nov 15 09:17:46 : (bat-boy!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ***.***) hai

~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Tracking Number: 1Z X99 857 66 7925 164 1
~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Service Type: EXPRESS
~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Weight: 20.80 Lb
~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:31:48 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:31:50 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Package Progress:
~Tue Nov 22 22:31:52 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:31:54 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:31:56 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Date/
~Tue Nov 22 22:31:58 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) Time Location Activity
~Tue Nov 22 22:32:00 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it )
~Tue Nov 22 22:32:02 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) 22 Nov 2005
~Tue Nov 22 22:32:04 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) 10:36
~Tue Nov 22 22:32:06 : (karebet_kere!~ This e-mail address is being protected from spam bots, you need JavaScript enabled to view it ) ANCHORAGE, AK, US
These were located in the log files on the server. Note the date? 2005? A fed-ex tracking number? Interesting but very old, I could not locate a package trace.

Doing a google search for karebet, revealed a Yahoo! Home page, but no certainty that this is related.

In any case, this is a partial log from an IRC control/chat session with a bot in 2005. That would mean that this particular payload has been around for while!

In any case, the other logs, while more recent revealed an underground hacker/spammer group.

Some of the logs left in tact, revealed the ‘bot master’ to be somewhere in Brazil! Gee, I could think of a lot more fun things to do in Brazil! But…each to his own!

And surprisingly while I was rebuilding the box, the bad guy attempted to connect, and I caught him. I quickly added his IP’s (and others around him) to the .htaccess and blocked him.

I reinstalled J! 1.0.13, attempted to do a database restore, it was flakey so I had to reinstall 1 table at a time. This was going well until, yes. You guessed it. A virus/script issue in the database.

Had to start over. arrrghhh!

That brings us to now. The site is up, mostly, the client has moved to a new host (schew!) and is reporting no problems.

To recap here are the steps taken to detect and fix this issue:

1) FIRST capture by downloading any and all log files
2) If possible take the host completely offline (sometimes you can, sometimes you can’t)
3) Do a (CPanel) complete backup of everything and be sure its ZIPPED
4) Scan the zip for virus / evil and eliminate
5) Then review the logs to determine the method of break in, where files may be,etc
6) Restore.

There was a lot more gnashing of teeth than this however this captures the moment.

Steps to pre-empt

* Make a disaster recovery plan and follow it
* Review your log files DAILY
* Check for Open ports
->(Nmap is a great tool but be very, very careful as hosts do not care for that stuff)
* Make sure you have the latest and best security practices in place

If all else fails? Call me!

This is great reading material.
I have lost websites due to being hacked and not knowing how to find and remove the hacker code.
I had plenty of old backups, but hey had all been infested.
It had been in there for months before the hosting company reported it to me and all was lost.

I rebuilt the site from scratch.

I will take your advice and build my own contingency plan.
If you have a lot of sites, reviewing logs could be very time consuming.
Are there scripts available to run against the log files than can raise a flag or is it changing so frequent that you need to eyeball the logs manually to be certain.

Thanks Tom good stuff !!!

Hi Phil,

Good question, in my book "Dodging the bullets - a disaster recovery plan for Joomla! websites" I outline a terrific plan! - However if you have a non-joomla site, much of it will apply.

Let me recommend another book, "Disaster recovery Planning" By Jon Toigo - a much broader covering of the DR topic.

On the log front, the challenge with an on-board script is one of trust. How can you trust that if you were compromised, that the script would not have been modified. You cannot.

With log files you face two challenges, 1 - make it a habit to review logs regularly AND copy them down. Because like my client in the post, the files were wiped out. Leaving no trace of how he got on to the system. 2 - tools to review them.

In my upcoming book - Practical Joomla! security I devote an entire chapter to log files. With that, let me say that the log files (on linux/apache) are in a specific format, log files on Windows/Apache are in the same formal (common log format) - reading them is no problem, except for the AMOUNT of data that is there.

Lots of tools exist to review log files or in other words, help you sort through them. But log files are historical, not pre-emptive. If you run a joomla! site, consider purchasing the extension from ravenswoodit.co.uk - known as JCHECK. It does a really good job on alerting you IF something has changed. You can set it for hourly, daily, weekly, etc. You can exclude or include portions of your site. This is ONE method to help you sort through.

Another method you may consider is building out a CRON job (cron is a timer) that will copy your log files to an email (for instance) and email them too you. You could run a search with notepad for suspicious stuff, but of course, you'd need to know what that is to search for. :0

Hope that helps.