Main Nav

Hello Folks,

 

Requesting suggestions to diagnose an Apache issue (Ubuntu server 8.04 LTS with Apache 2.2.8 serving custom PHP application with MySQL backend).  Server runs normally for a few hours and then Apache locks up, logs entries simply halt. Strace on the Apache processes all look like this:

 

Process 31281 attached - interrupt to quit

     0.000000 restart_syscall(<... resuming interrupted call ...>) = 0

     0.538531 poll([{fd=15, events=POLLIN|POLLPRI}], 1, 0) = 0

     0.000100 gettimeofday({1324400675, 745453}, NULL) = 0

     0.000046 gettimeofday({1324400675, 745492}, NULL) = 0

     0.000032 gettimeofday({1324400675, 745524}, NULL) = 0

 

We identified that a website blog function had allowed for significant commercial blog spam to be posted on the site (nonsensical text with lots of links to commercial sites: “Ugg boots clearance”, “Denim jackets cheap”, etc.), those posts have been deleted and the blog mechanism has been secured. Reviewing the Apache logs and Wireshark captures, we see that we have a LOT of traffic trying to get to those unauthorized (and now unresolvable) blog entries. Many of the requesting IPs are reverse proxies and search engine bots who seem to be crawling those spam URLs very aggressively.

 

We have concluded that our site was leveraged for a search engine “optimization” campaign, but now it appears we are suffering from a denial of service condition that may not have been intentional (If we were selling Ugg boots, we would be rich by now). We have some leads on mitigation: blocking aggressive hosts, mod_security, etc., but on a more fundamental level we are hoping to use this opportunity to educate ourselves on what to look for (and how to look for it) when experiencing these sort of events.

 

Any hints on Wireshark log parsing options for diagnosing DOS? Any thoughts on this behavior and the underpinnings of unscrupulous SEO campaigns?

 

I’ll take this opportunity to thank everyone for their contributions to the list in 2011 and offer a toast to an equally productive 2012!

 

Cheers,

alex

 

 

Alex Keller
Systems Administrator
Academic Technology, San Francisco State University
☛Burk Hall 155 ☎ (415)338-6117 ✉alkeller@sfsu.edu

 

Comments

You might look for a non-php way to handle the unresolved page requests. I have apache redirect to a flat.html file on error. Otherwise you run the risk of exhausting system resources.

Randall Grimshaw rgrimsha@syr.edu
 
Message from alexander.s@mccd.edu

Wireshark and other sniffers are useful if you want to take a look at some individual connections to see what is happening, but not for detecting DoS or other malicious activity in the first place. Your better bet would be to record session data using Argus or Cisco Netflows. You can also deploy an IDS, but I'd focus on getting session data first. The session data will give you a short record of every session with a timestamp, source/destination IP, source/destination port, in/out packet count and/or in/out bytes transferred. If you identify a potential issue from session logs or web server logs (based on timing, volume, etc), you can grab a full packet sniff with Wireshark/tcpdump/tethereal to tell you exactly what is happening. If you can log full content all the time, that will be very helpful in following up on any suspicious activity but you'll never want to use it as your first source of information. You may also want to develop some scripts for scanning your web server logs. If you develop scripts to extract all entries corresponding to a specific IP, error code, or time frame it will make it much easier to review logs in future investigations. My $.02, Steven ________________________________
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thu, Jan 05, 2012 at 07:56:48PM +0000, Alexander Kurt Keller wrote: > We have concluded that our site was leveraged for a search engine > ???optimization??? campaign, but now it appears we are suffering from a denial of > service condition that may not have been intentional (If we were selling Ugg > boots, we would be rich by now). We have some leads on mitigation: blocking > aggressive hosts, mod_security, etc., but on a more fundamental level we are > hoping to use this opportunity to educate ourselves on what to look for (and > how to look for it) when experiencing these sort of events. On the mitigation front, specifically on reducing resource exhaustion, have you looked at using cache software like squid or nginx? It's possible you can serve the PHP content using nginx and cut out Apache completely. Empirical testing has shown hosting drupal sites using nginx to be *considerably* less resource intensive than using Apache. kmw -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) iEYEARECAAYFAk8HCZQACgkQsKMTOtQ3fKE8AQCgub5sM+0sgMI3YOXCPCPG5CbT xkMAn09gYQRn8ORkJCrrm8tEyLEMmKIC =tBW1 -----END PGP SIGNATURE-----
Thank you Steven, Randall, Kevin, and Kay for your thoughtful responses to my query about our DOS condition. We are continuing to experiment but have started to bring the situation under control by clearing the errant content, using mod_security to restrict aggressive hosts, and implementing a PHP timeout on stalled requests. The Google webmaster tools proved useful for understanding what content was being searched for by visitors directed to our site as a result of the SEO tomfoolery. I'll post back to the list as merited, but wanted to say thank you for the assistance! Best, alex Alex Keller Systems Administrator Academic Technology, San Francisco State University ☛Burk Hall 155 ☎ (415)338-6117 ✉alkeller@sfsu.edu
Close
Close


EDUCAUSE Connect
View dates and locations

Events for all Levels and Interests

Whether you're looking for a conference to attend face-to-face to connect with peers, or for an online event for team professional development, see what's upcoming.

Close

EDUCAUSE Institute
Leadership/Management Programs
Explore More

Career Center


Leadership and Management Programs

EDUCAUSE Institute
Project Management

 

 

Jump Start Your Career Growth

Explore EDUCAUSE professional development opportunities that match your career aspirations and desired level of time investment through our interactive online guide.

 

Close
EDUCAUSE organizes its efforts around three IT Focus Areas

 

 

Join These Programs If Your Focus Is

Close

Get on the Higher Ed IT Map

Employees of EDUCAUSE member institutions and organizations are invited to create individual profiles.
 

 

Close

2014 Strategic Priorities

  • Building the Profession
  • IT as a Game Changer
  • Foundations


Learn More >

Uncommon Thinking for the Common Good™

EDUCAUSE is the foremost community of higher education IT leaders and professionals.