Quote:
Originally Posted by carnation
I noticed that the majority of them were looking at the same thread.
|
Yeah, it was an old thread,
greeks in the family , only 14 pages... shouldn't take the bots 100+ thousand page views throughout the day to spider/scrape that thread. Somehow they sometimes get hung up and just keep going on the same thread.
Last summer was a similar issue. Bots from 2 IP addresses loading all the pages from the print version of the original "What do you feel like saying, right now?" thread. Those bots were loading something like 50 to 100 million pages per month from that thread. They were a major contributing factor causing the site to slow down so much for a while up until then.
The bot flood back in the fall I think was mostly scattered across many different pages.