Bots flooding GC

[cheerfulgreek]

Originally Posted by John

Bots are flooding GC again. Nearly 6,500 online users right now...

There are always bots, search spider bots are fine & not a problem. This current flood, though, is probably the AI scraper (data harvesting) type again.

Will try to get things sorted out with this batch of bots before they cause too much trouble...

Thank you for everything, John.

I thought you made it where it’d be harder for bots to attack the site. Did they manage to get around it?

[John]

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.

Originally Posted by cheerfulgreek

Thank you for everything, John.

Originally Posted by cheerfulgreek

I thought you made it where it’d be harder for bots to attack the site.

Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.

Originally Posted by cheerfulgreek

Did they manage to get around it?

Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

[carnation]

I noticed that the majority of them were looking at the same thread.

[John]

Originally Posted by carnation

I noticed that the majority of them were looking at the same thread.

Yeah, it was an old thread, greeks in the family , only 14 pages... shouldn't take the bots 100+ thousand page views throughout the day to spider/scrape that thread. Somehow they sometimes get hung up and just keep going on the same thread.

Last summer was a similar issue. Bots from 2 IP addresses loading all the pages from the print version of the original "What do you feel like saying, right now?" thread. Those bots were loading something like 50 to 100 million pages per month from that thread. They were a major contributing factor causing the site to slow down so much for a while up until then.

The bot flood back in the fall I think was mostly scattered across many different pages.

[cheerfulgreek]

Originally Posted by John

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.







Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.



Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

They seemed to be gone or not as active for a while. Like, when you made those changes last summer, that seemed to work well.