Bots flooding GC

[carnation]

How can scammers live with themselves--cheating people? What would their mothers think if they knew what they're really doing?

[AGDee]

Originally Posted by carnation

How can scammers live with themselves--cheating people? What would their mothers think if they knew what they're really doing?

For the types I'm talking about, this is their job. They go to a big warehouse every day filled with computers and phones and go through their scams. It's a legit job in their countries. Sometimes sponsored by the government.

[John]

Bots are flooding GC again. Nearly 6,500 online users right now...

There are always bots, search spider bots are fine & not a problem. This current flood, though, is probably the AI scraper (data harvesting) type again.

Will try to get things sorted out with this batch of bots before they cause too much trouble...

[cheerfulgreek]

Originally Posted by John

Bots are flooding GC again. Nearly 6,500 online users right now...

There are always bots, search spider bots are fine & not a problem. This current flood, though, is probably the AI scraper (data harvesting) type again.

Will try to get things sorted out with this batch of bots before they cause too much trouble...

Thank you for everything, John.

I thought you made it where it’d be harder for bots to attack the site. Did they manage to get around it?

[John]

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.

Originally Posted by cheerfulgreek

Thank you for everything, John.

Originally Posted by cheerfulgreek

I thought you made it where it’d be harder for bots to attack the site.

Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.

Originally Posted by cheerfulgreek

Did they manage to get around it?

Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

[carnation]

I noticed that the majority of them were looking at the same thread.

[John]

Originally Posted by carnation

I noticed that the majority of them were looking at the same thread.

Yeah, it was an old thread, greeks in the family , only 14 pages... shouldn't take the bots 100+ thousand page views throughout the day to spider/scrape that thread. Somehow they sometimes get hung up and just keep going on the same thread.

Last summer was a similar issue. Bots from 2 IP addresses loading all the pages from the print version of the original "What do you feel like saying, right now?" thread. Those bots were loading something like 50 to 100 million pages per month from that thread. They were a major contributing factor causing the site to slow down so much for a while up until then.

The bot flood back in the fall I think was mostly scattered across many different pages.

[cheerfulgreek]

Originally Posted by John

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.







Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.



Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

They seemed to be gone or not as active for a while. Like, when you made those changes last summer, that seemed to work well.

[John]

We've hit a new record of bots on GC (unfortunately).

From the Online Users page:

"Most users ever online was 23274, Today at 08:10 AM."

That stat is over a 15 minute period. So 23,274 "users" from 7:55 to 8:10 AM today. Probably 99% bots.

The previous record was 15,051 back in August 2024.

Originally Posted by cheerfulgreek

They seemed to be gone or not as active for a while. Like, when you made those changes last summer, that seemed to work well.

Some bots are more difficult to deal with than others. Over time some are also getting better at disguising themselves, making it increasingly more difficult to manage.

The bots causing the massive slowdowns on GC leading up to last summer were originating from two specific IP addressed at a cloud web services provider. Those IPs I redirected in a way where it would not be clearly obvious that they were redirected and no longer on GC.

Since then I've redirected many other bots/IPs in a similar way.

Most seem to be originating from China, Singapore and now Brazil.

Not sure yet how I'll be managing this most recent onslaught of bots. The bulk of them seem to have gone away, but could be back at any moment.