Bots flooding GC

[cheerfulgreek]

Originally Posted by John

Bots are flooding GC again. Nearly 6,500 online users right now...

There are always bots, search spider bots are fine & not a problem. This current flood, though, is probably the AI scraper (data harvesting) type again.

Will try to get things sorted out with this batch of bots before they cause too much trouble...

Thank you for everything, John.

I thought you made it where it’d be harder for bots to attack the site. Did they manage to get around it?

[John]

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.

Originally Posted by cheerfulgreek

Thank you for everything, John.

Originally Posted by cheerfulgreek

I thought you made it where it’d be harder for bots to attack the site.

Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.

Originally Posted by cheerfulgreek

Did they manage to get around it?

Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

[carnation]

I noticed that the majority of them were looking at the same thread.

[John]

Originally Posted by carnation

I noticed that the majority of them were looking at the same thread.

Yeah, it was an old thread, greeks in the family , only 14 pages... shouldn't take the bots 100+ thousand page views throughout the day to spider/scrape that thread. Somehow they sometimes get hung up and just keep going on the same thread.

Last summer was a similar issue. Bots from 2 IP addresses loading all the pages from the print version of the original "What do you feel like saying, right now?" thread. Those bots were loading something like 50 to 100 million pages per month from that thread. They were a major contributing factor causing the site to slow down so much for a while up until then.

The bot flood back in the fall I think was mostly scattered across many different pages.

[cheerfulgreek]

Originally Posted by John

The bots that were causing trouble today were sorted out earlier this evening. Highest I saw peak online hit today was just under 8,000.







Partially, but not completely. It is harder for bots to continually cause trouble, but still plenty of room for improvement. When I spend more time working on GC projects again I'll button things down even more with regards to bot mitigation.



Sort of. We always have bots on GC. Some aren't programmed well, though, maybe due to software bugs, and they go a bit haywire while scraping or spidering sites. We're not blocking all bots but when they start causing problems I now at least have some ways of dealing with it. Imperfect solution that will be improved in the future.

Not sure yet but I've been thinking about spending more time in May and/or June on GC projects again, maybe improving bot mitigation will be included if I do that.

They seemed to be gone or not as active for a while. Like, when you made those changes last summer, that seemed to work well.

[John]

We've hit a new record of bots on GC (unfortunately).

From the Online Users page:

"Most users ever online was 23274, Today at 08:10 AM."

That stat is over a 15 minute period. So 23,274 "users" from 7:55 to 8:10 AM today. Probably 99% bots.

The previous record was 15,051 back in August 2024.

Originally Posted by cheerfulgreek

They seemed to be gone or not as active for a while. Like, when you made those changes last summer, that seemed to work well.

Some bots are more difficult to deal with than others. Over time some are also getting better at disguising themselves, making it increasingly more difficult to manage.

The bots causing the massive slowdowns on GC leading up to last summer were originating from two specific IP addressed at a cloud web services provider. Those IPs I redirected in a way where it would not be clearly obvious that they were redirected and no longer on GC.

Since then I've redirected many other bots/IPs in a similar way.

Most seem to be originating from China, Singapore and now Brazil.

Not sure yet how I'll be managing this most recent onslaught of bots. The bulk of them seem to have gone away, but could be back at any moment.

[honeychile]

Thank you so very much, John, for keeping GC up to date with the Battle Against the Bots!

[cheerfulgreek]

Originally Posted by John

We've hit a new record of bots on GC (unfortunately).

From the Online Users page:

"Most users ever online was 23274, Today at 08:10 AM."

That stat is over a 15 minute period. So 23,274 "users" from 7:55 to 8:10 AM today. Probably 99% bots.

The previous record was 15,051 back in August 2024.



Some bots are more difficult to deal with than others. Over time some are also getting better at disguising themselves, making it increasingly more difficult to manage.

The bots causing the massive slowdowns on GC leading up to last summer were originating from two specific IP addressed at a cloud web services provider. Those IPs I redirected in a way where it would not be clearly obvious that they were redirected and no longer on GC.

Since then I've redirected many other bots/IPs in a similar way.

Most seem to be originating from China, Singapore and now Brazil.

Not sure yet how I'll be managing this most recent onslaught of bots. The bulk of them seem to have gone away, but could be back at any moment.

Oh wow! How can you tell which countries they originate from?

John, where’s the coffee thread? I want to buy more coffee. You work so hard to keep this site going and I just want to thank you in the best way I possibly can.

[John]

Originally Posted by honeychile

Thank you so very much, John, for keeping GC up to date with the Battle Against the Bots!

You're welcome and thank you for helping to support the efforts with your GC subscription.

"Battle Against the Bots" for some reason reminded me of a TV show I used to watch some years ago called BattleBots: https://battlebots.com/ I used to love that show. Maybe I'll start watching again and when they're bashing each other to pieces I'll imagine they are the internet bots that were attacking GC.

Originally Posted by cheerfulgreek

Oh wow! How can you tell which countries they originate from?

Usually when there's a number of similar IPs on GC that look suspicious I'll run a few through the search at https://www.arin.net/ and ban the entire IP range.

Sometimes the top level domain country code is in the IPs reverse DNS as well.

However, there's no guarantee that the actual source is from those countries. Could be they are just renting servers from datacenters / cloud server providers in those countries or those IPs could be from some sort of bulk proxy / VPN type services with IP addresses assigned to organizations in those countries.

I suppose it may also be possible that those IPs are registered to orgs in those countries but being used somewhere else.

Originally Posted by cheerfulgreek

John, where’s the coffee thread? I want to buy more coffee. You work so hard to keep this site going and I just want to thank you in the best way I possibly can.

Thanks, I appreciate it.

The original "Buy GreekChat a Coffee" thread is at https://greekchat.com/gcforums/showthread.php?t=248704

And the direct link to the Buy Me A Coffee page for GC is https://buymeacoffee.com/greekchat

[SAEalumnus]

Originally Posted by honeychile

Thank you so very much, John, for keeping GC up to date with the Battle Against the Bots!

Should we start referring to all of these bots now as Decepticons? Seems fitting.

[cheerfulgreek]

Originally Posted by John

You're welcome and thank you for helping to support the efforts with your GC subscription.

"Battle Against the Bots" for some reason reminded me of a TV show I used to watch some years ago called BattleBots: https://battlebots.com/ I used to love that show. Maybe I'll start watching again and when they're bashing each other to pieces I'll imagine they are the internet bots that were attacking GC.



Usually when there's a number of similar IPs on GC that look suspicious I'll run a few through the search at https://www.arin.net/ and ban the entire IP range.

Sometimes the top level domain country code is in the IPs reverse DNS as well.

However, there's no guarantee that the actual source is from those countries. Could be they are just renting servers from datacenters / cloud server providers in those countries or those IPs could be from some sort of bulk proxy / VPN type services with IP addresses assigned to organizations in those countries.

I suppose it may also be possible that those IPs are registered to orgs in those countries but being used somewhere else.

Aye yai yai, so much terminology. Do you know what it is that they want from this website?

[John]

Originally Posted by cheerfulgreek

Aye yai yai, so much terminology. Do you know what it is that they want from this website?

Data, for the most part. My guess is most of the current bots are here to harvest data for AI training.

There are also spambots as well as they still seem to be registering accounts but cannot jump through all the hoops I setup so are unable to post their spam. That setup makes it a pain for new GCers to join us, but it helps block most of the spam.

Originally Posted by SAEalumnus

Should we start referring to all of these bots now as Decepticons? Seems fitting.

Yeah, we probably should.

Autobots, roll out.