Hi all,
For reasons unclear, some bot has been absolutely hammering this site all month, consuming vast amounts of bandwidth. It's probably some AI model, I suspect.
I have made some changes to try and deny it access. If you encounter any weirdness with your access to the site, please let me know.
(And yes, I appreciate depending on the weirdness, you might not be able to access the site to let me know.)
Apologies in advance if there are any unforeseen impacts, but the current consumption is absolutely unsustainable.
cheers
RET
. . . that'd be why I've been seeing Error 509 Bandwidth Limit Exceeded from time to time.
I will exercise patience.
Rob
Well, that's the effect, the "nbot" crawler is the cause.
The forum usually runs to about 20GB/month bandwidth. This month it's pushing 60GB.
Despite pushing the bandwidth limit on this site up from 30GB to 60GB, it still blew its top on 31 May. Apologies anyone who tried to access the site yesterday and couldn't, but I was unable to resolve it on the spot.
I've implemented some changes to try and block the bots, so we'll see how effective that is this month, I guess.
Just an update for the nerds interested.
There were two culprits:
- The one that looked like 'nbot' turned out to be Amazon, indexing the content for its stupid Alexa gizmo. If anyone has one of these devices and is asking it questions about FE/FC Holdens, sorry about that. It's no longer crawling the site.
- Facebook - seems to be related to people posting links to stuff on the forum to Facebook groups, although the rate that occurs (that I see) doesn't scale with the amount of traffic. Anyway, they're blocked too.
It doesn't look like the bandwidth limit will blow this month.