The worrying thing here is that so many people accepted outsourcing the decision on who can see their website to a large company (Cloudflare). If the company decides that a certain user should not see the website, the user will not see the website, and no one will know about it, and the user will have no recourse. That is not the open web that I would like to see.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
because they maintain the websites for social status, if the user never sees the specific website or knows they used it, you can't gain any social status
The social contract of accepting scraping for visibility was always tenuous and is now fully dead. But it’s shocking the number of people who are essentially victim blaming here. You should spend YOUR time to optimize your free website so MY use case is not impacted. How is that not incredibly selfish on its face?
Exactly and Cloudflare (there should be others out there too - open source options) has proposed that AI bots need to pay for access to our websites. If they and or others could pull off blocking AI access until it pays creators then AI is forced to pay as it should and always should've!
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
> the real problem is that the website is poorly built
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
We tried to deploy private cloud github enterprise node back in 2018. It was pure garbage without CI integration. I am happy microsoft took harder long term decision!
> My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
There's a lot of people who host their site at extremely expensive places and then do everything they can to minimise unneeded traffic - instead of just moving to a cheaper host. Vercel is another popular extremely expensive host.
I'm always flabbergasted when I see what people pay and how much effort they need to invest to keep their cloud websites from eating them alive.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
Thank you! I need to tighten up my KV compression, which is actually carrying a lot of D1's load otherwise. We also had some bad queries some months, as the pages and database grew, that were counting the wrong things (or extremely inefficiently) and those have since been fixed.
Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers.
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.
It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
It does not even require the PoW thing Anubis does. I've setup a simple logic that just:
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots.
Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
I once discover you can put escaped XML or json in css content. The purpose was to have static data sets that work cross domain. No headers to configure no letting strangers run all you can eat malicious js on your site.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
Similar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
Can someone help me understand the underlying motivation behind this?
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.
I'm in a similar boat. Probably 99.999% is bots. I have nearly 1m unique "visitors" according to cloudflare and my real users are in the dozens a day. That being said, I love the open internet and am holding on to keeping as much open as I can.
It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
> There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
I despise cloudflare. They've decided that my home IP address is bad, so I have to capchas for most websites. Sometimes on infinite loop and I never get to the website. I even reset my home IP address more than once, but it instantly continues. Especially if I use any VPN, even my work VPN.
I run scripts on my servers on an hourly basis to check which are the top 25 IPs visiting the server (aggregated by /24). If anyone in those top 25 IPs are from China, Vietnam, etc, or from Alibaba/Amazon/etc, the /24 gets blocked by iptables.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
Is there any cheap way to run personal websites without getting cooked by bots these days, degrading the performance? A $5/month VPS won't cut it anymore.
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
It's almost always the DBMS that's the bottleneck when websites drop from traffic. As long as you don't do anything fancier than primary key look-ups you're probably fine though.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
For a “personal website” or any relatively simple static content site, the DB should not be getting hit much at all. Cache is your friend. For my sites I have a map of all relevant queries and when we do updates we run the relevant queries one time and cache them to both memory and file system.
Should being the operative word. In practice it's both easy and common to have a database do unnecessary work when generating a page, a problem that often only rears its head when under unusual load.
Yeah, IIRC my django site was 3 queries, all correctly indexed, for a main page hit, and 2 for the actual posts. I don't recall the exact perf numbers, but I'm pretty sure it was easily in the 50/rps range for a small dual-CPU host... which doesn't sound like much in "requests per second" but is enough to cover a front-page-HN'ing just fine. And that "rps" was just the database-backed pages, all the static content was served over nginx, so that 50rps is a "real", 50 humans per second rps, not something getting consumed by only two or three humans.
I host a couple WordPress blogs on Lithium Hosting. It's like $5/mo, fixed cost, and I've never had any issues. If I somehow hit a transfer usage cap, they would just shut my site down until the next month, but I've never had that happen.
Static files on literally any hardware from the last 15 years on modern server software simply won't get cooked by bots. The network switch will bottleneck you before the server will. Your ephemeral port range will run out before the server will.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
Static site. If you do go for a WCM of some sort you should make sure that if it has a calendar view of some sort it isn't unbounded but that it only goes as far back as your first post, and only goes as far forward as your latest. That's one of the many ways the poorly-implemented AI systems get stuck.
Cloudflare offers a free plan that's fantastic. The free plan gives you effectively unlimited DNS/CDN traffic for a normal site, while the main practical cap is 100,000 Worker invocations per day, with 10 ms CPU per Worker request and a 100 MB request body limit. (If that means anything to you).
The next tier up from free is $25/month or $240 per year.
My site's built in Go on a reasonably priced VPS and holds up well. I do geoblock traffic from China and a few other countries as well as IP ranges maintained by Spamhaus.
Yeah, you just make sure your site is fast enough to handle more than 1 RPS.
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
I run my personal website (and a bunch of other websites and services) off a somewhat more expensive but still reasonable VPS (I think 20€ at TransIP - it's so little that I forgot). Load is basically nil most of the time anyway. I think the bot problem is not so bad for personal websites.
A $5/month VPS is fine for mostly static content. I’ve got a site that has been pulling millions of viewers and sits at the top of google for its niche for a decade. It is now hammered with bot traffic constantly. But everything is cached so it’s fine.
I have a low cost VPS (actually two in two different pre-AI datacenters) and it runs fine. I use nginx to serve the web pages, and the content is about 99% static content.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
Cloudfare has a very generous plan usually for hobby projects. The article did mention that cloudfare did not work for them, but they have recently introduced few features to block AI crawling as well.
All of my boxes are cheap VPS. Highly recommend people throw Cloudflare in front of their stuff. I switched all of my load balancers over to there and all of that bot crap went away. That combined with a proper UFW setup keeps the weather clear for me.
I'm on a $4/mo droplet on Digital Ocean and my static site has been just fine. I'm using Caddy and it seems to handle the load like a champ. My site is very lightweight though, so ymmv.
As someone who is moving my static sites over to GitHub pages: While they are free and work really nice, the problem is that GitHub frequently doesn’t deploy updates to the pages.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:
2 access.log [1]
40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.
Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
I thought about that but to your point CSS is cachable. In fact I made mine immutable. No I just visually spot patterns and use that to study other facets of the agent, other client headers or lack thereof, supported protocol, accepted encoding and so on.
Maybe it is enough to include some css/js which is served without cache and is loaded after all user visible css/js is loaded. Make it small enough to not cause too much bandwidth for the server and legitimate clients. Then anyone who doesn’t hit that CSS file gets banned.
blocking by geo yeah might work - but what happens when someone is traveling abroad ? they've to use a VPN to access your site ?
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
I'm working on a web app right now, with the intention of it going to be 100% paywalled. It's 95% complete, but the remaining 5% is just implementing the paywall. In the meantime, the app is live and operational with a fully functional signup. I am deleting about 100 new bot signups per day right now, it is crazy out there.
Sigh .. same here. I don't write blog posts often (enough) but the ones I do write are from personal experiences and I take a lot of care with them. I look at my logs snd see bots everywhere, but now I just let them get on with it. AI scrapers are different though; they get to read my content which, just for them contains a smsttering of finest digital toxin. A pox on your datasets!
I'm blocking the Amazon search crawler, anything coming from Googleusercontent, and limiting AI crawlers to search rather than allowing AI assistants. The residential proxy waves are something to behold, but Cloudflare does an okay job catching those in the AI labyrinth. Still, it's all a bit silly, and I can't imagine what large sites deal with when I'm tangoing with this much nonsense on a small tech blog.
The annoying part is a large percentage of misbehaving bots (not obeying robots.txt for example) are via end user proxies across the world. However most of these aren't doing full-browser loop, so if you are behind cloudflare, you can do non-interactive challenge and that can help quite a bit.
Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club, there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in the DOJ ordering Google to share their crawl data with qualified competitors (a work in progress, but it's progressing).
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.
It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
This is a static website running on Cloudflare infra. What on earth costs $90 per month? First optimize your infra before throwing up rules in front of your visitors. I have several websites on Cloudflare too and I don't even check how many million requests I get. Because it does not cost me anything.
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer. That's usually the easiest way I spot a bot.
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
Glad to hear you’re willing to accept feedback. You maybe don’t have ADHD and sorry for continued advice but you shouldn’t assume you have undiagnosed conditions IMO as it allows you to defer your mistakes from the self. Even if you do have ADHD you can still correct and understand good article structure. I highlighted the last sentence of that paragraph because what is clear from the split sentence style is that you’re writing for impact, this lends well to 140 characters but falls apart in longer form writing but as I stated the sentence is still impactful as you’re saying “but wait…there’s more to this!” Which is very intriguing. I think if you’re comfortable writing that way and it helps you split your ideas and sentences up so each one is impactful, that’s a good thing. But consider that style as a draft and then you can go back and group up your impactful ideas into paragraphs very easily.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
That's the thing: even if you don't use AI, if most of what you read is AI, soon this is what you'll sound like, just because it's so much of the training data your own brain's model has.
Using AI to write blogs doesn’t bother me in principle, but I hate the prose that gets left in. Even if it’s not AI writing, being human doesn’t give you a pass for writing like a self-help guru. This sort of grammar is straight out of Claude:
> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.
I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The <adjective> <noun phrase>”.
Using a global agent rule with something like the following makes agent output much more tolerable for me:
"Use only ASD-STE100 Simplified Technical English when communicating with me or writing documents, comments, and other communications intended for humans."
As a result, I've never seen Claude or Codex use it's weird terms like "load-bearing".
As odd as this sounds, and as odd as the world gets, I still find a small comfort in the fact that there is enough stability in this world that I can count on one thing: someone always accusing a blog post on HN as being AI generated.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
No; in this case you are not a user, you are a bot user.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
And if not, what if it was a custom browser and not Chrome/Firefox/Edge?
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
I feel strongly about this topic and wrote about it in June on my Substack (even wrote an AI protest real song about it) https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
[0]: https://www.githubstatus.com/incidents/qcvjkzcs7j74
Yes, but no.
you literally can turn off cloudflare and use your own solution
Cloudflare is opt-in so I don't see that being an issue (yet).
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
[1] https://anubis.techaro.lol/
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
Then they'd be blocking themselves from the website.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver.
[1] https://news.ycombinator.com/item?id=48869064
Loot first, ask questions later.
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.
And a lot of bad scrapers.
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
https://commoncrawl.org/
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
[1] https://news.ycombinator.com/item?id=28550764
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
If you can't, well then... We're stuck.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
[1] - https://nochan.net/logs/access.log
[2] - https://nochan.net/logs/botpoop.log
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
https://developers.cloudflare.com/bots/additional-configurat...
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
GPTBot is OpenAI, ClaudeBot is Anthropic, CCBot is Common Crawl, Google-Ext is a way for website owners to indicate they don't want their content to be used for AI, Bytespider is Bytedance, Bing and Google are the last two. Take these numbers with a truck of salt, haven't had time to verify them.It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.
Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
https://patronview.com/news/99-percent-of-my-website-traffic...
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
https://news.ycombinator.com/item?id=49035436
Anyways I will finish reading your article now since you’re so wonderful to take a bit of feedback and be proactive.
Best thing I've read on HN so far this year.
So this is how AI wins, humans kill each other off because we might be bots and the bots inherit the earth.
(This isn't a condemnation. AI can often do a better job of representing thoughts than humans.)
Examples:
- The article organization
- The general language flow
- The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats).
- The images are almost certainly AI generated. They look AI generated.
Using it slowly sucks the uniqueness out of you.
> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.
I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The <adjective> <noun phrase>”.
"Use only ASD-STE100 Simplified Technical English when communicating with me or writing documents, comments, and other communications intended for humans."
As a result, I've never seen Claude or Codex use it's weird terms like "load-bearing".
(i skimmed the whole post, there are none)
Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.
Never fails.
The OCD part of me can now go about my day.