The Tollbooth Always Wins: How Open Protocols Close Without Anyone Noticing

You can still run your own email server. It just won't reach anyone.

I walked a client through this last year. Mid-sized professional services firm, wanted to bring email in-house for compliance reasons. We built it right—proper authentication, clean IPs, the works. Technically flawless. Gmail routed 60% of their outbound to spam anyway. Microsoft was worse. Three weeks later they were renting deliverability from Google Workspace, paying the company that had just blocked their mail.

The protocol is still open. The access is gone. And what just happened to email is now happening to the rest of the open internet, just faster.

Cloudflare Just Closed the Web to AI Crawlers

On July 1, 2025, Cloudflare flipped a switch. AI crawlers—the bots that training models rely on—went from default-allowed to default-blocked for every new site and every free-tier customer. They stood up a marketplace where bot operators pay per crawl.

This isn't a feature. It's policy. Cloudflare sits in front of roughly 20% of the web. When they change the default, they're not offering a configuration option—they're rewriting the rules of access for a fifth of all websites.

The email parallel isn't subtle. SMTP is still an open protocol. Anyone can run a mail server. But Gmail and Microsoft gatekeep delivery now. They demand authentication (SPF, DKIM, DMARC), they track sender reputation, and as of last year they reject non-compliant mail before it arrives. So everyone serious rents their deliverability from a provider.

Every open protocol that gets abused at scale eventually grows a tollbooth. Email took twenty years to lock down. Web crawling is moving in months.

The Operator Is Never the Content Owner

Here's the part that should make you uncomfortable: the people installing the tollbooth aren't the ones who created the value being protected.

Email server operators didn't ask Google to police spam on their behalf. Website owners didn't petition Cloudflare to monetize their crawl traffic. But once abuse hits scale—once spam overwhelms inboxes or AI scrapers hammer servers—someone has to solve it. And the entity positioned at the choke point gets to define the solution.

The content owner gets protected. The intermediary gets paid. The protocol stays "open" in theory while access closes in practice.

I watched this happen with payment processing in the early 2000s. Technically, anyone could accept credit cards. Practically? Visa and Mastercard set the rules, processors collected the fees, and small merchants either paid up or went cash-only. The rails were "open." The access required permission. Same movie, different industry.

The Uncomfortable Question About Who Gets In

Once access gets metered, the incentive structure inverts.

When email was truly open, a teenager could spin up a mail server in their bedroom and reach any inbox on the internet. When crawling was truly open, a researcher with a laptop could index a meaningful slice of the web. Open protocols democratized access. Tollbooths sort by ability to pay.

Cloudflare's marketplace won't keep out OpenAI or Google. They'll pay. The companies training frontier models have capital and compute budgets that make per-crawl fees a rounding error.

The researcher building a specialized model for radiology reports? The startup trying to index public records? The non-profit archiving climate data? They get a 403 error—or they get a bill they can't afford.

I'm not saying Cloudflare is wrong here. Server costs are real. Scraper abuse is real. The tragedy of the commons is real. But let's be clear-eyed about the second-order effect: every access control mechanism favors incumbents over newcomers. That's not a bug. That's how tollbooths work.

The Faster Clock

What took email twenty years is happening to web crawling in months.

The gap between "open protocol" and "metered access" used to play out over a generation. Long enough that we'd forget the before-state. Long enough that the shift felt inevitable rather than chosen.

AI collapsed the timeline. Training runs need petabytes of text. Crawlers hit rate limits designed for human browsing. Infrastructure buckles. Intermediaries step in. Tollbooths appear.

I've now watched this cycle four times: email in the 2000s, API access in the 2010s, payment rails before that, and now crawling. The clock keeps accelerating. The pattern doesn't change—abuse at scale forces gatekeeping—but the window between "open" and "closed" keeps shrinking.

Which means if your business model quietly depends on open access to some protocol or platform, you have less time than you think to build an alternative or negotiate terms.

The Next Tollbooth in Line

So here's the question I've been asking clients: which open channel does your business rely on that's next in line?

RSS feeds that aggregate data for your dashboards? Public APIs you scrape for competitive intelligence? Blockchain nodes you query without running infrastructure? Open-source models you fine-tune without paying license fees?

Every one of those channels works—until abuse hits scale, until the economics shift, until the intermediary that sits at the choke point decides the old rules don't work anymore.

The protocol stays open. Access becomes conditional. And if you can't pay the toll or don't have the negotiating leverage, you're building on a platform that just became someone else's business model.

What to Do Monday Morning

I'm not here to tell you Cloudflare is the villain. They're solving a real problem. I'm here to tell you the solution follows a pattern, and the pattern has predictable consequences.

Here's what to ask your team this week:

• What external services do we depend on that we don't pay for? Free tiers are options, not promises.

• Where are we downstream of an intermediary with market power? If one company controls the choke point, they control the terms.

• What's our plan if access gets metered or cut off? Hope is not a strategy. Diversification is.

The open internet isn't dying. But openness without enforcement creates conditions for abuse, and abuse at scale creates conditions for gatekeeping.

Nobody gets fired the day the tollbooth goes up. Your emails just stop landing. Your crawlers start getting blocked. The service you built on slowly stops working.

Pay attention to where the tollbooths are being built. By the time you see the gate come down, the terms have already been set.

If you've seen this pattern play out in another protocol or platform, I want to hear about it. What open system did your business rely on that quietly closed? Reach out on LinkedIn or reply here—I'm building a catalog of these transitions because the next one is already in motion.