Own Your AI Systems or Become Their Tenant

The Security Check That Protected Someone Else's Files

Two years ago, I stopped renting my AI and built my own. This week I discovered it had been quietly protecting the wrong person for months.

I run my personal productivity stack on an open-source AI system I forked in 2022. The memory layer, the agent logic, the decision tree that determines what my assistant actually does—I own all of it. Not because I'm an AI purist, but because I got tired of SaaS products changing their terms, sunsetting features, and making decisions about my data that I only learned about from changelog emails.

When the upstream project shipped a new release this week, I did what you'd do with any vendor upgrade: audited the diff to see what improvements I could backport. I expected to find a dozen ideas worth stealing.

I read through 14,000 files and couldn't find a single one.

Not because the original developers aren't talented—they are. But because over two years of daily use, every time something broke on a Tuesday morning, I had to crack it open and actually understand it to fix it. That forced intimacy made my fork better for me than a system engineered by people smarter than me but not living in my specific workflow.

Then I found the part that matters for this post.

The Inherited Default Nobody Audited

Buried in a module I'd modified six months ago: a security check validating file permissions. Very thorough. Logged detailed access controls. Protected the original author's name and directory structure.

Not mine. His.

I'd been running production security code designed to protect someone else's assets. It looked like protection. It compiled cleanly. It never threw an error. It just wasn't doing anything useful for me—because I'd inherited it as a default and never interrogated what it actually secured.

That's the expensive kind of assumption. The kind that looks fine until the day you actually need it.

Every Firm Made This Bet with the Cloud

I've watched this movie before. In 2008, I was advising financial services clients through their first cloud migrations. The pitch was elegant: let someone else handle infrastructure so you can focus on your business. Rent the servers, rent the security, rent the expertise.

Fifteen years later, those same clients call me when their cloud bill quintuples overnight because a developer spun up resources nobody understood, or when they discover compliance controls they assumed were active were actually optional features nobody configured.

Nobody gets fired the day you move to the cloud. The bill just quietly grows while your understanding quietly shrinks.

The railroad analogy fits. Towns that built next to the tracks thrived. Towns that assumed the railroad would always stop there—just because it did last year—eventually discovered that infrastructure owners optimize for their economics, not yours. The defaults that worked when you were strategically important stop working when you're not. But you don't find out until the train stops stopping.

Cloud taught us to trade understanding for convenience. AI is the same trade, just faster.

The Demo That Hides the Defaults

Right now, you're evaluating AI tools that will route customer inquiries, draft contract language, flag compliance exceptions. The demos look great. The accuracy benchmarks are impressive. The UI is cleaner than anything your team could build in-house.

So you buy it. You integrate it. You assume.

Here's what you probably don't know:

• What training data the model excludes (and whether that creates blind spots in your domain)

• How the system behaves when it encounters ambiguous input that looks clear

• What the fallback logic does when confidence scores drop below threshold

• Whether the "explainability" feature is showing you actual reasoning or plausible post-hoc justification

I'm not arguing you should build your own large language models. I'm saying a system you don't understand isn't working for you—it's just working near you.

That security check in my code wasn't malicious. It was just optimized for someone else's threat model. The risk isn't that your AI tools are designed badly. It's that they're designed well—for use cases and constraints that don't perfectly map to yours.

And you won't know where the gaps are until you go looking.

What Happens When Nobody's Watching?

Last month I worked with a client deploying an AI assistant to help their audit team flag unusual transactions. Smart tool, reputable vendor, solid accuracy in testing. Three weeks into production, someone noticed it was systematically under-flagging transactions in a specific subsidiary.

Turns out the training data had very few examples from that entity type. Not zero—enough that nobody noticed in QA. Just sparse enough that the model had learned conservative thresholds. It was making fewer mistakes, technically. Also catching fewer problems.

The system was working exactly as designed. The design just didn't fit the actual risk profile.

They only caught it because someone manually audited a week's worth of flagged items and got curious about what didn't get flagged. How many firms have that person? How many have the time?

When I built my AI fork, I broke it constantly. Every modification introduced new failure modes I had to understand to fix. Painful, time-consuming, inefficient by every productivity metric.

Also the reason I trust it.

The Tenant Living in Your Stack

Here's the uncomfortable question I don't have a clean answer for: how do you validate systems too complex to fully audit but too critical to blindly trust?

You can't realistically fork and self-host every AI service you use. You can't hire enough ML engineers to reproduce vendor capabilities in-house. The "just build it yourself" advice doesn't scale beyond personal productivity tools and niche applications.

But you also can't just assume that vendor defaults align with your risk tolerance, your compliance obligations, your operational reality.

The middle path—the one nobody's figured out yet—is building organizational muscle for interrogating black boxes without opening them. That means:

Adversarial testing. Not "does this work?" but "how does this fail?"—and specifically testing the edge cases that look like your actual data, not the vendor's benchmarks.

Instrumentation you own. If you can't see inside the model, instrument everything around it. Log the inputs, log the outputs, log the contexts where human operators override the AI. Build your own understanding from behavioral patterns.

Humans who've earned skepticism. Put people who've survived previous automation cycles in the review chain. Not to slow things down—to ask the annoying questions about inherited defaults before they're buried in production.

The last part matters most. The person who found that audit gap? She'd been manually flagging transactions for eight years before the AI showed up. She knew what normal looked like.

But what do I know—I've only watched firms rent their infrastructure three times and promise this time they'll maintain the expertise to audit it.

What to Actually Do Monday Morning

If you're deploying AI systems in production, here's what to ask your team:

"Show me what this does when it's wrong." Not the accuracy rate. The actual failure mode. What does a missed fraud alert look like? What happens when the contract AI halves a term instead of flagging it as ambiguous?

"What did we inherit that we haven't validated?" Every AI tool ships with default thresholds, training exclusions, fallback behaviors. Which ones did you consciously choose, and which did you just accept?

"Who on our team can explain why this made that decision?" Not how transformers work. Why this specific output happened for this specific input. If the answer is "we'd have to ask the vendor," you have a tenant, not a tool.

I'm not anti-AI. I run my entire workflow on it. I'm anti-assumption.

That security check protecting someone else's files wasn't a catastrophic failure. It was a small thing that didn't matter—until someday it would. That's what inherited defaults do. They look like protection and do nothing for you, right up until the moment you actually need them.

The question isn't whether to use AI. It's whether you're going to understand the systems making decisions in your name—or just assume someone else thought through the edge cases.

I know which bet I'd make. I also know which bet most firms are actually making.

The gap between those two is where the next expensive lesson lives.