I have been building platform teams for a while now, long enough to have lived through the shift from DevOps to Platform Engineering firsthand. I think the same thing is starting to happen again, and this time it is bigger.

Platform Engineering is splitting, and the new piece is something I have been calling AI Platform Engineering. If you work in a regulated industry like healthcare, it is something you can’t really afford to ignore.

#We have seen this movie before

Remember when “DevOps” was the answer to everything? Every ops engineer became a DevOps engineer overnight, and for a while the title changed but the work stayed the same. Then the real shift happened slowly, as teams started building internal developer platforms with golden paths and self-service infrastructure, and the role turned into something new.

I am seeing that same pressure build again. AI brings a different set of problems than the workloads we are used to deploying: model governance, prompt injection, data privacy boundaries, inference costs, and compliance frameworks that regulators are still writing as we go.

Most traditional platform teams aren’t set up to handle all of that, and I don’t think they should have to be.

#What AI Platform Engineering actually looks like

An AI Platform team doesn’t train models, so it isn’t a machine learning team. It builds the guardrails, pipelines, and self-service tooling that let every other engineering team use AI safely and quickly.

In practice, that means owning things like:

• LLM gateways: a central proxy that handles authentication, rate limiting, cost tracking, and content filtering for every model call in the org, so engineers aren’t copy-pasting API keys into random services.
• Prompt management and versioning: treating prompts like infrastructure that is version controlled, tested, and reviewed instead of scattered across application code.
• Guardrails as code: input and output validation that is defined in one place and enforced automatically. In healthcare this is required, because a model hallucinating medical advice is a compliance nightmare.
• Model routing and fallback: hiding which model is being called so teams can swap providers, manage costs, and ride out outages without code changes.
• Audit and observability: logging every model interaction so it can be traced and queried. When a regulator asks what your AI told a patient, you need an answer in minutes, not weeks.

I am seeing teams try to solve all of these right now with duct tape and hope.

#Start with the LLM gateway

If you only build one thing, build the gateway. It covers most of that list in a single layer, and it is the same pattern we used with API gateways a decade ago, just pointed at model access.

The flow is pretty simple. An engineer’s application sends its request to the gateway instead of directly to OpenAI or Anthropic, and the gateway handles everything in between.

On the way in, it authenticates the caller, checks rate limits, scans the input for PII or PHI, and applies content policies. Then it picks which model to send the request to based on the use case, cost, or availability. On the way out, it filters the response, logs everything, and tracks token usage by team. That one config replaces dozens of decisions every team would otherwise be making on their own.

The tooling is getting better quickly too. There are open source LLM gateways that run natively in Kubernetes, implement the Kubernetes Gateway API, and handle agent-to-tool traffic (via MCP) and agent-to-agent traffic (via A2A) as well as agent-to-LLM calls. The better ones let you define routes so different teams hit the same gateway but get different policies. Your clinical team’s traffic can go through a PHI guardrail before it ever reaches the model, while your internal tooling team gets a lighter policy. API keys live in Kubernetes secrets instead of application code, every interaction is traced with OpenTelemetry, and if one provider goes down, traffic fails over to another.

So with one gateway you get authentication, model routing with fallback, content guardrails, and a full audit trail. Nobody is managing their own API keys or writing their own retry logic, and nobody is accidentally sending PHI to a third-party model without filtering it first.

#Why regulated industries cannot wait

Earlier this year, Chipotle’s customer service AI agent got jailbroken. Someone nudged it off task and had it answering coding questions instead of helping with burrito orders. For Chipotle it was a minor embarrassment and some wasted tokens, and they patched it and moved on.

Now picture that same failure in healthcare. A patient-facing AI agent with no guardrails gets talked into giving medical advice it was never designed to give. That would be a HIPAA violation, a liability problem, and possibly a patient safety issue, and nobody would be sharing it as a funny LinkedIn post.

That is why I think AI Platform Engineering matters most in regulated environments. The guardrails can’t be bolted on after the product team ships something. They need to be part of the platform from day one. Chipotle learned that lesson with a PR blip, and in healthcare it would cost a lot more.

Teams that figure this out early end up moving faster, because their engineers aren’t scared to use AI. They know what is allowed, what is logged, and what is blocked.

Teams that wait end up with shadow AI: engineers using personal API keys, and prompts with PHI flowing through third-party services nobody vetted. It is the same shadow IT problem we dealt with a decade ago, with higher stakes.

#Where this is heading

By 2027, I think these circles will have pulled a lot closer together. Security teams will need to understand AI-specific threat models, AI Platform teams will need deep infrastructure and networking knowledge, and traditional platform engineers will need to understand model serving, GPU scheduling, and inference optimization. The teams that see that coming and start cross-training now will be in a much better spot.

#What to do about it

If you are leading a platform team today, start by treating AI infrastructure as its own problem space instead of handing it to your existing platform engineers and hoping for the best.

Start small. Stand up an LLM gateway, centralize your API keys, build an audit trail, and get your security team involved early. None of that is a huge investment, but it gives you something to build on.

If you are in healthcare or another regulated space, I would treat this as urgent. The rules for AI compliance are being written right now, and the organizations that get the patterns right early will end up shaping what everyone else follows.

#Where does your team stand?

Here are the questions I would ask about my own team:

• Do you have a centralized way to manage LLM API keys, or are they scattered across teams and services?
• Can you tell a regulator exactly what your AI said to a user last Tuesday at 2pm?
• Do you have input/output guardrails that are enforced automatically, or is each team rolling their own?
• If your primary model provider goes down, can your applications fail over without a code change?
• Do you know how much you spent on inference last month, broken down by team?
• Has your security team reviewed your AI attack surface, like prompt injection, data exfiltration, and model abuse?
• Can a new engineer on your team ship an AI-powered feature without copy-pasting API keys or writing their own retry logic?

If you answered no to more than two of these, what you have is a collection of individual experiments more than an AI platform. That is fine, and most teams are there right now, but the gap between experimenting and actually governing it is where the risk is.

The move from DevOps to Platform Engineering took years. I expect this one to go faster, because AI adoption is moving too quickly for it not to.