Skip to content
HomeAt Splunk, that means pairing React product work with Python services, operational data, permissions, evaluation, and release controls. Earlier at Deloitte, I built extraction, storage, API, and visualization systems; two removed more than 36,000 hours of annual manual work and cut investment-processing time by 66%.WritingTechnical writing on software engineering, AI engineering, system design, and the trade-offs behind production systems.ProjectsA growing collection of systems and products I’ve built—from current infrastructure work to earlier projects that shaped how I build today.Abdul AhadI’m an AI full-stack engineer with more than five years of experience building data-intensive enterprise software and production AI systems. At Splunk, I drove AI Service and KPI Discovery from an early prototype through technical design, validation, and production delivery. The capability analyzes operational data and recommends service models and health signals within Splunk IT Service Intelligence.Let’s talk.If you’re hiring, building something interesting, or want to compare notes on a difficult engineering problem, send me a message. A few lines is plenty.ColophonHow this portfolio handles static publishing, search, contact delivery, analytics, and releases.Capacity Planning Before Architecture DiagramsA practical method for turning workload assumptions into request, storage, bandwidth, concurrency, and failure-capacity budgets before choosing an architecture.Design Modules Around Change, Not LayersA practical guide to information hiding, volatile decisions, coupling, cohesion, dependency direction, and avoiding abstractions that preserve the wrong boundaries.From Model Call to Reliable AI WorkflowHow one model request grows into retrieval, tools, deterministic workflows, agents, evaluation, observability, and human escalation.Queues, Backpressure, and Idempotency: Designing for FailureHow bounded queues, admission control, idempotency keys, retry budgets, and dead-letter handling turn asynchronous delivery into a controlled reliability system.Refactoring a Legacy Backend Without a Big-Bang RewriteA staged method for understanding, testing, isolating, replacing, observing, and finally deleting legacy backend behavior without one irreversible cutover.MCP in Production: Trust Boundaries, Permissions, and Tool DesignA production-focused guide to MCP hosts, clients, servers, capability discovery, authorization, consent, token audiences, and safely designed tools.How this portfolio worksA walkthrough of how an edit becomes a published page, what happens after someone presses Send, and how the site avoids losing or saving the same message twice.ThreadsPosts, nested replies, profiles, and communities share the same records. This follows one conversation from creation to deletion.Prompt NexusGoogle sign-in leads to a local user, then one saved prompt powers the feed, profile, search, and edit screens.SeedEditors publish articles in HyGraph, readers receive prepared pages, and new comments remain private until they are approved.CactaOne published video creates a media asset, a post record, and temporary browser state. This follows that upload across the product.System DesignCapacity, queues, failure handling, and distributed-system trade-offs.AI EngineeringLLM workflows, tools, evaluation, permissions, and operations.Software EngineeringModules, testing, refactoring, interfaces, and maintainability.System Design from First PrinciplesCapacity planning, queues, backpressure, retries, and failure handling.Production AI EngineeringModel calls, workflows, tools, evaluation, permissions, and operations.

← Back to Writing
AI Engineeringarticleintermediate10 min read

MCP in Production: Trust Boundaries, Permissions, and Tool Design

A production-focused guide to MCP hosts, clients, servers, capability discovery, authorization, consent, token audiences, and safely designed tools.

MCPAIAgentic AISecurityArchitecture

Series: Production AI Engineering2/2

On this page

The Model Context Protocol standardizes how AI applications discover and invoke external capabilities. Standardization reduces integration work; it does not reduce the authority those integrations carry. An MCP server can expose customer records, issue tracker data, file operations, or production actions. Connecting it to an AI host creates a security boundary that deserves the same discipline as any other privileged API.

The useful question is not “Does an MCP server exist for this system?” It is “What precise authority should this host receive, how is that authority enforced, and how can an operator reconstruct what happened?”

The architecture in one view#

MCP uses a host-client-server model:

  • The host is the AI application responsible for user interaction, model orchestration, consent, and policy.
  • The host creates an MCP client for each server connection.
  • An MCP server exposes capabilities through protocol primitives.
View diagram source
flowchart LR
    U[User] --> H[AI host]
    H --> M[Model and policy layer]
    H --> C1[MCP client: issue tracker]
    H --> C2[MCP client: database]
    C1 --> S1[MCP server A]
    C2 --> S2[MCP server B]
    S1 --> A1[Issue tracker API]
    S2 --> A2[Read replica]

Each client maintains a dedicated connection to its server. The host combines the discovered capabilities into the model’s available context, but the servers remain separate trust domains.

Understand the primitives#

MCP servers can expose three core server primitives:

  • Tools: executable operations the model may request.
  • Resources: contextual data the application can retrieve.
  • Prompts: reusable interaction templates.

These primitives have different risk profiles. Reading a public schema is not equivalent to executing a deployment. Treating every capability as a generic “tool” loses information that should drive consent and policy.

Capability discovery is not authorization. tools/list tells a client what a server says it can do. The server must still authenticate and authorize every actual request.

Draw the trust boundaries first#

A production design commonly includes five principals:

  1. The human user.
  2. The AI host.
  3. The model provider or runtime.
  4. The MCP server operator.
  5. The upstream system behind the server.
View diagram source
flowchart TB
    subgraph UserDevice[User-controlled boundary]
        U[User]
        H[Host]
        C[MCP client]
    end
    subgraph ServerBoundary[MCP server boundary]
        S[Server]
        P[Policy enforcement]
        L[Audit log]
    end
    subgraph Upstream[Upstream system]
        R[Protected resources]
    end
    U --> H
    H --> C
    C -->|scoped token + request| S
    S --> P
    P --> R
    P --> L

For each arrow, document:

  • The identity being represented.
  • The data crossing the boundary.
  • The credential used.
  • The policy enforcement point.
  • The audit record created.
  • The behavior when authorization or consent is absent.

Authentication belongs at the server#

Remote MCP servers using HTTP should follow the protocol’s authorization specification. The current specification builds on OAuth and protected-resource metadata so clients can discover the appropriate authorization server.

The MCP server is a resource server. It must validate that an access token:

  • Is valid and unexpired.
  • Was issued for this resource server.
  • Contains the required scope or authorization context.
  • Represents a principal permitted to perform the requested operation.

The host’s system prompt is not an authorization layer. A prompt saying “only read tickets” cannot prevent a server from accepting an unauthorized write request.

Never pass tokens through#

Token passthrough occurs when an MCP server receives a token intended for itself and forwards that token unchanged to an upstream API. This can create a confused-deputy vulnerability: the upstream system may accept a credential outside the context in which it was issued.

Use separate audience-bound credentials:

text
host → MCP server: token audience = https://mcp.example.com
MCP server → upstream API: separate token audience = https://api.example.com

The server validates the inbound token, derives the authorized principal, applies policy, and uses its own upstream authorization flow. It must not turn a credential intended for one resource into a bearer credential for another.

Consent should describe the actual effect, not merely the tool name. Compare:

text
Run tool: update_record

with:

text
Change ticket ENG-402 from "Open" to "Resolved" and add the displayed comment.

Useful consent is:

  • Specific about resource, action, and relevant arguments.
  • Requested close to the action.
  • Different for reads, drafts, writes, and destructive operations.
  • Revocable and visible in the host.
  • Not silently expanded when a server advertises new capabilities.

The host should make tool requests inspectable before consequential execution. The server must enforce policy even after the host receives consent, because client behavior can be incorrect or compromised.

Tools should be narrow and typed#

Prefer domain operations:

json
{
    "name": "ticket_add_internal_note",
    "description": "Add a non-customer-visible note to one ticket",
    "inputSchema": {
        "type": "object",
        "properties": {
            "ticketId": { "type": "string", "pattern": "^ENG-[0-9]+$" },
            "note": { "type": "string", "maxLength": 2000 }
        },
        "required": ["ticketId", "note"],
        "additionalProperties": false
    }
}

Avoid generic capabilities such as execute_sql, run_shell, or call_api unless the product explicitly requires that authority and can sandbox it. Narrow tools improve authorization, evaluation, observability, and recovery.

Tool descriptions are untrusted metadata from another system. They help the model choose a capability but should not override host policy. Names should be namespaced and unambiguous when multiple servers expose similar operations.

Derive identity from authorization context#

Do not accept a user ID supplied by the model as proof of identity:

json
{
    "userId": "admin",
    "ticketId": "ENG-402"
}

The server should derive the user or tenant from the validated token and treat tool arguments only as requested resources. Every database query and upstream call must remain tenant scoped.

This matters for session state as well. If a server uses sessions, bind session identifiers to the authorized user context and generate them with cryptographically secure randomness. A guessed session ID must not permit cross-user event injection.

Separate read, draft, and commit#

High-impact operations benefit from a staged interface:

  1. deployment_plan computes and returns the intended changes.
  2. The host presents the plan and requests approval.
  3. deployment_apply accepts a short-lived plan identifier.
  4. The server revalidates identity, permissions, plan freshness, and current state.

This keeps the model from reconstructing arbitrary write arguments after approval. It also supports an audit trail showing exactly what was reviewed.

For irreversible actions, consider requiring an independent approval mechanism outside the model conversation.

Keep local servers constrained too#

Local STDIO servers do not use the remote HTTP authorization flow, but they can be more dangerous because they run with local process permissions.

Apply operating-system controls:

  • Run with the least-privileged user.
  • Restrict filesystem roots.
  • Pass only required environment variables.
  • Avoid inheriting broad cloud credentials.
  • Pin and verify packages rather than executing unknown latest versions.
  • Isolate untrusted code in a sandbox.
  • Make network access explicit.

A local server launched through npx -y is executable supply-chain input. Convenience does not make it trustworthy.

Treat returned content as untrusted#

Resources and tool results may contain malicious or accidental instructions. The host should preserve provenance and keep data distinct from policy:

text
trusted: system policy and application rules
untrusted: resource text, tool output, issue comments, web pages

Do not concatenate all content into one undifferentiated prompt. Label sources, minimize returned fields, and apply output encoding before rendering. Sensitive resources should be filtered before they reach a third-party model if policy requires local handling.

Audit the operation, not hidden reasoning#

A useful MCP audit record contains:

  • Authenticated principal and tenant.
  • Host/client identity.
  • Server and protocol versions.
  • Tool or resource name.
  • Sanitized arguments or argument fingerprint.
  • Authorization and consent decision.
  • Start time, duration, and outcome.
  • Upstream request identifier.
  • Stable operation and attempt identifiers.

Do not log access tokens or unnecessary sensitive payloads. The purpose is to reconstruct effects and decisions, not to store every byte that crossed the system.

Plan for capability drift#

Servers evolve. A new tool, changed schema, or expanded behavior can invalidate host assumptions.

At initialization:

  • Negotiate protocol versions and capabilities.
  • Validate schemas before exposing them to the model.
  • Apply an allowlist or approval workflow for new write capabilities.
  • Fail closed when a required security capability is absent.
  • Record server version and capability changes.

Pinning a server version reduces surprise but does not replace compatibility checks. Remote servers may change independently of the host.

Production readiness checklist#

  • Identify the user, host, server, and upstream trust domains.
  • Use audience-bound authorization for remote servers.
  • Never pass client tokens through to upstream APIs.
  • Derive identity and tenant from validated authorization context.
  • Separate read, draft, write, and destructive capabilities.
  • Use narrow tools with strict schemas and bounded output.
  • Request effect-specific consent for consequential operations.
  • Treat descriptions, resources, and results as untrusted input.
  • Constrain local servers with OS and sandbox permissions.
  • Log authorization decisions and external effects without secrets.
  • Detect capability and schema changes.
  • Provide revocation, cancellation, and recovery paths.

MCP makes integrations composable. Production safety still comes from explicit authority, narrow interfaces, enforced boundaries, and evidence that operators can inspect.

Review the system as an authorization graph#

A final architecture review should trace each possible effect from the human principal to the protected resource. For every edge, ask which component authenticated the caller, which audience the credential names, which scopes constrain it, and where consent becomes a committed effect. This graph often reveals that a component described as “just a connector” is actually making an authorization decision.

Test negative paths as deliberately as successful calls. A server should reject a token issued for another audience, a missing scope, an expired grant, an unregistered redirect URI, and parameters outside the authorized resource. A host should display a comprehensible failure instead of encouraging the model to retry authorization errors. Repeated retries can turn a permission bug into account lockout, alert noise, or unintended load.

Revocation deserves an end-to-end exercise. Remove a grant while a session is active, then verify that cached capability metadata does not preserve access, refresh cannot silently recreate authority, and subsequent calls produce a bounded failure. Removing a server should eliminate its tools and stored credentials without damaging unrelated integrations.

Transport security is necessary but insufficient. Local transports still cross process boundaries, and remote transports still require application-level authorization. Log connection identity and operation metadata without retaining bearer tokens, arbitrary prompt content, or returned secrets. Audit data should answer who authorized which effect and when without becoming a copy of everything handled.

Production approval should therefore attach to a concrete server version, tool schema, authorization policy, and deployment identity. Re-review when any changes. MCP simplifies discovery; it does not make delegated authority static.

Sources and further reading#

Related writing