Why secure ML deployment is different

Traditional app security is necessary, but it’s not sufficient for model-serving. The model itself is an asset, and the ML lifecycle introduces new attack surfaces (training pipelines, artifacts, dependencies, drift, and feedback loops).

Here are the failures that actually hurt in production:

• Model theft / extraction: attackers replicate your model via repeated queries or leak artifacts from storage.
• Unauthorized inference: anyone who finds the endpoint can rack up cost or abuse outputs.
• Tampering: swapping weights, poisoning inputs, or sneaking changes into container images/artifacts.
• Data leakage: sensitive prompts/features/records logged or returned in responses.
• Silent regression: drift changes outcomes while dashboards stay “green.”

The 4-layer security blueprint (the CTO version)

If you can answer “what’s running, who can call it, what it’s doing right now, and what secrets it can touch,” you’re already ahead. The rest is execution and discipline.

Jump to the deep dive

Container security: practical examples

Example: CI image scan (Trivy)

# GitHub Actions (example)
- name: Build image
  run: docker build -t myorg/model:${{ github.sha }} .

- name: Scan image (fail on High/Critical)
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myorg/model:${{ github.sha }}
    severity: HIGH,CRITICAL
    exit-code: 1

Example: Kubernetes securityContext

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]

Example: NetworkPolicy (only gateway can call the model)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-only-api-gateway
spec:
  podSelector:
    matchLabels:
      app: model-serving
  policyTypes: ["Ingress"]
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway
    ports:
    - protocol: TCP
      port: 8080

API protection: practical examples

Example: NGINX Ingress rate limiting

metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "20"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "3"

Example: Enforce request size limits (stop “giant payload” attacks)

metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "2m"

Example: Put schema validation at the gateway

Use an API gateway (Kong / Apigee / AWS API Gateway) with OpenAPI-based request validation
so invalid payloads never reach your model server. This is one of the simplest “high leverage” controls you can add.

Monitoring: practical examples

Example: Export app metrics + alert on P95 latency

# Prometheus alert rule (example concept)
- alert: ModelServingHighLatency
  expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.75
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "Model serving P95 latency is too high for 10m"

Also add a “model quality” signal if you can (delayed labels, canary set, or business KPI proxy).
That’s how you catch silent degradation before customers do.

Secrets: practical examples

Example: Vault Agent Injector (concept)

metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "model-serving"
    vault.hashicorp.com/agent-inject-secret-db: "secret/data/prod/db"

Example: External Secrets Operator (concept)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: model-serving-secrets
spec:
  secretStoreRef:
    name: aws-secretsmanager
    kind: ClusterSecretStore
  target:
    name: model-serving-secrets
  data:
  - secretKey: OPENAI_API_KEY
    remoteRef:
      key: prod/openai

The pattern is the same no matter the tool: authenticate the workload, fetch secrets at runtime, audit access, rotate keys.