Skip to main content

CMP Cilium Gateway & HTTPRoute Setup

Last updated: 2025-11-20 Status: Production (k8s cluster with Cilium Gateway API)

This document describes the CMP routing architecture using Kubernetes Gateway API with Cilium, replacing the deprecated legacy ingress pattern.

Architecture Overview​

CMP services use Cilium Gateway with HTTPRoute resources for traffic routing instead of legacy Ingress or in-container proxy configs. This provides:

  • Declarative L7 routing at the Gateway layer
  • Cross-namespace service references with ReferenceGrants
  • Path-based routing for API delegation
  • Unified observability through Envoy metrics

Service Topology​

External Traffic (HTTPS)
    ↓
Cilium Gateway (gateway-system/external-gateway)
    ↓
HTTPRoute (cmp/cmp-portal) — Path-based routing:
    │
    ├─→ /api/auth/*   → idp-backend.idp:80       (Auth/OIDC)
    ├─→ /api/*        → cmp-registry.cmp:80      (Registry API)
    ├─→ /health/*     → cmp-registry.cmp:80      (Health endpoints)
    └─→ /*            → cmp-portal.cmp:80        (Frontend SPA)

HTTPRoute Configuration​

Portal (kubernetes/cmp/portal/httproute.yaml)​

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cmp-portal
  namespace: cmp
  annotations:
    external-dns.alpha.kubernetes.io/exclude: "true"
spec:
  hostnames:
    - cmp-portal.uat.digiwedge.com
  parentRefs:
    - name: external-gateway
      namespace: gateway-system
  rules:
    # Route /api/auth/* to IDP backend (cross-namespace)
    - matches:
        - path:
            type: PathPrefix
            value: /api/auth/
      backendRefs:
        - name: idp-backend
          namespace: idp
          port: 80
    # Route /api/* to CMP registry backend
    - matches:
        - path:
            type: PathPrefix
            value: /api/
      backendRefs:
        - name: cmp-registry
          port: 80
    # Route /health/* to CMP registry backend
    - matches:
        - path:
            type: PathPrefix
            value: /health/
      backendRefs:
        - name: cmp-registry
          port: 80
    # Route everything else to portal frontend
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: cmp-portal
          port: 80

Key Points:

  • Rules are evaluated in order (most specific first)
  • /api/auth/ must come before /api/ to avoid shadowing
  • Cross-namespace references require a ReferenceGrant

ReferenceGrant (kubernetes/idp/cmp-to-idp-referencegrant.yaml)​

Required for CMP namespace to reference IDP backend service:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: cmp-to-idp-backend
  namespace: idp
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      namespace: cmp
  to:
    - group: ""
      kind: Service
      name: idp-backend

Other CMP HTTPRoutes​

  • cmp-registry: Direct access to registry API

    • Hostname: cmp-registry.uat.digiwedge.com
    • Backend: cmp-registry:80

  • cmp-scan-api: Scanner API endpoint

    • Hostname: cmp-scan-api.uat.digiwedge.com
    • Backend: cmp-scan-api:80

  • cmp-docs-internal: Internal documentation

    • Hostname: cmp-docs-internal.uat.digiwedge.com
    • Backend: cmp-docs-internal:80

  • cmp-docs-public: Public documentation

    • Hostname: cmp-docs-public.uat.digiwedge.com
    • Backend: cmp-docs-public:80

Portal Dockerfile Simplification​

With routing handled by Cilium Gateway, the portal Dockerfile only serves static files:

FROM caddy:2-alpine

COPY dist/apps/cmp/portal /usr/share/caddy

COPY <<'CADDY' /etc/caddy/Caddyfile
:80
root * /usr/share/caddy
try_files {path} /index.html
file_server
CADDY

EXPOSE 80
CMD ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]

No in-container proxy configuration needed — routing is declarative at the Gateway layer.

Deployment Process​

  1. Deploy ReferenceGrant (one-time per namespace pair):

    kubectl apply -f kubernetes/idp/cmp-to-idp-referencegrant.yaml

  2. Deploy HTTPRoutes:

    kubectl apply -f kubernetes/cmp/portal/httproute.yaml
    kubectl apply -f kubernetes/cmp/registry/httproute.yaml
    kubectl apply -f kubernetes/cmp/scan-api/httproute.yaml
    kubectl apply -f kubernetes/cmp/docs-internal/httproute.yaml
    kubectl apply -f kubernetes/cmp/docs-public/httproute.yaml

  3. Verify HTTPRoute Status:

    kubectl get httproute -n cmp cmp-portal -o yaml | grep -A 20 "status:"

    Look for:

    • Accepted: True
    • ResolvedRefs: True
    • No error messages

  4. Test Endpoints:

    # Auth endpoint (→ IDP)
    curl -i https://cmp-portal.uat.digiwedge.com/api/auth/csrf

    # Registry API (→ Registry)
    curl -s https://cmp-portal.uat.digiwedge.com/api/health | jq .

    # Frontend (→ Portal)
    curl -s https://cmp-portal.uat.digiwedge.com/ | grep -o '<title>.*</title>'

Troubleshooting​

HTTPRoute Not Accepting​

Symptom: Accepted: False in status

Check:

kubectl describe httproute -n cmp cmp-portal

Common causes:

  • Parent Gateway doesn't exist or is in different namespace
  • Invalid backend service name/port
  • Missing ReferenceGrant for cross-namespace refs

404 on Cross-Namespace Routes​

Symptom: /api/auth/* returns 404

Solution:

  1. Verify ReferenceGrant exists:

    kubectl get referencegrant -n idp cmp-to-idp-backend

  2. Check IDP service exists:

    kubectl get svc -n idp idp-backend

  3. Verify HTTPRoute references correct namespace:

    kubectl get httproute -n cmp cmp-portal -o yaml | grep -A 5 "namespace: idp"

Route Shadowing​

Symptom: More specific routes not matching

Solution: Ensure rule order is most-specific-first:

  1. /api/auth/ (most specific)
  2. /api/
  3. /health/
  4. / (catch-all)

Migration from legacy Ingress​

If migrating from legacy Ingress:

  1. Remove Ingress resources
  2. Remove legacy proxy config from Dockerfiles
  3. Create HTTPRoute and ReferenceGrant resources
  4. Update documentation and runbooks
  5. Rebuild images without legacy proxy config
  6. Deploy new images
  7. Apply HTTPRoutes
  8. Test all endpoints

Monitoring​

Gateway metrics are exposed via Envoy:

# Request rate by route
rate(envoy_http_downstream_rq_completed[5m])

# Latency p95
histogram_quantile(0.95, rate(envoy_http_downstream_rq_time_bucket[5m]))

# Error rate
rate(envoy_http_downstream_rq_xx{envoy_response_code_class="5"}[5m])

References​

See Also​

  • CMP Operator Checklist: docs/cmp/OPERATOR_CHECKLIST.md
  • Kubernetes Migration: docs/internal/k8s/infrastructure-migration-complete.md
  • CMP Portal: docs/cmp/portal.md
  • CMP Registry: docs/cmp/registry.md