Blog › Field Notes › Eight Commits to Fix a Sub-Path: Nginx Ingress Rewrites for Grafana, Prefect, MLflow, and ArgoCD
Debug #nginx#ingress#kubernetes#grafana#prefect#mlflow#argocd#reverse-proxy#path-rewrite

Eight Commits to Fix a Sub-Path: Nginx Ingress Rewrites for Grafana, Prefect, MLflow, and ArgoCD

Routed six services under one IP using Nginx ingress path matching, rewrite-target annotations, and application-level sub-path configuration on AKS.

· Gideon Warui
ON THIS PAGE

I was routing six services through a single Nginx ingress controller on AKS — Grafana, MLflow, Prefect, ArgoCD, FastAPI, and its API endpoints. Each service needed its own sub-path: /dashboards, /intelligence, /pipelines, /deployments, and / as the catch-all. The Prefect UI alone took eight commits to get right.

This documents the three routing patterns I used, why Prefect was the hardest, and the exact annotations and environment variables that made each service work.

The Routing Problem

One public IP. Six services. No DNS subdomains — just path-based routing. The layout:

PathServicePort
/dashboards/*Grafana3000
/intelligence/*MLflow5000
/pipelines/*Prefect UI4200
/pipelines/api/*Prefect API4200
/deployments/*ArgoCD80
/*FastAPI8000

The ingress controller is nginx. Each service gets its own Ingress resource — not one monolithic resource with all paths — because each service needs different timeout and rewrite settings.

Pattern 1: No Rewrite — App Handles the Prefix

Grafana, MLflow, and ArgoCD all have native sub-path support. The strategy: pass the full path through to the backend and configure the application to expect it.

Grafana

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <namespace>-ingress
  namespace: <namespace>
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /dashboards(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: grafana
            port:
              number: 3000

Grafana receives /dashboards/d/abc-123 as-is. It knows about the prefix because of two environment variables:

env:
- name: GF_SERVER_ROOT_URL
  value: "%(protocol)s://%(domain)s/dashboards"
- name: GF_SERVER_SERVE_FROM_SUB_PATH
  value: "true"

GF_SERVER_ROOT_URL tells Grafana where it’s externally accessible. GF_SERVER_SERVE_FROM_SUB_PATH tells it to strip the prefix when matching internal routes and to include it in all generated URLs — CSS links, API calls, JavaScript paths. Without both, Grafana serves HTML that references /public/build/... instead of /dashboards/public/build/..., and every asset 404s.

MLflow

Same pattern, different config mechanism:

- path: /intelligence(/|$)(.*)
  backend:
    service:
      name: mlflow
      port:
        number: 5000

MLflow’s sub-path support is a CLI flag:

command: ["mlflow", "server"]
args:
- --host=0.0.0.0
- --port=5000
- --static-prefix=/intelligence

--static-prefix rewrites all internal asset URLs. The MLflow UI at /intelligence loads JavaScript from /intelligence/static/... and makes API calls to /intelligence/api/.... This is the cleanest integration — a single flag handles everything.

ArgoCD

ArgoCD uses a ConfigMap for its sub-path:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"
  server.rootpath: "/deployments"

ArgoCD lives in its own namespace (argocd), so it gets a separate Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-ingress
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /deployments(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: argocd-server
            port:
              number: 80

The backend-protocol: HTTP annotation is necessary because ArgoCD server defaults to gRPC/HTTPS internally. Without it, nginx tries to connect with the wrong protocol and returns 502.

Pattern 2: Rewrite-Target — Transform the Path

Prefect’s API server expects requests at /api/.... The UI is configured to call /pipelines/api/.... These don’t match. Nginx bridges the gap.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prefect-api-ingress
  namespace: <namespace>
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /api/$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /pipelines/api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: prefect-server
            port:
              number: 4200

The regex captures two groups:

  • Group 1: (/|$) — the trailing slash or end of string
  • Group 2: (.*) — everything after

rewrite-target: /api/$2 sends /pipelines/api/v1/flows to the backend as /api/v1/flows. Prefect’s API handler recognizes it.

The Prefect UI is a separate Ingress (no rewrite) because the UI static assets use PREFECT_SERVER_UI_SERVE_BASE:

env:
- name: PREFECT_SERVER_UI_SERVE_BASE
  value: "/pipelines/"
- name: PREFECT_UI_API_URL
  value: "/pipelines/api"

The Prefect Sub-Path Saga

The git log tells the story:

6a33aae fix: inject base href for Prefect UI sub-path via sub_filter
45a2e73 fix: rewrite Prefect absolute asset paths via sub_filter
d58dce0 fix: disable backend gzip for Prefect sub_filter to work
4e2725f fix: route Prefect static assets via dedicated ingress rules
8933e0c fix: use PREFECT_SERVER_UI_SERVE_BASE for native sub-path support
a084fae fix: set Prefect API base path to /pipelines for consistent sub-path
49f5061 fix: PREFECT_SERVER_API_BASE_PATH replaces /api, not appends to it
567c286 fix: split Prefect ingress — rewrite /pipelines/api, passthrough UI

Eight commits. Here’s what happened.

Attempt 1: Ingress rewrite only. I set rewrite-target: /$2 on the Prefect ingress. The API worked but the UI served a white screen. Prefect’s single-page app loads JavaScript that makes API calls to /api/... — hardcoded absolute paths. The ingress rewrite only changes the inbound request; it can’t rewrite JavaScript served by the backend.

Attempt 2: sub_filter injection. I added nginx sub_filter annotations to inject a <base href="/pipelines/"> tag into the HTML response. This required disabling backend gzip (nginx.ingress.kubernetes.io/configuration-snippet with proxy_set_header Accept-Encoding "";) because sub_filter operates on uncompressed text. The base tag partially worked — relative asset paths resolved, but the Prefect UI has JavaScript that constructs API URLs with absolute paths. sub_filter can’t rewrite JavaScript strings in-flight.

Attempt 3: Dedicated asset routes. I created separate ingress rules for /pipelines/static/... and /pipelines/assets/... to handle static file routing. This got CSS loading but the API calls still failed.

Attempt 4: PREFECT_SERVER_UI_SERVE_BASE. The actual solution. Prefect 3 supports PREFECT_SERVER_UI_SERVE_BASE — an environment variable that tells the build process to prefix all asset URLs. Combined with PREFECT_UI_API_URL for API routing, the entire sub-path problem is handled at the application level. No sub_filter, no gzip hacking, no dedicated asset routes.

The final architecture: two Ingress resources (one for UI passthrough, one for API rewrite) plus two environment variables. Everything else was noise.

Pattern 3: Catch-All — FastAPI Gets the Rest

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <client>-api-ingress
  namespace: <namespace>
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: <client>-api
            port:
              number: 8000

This is the lowest-priority rule. Kubernetes evaluates ingress paths by specificity — longer, more specific paths win. /dashboards/..., /intelligence/..., /pipelines/... all match before /(.*).

FastAPI receives the full path and handles routing internally — portal pages (/, /login, /portal/analytics), API endpoints (/api/v1/...), and Swagger docs (/docs).

Why Separate Ingress Resources

A single Ingress with all paths would inherit the same annotations everywhere. But Grafana needs proxy-read-timeout: 3600 for long-running dashboard queries, while FastAPI needs proxy-body-size: 50m for file uploads. Prefect API needs rewrite-target but Prefect UI does not.

Separate resources let each service declare its own:

  • Timeout values
  • Body size limits
  • Rewrite rules
  • Backend protocol
  • TLS settings

It also makes debugging cleaner. When Prefect returns 502, I check kubectl describe ingress prefect-api-ingress — not a monolithic resource with 15 paths.

The Regex

Every path uses the same regex shape: /prefix(/|$)(.*). Breaking it down:

/dashboards(/|$)(.*)

/dashboards   — literal match
(/|$)         — either "/" or end of string (matches both /dashboards and /dashboards/)
(.*)          — everything else

The (/|$) group is important. Without it, /dashboardsettings would match the /dashboards prefix. The group ensures a clean boundary — only a slash or string termination after the prefix.

pathType: ImplementationSpecific tells the ingress controller to handle the path however it wants (as opposed to Prefix or Exact). With nginx and use-regex: "true", this means full PCRE regex support.

Authentication Architecture

Auth is enforced at the FastAPI application level, not at the ingress. The portal uses JWT tokens stored in HTTP-only cookies:

response.set_cookie("access_token", token, httponly=True, max_age=28800)

Protected pages check the cookie:

def require_auth(request: Request):
    token = request.cookies.get("access_token")
    if not token:
        return RedirectResponse(url="/", status_code=302)
    try:
        decode_token(token)
    except HTTPException:
        return RedirectResponse(url="/", status_code=302)
    return None

External services (Grafana, MLflow, Prefect, ArgoCD) have their own auth mechanisms. Grafana uses admin/<redacted-token>. ArgoCD has its own login. These are accessible directly via their sub-paths — the portal provides convenient links but doesn’t gate them.

For production, I would add nginx.ingress.kubernetes.io/auth-url annotations pointing to an OAuth2 proxy or an external auth service. This centralizes authentication at the ingress layer rather than relying on each service to implement it.

Sub-Path Routing Rules

Check for native sub-path support before writing rewrite rules

Grafana has GF_SERVER_SERVE_FROM_SUB_PATH. MLflow has --static-prefix. Prefect has PREFECT_SERVER_UI_SERVE_BASE. ArgoCD has server.rootpath. If the application can handle the prefix, let it — rewrite rules add complexity and break when the application generates absolute URLs in JavaScript.

sub_filter is a trap for single-page apps

It works for rewriting HTML responses, but modern SPAs generate URLs in JavaScript at runtime. You can’t sub_filter JavaScript string concatenation. The application itself needs to know its base path.

Split API and UI Ingress resources when they need different rewrite behavior

Prefect’s UI needs passthrough (no rewrite) while its API needs /pipelines/api -> /api. One Ingress resource can’t express both.

Regex capture groups count from left

$1 is the first (...), $2 is the second. When using rewrite-target: /api/$2, make sure $2 captures the right segment. Off-by-one here sends traffic to the wrong path.

backend-protocol: HTTP matters for services that default to HTTPS/gRPC

ArgoCD server runs gRPC by default. Without the annotation, nginx tries HTTP/1.1, gets a protocol mismatch, and returns 502. The annotation forces HTTP communication between nginx and the backend.

#nginx#ingress#kubernetes#grafana#prefect#mlflow#argocd#reverse-proxy#path-rewrite

Share

Post on X LinkedIn

Discussion