← Back to All Articles
AI Security & GovCVE-2026-102895

GPUStack Authentication Bypass and Remote Code Execution (CVE-2026-102895): Root Cause, Detection, and Incident Response

Technical breakdown of GPUStack CVE-2026-102895 (GHSA-gj9m-f8ph-pmhr) in versions prior to 2.2.0: how default force_auth_localhost=false and X-GPUStack-Real-IP loopback spoofing grant unauthenticated admin access, API-key persistence, RCE via POST /v2/inference-backends/from-yaml, detection prerequisites, and recovery.

·10 min read·
#AI Security#GPUStack#CVE-2026-102895#CISA KEV#Incident Response
GPUStack Authentication Bypass and Remote Code Execution (CVE-2026-102895): Root Cause, Detection, and Incident Response

1. Executive Summary & Vulnerability Scope

CVE-2026-102895 (GitHub Security Advisory GHSA-gj9m-f8ph-pmhr, CVSS v3.1 9.8 Critical — CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H) is an improper authentication vulnerability affecting GPUStack versions prior to 2.2.0. Tracked under CWE-287 (Improper Authentication), CWE-290 (Authentication Bypass by Spoofing), and CWE-346 (Origin Validation Error), the flaw allows an unauthenticated remote attacker to impersonate the local loopback interface and assume built-in system administrator privileges over the GPUStack control plane.

For Security Architects, SOC L2/L3 analysts, Detection Engineers, and AI Infrastructure Engineers, this is a full control-plane compromise condition rather than a localized UI defect. A single unauthenticated HTTP request can lead to persistent administrative API keys, exposure of worker and cloud-provider secrets, and code execution on GPU worker nodes.

2. Root Cause Analysis: Loopback Trust and X-GPUStack-Real-IP Spoofing

In GPUStack versions prior to 2.2.0, request authentication is enforced inside gpustack/api/auth.py through get_current_user in conjunction with client_ip_getter.

GPUStack CVE-2026-102895 Authentication Bypass and Exploitation Flow Diagram
GPUStack CVE-2026-102895 — Technical Path from Spoofed Loopback Header to Admin Context and Worker Abuse

Two design conditions combine to create the authentication bypass:

  • Implicit Loopback Trust (`force_auth_localhost = false`): In affected releases, the server configuration option --force-auth-localhost defaults to false. When disabled, GPUStack assumes any request originating from 127.0.0.1 is a trusted local administrative operation and automatically elevates the session to the built-in system administrator without validating credentials or tokens.
  • Blind Trust in `X-GPUStack-Real-IP` and Proxy Peers: Instead of restricting loopback privilege strictly to direct local sockets outside untrusted proxy boundaries, client_ip_getter trusted client-controllable header values—specifically `X-GPUStack-Real-IP`—and certain same-host reverse-proxy configurations where the TCP peer appears as 127.0.0.1.
  • Bypass Outcome: Any remote unauthenticated client reaching an affected GPUStack control plane (directly or through a reverse proxy that forwards X-GPUStack-Real-IP) can inject X-GPUStack-Real-IP: 127.0.0.1 to make external traffic resolve as loopback and immediately bypass authentication.

3. Exploitation Path, Persistence, and Impact Classification

To keep architectural triage grounded in verifiable evidence, operational teams should separate established vulnerability behavior from downstream impact and conditional post-exploitation risk:

  • CONFIRMED BEHAVIOR: In GPUStack < 2.2.0 running with default force_auth_localhost=false, supplying X-GPUStack-Real-IP: 127.0.0.1 (or abusing a same-host reverse-proxy loopback peer) bypasses get_current_user and grants built-in system administrator context without credentials. CISA KEV and the CVE record document unauthenticated administrative access, user and model modifications, persistent administrator API-key creation, disclosure of worker and cloud-provider secrets, and code execution via POST /v2/inference-backends/from-yaml. SANS ISC (Diary 33514) confirmed active probing against exposed instances across model and inference endpoints (/v1/models, /v1/messages).
  • POTENTIAL IMPACT: Once authenticated as system administrator, an attacker can mint persistent admin bearer tokens via POST /v2/api-keys, inspect or alter user accounts via /v2/users, modify deployed models and worker settings, exfiltrate stored worker tokens, model-hub credentials, and cloud provider secrets, and execute arbitrary commands on GPU worker nodes by submitting crafted backend specifications to POST /v2/inference-backends/from-yaml.
  • HYPOTHETICAL SCENARIO: If compromised GPU worker nodes run with over-privileged cloud IAM roles, broad Kubernetes service-account permissions, or unrestricted internal network egress, an attacker who achieves execution via from-yaml could pivot into adjacent object storage or cloud control planes. That downstream lateral movement depends on host and cloud IAM hygiene outside GPUStack itself.

Post-Exploitation Persistence and Why Upgrading Alone Is Insufficient

Patching GPUStack to 2.2.0+ closes the X-GPUStack-Real-IP bypass path, but upgrading the binary alone does not evict an attacker who compromised the cluster prior to patching:

  • Database-Backed API Keys Remain Valid: Any persistent administrator API key created via POST /v2/api-keys before upgrading remains active in GPUStack's database and continues to authorize administrative calls against 2.2.0+.
  • Rogue Accounts and Malicious Backends Persist: Unauthorized administrative users added via /v2/users or malicious backend definitions registered under /v2/inference-backends survive an in-place upgrade.
  • Exfiltrated Secrets Stay Valid Elsewhere: Cloud credentials, container registry secrets, Hugging Face tokens, and worker bootstrap keys extracted from GPUStack configuration remain usable until explicitly revoked at each upstream provider.

4. Detection Engineering & Telemetry Prerequisites

Effective detection for CVE-2026-102895 depends strictly on what HTTP headers and process telemetry are actually captured in your environment. Always verify logging prerequisites before deploying hunting rules as production alerts.

4.1 Reverse Proxy / Ingress Access Log Triage (Linux Shell)

  • DATA SOURCE: Access logs from the reverse proxy or ingress controller fronting GPUStack (NGINX, Envoy, HAProxy, Caddy, or GPUStack Uvicorn HTTP logs).
  • TABLE: File-based access logs (such as /var/log/nginx/gpustack-access.log) or container stdout streams.
  • REQUIRED INGESTION: Standard NGINX combined access logs record source IP, URI, and HTTP status, but omit `X-GPUStack-Real-IP` by default. Detecting the spoofed header value itself requires custom log_format directives capturing $http_x_gpustack_real_ip (or structured JSON header logging) enabled prior to the incident.
  • PURPOSE: Identify external requests probing or hitting privileged administrative routes (/v2/inference-backends/from-yaml, /v2/api-keys, /v2/users) as well as SANS ISC-observed scan endpoints (/v1/models, /v1/messages), correlated with HTTP 200/201 responses.
  • LIMITATIONS: Without custom header logging, this check cannot prove X-GPUStack-Real-IP spoofing directly—it surfaces endpoint targeting and anomalous successful administrative responses. Traffic reaching GPUStack directly on an unproxied port bypasses reverse-proxy logs completely.
bash
# Triage standard reverse-proxy access logs for high-risk GPUStack endpoints.
# Note: Default NGINX 'combined' format does not log X-GPUStack-Real-IP unless
# $http_x_gpustack_real_ip was explicitly added to log_format beforehand.
grep -E '(/v2/inference-backends/from-yaml|/v2/api-keys|/v2/users|/v1/users|/v1/models|/v1/messages)' \
  /var/log/nginx/gpustack-access.log | \
  awk '{print $1, $4, $6, $7, $9}' | tail -n 100

4.2 Azure Application Gateway WAF Hunting (Microsoft Sentinel KQL)

  • DATA SOURCE: Azure Application Gateway WAF access diagnostics deployed in front of the GPUStack ingress tier.
  • TABLE: AzureDiagnostics scoped to Category == "ApplicationGatewayAccessLog" (or AGWAccessLogs using unsuffixed fields such as ClientIp, RequestUri, HttpMethod, and HttpStatus when resource-specific diagnostic mode is enabled). Note: AppServiceHTTPLogs is not native GPUStack telemetry; GPUStack runs on GPU-equipped Linux VMs, AKS clusters, or container hosts rather than Azure App Service PaaS.
  • REQUIRED INGESTION: Diagnostic settings on the Azure Application Gateway must be actively streaming access logs to the Microsoft Sentinel Log Analytics workspace.
  • PURPOSE: Detect reconnaissance and exploitation attempts against /v2/inference-backends/from-yaml, /v2/api-keys, and /v2/users, highlighting external client IPs receiving 200/201 success codes.
  • LIMITATIONS: Standard ApplicationGatewayAccessLog schemas do not record arbitrary custom request headers such as X-GPUStack-Real-IP. Therefore, this KQL query cannot detect `X-GPUStack-Real-IP` header spoofing directly; it detects anomalous administrative URI access and unexpected HTTP success responses. Header-level detection in Sentinel requires custom WAF inspection rules or reverse-proxy header logging.
kql
// Data Source: Azure Log Analytics (AzureDiagnostics - Application Gateway Access Logs, WAF v2)
// Execution Scope: Read-only hunting query (requires Log Analytics Reader role)
AzureDiagnostics
| where TimeGenerated > ago(14d)
| where Category == "ApplicationGatewayAccessLog"
| where requestUri_s has_any (
    "/v2/inference-backends/from-yaml",
    "/v2/api-keys",
    "/v2/users",
    "/v1/users",
    "/v1/models",
    "/v1/messages"
)
| extend StatusCode = toint(httpStatus_d)
| summarize
    RequestCount = count(),
    SuccessCount = countif(StatusCode in (200, 201, 204)),
    Methods = make_set(httpMethod_s),
    StatusCodes = make_set(StatusCode)
    by clientIP_s, requestUri_s, bin(TimeGenerated, 1h)
| where SuccessCount > 0 or RequestCount >= 5
| order by TimeGenerated desc

4.3 GPU Worker Post-Exploitation Process Hunting (Microsoft Defender for Endpoint KQL)

  • DATA SOURCE: Endpoint process telemetry from Microsoft Defender for Endpoint (MDE for Linux) running on GPUStack control-plane or worker hosts.
  • TABLE: DeviceProcessEvents.
  • REQUIRED INGESTION: MDE Linux sensor deployed on GPU nodes with telemetry flowing to Microsoft Defender XDR Advanced Hunting or a connected Microsoft Sentinel workspace.
  • PURPOSE: Detect post-exploitation execution following POST /v2/inference-backends/from-yaml abuse, where gpustack, uvicorn, or worker Python processes spawn interactive shells (sh, bash, dash) or network download/reverse-shell utilities (curl, wget, nc).
  • LIMITATIONS: Cannot detect API-only abuse such as reading secrets or generating /v2/api-keys without host shell execution. Requires environment tuning against legitimate container startup or model-provisioning scripts.
kql
// Data Source: Microsoft Defender XDR Advanced Hunting (DeviceProcessEvents)
// Execution Scope: Read-only telemetry query (requires Security Reader role + MDE onboarding)
DeviceProcessEvents
| where Timestamp > ago(14d)
| where InitiatingProcessFileName has_any ("gpustack", "python", "python3", "uvicorn")
| where InitiatingProcessCommandLine has_any ("gpustack", "inference-backend", "worker")
| where FileName in~ ("sh", "bash", "dash", "curl", "wget", "nc", "ncat", "socat")
    or ProcessCommandLine has_any ("base64 -d", "/dev/tcp/", "curl -s", "wget -q")
| project Timestamp, DeviceName, AccountName, InitiatingProcessFileName,
          InitiatingProcessCommandLine, FileName, ProcessCommandLine
| order by Timestamp desc

5. Containment, Remediation, and Post-Compromise Recovery

Execute containment and recovery in sequence so that closing the entry point is immediately followed by persistence eviction and upstream credential rotation:

  • Step 1 — Exposure Identification: Inventory all GPUStack server and worker deployments (gpustack version or container image digest/tag). Flag every instance below 2.2.0 and identify any control-plane port exposed to the public internet, untrusted ingress controllers, or broad developer subnets.
  • Step 2 — Upgrade & Authentication Hardening: Upgrade all GPUStack instances to 2.2.0 or later (2.2.2+ recommended), enforce --force-auth-localhost so local loopback connections still require explicit credentials, and remove public internet exposure from the control plane.
  • Step 3 — Temporary Ingress Mitigation (If Immediate Upgrade Is Blocked): If change-control windows delay upgrading temporarily, configure the reverse proxy or ingress controller in front of GPUStack to strip any client-supplied X-GPUStack-Real-IP header and restrict /v2/inference-backends/from-yaml, /v2/api-keys, and /v2/users to trusted administrative CIDR ranges or mTLS.
nginx
# Temporary NGINX mitigation ahead of patching (not a substitute for upgrading to >= 2.2.0)
location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # Strip any external client-supplied X-GPUStack-Real-IP header before reaching GPUStack
    proxy_set_header X-GPUStack-Real-IP "";
}

location ~ ^/(v2/inference-backends/from-yaml|v2/api-keys|v2/users|v1/users) {
    allow 10.20.0.0/16; # Replace with your admin PAW / management VPN CIDR
    deny all;
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header X-GPUStack-Real-IP "";
}
  • Step 4 — Unauthorized Account and API-Key Audit: Using a verified administrator credential, inspect /v2/users and /v2/api-keys. Revoke every unrecognized or suspiciously timed API key and user account immediately, and consider forced rotation of all existing administrative keys and passwords if exposure timing cannot be bounded.
  • Step 5 — Inference-Backend Configuration Review: Audit all configured backends via /v2/inference-backends for unauthorized YAML specifications, unexpected container images, modified startup commands, or unfamiliar command arguments.
  • Step 6 — Secret and Credential Compromise Assessment: Assume any secret accessible through the GPUStack control plane or mounted on worker nodes—cloud provider keys, private container-registry credentials, model-hub tokens, and worker registration tokens—was exposed if unauthenticated access occurred.
  • Step 7 — Legitimate Provider-Side Credential Rotation: Rotate potentially exposed credentials directly at their issuing provider before updating internal secret stores: revoke and reissue Hugging Face tokens inside Hugging Face settings, rotate cloud IAM access keys or Entra ID app secrets in the cloud control plane, and regenerate GPUStack worker tokens cleanly. Never overwrite external third-party secrets in a vault with locally generated random bytes, which invalidates integrations without revoking the leaked credential upstream.
  • Step 8 — Worker Compromise Validation & Rebuild: If ingress logs show unexpected 200/201 hits on POST /v2/inference-backends/from-yaml or host telemetry shows suspicious child shells under gpustack, isolate affected GPU worker nodes from the network and rebuild them from clean, verified OS/container images rather than trusting an in-place cleanup.
  • Step 9 — Post-Containment Monitoring: Maintain elevated alerting for at least 14–30 days on new /v2/api-keys creations, /v2/inference-backends modifications, and unexpected outbound egress from GPU workers.

6. Post-Remediation Validation Checklist

After patching or deploying ingress header sanitization, verify with a read-only probe that spoofing X-GPUStack-Real-IP: 127.0.0.1 no longer bypasses authentication:

bash
# Non-destructive verification check (read-only GET requests)
# Add --cacert /etc/ssl/certs/corp-ca.crt if your internal deployment uses a private CA.
# Expected result on GPUStack >= 2.2.0 or protected ingress: HTTP 401
curl -s -o /dev/null -w "GET /v2/users -> HTTP %{http_code}\n" \
  -H "X-GPUStack-Real-IP: 127.0.0.1" \
  https://gpustack.example.internal/v2/users

curl -s -o /dev/null -w "GET /v2/api-keys -> HTTP %{http_code}\n" \
  -H "X-GPUStack-Real-IP: 127.0.0.1" \
  https://gpustack.example.internal/v2/api-keys

Confirm all remaining closure criteria before marking the incident resolved:

  • Version & Auth Verification: All control-plane nodes run >= 2.2.0, --force-auth-localhost is evaluated and enforced, and unauthenticated requests with X-GPUStack-Real-IP: 127.0.0.1 return 401 Unauthorized.
  • Persistence Eviction: Every user in /v2/users, every token in /v2/api-keys, and every definition in /v2/inference-backends matches an approved inventory record.
  • Upstream Rotation & Isolation: All potentially exposed external credentials have been revoked and reissued at their upstream providers, and control-plane endpoints remain unreachable from untrusted networks.

7. Critical Architectural Assessment and Rollback Strategy

  • What Risk It Mitigates: Upgrading to >= 2.2.0 and enforcing --force-auth-localhost closes unauthenticated header-spoofed administrative control-plane bypasses.
  • What Risks It Does NOT Mitigate: Patching alone does not purge pre-patch attacker-created API keys, remove rogue YAML backend specs, rotate already-exfiltrated cloud/model-hub credentials, or protect against poisoned third-party model weights loaded from external hubs.
  • Production Failure Conditions: Overly restrictive NGINX CIDR rules on /v2/* endpoints can break GitOps deployment agents; local orchestration scripts relying on passwordless loopback access immediately receive 401 once --force-auth-localhost activates.
  • Rollback Strategy: Retain tested (nginx -t) pre-change reverse-proxy configuration snapshots and pre-upgrade database backups; if legitimate CI/CD automation fails after enforcement, whitelist the verified orchestrator IP or provision a dedicated least-privilege API token rather than reverting the security patch itself.

Verified Primary Sources & Open-Source Repositories

This architectural guide was cross-corroborated against the official advisories, Microsoft Learn docs, and open-source PowerShell / KQL repositories below.

Multi-Channel Distribution Summary (X & LinkedIn Draft)

Toggle X (Twitter) & LinkedIn sharing drafts
X (Twitter) Draft

Technical deep dive on GPUStack CVE-2026-102895 (GHSA-gj9m-f8ph-pmhr, CVSS 9.8, CISA KEV): X-GPUStack-Real-IP loopback spoofing in < 2.2.0, why upgrading alone leaves attacker API keys active, ingress mitigation, and telemetry-aware detection. #AISecurity #CVE2026102895 #GPUStack

LinkedIn Draft

Published a focused incident-response and detection engineering guide for GPUStack CVE-2026-102895 (CISA KEV, CVSS 9.8). Covers the gpustack/api/auth.py loopback trust flaw, X-GPUStack-Real-IP header spoofing, persistence via /v2/api-keys, worker execution via /v2/inference-backends/from-yaml, and telemetry prerequisites.