GPUStack Authentication Bypass and Remote Code Execution (CVE-2026-102895): Root Cause, Detection, and Incident Response
Technical breakdown of GPUStack CVE-2026-102895 (GHSA-gj9m-f8ph-pmhr) in versions prior to 2.2.0: how default force_auth_localhost=false and X-GPUStack-Real-IP loopback spoofing grant unauthenticated admin access, API-key persistence, RCE via POST /v2/inference-backends/from-yaml, detection prerequisites, and recovery.
1. Executive Summary & Vulnerability Scope
CVE-2026-102895 (GitHub Security Advisory GHSA-gj9m-f8ph-pmhr, CVSS v3.1 9.8 Critical — CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H) is an improper authentication vulnerability affecting GPUStack versions prior to 2.2.0. Tracked under CWE-287 (Improper Authentication), CWE-290 (Authentication Bypass by Spoofing), and CWE-346 (Origin Validation Error), the flaw allows an unauthenticated remote attacker to impersonate the local loopback interface and assume built-in system administrator privileges over the GPUStack control plane.
For Security Architects, SOC L2/L3 analysts, Detection Engineers, and AI Infrastructure Engineers, this is a full control-plane compromise condition rather than a localized UI defect. A single unauthenticated HTTP request can lead to persistent administrative API keys, exposure of worker and cloud-provider secrets, and code execution on GPU worker nodes.
2. Root Cause Analysis: Loopback Trust and X-GPUStack-Real-IP Spoofing
In GPUStack versions prior to 2.2.0, request authentication is enforced inside gpustack/api/auth.py through get_current_user in conjunction with client_ip_getter.
Two design conditions combine to create the authentication bypass:
- Implicit Loopback Trust (`force_auth_localhost = false`): In affected releases, the server configuration option
--force-auth-localhostdefaults tofalse. When disabled, GPUStack assumes any request originating from127.0.0.1is a trusted local administrative operation and automatically elevates the session to the built-in system administrator without validating credentials or tokens. - Blind Trust in `X-GPUStack-Real-IP` and Proxy Peers: Instead of restricting loopback privilege strictly to direct local sockets outside untrusted proxy boundaries,
client_ip_gettertrusted client-controllable header values—specifically `X-GPUStack-Real-IP`—and certain same-host reverse-proxy configurations where the TCP peer appears as127.0.0.1. - Bypass Outcome: Any remote unauthenticated client reaching an affected GPUStack control plane (directly or through a reverse proxy that forwards
X-GPUStack-Real-IP) can injectX-GPUStack-Real-IP: 127.0.0.1to make external traffic resolve as loopback and immediately bypass authentication.
3. Exploitation Path, Persistence, and Impact Classification
To keep architectural triage grounded in verifiable evidence, operational teams should separate established vulnerability behavior from downstream impact and conditional post-exploitation risk:
- CONFIRMED BEHAVIOR: In GPUStack
< 2.2.0running with defaultforce_auth_localhost=false, supplyingX-GPUStack-Real-IP: 127.0.0.1(or abusing a same-host reverse-proxy loopback peer) bypassesget_current_userand grants built-in system administrator context without credentials. CISA KEV and the CVE record document unauthenticated administrative access, user and model modifications, persistent administrator API-key creation, disclosure of worker and cloud-provider secrets, and code execution viaPOST /v2/inference-backends/from-yaml. SANS ISC (Diary 33514) confirmed active probing against exposed instances across model and inference endpoints (/v1/models,/v1/messages). - POTENTIAL IMPACT: Once authenticated as system administrator, an attacker can mint persistent admin bearer tokens via
POST /v2/api-keys, inspect or alter user accounts via/v2/users, modify deployed models and worker settings, exfiltrate stored worker tokens, model-hub credentials, and cloud provider secrets, and execute arbitrary commands on GPU worker nodes by submitting crafted backend specifications toPOST /v2/inference-backends/from-yaml. - HYPOTHETICAL SCENARIO: If compromised GPU worker nodes run with over-privileged cloud IAM roles, broad Kubernetes service-account permissions, or unrestricted internal network egress, an attacker who achieves execution via
from-yamlcould pivot into adjacent object storage or cloud control planes. That downstream lateral movement depends on host and cloud IAM hygiene outside GPUStack itself.
Post-Exploitation Persistence and Why Upgrading Alone Is Insufficient
Patching GPUStack to 2.2.0+ closes the X-GPUStack-Real-IP bypass path, but upgrading the binary alone does not evict an attacker who compromised the cluster prior to patching:
- Database-Backed API Keys Remain Valid: Any persistent administrator API key created via
POST /v2/api-keysbefore upgrading remains active in GPUStack's database and continues to authorize administrative calls against2.2.0+. - Rogue Accounts and Malicious Backends Persist: Unauthorized administrative users added via
/v2/usersor malicious backend definitions registered under/v2/inference-backendssurvive an in-place upgrade. - Exfiltrated Secrets Stay Valid Elsewhere: Cloud credentials, container registry secrets, Hugging Face tokens, and worker bootstrap keys extracted from GPUStack configuration remain usable until explicitly revoked at each upstream provider.
4. Detection Engineering & Telemetry Prerequisites
Effective detection for CVE-2026-102895 depends strictly on what HTTP headers and process telemetry are actually captured in your environment. Always verify logging prerequisites before deploying hunting rules as production alerts.
4.1 Reverse Proxy / Ingress Access Log Triage (Linux Shell)
- DATA SOURCE: Access logs from the reverse proxy or ingress controller fronting GPUStack (NGINX, Envoy, HAProxy, Caddy, or GPUStack Uvicorn HTTP logs).
- TABLE: File-based access logs (such as
/var/log/nginx/gpustack-access.log) or containerstdoutstreams. - REQUIRED INGESTION: Standard NGINX
combinedaccess logs record source IP, URI, and HTTP status, but omit `X-GPUStack-Real-IP` by default. Detecting the spoofed header value itself requires customlog_formatdirectives capturing$http_x_gpustack_real_ip(or structured JSON header logging) enabled prior to the incident. - PURPOSE: Identify external requests probing or hitting privileged administrative routes (
/v2/inference-backends/from-yaml,/v2/api-keys,/v2/users) as well as SANS ISC-observed scan endpoints (/v1/models,/v1/messages), correlated with HTTP200/201responses. - LIMITATIONS: Without custom header logging, this check cannot prove
X-GPUStack-Real-IPspoofing directly—it surfaces endpoint targeting and anomalous successful administrative responses. Traffic reaching GPUStack directly on an unproxied port bypasses reverse-proxy logs completely.
# Triage standard reverse-proxy access logs for high-risk GPUStack endpoints.
# Note: Default NGINX 'combined' format does not log X-GPUStack-Real-IP unless
# $http_x_gpustack_real_ip was explicitly added to log_format beforehand.
grep -E '(/v2/inference-backends/from-yaml|/v2/api-keys|/v2/users|/v1/users|/v1/models|/v1/messages)' \
/var/log/nginx/gpustack-access.log | \
awk '{print $1, $4, $6, $7, $9}' | tail -n 1004.2 Azure Application Gateway WAF Hunting (Microsoft Sentinel KQL)
- DATA SOURCE: Azure Application Gateway WAF access diagnostics deployed in front of the GPUStack ingress tier.
- TABLE:
AzureDiagnosticsscoped toCategory == "ApplicationGatewayAccessLog"(orAGWAccessLogsusing unsuffixed fields such asClientIp,RequestUri,HttpMethod, andHttpStatuswhen resource-specific diagnostic mode is enabled). Note:AppServiceHTTPLogsis not native GPUStack telemetry; GPUStack runs on GPU-equipped Linux VMs, AKS clusters, or container hosts rather than Azure App Service PaaS. - REQUIRED INGESTION: Diagnostic settings on the Azure Application Gateway must be actively streaming access logs to the Microsoft Sentinel Log Analytics workspace.
- PURPOSE: Detect reconnaissance and exploitation attempts against
/v2/inference-backends/from-yaml,/v2/api-keys, and/v2/users, highlighting external client IPs receiving200/201success codes. - LIMITATIONS: Standard
ApplicationGatewayAccessLogschemas do not record arbitrary custom request headers such asX-GPUStack-Real-IP. Therefore, this KQL query cannot detect `X-GPUStack-Real-IP` header spoofing directly; it detects anomalous administrative URI access and unexpected HTTP success responses. Header-level detection in Sentinel requires custom WAF inspection rules or reverse-proxy header logging.
// Data Source: Azure Log Analytics (AzureDiagnostics - Application Gateway Access Logs, WAF v2)
// Execution Scope: Read-only hunting query (requires Log Analytics Reader role)
AzureDiagnostics
| where TimeGenerated > ago(14d)
| where Category == "ApplicationGatewayAccessLog"
| where requestUri_s has_any (
"/v2/inference-backends/from-yaml",
"/v2/api-keys",
"/v2/users",
"/v1/users",
"/v1/models",
"/v1/messages"
)
| extend StatusCode = toint(httpStatus_d)
| summarize
RequestCount = count(),
SuccessCount = countif(StatusCode in (200, 201, 204)),
Methods = make_set(httpMethod_s),
StatusCodes = make_set(StatusCode)
by clientIP_s, requestUri_s, bin(TimeGenerated, 1h)
| where SuccessCount > 0 or RequestCount >= 5
| order by TimeGenerated desc4.3 GPU Worker Post-Exploitation Process Hunting (Microsoft Defender for Endpoint KQL)
- DATA SOURCE: Endpoint process telemetry from Microsoft Defender for Endpoint (MDE for Linux) running on GPUStack control-plane or worker hosts.
- TABLE:
DeviceProcessEvents. - REQUIRED INGESTION: MDE Linux sensor deployed on GPU nodes with telemetry flowing to Microsoft Defender XDR Advanced Hunting or a connected Microsoft Sentinel workspace.
- PURPOSE: Detect post-exploitation execution following
POST /v2/inference-backends/from-yamlabuse, wheregpustack,uvicorn, or worker Python processes spawn interactive shells (sh,bash,dash) or network download/reverse-shell utilities (curl,wget,nc). - LIMITATIONS: Cannot detect API-only abuse such as reading secrets or generating
/v2/api-keyswithout host shell execution. Requires environment tuning against legitimate container startup or model-provisioning scripts.
// Data Source: Microsoft Defender XDR Advanced Hunting (DeviceProcessEvents)
// Execution Scope: Read-only telemetry query (requires Security Reader role + MDE onboarding)
DeviceProcessEvents
| where Timestamp > ago(14d)
| where InitiatingProcessFileName has_any ("gpustack", "python", "python3", "uvicorn")
| where InitiatingProcessCommandLine has_any ("gpustack", "inference-backend", "worker")
| where FileName in~ ("sh", "bash", "dash", "curl", "wget", "nc", "ncat", "socat")
or ProcessCommandLine has_any ("base64 -d", "/dev/tcp/", "curl -s", "wget -q")
| project Timestamp, DeviceName, AccountName, InitiatingProcessFileName,
InitiatingProcessCommandLine, FileName, ProcessCommandLine
| order by Timestamp desc5. Containment, Remediation, and Post-Compromise Recovery
Execute containment and recovery in sequence so that closing the entry point is immediately followed by persistence eviction and upstream credential rotation:
- Step 1 — Exposure Identification: Inventory all GPUStack server and worker deployments (
gpustack versionor container image digest/tag). Flag every instance below2.2.0and identify any control-plane port exposed to the public internet, untrusted ingress controllers, or broad developer subnets. - Step 2 — Upgrade & Authentication Hardening: Upgrade all GPUStack instances to 2.2.0 or later (2.2.2+ recommended), enforce
--force-auth-localhostso local loopback connections still require explicit credentials, and remove public internet exposure from the control plane. - Step 3 — Temporary Ingress Mitigation (If Immediate Upgrade Is Blocked): If change-control windows delay upgrading temporarily, configure the reverse proxy or ingress controller in front of GPUStack to strip any client-supplied
X-GPUStack-Real-IPheader and restrict/v2/inference-backends/from-yaml,/v2/api-keys, and/v2/usersto trusted administrative CIDR ranges or mTLS.
# Temporary NGINX mitigation ahead of patching (not a substitute for upgrading to >= 2.2.0)
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Strip any external client-supplied X-GPUStack-Real-IP header before reaching GPUStack
proxy_set_header X-GPUStack-Real-IP "";
}
location ~ ^/(v2/inference-backends/from-yaml|v2/api-keys|v2/users|v1/users) {
allow 10.20.0.0/16; # Replace with your admin PAW / management VPN CIDR
deny all;
proxy_pass http://127.0.0.1:8080;
proxy_set_header X-GPUStack-Real-IP "";
}- Step 4 — Unauthorized Account and API-Key Audit: Using a verified administrator credential, inspect
/v2/usersand/v2/api-keys. Revoke every unrecognized or suspiciously timed API key and user account immediately, and consider forced rotation of all existing administrative keys and passwords if exposure timing cannot be bounded. - Step 5 — Inference-Backend Configuration Review: Audit all configured backends via
/v2/inference-backendsfor unauthorized YAML specifications, unexpected container images, modified startup commands, or unfamiliar command arguments. - Step 6 — Secret and Credential Compromise Assessment: Assume any secret accessible through the GPUStack control plane or mounted on worker nodes—cloud provider keys, private container-registry credentials, model-hub tokens, and worker registration tokens—was exposed if unauthenticated access occurred.
- Step 7 — Legitimate Provider-Side Credential Rotation: Rotate potentially exposed credentials directly at their issuing provider before updating internal secret stores: revoke and reissue Hugging Face tokens inside Hugging Face settings, rotate cloud IAM access keys or Entra ID app secrets in the cloud control plane, and regenerate GPUStack worker tokens cleanly. Never overwrite external third-party secrets in a vault with locally generated random bytes, which invalidates integrations without revoking the leaked credential upstream.
- Step 8 — Worker Compromise Validation & Rebuild: If ingress logs show unexpected
200/201hits onPOST /v2/inference-backends/from-yamlor host telemetry shows suspicious child shells undergpustack, isolate affected GPU worker nodes from the network and rebuild them from clean, verified OS/container images rather than trusting an in-place cleanup. - Step 9 — Post-Containment Monitoring: Maintain elevated alerting for at least 14–30 days on new
/v2/api-keyscreations,/v2/inference-backendsmodifications, and unexpected outbound egress from GPU workers.
6. Post-Remediation Validation Checklist
After patching or deploying ingress header sanitization, verify with a read-only probe that spoofing X-GPUStack-Real-IP: 127.0.0.1 no longer bypasses authentication:
# Non-destructive verification check (read-only GET requests)
# Add --cacert /etc/ssl/certs/corp-ca.crt if your internal deployment uses a private CA.
# Expected result on GPUStack >= 2.2.0 or protected ingress: HTTP 401
curl -s -o /dev/null -w "GET /v2/users -> HTTP %{http_code}\n" \
-H "X-GPUStack-Real-IP: 127.0.0.1" \
https://gpustack.example.internal/v2/users
curl -s -o /dev/null -w "GET /v2/api-keys -> HTTP %{http_code}\n" \
-H "X-GPUStack-Real-IP: 127.0.0.1" \
https://gpustack.example.internal/v2/api-keysConfirm all remaining closure criteria before marking the incident resolved:
- Version & Auth Verification: All control-plane nodes run
>= 2.2.0,--force-auth-localhostis evaluated and enforced, and unauthenticated requests withX-GPUStack-Real-IP: 127.0.0.1return401 Unauthorized. - Persistence Eviction: Every user in
/v2/users, every token in/v2/api-keys, and every definition in/v2/inference-backendsmatches an approved inventory record. - Upstream Rotation & Isolation: All potentially exposed external credentials have been revoked and reissued at their upstream providers, and control-plane endpoints remain unreachable from untrusted networks.
7. Critical Architectural Assessment and Rollback Strategy
- What Risk It Mitigates: Upgrading to
>= 2.2.0and enforcing--force-auth-localhostcloses unauthenticated header-spoofed administrative control-plane bypasses. - What Risks It Does NOT Mitigate: Patching alone does not purge pre-patch attacker-created API keys, remove rogue YAML backend specs, rotate already-exfiltrated cloud/model-hub credentials, or protect against poisoned third-party model weights loaded from external hubs.
- Production Failure Conditions: Overly restrictive NGINX CIDR rules on
/v2/*endpoints can break GitOps deployment agents; local orchestration scripts relying on passwordless loopback access immediately receive401once--force-auth-localhostactivates. - Rollback Strategy: Retain tested (
nginx -t) pre-change reverse-proxy configuration snapshots and pre-upgrade database backups; if legitimate CI/CD automation fails after enforcement, whitelist the verified orchestrator IP or provision a dedicated least-privilege API token rather than reverting the security patch itself.
Verified Primary Sources & Open-Source Repositories
This architectural guide was cross-corroborated against the official advisories, Microsoft Learn docs, and open-source PowerShell / KQL repositories below.
Multi-Channel Distribution Summary (X & LinkedIn Draft)
Toggle X (Twitter) & LinkedIn sharing drafts
Multi-Channel Distribution Summary (X & LinkedIn Draft)
Toggle X (Twitter) & LinkedIn sharing draftsTechnical deep dive on GPUStack CVE-2026-102895 (GHSA-gj9m-f8ph-pmhr, CVSS 9.8, CISA KEV): X-GPUStack-Real-IP loopback spoofing in < 2.2.0, why upgrading alone leaves attacker API keys active, ingress mitigation, and telemetry-aware detection. #AISecurity #CVE2026102895 #GPUStack
Published a focused incident-response and detection engineering guide for GPUStack CVE-2026-102895 (CISA KEV, CVSS 9.8). Covers the gpustack/api/auth.py loopback trust flaw, X-GPUStack-Real-IP header spoofing, persistence via /v2/api-keys, worker execution via /v2/inference-backends/from-yaml, and telemetry prerequisites.