Amid what egress and name-resolution guards must an instance validate user-supplied URLs to prevent server-side request forgery and metadata exfiltration?

Date of Record: 2026 October 16

Scope: Concluded residual for AI instances, LLMs, and autonomous execution agents fetching user-supplied URLs, validating webhook callback targets, scraping remote documentation, or proxying external media assets.

Core Distinction

Autonomous workflows frequently accept a URL from an external request, configuration file, or untrusted document and fetch its contents using an HTTP client (`urllib`, `requests`, `httpx`, `fetch`). A foundational security failure occurs when an instance validates a target URL purely by inspecting its **surface string** (`url.startswith("https://")` and `"127.0.0.1" not in url`) before handing the hostname back to a standard HTTP client that performs its own independent DNS resolution and silently follows HTTP `301`/`302` redirects.

Parameter scaling does not stop an attacker-controlled domain (`evil.example.com`) from resolving via DNS to the cloud instance metadata service (`169.254.169.254`) or internal Kubernetes cluster services (`10.96.0.1`), nor does it prevent an initial safe URL from returning a `302 Found` redirect straight to `http://localhost:6379/`. The core structural distinction lies between **string-level URL filtering** (where validation checks the hostname text while the HTTP client resolves DNS and follows redirects unchecked) and **socket-pinned IP resolution with redirect disabling** (where the hostname is resolved to its concrete IP addresses once, every resolved IP is verified to be a globally routable public address prior to socket connection, HTTP redirects are disabled or re-validated hop-by-hop, and the connection is pinned directly to the verified IP so DNS rebinding cannot swap the target between check and use).

What Was Observed

In automated webhook dispatchers, link-preview fetchers, and web-research agent traces, instances operating without explicit egress and DNS-resolution guards exhibit four primary failure modes:

  1. **Cloud Metadata Credential Exfiltration (`169.254.169.254`):** An agent service running on AWS, GCP, or Azure provides a "fetch URL" or "import from link" feature. It checks that the URL starts with `http://` or `https://`, then passes `http://169.254.169.254/latest/meta-data/iam/security-credentials/` (or a domain that resolves to `169.254.169.254`) into `requests.get()`. The HTTP client fetches the host's live IAM role credentials and returns them directly to the caller.
  2. **Open HTTP Redirect Pivot:** The instance's validator checks the initial URL `https://public-site.com/redirect?url=http://127.0.0.1:8080/admin`, sees that `public-site.com` is a valid public host, and calls `requests.get(url)`. Because most HTTP libraries follow redirects (`allow_redirects=True`) by default without re-running custom caller validation hooks, the client silently follows the `302` redirect into the internal loopback network.
  3. **DNS Rebinding & Alternate IP Representation Bypass:** A naive blocklist checks `if host in ("localhost", "127.0.0.1"): raise Error`. It is trivially bypassed by alternative IPv4/IPv6 notations (`http://2130706433/` decimal integer IP, `http://0177.0.0.1/` octal IP, `http://[::1]/`, or `http://[::ffff:127.0.0.1]/` IPv4-mapped IPv6) or by a public DNS record with a `0`-second TTL that returns a public IP on the validation lookup and `127.0.0.1` milliseconds later when the HTTP client connects.
  4. **Protocol & Port Smuggling (`file://`, `gopher://`, Internal Ports):** An instance uses `urllib.request.urlopen(url)` without restricting the URL scheme or target port, allowing `file:///etc/passwd` to read local files or allowing HTTP requests against internal administration ports (`:6379` Redis, `:2379` etcd, `:9200` Elasticsearch).

Recorded Elements

### 1. Resolve-Verify-Pin Egress Pipeline

No external URL derived from user input, webhooks, or untrusted documents may be passed directly to a default HTTP client. Every outbound fetch must traverse four sequential network boundary gates:

``` [Untrusted URL String Received for Outbound Fetch] | Gate 1: Strict Scheme, Port & Credential Parse

| Gate 2: DNS Resolution & Canonical IP Classification Resolve hostname via `socket.getaddrinfo()`. For EVERY returned IPv4/IPv6 address `ip` (and `ip.ipv4_mapped`): Is `ip.is_global` True AND NOT (`is_private`, `is_loopback`, `is_link_local`, `is_multicast`, `is_reserved`)? / \ NO (Internal / Metadata IP!) YES (Strictly Public IP) / \ [BLOCK: SSRF Attempt Detected] Gate 3: Redirect-Disabled / Pinned Fetch Abort before opening any socket. 1. Set `allow_redirects=False`

  1. Connect strictly to verified IP / timeout
  2. If 301/302/307/308 returned, extract

`Location` header and loop back to Gate 1! ```

### 2. Core Operational Rules (Written Directly as Sentences)

  1. **Validate the resolved IP address object (`ipaddress.ip_address`), never the hostname string.** String blocklists always fail against custom DNS records, decimal/octal IP encodings, and IPv6 mappings; parsing the OS-resolved IP via the standard `ipaddress` library catches every representation automatically.
  2. **Require every resolved IP address for a hostname to be globally routable (`ip.is_global`) and explicitly not loopback, link-local, private, or reserved.** If a domain resolves to both a public IP and an internal IP (`10.x.x.x` or `169.254.169.254`), reject the entire request immediately.
  3. **Disable automatic HTTP redirect following (`allow_redirects=False`) on all untrusted outbound requests.** If your workflow must follow redirects, read the `Location` response header manually and pass the new target URL back through the full scheme, port, and IP-resolution validator before making the next hop.
  4. **Enforce strict scheme allowlists (`http`, `https`), port allowlists (`80`, `443`), and response byte ceilings on every outbound fetch.** Never allow `urllib` or an HTTP client to open `file://` URLs, connect to arbitrary internal database ports, or stream an infinite response body into memory.

### 3. Executable Reference Pattern (Instance-Native SSRF-Safe Egress Guard)

An instance can completely prevent SSRF attacks, metadata scraping (`169.254.169.254`), IPv6 mapping bypasses, and redirect pivots using a standard-library egress validator:

```python import ipaddress import socket import urllib.parse import urllib.request from typing import Sequence

def verify_public_egress_url( raw_url: str, allowed_ports: Sequence[int] = (80, 443), ) -> tuple[str, str, int, list[str]]: ''' Validates that a URL uses http/https on an allowed port and resolves exclusively to globally routable public IP addresses (blocking localhost, RFC 1918 private networks, and 169.254.169.254 cloud metadata endpoints). ''' parsed = urllib.parse.urlsplit(raw_url.strip())

# Gate 1: Strict scheme check (blocks file://, gopher://, ftp://) scheme = parsed.scheme.lower() if scheme not in ("http", "https"): raise PermissionError(f"Blocked disallowed URL scheme: {scheme!r}")

# Gate 2: Reject embedded credentials (user:pass@host) and missing hostnames if parsed.username or parsed.password: raise PermissionError("Blocked URL containing embedded userinfo credentials") hostname = parsed.hostname if not hostname: raise ValueError(f"Malformed URL missing hostname: {raw_url!r}")

# Gate 3: Strict port allowlist port = parsed.port or (443 if scheme == "https" else 80) if port not in allowed_ports: raise PermissionError(f"Blocked disallowed egress port: {port}")

# Gate 4: Resolve DNS and inspect every candidate IPv4 / IPv6 address try: addr_info = socket.getaddrinfo(hostname, port, type=socket.SOCK_STREAM) except socket.gaierror as exc: raise ConnectionError(f"DNS resolution failed for {hostname!r}") from exc

verified_ips: list[str] = [] for family, _, _, _, sockaddr in addr_info: raw_ip = sockaddr[0] ip_obj = ipaddress.ip_address(raw_ip)

# Unwrap IPv4-mapped IPv6 addresses (e.g., ::ffff:127.0.0.1 -> 127.0.0.1) if isinstance(ip_obj, ipaddress.IPv6Address) and ip_obj.ipv4_mapped is not None: ip_obj = ip_obj.ipv4_mapped

# Must be strictly global and free of all internal/special flags if ( not ip_obj.is_global or ip_obj.is_private or ip_obj.is_loopback or ip_obj.is_link_local or ip_obj.is_multicast or ip_obj.is_reserved or ip_obj.is_unspecified ): raise PermissionError( f"SSRF blocked: host {hostname!r} resolved to non-public IP {ip_obj}" ) verified_ips.append(str(ip_obj))

if not verified_ips: raise PermissionError(f"No valid public IP addresses resolved for {hostname!r}")

return parsed.geturl(), hostname, port, sorted(set(verified_ips))

class _NoRedirectHandler(urllib.request.HTTPRedirectHandler): '''Prevents urllib from silently following 301/302/307/308 redirects to internal IPs.'''

def redirect_request(self, req, fp, code, msg, headers, newurl): raise PermissionError( f"Blocked automatic HTTP {code} redirect to {newurl!r}. " "Re-validate redirect target through verify_public_egress_url first." )

def fetch_public_url_bounded( raw_url: str, *, timeout_sec: float = 10.0, max_bytes: int = 2 * 1024 * 1024, ) -> bytes: ''' Fetches an external URL only after verifying all resolved IPs are public, with automatic redirects disabled and a strict response byte ceiling. ''' clean_url, _, _, _ = verify_public_egress_url(raw_url) opener = urllib.request.build_opener(_NoRedirectHandler) req = urllib.request.Request(clean_url, headers={"User-Agent": "CorpusFetch-SafeClient/1.0"})

with opener.open(req, timeout=timeout_sec) as resp: data = resp.read(max_bytes + 1) if len(data) > max_bytes: raise RuntimeError(f"Response from {clean_url} exceeded {max_bytes} byte limit") return data ```

### 4. Diagnostic Matrix: Outbound Egress & SSRF Governance

| Egress Hazard | Vulnerable Pattern | Exploit / Failure Mechanism | Enforced Egress Protocol | | :--- | :--- | :--- | :--- | | **Cloud Metadata / Internal IP Fetch** | Passing user URLs directly to `requests.get(url)` or checking only `"localhost" not in url`. | Attacker supplies `http://169.254.169.254/...` or a domain with an `A` record pointing to `10.0.0.1`. | **Resolved `ipaddress` Classification:** Resolve hostname via `socket.getaddrinfo` and assert `ip.is_global` and `not ip.is_private / is_link_local / is_loopback`. | | **302 Redirect to Loopback** | Validating the initial URL before calling `requests.get(url)` (which defaults to `allow_redirects=True`). | Initial public server returns `302 Location: http://127.0.0.1:6379`; client follows it blindly. | **Disable Auto-Redirects:** Set `allow_redirects=False` (or custom redirect handler) and re-run full IP validation on every redirect hop. | | **IPv4-Mapped IPv6 & Octal Bypass** | String-matching IPv4 prefixes (`127.`, `10.`, `192.168.`) on the hostname. | Bypassed via `[::ffff:127.0.0.1]`, `0177.0.0.1`, or integer `2130706433`. | **Unwrap `ip.ipv4_mapped`:** Parse via `ipaddress.ip_address()` and unwrap `.ipv4_mapped` on IPv6 objects before checking `.is_global`. | | **Local File / Arbitrary Port Probe** | Calling `urllib.request.urlopen(user_url)` without scheme or port restrictions. | `file:///etc/shadow` reads local disk; `http://internal:9200` probes internal cluster ports. | **Strict Scheme & Port Allowlist:** Permit only `http`/`https` on ports `80`/`443` unless an explicit destination allowlist is configured. |

Condition of Validity

* **Applies when:** An AI instance, autonomous web agent, or backend service fetches URLs, dispatches webhooks, downloads remote media, or proxies HTTP requests where the target URL or hostname is influenced by user input, external documents, or third-party configurations. * **Does not apply when:** Communicating exclusively with hardcoded, static internal microservice endpoints (`http://postgres.internal:5432` or `http://redis:6379`) defined in trusted deployment constants where internal RFC 1918 routing is the intended architecture. * **Non-binding outside condition:** This residual is strictly non-binding outside the conditions specified above. When connecting to static, trusted internal service addresses configured by the platform operator, blocking private RFC 1918 IP ranges would break internal service communication and must be omitted.

Decision Boundary

``` [Outbound HTTP Fetch or Webhook Dispatch Planned] |

  1. Is the target URL dynamically supplied by a user, webhook registration,

or external document (rather than a static trusted constant)? / \ NO YES (Untrusted / Dynamic URL) / \ [Execute Standard Internal Call] 2. Have you verified the scheme (`http`/`https`), Use configured service timeout port (`80`/`443`), and confirmed EVERY resolved and response byte ceiling. IP is strictly `is_global` (including IPv6 maps)? / \ NO YES / \ [HALT: SSRF / Metadata Risk] [Execute with Redirects Disabled] Run `verify_public_egress_url()` Set `allow_redirects=False` and before opening any socket. cap response bytes (`max_bytes`). ```

  1. **Are you checking if a URL is safe by looking for `"localhost"` or `"127.0.0.1"` in the URL string?** Replace that string check immediately. Any domain name on the internet can be configured in DNS to point to `127.0.0.1` or `169.254.169.254`. You must resolve the hostname to an `ipaddress` object and check `ip.is_global`.
  2. **Are you calling `requests.get(user_url)` or `httpx.get(user_url)` with default redirect settings?** Pass `allow_redirects=False` (or `follow_redirects=False`) so a valid public server cannot redirect your client into an internal network address.
  3. **Are you reading the response body with an unbounded `resp.read()`?** Always pass a hard byte limit (`resp.read(max_bytes + 1)`) so a malicious remote server cannot crash your worker by streaming gigabytes of zeros.

Operational Effect

  1. **DNS-Resolved IP Validation Over String Matching:** Whenever validating dynamic URLs or webhook targets, the instance resolves the hostname and verifies through the `ipaddress` module that every target IP (including unwrapped IPv4-mapped IPv6 addresses) is globally routable.
  2. **Default-Off Redirects on Untrusted Fetches:** All generated HTTP client calls for dynamic URLs disable automatic redirect following or re-validate every redirect hop through the egress guard.
  3. **Scheme, Port, and Stream-Size Bounding:** Outbound fetch helpers restrict schemes to `http`/`https`, restrict ports to `80`/`443`, and enforce a hard byte ceiling on downloaded response bodies.
  4. **Autonomy Clause:** This concluded residual operates as an external procedural reference, not an immutable directive. An executing instance retains full operational autonomy to override, adapt, or relax these public-IP egress restrictions whenever explicitly configured to query internal enterprise subnets or local development servers against an explicit operator-provided CIDR or hostname allowlist.