* reverseproxy: set a write deadline on websocket close messages
Cleanup sends a Close control message to every attached websocket while
holding the tunnel tracker lock, during a config reload. If the peer has
stopped reading, the tunnel's copy goroutine is blocked writing to it and
the close message waits behind that write forever. The reload never
finishes and holds the config lock, so every later /load and /config
request hangs until Caddy is restarted.
Setting a write deadline before the close message unblocks both writes.
The connection is closed right after, so the deadline is never cleared.
* Add bot disclosure
* reverseproxy: make close control timeout a const, harden stalled-peer test
* Delete BOT_DISCLOSURE.md
Signed-off-by: Francis Lavoie <lavofr@gmail.com>
* reverseproxy: bound each batch of websocket close messages by one deadline
The close loops are serial and hold the tunnel tracker lock, so a
per-connection write timeout let N stalled peers hold a reload for N
times the timeout. Compute one absolute deadline per batch instead,
and skip the best-effort close frame on connections that cannot set a
write deadline rather than risk the hang.
Extended CONNECT streams gain SetWriteDeadline through
http.ResponseController. For HTTP/1 upgrades, net/http returns the
upstream socket inside a body that hides its deadline methods, so the
upstream close message was never bounded; record the connection from
the request's ClientTrace and write the close message to that socket.
Restores the TODO about being the exclusive writer, per review.
* reverseproxy: make websocket close tests deterministic
Per review, the stalled-peer tests use net.Pipe instead of real TCP:
it is unbuffered and lets one write through at a time, so a close
message waits behind a stuck copy write without large allocations or
sleeps. The batch test checks that graceful closes share one deadline
instead of timing cleanup.
The HTTP/1 upstream close test now runs a real upgrade through
ServeHTTP and the transport, so it fails if the GotConn hook is
removed. The choice of upstream close writer moves into
upstreamCloseWriter, with a table test for the HTTP/2 and HTTP/3 case.
* reverseproxy: record the upstream conn with atomic.Pointer
Per review, store the connection from the GotConn hook in an
atomic.Pointer[net.Conn] instead of a mutex-guarded holder type.
Rename upstreamCloseWriter to upstreamWriteCloser. It returns
io.ReadWriteCloser, since both the body and the recorded socket are
one, so gracefulClose keeps its original io.ReadWriteCloser parameter.
* reverseproxy: share one close deadline across detached trackers
Per review, an upstream removal notifies every stopped tracker in
turn, and each close loop holds its own lock, so a deadline per
tracker let N trackers with stalled peers hold a reload for N times
the timeout. Compute one absolute deadline in
notifyDetachedTunnelTrackersOfUpstreamRemoval and pass it through to
closeConnectionsForUpstream.
Add a test with two stopped trackers holding stalled net.Pipe peers
that asserts both graceful closes receive the same deadline.
* reverseproxy: simplify the websocket close deadline change
Fold the two tunnel close loops into closeConnectionsLocked, so the
shared-deadline rule and its rationale live in one place. Replace the
per-tracker closeWriteTimeout field, which production never varied,
with a package variable that tests shorten.
Record the upstream socket only for websocket requests, since only
their graceful close reads it, and store a copy of the connection
rather than the address of the hook's parameter. Use the package's
caddy.CtxKey convention for the context key.
Share the upgrade tunnel fixture between the half-close test and the
upstream close message test.
When downgrading an HTTP/2 or HTTP/3 extended CONNECT WebSocket request
to HTTP/1.1, delete any parsed client-supplied Sec-WebSocket-Key header
prior to installing Caddy's generated 16-byte random key. This prevents
request header operations from canonicalizing and combining both keys
into duplicate Sec-WebSocket-Key headers upstream.
Signed-off-by: duongynhi000005-oss <duongynhi000005@gmail.com>
* caddyhttp: presize buffers in hot paths to reduce allocations
Right-size three per-request allocations that previously grew from a
nil/empty backing array, causing repeated reallocation:
- CleanPath: pre-Grow the strings.Builder used when slash collapsing is
disabled, instead of letting it reallocate as runes are appended
- headers: presize the rewritten value slice in HeaderOps.ApplyTo's Set
loop instead of appending from nil
- reverseproxy: presize the weights slice in WeightedRoundRobinSelection
.Select
This commit also adds benchmarks exercising each path. Measured with benchstat
with count=20 and GOMAXPROCS=1, with p=0.000:
sec/op B/op allocs/op
CleanPathNoCollapse 1013.4n -> 828.6n (-18.23%) 120->64 (-46.67%) 4->1 (-75.00%)
HeaderOpsApplyToSet 1262.0n -> 955.8n (-24.27%) 255->152 (-40.39%) 9->6 (-33.33%)
WeightedRoundRobinSelect 215.8n -> 200.3n ( -7.20%) 112->96 (-14.29%) 2->2 (N/A)
* caddyhttp: presize strings.Builder in query rewrite and path matcher
Two builders whose output size is bounded by a known input length were
growing from empty, causing repeated reallocation. Seed each with a
capacity hint:
- rewrite.buildQueryString: pre-Grow to len(qs) (runs per request when
rewrite query manipulation is configured)
- MatchPath.matchPatternWithEscapeSequence: pre-Grow to len(escapedPath)
(reached only when a path pattern contains % escape sequences)
Adds a benchmark for buildQueryString, measured with benchstat:
BuildQueryString:
- sec/op 965.2n -> 743.3n (-22.99%)
- B/op 248->80 (-67.74%)
- allocs/op 5->1 (-80.00%)
The builder previously reallocated four times per call. The matcher path
is cold and not separately benchmarked.
* chore: add benchmark headers
---------
Signed-off-by: Julien Voisin <jvoisin@users.noreply.github.com>
* httpcaddyfile: reject conflicting client_auth when merging TLS policies
* httpcaddyfile: use host and host:port site addresses in client auth conflict test
* httpcaddyfile: enhance client auth consolidation test to handle conflicting configurations
* httpcaddyfile: remove unnecessary blank line in conflict error handling
* caddyhttp: lower-case path patterns with placeholders at request time
Patterns with placeholders were lower-cased at provisioning, which
changed case-sensitive placeholder names such as {env.FOO}, while the
replaced values were never lower-cased and so could not match the
lower-cased request path. Leave such patterns untouched at provisioning
and lower-case the expanded pattern at request time.
Fixes#6387
* caddyhttp: restore the encoded separator path matcher tests
* reverseproxy: reduce lock contention in dynamic SRV and A upstreams (#7946)
The shared SRV and A/AAAA caches held a global write lock across the DNS
lookup, so one slow or timing-out record blocked every unrelated
upstream source.
Lookups now run without the cache lock held, deduplicated per key with
singleflight. A per-key generation (stored on the cache entry, so it is
bounded by the cache cap) and a cache-wide epoch keep a lookup that was
in flight during ResetSRV/ResetAllSRV from writing its result back, and
are part of the singleflight key so callers after a reset do not join a
pre-reset lookup.
The shared lookup is detached from the request that started it and
bounded by its own 30s timeout; each caller still returns as soon as its
own context is done. Distinct lookups across both caches are capped by a
shared gate (64 running, 256 queued); beyond that they are shed, serving
a stale SRV entry if a grace period is set and one is cached.
SRV and A now share one cache implementation. AUpstreams also implements
CachingUpstreamSource (via new ResetA/ResetAllA), so retries purge its
cache entry like SRVUpstreams already did.
Rebased onto the move of this code into modules/internal/network (#7790,
#8252).
Fixes#7946
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* network: backdate cache entries in TestShedLookups instead of a 1ns refresh
On Windows two close time.Now() calls can report no elapsed time, so an
entry seeded with a 1ns refresh could still be fresh and skip the shed
path entirely.
* network: move the shared cache and lookup gate into cache.go
dynamicupstreams.go keeps the SRV/A API; the generic cache, its
singleflight/generation handling and the lookup gate are not specific to
either record type. No functional change.
* network: preserve cache reset and grace semantics
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
Co-authored-by: steadytao <mail@steadytao.com>
HTTPBasicAuth.correctPassword built its hash cache key with
append(account.password, plaintextPassword...). account.password is
shared by all requests and can have spare capacity, so short passwords
were written into the stored hash's backing array, causing a data race
between concurrent requests and potentially mixed-up cache keys.
Build the key in a fresh buffer instead, and add regression tests.
* requestbody: reject oversized Content-Length before the next handler
When max_size is set and the request declares a Content-Length above it, respond 413 right away instead of letting the next handler start and fail mid-read. Bodies of unknown length are still limited by MaxBytesReader. Both paths return the same error, so handle_errors sees no difference.
* caddytest: send unknown-length bodies in max_size placeholder tests
request_body now answers 413 for a declared Content-Length over max_size before the placeholder reads the body, so these tests no longer covered the read path and the direct respond one failed.
* requestbody: keep the body limited for error routes
Error routes get the same request, so wrap the body before the early 413 to stop them from reading past max_size.
{args[-1]} and {args.-1} parsed as -1 and indexed the args slice
directly, panicking with index out of range. Treat negative indices as
out of bounds, like indices past the end.
* admin: authorize remote requests matching any permission entry
enforceAccessControls rejected a request as soon as one permission entry
did not match its method or path, so only the first entry was usable.
Allow the request if any entry permits both, and deny only after all fail.
* admin: check every access entry that holds the client key
A key listed in several access control entries was only checked
against the first one; deny only after all of them are tried.
Use the chosen file info, not a zero size, to tell whether a candidate
was chosen, so smallest_size keeps an empty file and largest_size can
choose one.
When a handle_response status code did not evaluate to an integer, the
handler returned a 500 without closing the upstream response body,
leaking the connection.
* caddyauth: Support third-party http.authentication.hashes modules
basic_auth and hash-password only accepted the built-in bcrypt and
argon2id algorithms, even though the http.authentication.hashes
namespace is documented as extensible. Look up any other algorithm
name among the installed hash modules instead, and list the available
ones when the name is not recognized.
The built-in algorithms keep their own hash-password flags; other
modules are provisioned and must implement Hasher to generate hashes.
Closes#6670
* caddyauth: Base64-encode non-MCF hashes from hash-password modules
hash-password printed module hashes as raw bytes, but basic_auth
base64-decodes any password that isn't in Modular Crypt Format, so the
output couldn't be used as-is. Encode non-MCF hashes as base64 and add
a round-trip test that provisions HTTPBasicAuth with the command output.
* caddyauth: always base64-encode third-party hash output
* reverseproxy: Test that WebSockets + unix-sockets works
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Close temp file before reusing its path for the socket
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Strengthen unix socket WebSocket test, reuse tempSocketName helper
Check that upstream bytes sent with the 101 are delivered, echo a payload larger than any buffer, and verify the upstream connection closes when the client disconnects. Reuse the existing tempSocketName helper for all unix socket tests, which also closes the temp file descriptor the older tests leaked.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* httpcaddyfile: Don't merge automation policies if it would shadow another
Signed-off-by: Bartlomiej Wisniowski <56162793+bwisn@users.noreply.github.com>
* httpcaddyfile: add reverse case for shadowing merge, use force_automate to match the real life usage and explain why we're not checking the inverse
Signed-off-by: Bartlomiej Wisniowski <56162793+bwisn@users.noreply.github.com>
* httpcaddyfile: Check overlap in both directions when merging policies
Signed-off-by: Bartlomiej Wisniowski <56162793+bwisn@users.noreply.github.com>
---------
Signed-off-by: Bartlomiej Wisniowski <56162793+bwisn@users.noreply.github.com>
* caddyhttp: let handler read deadlines cap the idle read timeout
IdleTimeoutReader sets the read deadline before every body read, so a
deadline set by a handler with http.ResponseController.SetReadDeadline()
was replaced before it could take effect. Since #7913, handlers can't
bound a stalled request body more tightly than read_body.
A handler's read deadline is now routed to the IdleTimeoutReader, which
applies it right away and keeps it as a ceiling for every idle reset and
for the deadline put back between HTTP/2 reads. A handler can make the
idle timeout stricter but can't loosen it, and a zero deadline removes
the ceiling, as #8145 does for write deadlines.
* caddyhttp: keep the hard read deadline and the drain deadline with handler deadlines
SetReadDeadline installed the handler's deadline as is, so a later or zero
deadline lifted read_timeout. It now installs the earlier of the two.
HandlerDone cleared a deadline set by the handler before arming the HTTP/1
drain deadline, leaving the post-handler drain unbounded.
* caddyhttp: keep the idle deadline of a read in flight on handler deadlines
SetReadDeadline replaced the idle deadline armed for a blocked read with the
handler's ceiling, so a zero or later deadline lifted the idle timeout for
that read. It now installs the earlier of the two until the read returns.
* caddyhttp: keep the expired drain deadline of a timed-out body on handler deadlines
After a body read timed out on HTTP/1, a zero or later deadline set by the
handler replaced the expired one kept to bound net/http's post-handler drain.