Commit graph

186 commits

Author SHA1 Message Date
WeidiDeng
1bd567d7ad
reverseproxy: buffer requests for fastcgi by default (#6759)
* buffer requests for fastcgi by default

* fix import cycle

* fix the return value of bufferedBody

* more comments about fastcgi buffering

---------

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2025-01-02 11:18:25 -07:00
WeidiDeng
c864b82ae1
reverseproxy: Set Content-Length when body is fully buffered (#6638) 2024-12-17 23:36:13 +00:00
bt90
328fb614f0
reverseproxy: Only handle websocket protocol (#6740) 2024-12-11 11:17:05 -07:00
WeidiDeng
9c0c71e577
reverseproxy: Rewrite requests and responses for websocket over http2 (#6567)
* reverse proxy: rewrite requests and responses for websocket over http2

* delete protocol pseudo-header

* modify cloned requests

* set request variable to track if it's a h2 websocket

* use request bodu

* rewrite request body

* use WebSocket instead of Websocket in the headers

* use logger check for zap loggers

* fix lint
2024-12-06 13:23:27 -07:00
Francis Lavoie
315715e90f
core: Implement FastAbs to avoid repeated os.Getwd calls (#6687)
* core: Implement FastAbs to avoid repeated os.Getwd calls

* Lint

* Rename files
2024-11-13 03:55:51 -05:00
Matthew Holt
238f1108e6
reverseproxy: Revert #4952 - don't ignore context cancellation in stream mode
i.e. Revert commit f5dce84a70

Two years ago, the patch in #4952 was a seemingly necessary way to fix an issue (sort of an edge case), but it broke other more common use cases (see #6666).

Now, as of #6669, it seems like the original issue can no longer be replicated, so we are reverting that patch, because it was incorrect anyway.

If it turns out the original issue returns, a more proper patch may be in #6669 (even if used as a baseline for a future fix). A potential future fix could be an opt-in setting.
2024-11-12 13:43:34 -07:00
Francis Lavoie
09b2cbcf4d
caddyhttp: Add MatchWithError to replace SetVar hack (#6596)
* caddyhttp: Add `MatchWithError` to replace SetVar hack

* Error in IP matchers on TLS handshake not complete

* Use MatchWithError everywhere possible

* Move implementations to MatchWithError versions

* Looser interface checking to allow fallback

* CEL factories can return RequestMatcherWithError

* Clarifying comment since it's subtle that an err is returned

* Return 425 Too Early status in IP matchers

* Keep AnyMatch signature the same for now

* Apparently Deprecated can't be all-uppercase to get IDE linting

* Linter
2024-11-04 23:18:50 +00:00
Francis Lavoie
fbf0f4c425
reverseproxy: Sync changes from stdlib for 1xx handling (#6656)
* reverseproxy: Sync changes from stdlib for 1xx handling

Sourced from 960654be0c

* Use clear()

3bc28402fa
2024-10-22 12:10:46 -06:00
WeidiDeng
48ce47f1d4
reverseproxy: Use correct cases for websocket related headers (#6621)
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2024-10-11 09:02:23 +00:00
Kévin Dunglas
f4bf4e0097
perf: use zap's Check() to prevent useless allocs (#6560)
* perf: use zap's Check() to prevent useless allocs

* fix

* fix

* fix

* fix

* restore previous replacer behavior

* fix linter
2024-09-13 11:16:37 -06:00
Steffen Busch
c050a37e1c
reverse_proxy: add placeholder http.reverse_proxy.retries (#6553)
* Add placeholder http.reverse_proxy.lb.retries

* Renamed placeholder to http.reverse_proxy.retries
2024-08-30 11:53:56 -06:00
Jens-Uwe Mager
2bb2ecc549
reverseproxy: Change errors writing the response to warning. (#6532)
Most of the errors that can be seen here are write errors due to clients
aborting the request from their side. Often seen ones include:

	* writing: ... write: broken pipe
	* writing: ... connection timed out
	* writing: http2: stream closed
	* writing: timeout...
	* writing: h3 error...

Most of these errors are beyond of the control of caddy on the client side,
probably nothing can be done on the server side. It still warrants
researching when these errors occur very often, so a change in level from
error to warn is better here to not polute the logs with errors in the
normal case.
2024-08-21 11:39:20 -06:00
Matt Holt
c3fb5f4d3f
caddyhttp: Reject 0-RTT early data in IP matchers and set Early-Data header when proxying (#6427)
* caddyhttp: Reject 0-RTT early data in IP matchers and set Early-Data header when proxying

See RFC 8470: https://httpwg.org/specs/rfc8470.html

Thanks to Michael Wedl (@MWedl)  at the University of Applied Sciences St. Poelten for reporting this.

* Don't return value for {remote} placeholder in early data

* Add Caddyfile support
2024-07-05 10:46:20 -06:00
Matthew Holt
9f97df2275
reverseproxy: Remove long-deprecated buffering properties
They've been deprecated for over a year and we printed warnings during that time.
2024-04-22 15:34:14 -06:00
Matthew Holt
d93e027e01
reverseproxy: Reuse buffered request body even if partially drained
Previous commit only works when the backends don't read any of the body first.
2024-04-22 15:22:50 -06:00
Matthew Holt
613d544a47 reverseproxy: Accept EOF when buffering
Before this change, a read of size (let's say) < 10, into a buffer of size 10, will return EOF because we're using CopyN to limit to the size of the buffer. That resulted in the body being read from later, which should only happen if it couldn't fit in the buffer.

With this change, the body is properly NOT set when it can all fit in the buffer.
2024-04-22 13:12:10 -06:00
WeidiDeng
b40cacf5ce
reverseproxy: Wait for both ends of websocket to close (#6175) 2024-04-15 11:37:37 -06:00
WeidiDeng
53f7035299
reverseproxy: use context.WithoutCancel (#6116) 2024-02-19 20:25:02 -07:00
Yolan Romailler
2fe69a828f
chore: enabling a few more linters (#5961)
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2024-01-25 15:24:58 +00:00
Fred Cox
d9ff7b1872
reverseproxy: Only change Content-Length when full request is buffered (#5830)
fixes: https://github.com/caddyserver/caddy/issues/5829

Signed-off-by: Fred Cox <mcfedr@gmail.com>
2024-01-09 12:59:30 -07:00
WeidiDeng
4de2c1c65e
panic when reading from backend failed to propagate stream error (#5952) 2023-11-23 03:18:18 -05:00
Matt Holt
3a3182fba3
reverseproxy: Add more debug logs (#5793)
* reverseproxy: Add more debug logs

This makes debug logging very noisy when reverse proxying, but I guess
that's the point.

This has shown to be useful in troubleshooting infrastructure issues.

* Update modules/caddyhttp/reverseproxy/streaming.go

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Update modules/caddyhttp/reverseproxy/streaming.go

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Add opt-in `trace_logs` option

* Rename to VerboseLogs

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2023-10-11 13:36:20 -06:00
Francis Lavoie
2a6859a5e4
reverseproxy: Fix retries on "upstreams unavailable" error (#5841) 2023-10-10 22:07:20 +00:00
Patrick Koenig
4feac4d83c
reverseproxy: Allow fallthrough for response handlers without routes (#5780) 2023-10-05 23:15:26 -04:00
Matt Holt
936ee918ee
reverseproxy: Always return new upstreams (fix #5736) (#5752)
* reverseproxy: Always return new upstreams (fix #5736)

* Fix healthcheck logger race
2023-08-17 11:33:40 -06:00
Jacob Gadikian
d6f86cccf5
ci: use gci linter (#5708)
* use gofmput to format code

* use gci to format imports

* reconfigure gci

* linter autofixes

* rearrange imports a little

* export GOOS=windows golangci-lint run ./... --fix
2023-08-14 09:41:15 -06:00
Jacob Gadikian
b32f265eca
ci: Use gofumpt to format code (#5707) 2023-08-07 19:40:31 +00:00
Francis Lavoie
cd486c25d1
caddyhttp: Make use of http.ResponseController (#5654)
* caddyhttp: Make use of http.ResponseController

Also syncs the reverseproxy implementation with stdlib's which now uses ResponseController as well 2449bbb5e6

* Enable full-duplex for HTTP/1.1

* Appease linter

* Add warning for builds with Go 1.20, so it's less surprising to users

* Improved godoc for EnableFullDuplex, copied text from stdlib

* Only wrap in encode if not already wrapped
2023-08-02 20:03:26 +00:00
mmm444
da23501457
reverseproxy: Connection termination cleanup (#5663) 2023-08-01 14:01:12 +00:00
Marten Seemann
f45a6de20d
go.mod: Update quic-go to v0.37.0, bump to Go 1.20 minimum (#5644)
* update quic-go to v0.37.0

* Bump to Go 1.20

* Bump golangci-lint version, yml syntax consistency

* Use skip-pkg-cache workaround

* Workaround needed for both?

* Seeding weakrand is no longer necessary

---------

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2023-07-21 22:00:48 -06:00
mmm444
424ae0f420
reverseproxy: Experimental streaming timeouts (#5567)
* reverseproxy: WIP streaming timeouts

* More verbose logging by using the child logger

* reverseproxy: Implement streaming timeouts

* reverseproxy: Refactor cleanup

* reverseproxy: Avoid **time.Timer

---------

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2023-06-19 15:54:43 -06:00
Corin Langosch
2ddb717144
reverseproxy: Fix parsing of source IP in case it's an ipv6 address (#5569) 2023-06-12 09:35:22 -06:00
Francis Lavoie
335cd2e8a4
reverseproxy: Fix active health check header canonicalization, refactor (#5446) 2023-05-05 15:19:22 -06:00
Francis Lavoie
4636109ce1
reverseproxy: Remove deprecated lookup_srv (#5396) 2023-04-10 20:08:40 +00:00
Corin Langosch
b6fe5d4b41
proxyprotocol: Add PROXY protocol support to reverse_proxy, add HTTP listener wrapper (#5424)
Co-authored-by: WeidiDeng <weidi_deng@icloud.com>
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2023-03-31 17:44:53 -04:00
Trea Hauet
2182270a2c
reverseproxy: Reset Content-Length to prevent FastCGI from hanging (#5435)
Fixes: https://github.com/caddyserver/caddy/issues/5420
2023-03-16 11:42:16 -06:00
Matt Holt
4b119a475f
reverseproxy: Don't buffer chunked requests (fix #5366) (#5367)
* reverseproxy: Don't buffer chunked requests (fix #5366)

Mostly reverts 845bc4d50b (#5289)

Adds warning for unsafe config.

Deprecates unsafe properties in favor of simpler, safer designed ones.

* Update modules/caddyhttp/reverseproxy/caddyfile.go

Co-authored-by: Y.Horie <u5.horie@gmail.com>

* Update modules/caddyhttp/reverseproxy/reverseproxy.go

Co-authored-by: Y.Horie <u5.horie@gmail.com>

* Update modules/caddyhttp/reverseproxy/reverseproxy.go

Co-authored-by: Y.Horie <u5.horie@gmail.com>

* Remove unused code

---------

Co-authored-by: Y.Horie <u5.horie@gmail.com>
2023-02-11 17:25:29 -07:00
Francis Lavoie
223cbe3d0b
caddyhttp: Add server-level trusted_proxies config (#5103) 2023-01-10 00:08:23 -05:00
Y.Horie
845bc4d50b
reverseproxy: Fix hanging for Transfer-Encoding: chunked (#5289)
* Fixes #5236
* enable request body buffering in reverse proxy
  when the request header has Transfer-Encoding: chunked
2023-01-09 00:13:34 -07:00
Matthew Holt
fcdbc69fab
Fix comment
I apparently read the diff backwards in
2a8c458ffe
2022-09-29 12:38:36 -06:00
Matthew Holt
2a8c458ffe
reverseproxy: Parse humanized byte size (fix #5095) 2022-09-29 12:37:06 -06:00
Matt Holt
e2991eb019
reverseproxy: On 103 don't delete own headers (#5091)
See #5074
2022-09-29 08:19:56 -06:00
Matthew Holt
e43b6d8178 core: Variadic Context.Logger(); soft deprecation
Ideally I'd just remove the parameter to caddy.Context.Logger(), but
this would break most Caddy plugins.

Instead, I'm making it variadic and marking it as partially deprecated.
In the future, I might completely remove the parameter once most
plugins have updated.
2022-09-16 16:55:36 -06:00
Matt Holt
66476d8c8f
reverseproxy: Close hijacked conns on reload/quit (#4895)
* reverseproxy: Close hijacked conns on reload/quit

We also send a Close control message to both ends of
WebSocket connections. I have tested this many times in
my dev environment with consistent success, although
the variety of scenarios was limited.

* Oops... actually call Close() this time

* CloseMessage --> closeMessage

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* Use httpguts, duh

* Use map instead of sync.Map

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2022-09-02 17:01:55 -06:00
Francis Lavoie
d4d8bbcfc6
events: Implement event system (#4912)
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2022-08-31 15:01:30 -06:00
Matthew Holt
8cb3cf540c Minor cleanup, resolve a couple lint warnings 2022-08-29 12:31:56 -06:00
Abirdcfly
e1801fdb19
Remove duplicate words in comments (#4986) 2022-08-27 14:39:26 -06:00
Dávid Szabó
0c57facc67
reverseproxy: Add upstreams healthy metrics (#4935) 2022-08-27 12:30:23 -06:00
WilczyńskiT
c7772588bd
core: Change net.IP to netip.Addr; use netip.Prefix (#4966)
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2022-08-17 16:10:57 -06:00
Matt Holt
f5dce84a70
reverseproxy: Ignore context cancel in stream mode (#4952) 2022-08-12 13:15:41 -06:00
Kévin Dunglas
085df25c7e
reverseproxy: Support 1xx status codes (HTTP early hints) (#4882) 2022-08-09 10:53:24 -06:00
Chirag Maheshwari
d26559316f
Replace strings.Index with strings.Cut (#4932) 2022-08-06 22:03:37 -06:00
Francis Lavoie
141872ed80
chore: Bump up to Go 1.19, minimum 1.18 (#4925) 2022-08-02 16:39:09 -04:00
Francis Lavoie
7d1f7771c9
reverseproxy: Implement retry count, alternative to try_duration (#4756)
* reverseproxy: Implement retry count, alternative to try_duration

* Add Caddyfile support for `retry_match`

* Refactor to deduplicate matcher parsing logic

* Fix lint
2022-07-13 14:15:00 -06:00
Matthew Holt
d6bc9e0b5c
reverseproxy: Err 503 if all upstreams unavailable 2022-07-08 13:01:32 -06:00
Francis Lavoie
98468af8b6
reverseproxy: Fix double headers in response handlers (#4847) 2022-06-22 15:10:14 -04:00
Matt Holt
5e729c1e85
reverseproxy: HTTP 504 for upstream timeouts (#4824)
Closes #4823
2022-06-03 14:13:47 -06:00
Francis Lavoie
f6900fcf53
reverseproxy: Support performing pre-check requests (#4739) 2022-05-06 10:50:26 -04:00
Francis Lavoie
3e3bb00265
reverseproxy: Add _ms placeholders for proxy durations (#4666)
* reverseproxy: Add `_ms` placeholders for proxy durations

* Add http.request.duration_ms

Also add comments, and change duration_sec to duration_ms

* Add response.duration_ms for consistency

* Add missing godoc comment

Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2022-04-11 13:04:05 -06:00
Francis Lavoie
c7d6c4cbb9
reverseproxy: copy_response and copy_response_headers for handle_response routes (#4391)
* reverseproxy: New `copy_response` handler for `handle_response` routes

Followup to #4298 and #4388.

This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream. 

Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch.

To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream.

Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works.

Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response.

* Delete the "Content-Length" header if we aren't copying

Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl:

```
curl: (18) transfer closed with 18 bytes remaining to read
```

To replicate:

```
{
	admin off
	debug
}

:8881 {
	reverse_proxy 127.0.0.1:8882 {
		@200 status 200
		handle_response @200 {
			header Foo bar
		}
	}
}

:8882 {
	header Content-Type application/json
	respond `{"hello": "world"}` 200
}
```

* Implement `copy_response_headers`, with include/exclude list support

* Apply suggestions from code review

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2022-03-09 11:00:51 -07:00
Matt Holt
ab0455922a
reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470)
* reverseproxy: Begin refactor to enable dynamic upstreams

Streamed here: https://www.youtube.com/watch?v=hj7yzXb11jU

* Implement SRV and A/AAA upstream sources

Also get upstreams at every retry loop iteration instead of just once
before the loop. See #4442.

* Minor tweaks from review

* Limit size of upstreams caches

* Add doc notes deprecating LookupSRV

* Provision dynamic upstreams

Still WIP, preparing to preserve health checker functionality

* Rejigger health checks

Move active health check results into handler-specific Upstreams.

Improve documentation regarding health checks and upstreams.

* Deprecation notice

* Add Caddyfile support, use `caddy.Duration`

* Interface guards

* Implement custom resolvers, add resolvers to http transport Caddyfile

* SRV: fix Caddyfile `name` inline arg, remove proto condition

* Use pointer receiver

* Add debug logs

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2022-03-06 17:43:39 -07:00
Francis Lavoie
c50094fc9d
reverseproxy: Implement trusted proxies for X-Forwarded-* headers (#4507) 2022-03-06 18:51:55 -05:00
Francis Lavoie
f5e104944e
reverseproxy: Make shallow-ish clone of the request (#4551)
* reverseproxy: Make shallow-ish clone of the request

* Refactor request cloning into separate function

Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2022-03-03 09:54:45 -07:00
Денис Телюх
2e46c2ac1d
admin, reverseproxy: Stop timers if canceled to avoid goroutine leak (#4482) 2022-01-04 12:14:18 -07:00
Francis Lavoie
5bf0adad87
caddyhttp: Make logging of credential headers opt-in (#4438) 2021-12-02 13:26:24 -07:00
Francis Lavoie
9ee68c1bd5
reverseproxy: Adjust defaults, document defaults (#4436)
* reverseproxy: Adjust defaults, document defaults

Related to some of the issues in https://github.com/caddyserver/caddy/issues/4245, a complaint about the proxy transport defaults not being properly documented in https://caddy.community/t/default-values-for-directives/14254/6.

- Dug into the stdlib to find the actual defaults for some of the timeouts and buffer limits, documenting them in godoc so the JSON docs get them next release.

- Moved the keep-alive and dial-timeout defaults from `reverseproxy.go` to `httptransport.go`. It doesn't make sense to set defaults in the proxy, because then any time the transport is configured with non-defaults, the keep-alive and dial-timeout defaults are lost!

- Sped up the dial timeout from 10s to 3s, in practice it rarely makes sense to wait a whole 10s for dialing. A shorter timeout helps a lot with the load balancer retries, so using something lower helps with user experience.

* reverseproxy: Make keepalive interval configurable via Caddyfile

* fastcgi: DialTimeout default for fastcgi transport too
2021-11-24 01:32:25 -05:00
Francis Lavoie
f73f55dba7
reverseproxy: Sanitize scheme and host on incoming requests (#4237)
* caddyhttp: Sanitize scheme and host on incoming requests

* reverseproxy: Sanitize the URL scheme and host before proxying

* Apply suggestions from code review

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2021-10-26 14:41:28 -06:00
Francis Lavoie
b092061591
reverseproxy: Prevent copying the response if a response handler ran (#4388) 2021-10-18 14:00:43 -04:00
Matthew Holt
059fc32f00
Revert 3336faf2 (close #4360)
Debug log is correct level for this
2021-09-27 12:06:06 -06:00
Matthew Holt
3336faf254
reverseproxy: Log error at error level (fix #4360) 2021-09-24 18:29:23 -06:00
Francis Lavoie
f70a7578fa
reverseproxy: Remove redundant flushing (#4299)
From reading through the code, I think this code path is now obsoleted by the changes made in https://github.com/caddyserver/caddy/pull/4266.

Basically, `h.flushInterval()` will set the flush interval to `-1` if we're in a bi-directional stream, and the recent PR ensured that `h.copyResponse()` properly flushes headers immediately when the flush interval is non-zero. So now there should be no need to call Flush before calling `h.copyResponse()`.
2021-08-23 11:54:28 -06:00
Francis Lavoie
e6c29ce081
reverseproxy: Incorporate latest proxy changes from stdlib (#4266)
I went through the commits that touched stdlib's `reverseproxy.go` file, and copied over all the changes that are to code that was copied into Caddy.

The commits I pulled changes from:

- 2cc347382f
- a5cea062b3
- ecdbffd4ec
- 21898524f6
-ca3c0df1f8
- 9c017ff30d

This may also fix https://github.com/caddyserver/caddy/issues/4247 because of the change to `copyResponse` to set `mlw.flushPending = true` right away.
2021-08-12 10:48:24 -06:00
Matthew Holt
7c68809f4e
reverseproxy: Fix overwriting of max_idle_conns_per_host (closes #4201)
Also split the Caddyfile subdirective keepalive_idle_conns into two properties so the conns and conns_per_host can be set separately.

This is technically a breaking change, but probably anyone who this breaks already had a broken config anyway, and silently fixing it won't help them fix their configs.
2021-06-15 14:54:48 -06:00
Matthew Holt
2a8109468c
reverseproxy: Always remove hop-by-hop headers
See golang/go#46313

Based on 950fa11c4c
2021-06-04 15:21:16 -06:00
Francis Lavoie
aef8d4decc
reverseproxy: Set the headers in the replacer before handle_response (#4165)
Turns out this was an oversight, we assumed we could use `{http.response.header.*}` but that doesn't work because those are grabbed from the response writer, and we haven't copied any headers into the response writer yet.

So the fix is to set all the response headers into the replacer at a new namespace before running the handlers.

This adds the `{http.reverse_proxy.header.*}` replacer.

See https://caddy.community/t/empty-http-response-header-x-accel-redirect/12447
2021-05-12 14:19:08 -06:00
Matthew Holt
a17c3b568d
reverseproxy: Minor logging improvements 2021-05-05 14:52:24 -06:00
Francis Lavoie
e4a22de9d1
reverseproxy: Add handle_response blocks to reverse_proxy (#3710) (#4021)
* reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710)

* reverseproxy: complete handle_response test

* reverseproxy: Change handle_response matchers to use named matchers

reverseproxy: Add support for changing status code

* fastcgi: Remove obsolete TODO

We already have d.Err("transport already specified") in the reverse_proxy parsing code which covers this case

* reverseproxy: Fix support for "4xx" type status codes

* Apply suggestions from code review

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* caddyhttp: Reorganize response matchers

* reverseproxy: Reintroduce caddyfile.Unmarshaler

* reverseproxy: Add comment mentioning Finalize should be called

Co-authored-by: Maxime Soulé <btik-git@scoubidou.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2021-05-02 12:39:06 -06:00
Francis Lavoie
75f797debd
reverseproxy: Implement health_uri, deprecate health_path, supports query (#4050)
* reverseproxy: Implement health_uri, replaces health_path, supports query

Also fixes a bug with `health_status` Caddyfile parsing , it would always only take the first character of the status code even if it didn't end with "xx".

* reverseproxy: Rename to URI, named logger, warn in Provision (for JSON)
2021-03-29 18:36:40 -06:00
Matthew Holt
ce5a0934a8
reverseproxy: Fix round robin data race (#4038) 2021-02-25 09:41:52 -07:00
Matt Holt
6722ae3a83
reverseproxy: Add duration/latency placeholders (close #4012) (#4013)
* reverseproxy: Add duration/latency placeholders (close #4012) (and #2268)

Adds 4 placeholders, one is actually outside reverse proxy though:

{http.request.duration} is how long since the server decoded the HTTP request (headers).
{http.reverse_proxy.upstream.latency} is how long it took a proxy upstream to write the response header.
{http.reverse_proxy.upstream.duration} is total time proxying to the upstream, including writing response body to client.
{http.reverse_proxy.duration} is total time spent proxying, including selecting an upstream and retries.

Obviously, most of these are only useful at the end of a request, like when writing response headers or logs.

See also: https://caddy.community/t/any-equivalent-of-request-time-and-upstream-header-time-from-nginx/11418

* Add new placeholders to documentation
2021-02-22 11:57:21 -07:00
Matthew Holt
5ef76ff3e6
reverseproxy: Response buffering & configurable buffer size
Proxy response bodies can now be buffered, and the size of the request body and
response body buffer can be limited. Any remaining content that doesn't fit in the
buffer will remain on the wire until it can be read; i.e. bodies are not truncated,
even if the buffer is not big enough.

This fulfills a customer requirement. This was made possible by their sponsorship!
2021-02-09 14:15:04 -07:00
Daniel Santos
53aa60afff
reverseproxy: Handle "operation was canceled" errors (#3816)
* fix(caddy): Avoid "operation was canceled" errors

- Also add error handling for StatusGatewayTimeout

* revert(caddy): Revert 504 handling

- This will potentially break load balancing and health checks

* Handle client cancellation as different error

Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2020-11-25 10:54:23 -07:00
Dave Henderson
bd17eb205d
ci: Use golangci's github action for linting (#3794)
* ci: Use golangci's github action for linting

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix most of the staticcheck lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the prealloc lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the misspell lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the varcheck lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the errcheck lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the bodyclose lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the deadcode lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the unused lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the gosec lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the gosimple lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the ineffassign lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Fix the staticcheck lint errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Revert the misspell change, use a neutral English

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Remove broken golangci-lint CI job

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Re-add errantly-removed weakrand initialization

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* don't break the loop and return

* Removing extra handling for null rootKey

* unignore RegisterModule/RegisterAdapter

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>

* single-line log message

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* Fix lint after a1808b0dbf209c615e438a496d257ce5e3acdce2 was merged

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Revert ticker change, ignore it instead

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Ignore some of the write errors

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Remove blank line

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Use lifetime

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* close immediately

Co-authored-by: Matt Holt <mholt@users.noreply.github.com>

* Preallocate configVals

Signed-off-by: Dave Henderson <dhenderson@gmail.com>

* Update modules/caddytls/distributedstek/distributedstek.go

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2020-11-22 14:50:29 -07:00
Dimitri Masson
6e0849d4c2
reverseproxy: Implement cookie hash selection policy (#3809)
* add CookieHashSelection for session affinity

* add CookieHashSelection for session affinity

* register module

* reverse_proxy: Add and fix cookie lb_policy

* reverse_proxy: Manage hmac.write error on cookie hash selection

* reverse_proxy: fix some comments

* reverse_proxy: variable `cookieValue` is inside the else block

* reverse_proxy: Abstract duplicate nuanced logic of reservoir sampling into a function

* reverse_proxy: Set a default secret is indeed useless

* reverse_proxy: add configuration syntax for cookie lb_policy

* reverse_proxy: doc typo and improvement

Co-authored-by: utick <123liuqingdong@163.com>
2020-11-20 12:39:26 -07:00
Matt Holt
c7efb0307d
reverseproxy: Fix dial placeholders, SRV, active health checks (#3780)
* reverseproxy: Fix dial placeholders, SRV, active health checks

Supercedes #3776
Partially reverts or updates #3756, #3693, and #3695

* reverseproxy: add integration tests

Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
2020-10-13 10:35:20 -06:00
Mohammed Al Sahaf
6722426f1a
reverseproxy: allow no port for SRV; fix regression in d55d50b (#3756)
* reverseproxy: fix breakage in handling SRV lookup introduced by 3695

* reverseproxy: validate against incompatible config options with lookup_srv

* reverseproxy: add integration test cases for validations involving lookup_srv

* reverseproxy: clarify the reason for skipping an iteration

* grammar.. Oxford comma

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

Fixes #3753
2020-10-01 14:05:39 -06:00
Aleksei
3b9eae70c9
reverseproxy: Change 500 error to 502 for lookup_srv config (#3771)
Fixes #3763
2020-10-01 14:02:31 -06:00
Mohammed Al Sahaf
bc453fa6ae
reverseproxy: Correct alternate port for active health checks (#3693)
* reverseproxy: construct active health-check transport from scratch (Fixes #3691)

* reverseproxy: do upstream health-check on the correct alternative port

* reverseproxy: add integration test for health-check on alternative port

* reverseproxy: put back the custom transport for health-check http client

* reverseproxy: cleanup health-check integration test

* reverseproxy: fix health-check of unix socket upstreams

* reverseproxy: skip unix socket tests on Windows

* tabs > spaces

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* make the linter (and @francislavoie) happy

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

* One more lint fix

Co-authored-by: Francis Lavoie <lavofr@gmail.com>

Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2020-09-17 10:25:34 -06:00
Mohammed Al Sahaf
d55d50b3b3
reverseproxy: Enforce port range size of 1 at provision (#3695)
* reverse_proxy: ensure upstream address has port range of only 1

* reverse_proxy: don't log the error if upstream range size is more than 1
2020-09-16 19:48:37 -06:00
Matthew Holt
e2f913bb7f
reverseproxy: Minor fixes and cleanup
Now use context cancellation to stop active health checker, which is
simpler than and just as effective as using a separate stop channel.
2020-08-07 18:02:24 -06:00
Kevin Lin
904f149e5b
reverse_proxy: fix bidirectional streams with encodings (fix #3606) (#3620)
* reverse_proxy: fix bi-h2stream breaking gzip encode handle(#3606).

* reverse_proxy: check http version of both sides to avoid affecting non-h2 upstream.

* Minor cleanup; apply review suggestions

Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2020-08-03 20:50:38 -06:00
Kevin Lin
e9b1d7dcb4
reverse_proxy: flush HTTP/2 response when ContentLength is unknown (#3561)
* reverse proxy: Support more h2 stream scenarios (#3556)

* reverse proxy: add integration test for better h2 stream (#3556)

* reverse proxy: adjust comments as francislavoie suggests

* link to issue #3556 in the comments
2020-07-20 12:14:46 -06:00
Matthew Holt
246a31aacd
reverseproxy: Restore request's original host and header (fix #3509)
We already restore them within the retry loop, but after successful
proxy we didn't reset them, so as handlers bubble back up, they would
see the values used for proxying.

Thanks to @ziddey for identifying the cause.
2020-07-17 17:54:58 -06:00
Matthew Holt
7a99835dab
reverseproxy: Enable changing only the status code (close #2920) 2020-06-04 12:06:38 -06:00
Matthew Holt
881b826fb5
reverseproxy: Pool copy buffers (minor optimization) 2020-05-27 11:42:19 -06:00
Matthew Holt
538ddb8587 reverseproxy: Enable response interception (#1447, #2920)
It's a raw, low-level implementation for now, but it's very flexible.
More sugar-coating can be added after error handling is more developed.
2020-05-27 10:17:45 -06:00
Matthew Holt
2a8a198568
reverseproxy: Don't overwrite existing X-Forwarded-Proto header
Correct behavior is not well defined because this is a non-standard
header field. This could be a "hop-by-hop" field much like
X-Forwarded-For is, but even our X-Forwarded-For implementation
preserves prior entries. Or, it could be best to preserve the original
value from the first hop, representing the protocol as facing the
client.

Let's try it the other way for a bit and see how it goes.

See https://caddy.community/t/caddy2-w-wordpress-behind-nginx-reverse-proxy/8174/3?u=matt
2020-05-20 11:33:17 -06:00
Matthew Holt
9ee01dceac
reverseproxy: Make debug log safe if error occurs 2020-05-18 14:08:11 -06:00
Matthew Holt
812278acd8 reverseproxy: Emit debug log before checking error (#3425)
This way the upstream request will always be available even if it failed
2020-05-18 13:50:46 -06:00
Matt Holt
90c7b4b0a1
reverseproxy: Apply response header ops before copying it (fix #3382) (#3401) 2020-05-13 09:52:20 -06:00