* docs: add Caddyfile lexer/parser/formatter unification design spec
Design for unifying the Caddyfile lexer/parser and the standalone formatter
by making the token pipeline drive formatting: raw-source-span capture,
comment tokens, separator-kind metadata, follow-imports and (default-off)
braced-wrap modes. Includes adversarial-review hardening and empirically-found
intentional improvements over the legacy formatter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: add phased implementation plan for Caddyfile formatter unification
Task-by-task TDD plan: Phase 1 (token model + formatter rewrite + fuzz
invariants + legacy oracle), Phase 2 (follow-imports mode + caddy fmt
--imports), Phase 3 (default-off braced-wrap).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: vendor legacy Caddyfile formatter as differential oracle
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: add format-mode Token fields and accessors
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: add Lex and LexOptions; Tokenize delegates to Lex
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: capture verbatim raw source spans in format mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: emit comment tokens in format mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: record token separator kind in format mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: split structural braces in format mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: rewrite Format as a token-based renderer (core layout)
Replace the rune-by-rune state machine with a renderer that consumes the
format-mode token stream (Lex with Comments+Raw). Adds FormatWithOptions and
FormatOptions{WrapUnbracedSite}; Format keeps its signature. Comment
placement, continuations, empty-block expansion, and the import/}-glue rules
land in Tasks 9-11; the six dependent TestFormatter cases are temporarily
skipped with annotations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: token renderer comments (I1/I3) and continuations (D1)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: token renderer empty blocks (I2) and post-} break (I4)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: import-brace exception, drop nesting cap, drop < quirk
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: fix formatter idempotency/semantic bugs, retire legacy oracle
Fix two fuzz-found bugs plus several related idempotency issues in the
token-based Caddyfile formatter, and retire the legacy byte-scanner
formatter used as a test oracle.
Bug B (formatter.go): a stray "}" at nesting 0 was glued to an unquoted
predecessor because precededBySpace is unreliable after an unquoted token.
Only suppress the separating space for an inline close brace when the
previous token is quoted (same guard as comment gluing), so "0 }" no longer
merges into "0}".
Bug A (lexer.go): an empty-bodied heredoc hitting EOF with no closing marker
was silently dropped, breaking Format idempotency. Surface it as an
"incomplete heredoc" error ONLY in format mode (opts.Comments || opts.Raw);
Tokenize/Parse keep the exact prior silent-drop behavior. Format then falls
back to preserving the trimmed input.
Additional format-mode-only fixes surfaced by property fuzzing: correct
line-break accounting for empty-bodied heredocs and continuation-framed
quoted tokens; normalize the raw slice / clear continuation on split and
standalone structural braces; guard the address/comment/brace fold so a
next-line "{" is not folded onto a comment trailing an opening brace; and a
fallback + idempotency backstop in FormatWithOptions for degenerate tokens
(ambiguous heredoc openers, unterminated/escaped quotes, dangling escapes,
messy/leading line continuations). The parse path is unchanged.
Property fuzz targets (formatter_test.go): keep FuzzFormatIdempotent,
FuzzFormatNoPanic, and a redesigned FuzzFormatSemanticPreserve that asserts
format-mode token-text preservation (robust to the parser's lenient grouping
of ambiguous brace layouts). Add regression tests
TestFormatStrayCloseBraceKeepsTokens and TestFormatIncompleteHeredocIdempotent.
Remove FuzzFormatParity, containsParityExclusion, maxNesting and delete
formatter_legacy_test.go (legacyFormat oracle + TestLegacyOracleMatchesCurrent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: seed corpus and FormattingDifference stability tests
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: extract pure import-glob resolution helper
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: import discovery for follow-imports formatting
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: FormatImports formats root plus imported files
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* cmd: add 'caddy fmt --imports' to format imported files
Adds a --imports flag to 'caddy fmt' that calls caddyfile.FormatImports
to format the root Caddyfile and all files it references via import
directives. Rejects --imports with stdin (no source directory to resolve
against). With --overwrite each file is written back; otherwise each
file's output is printed to stdout with a '# <path>' header, with
optional per-file diff via --diff.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: detect single unbraced site block
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: implement default-off unbraced->braced site wrapping
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: make WrapUnbracedSite decision on normalized stream (idempotency fix)
FormatWithOptions wrapped before formatting, deciding on the raw token
stream. isSingleUnbracedSite uses source-line gaps, but formatTokens
removes blank lines and folds dangling braces, so a shape read as "not a
single site" on pass 1 could read as a single unbraced site on pass 2 and
get wrapped, breaking Format(Format(x)) == Format(x).
Decide the wrap on the already-normalized stream instead: normalize, re-lex,
and wrap only if isSingleUnbracedSite holds there. tokens becomes the wrapped
stream so the backstop still compares like-for-like; once wrapped the output
is braced, so no double-wrap on re-runs. Format (zero options) still never
wraps.
Adds a regression reproducer and FuzzWrapUnbracedSiteIdempotent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: satisfy golangci-lint (rangeint + De Morgan)
Modernize three index loops to range-over-int and simplify a negated
boolean per staticcheck QF1001. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: modernize remaining index loops (rangeint)
Convert the remaining for-i index loops to range-over-int per golangci-lint
modernize. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* caddyfile: fix formatter and import correctness issues (#7881)
* caddyfile: fix formatter and import correctness issues
* noot
* noot
* caddyfile: document parser-driven import discovery
* test: make formatter import tests portable (#7882)
* caddyfile: propagate import observer into named route blocks
Named route blocks expand their imports through a separate parser built in
expandImportsInBlock, which copied only definedSnippets and importGraph. The
importObserver hook was left nil, so FormatImports never saw files imported
from inside a &(name) { ... } block and "caddy fmt --imports" silently skipped
them.
Carry the observer into that parser and cover it with a discovery test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* caddyfile: detect end-of-input-swallowing tokens from token shape
hasUnformattableToken documented an unterminated quote/backtick as a shape
that cannot be rendered idempotently, but never implemented the check. The
condition was only caught dynamically by trailingNewlineChangesTokens, which
returns early for input that already ends in a newline. That early return is
wrong for a token that swallowed the newline: once Format's mandatory trailing
newline sits inside an unterminated quote, appending another one no longer
changes the stream, so the second pass rendered what the first pass had
preserved verbatim.
Format(`"""`) => `"""` (fallback, preserved)
Format(`"""` + newline) => `"" "` (rendered, space inserted)
Decide it from the token instead: a non-quoted token whose verbatim source
ends in a newline swallowed it, since a newline otherwise terminates a token.
This also covers an escaped quote ("\"") running to end-of-input, which has
the same shape. A lone "\r" is kept in raw but does not terminate a token, so
it is deliberately excluded; heredoc tokens carry wasQuoted == '<'.
Master's rune-based formatter was idempotent on these inputs, so this restores
parity. Found by FuzzFormatIdempotent; the inputs are added as portable seeds
and table cases rather than testdata/fuzz files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* caddyfile: stop Format from mutating its input buffer
The three fallback paths returned append(bytes.TrimSpace(input), '\n').
TrimSpace returns a subslice sharing the caller's backing array, so the append
wrote the newline into the caller's buffer. os.ReadFile returns a slice with
one byte of spare capacity, which is exactly what the append needs, so this
reached real callers:
$ printf 'foo "unterminated \n\n' > Caddyfile
$ caddy fmt --diff
foo "unterminated <- reported unchanged; its trailing
- whitespace was in fact stripped
cmd/commandfuncs.go compares input against output after calling Format, and by
then input had been overwritten to match. The same aliasing corrupted
FuzzFormatIdempotent, which reported failures whose recorded inputs did not
reproduce.
Copy into a fresh buffer instead, and assert both non-mutation and
non-aliasing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* caddyfile: preserve repeated import expansions during formatting
Deduplicate formatted files without skipping imports with different arguments,
block substitutions, or lexical aliases. Track active physical-file ancestry
to prevent symlink cycles, including snippet and block expansion contexts.
Add regression coverage and align the upstream glued-brace expectation with
the token-based formatter's literal-token behavior.
Validation: go test -race -short ./..., go build ./..., and golangci-lint run
--timeout 10m all pass.
AI assistance: OpenAI Codex (GPT-6) assisted with implementation and testing.
* docs: remove formatter implementation plan and design spec
* caddyfile: require whitespace before line continuations
Preserve attached trailing backslashes as literal text, including Windows root paths. Require a space or tab before an unquoted continuation backslash, and preserve significant leading whitespace in formatter fallback output.
Add lexer, formatter, CRLF, adapter, and fuzz regression coverage. Validated with race tests, build, lint, formatter fuzzing, and the 165-example forum corpus.
AI assistance: OpenAI Codex (GPT-6) assisted with implementation, regression tests, and validation.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: a <a@tuxpa.in>
935 lines
28 KiB
Go
935 lines
28 KiB
Go
// Copyright 2015 Matthew Holt and The Caddy Authors
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package caddyfile
|
|
|
|
import (
|
|
"bytes"
|
|
"fmt"
|
|
"io"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
|
|
"go.uber.org/zap"
|
|
|
|
"github.com/caddyserver/caddy/v2"
|
|
)
|
|
|
|
// Parse parses the input just enough to group tokens, in
|
|
// order, by server block. No further parsing is performed.
|
|
// Server blocks are returned in the order in which they appear.
|
|
// Directives that do not appear in validDirectives will cause
|
|
// an error. If you do not want to check for valid directives,
|
|
// pass in nil instead.
|
|
//
|
|
// Environment variables in {$ENVIRONMENT_VARIABLE} notation
|
|
// will be replaced before parsing begins.
|
|
func Parse(filename string, input []byte) ([]ServerBlock, error) {
|
|
// unfortunately, we must copy the input because parsing must
|
|
// remain a read-only operation, but we have to expand environment
|
|
// variables before we parse, which changes the underlying array (#4422)
|
|
inputCopy := make([]byte, len(input))
|
|
copy(inputCopy, input)
|
|
|
|
tokens, err := allTokens(filename, inputCopy)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
p := parser{
|
|
Dispenser: NewDispenser(tokens),
|
|
importGraph: importGraph{
|
|
nodes: make(map[string]struct{}),
|
|
edges: make(adjacency),
|
|
},
|
|
}
|
|
return p.parseAll()
|
|
}
|
|
|
|
// allTokens lexes the entire input, but does not parse it.
|
|
// It returns all the tokens from the input, unstructured
|
|
// and in order. It may mutate input as it expands env vars.
|
|
func allTokens(filename string, input []byte) ([]Token, error) {
|
|
return Tokenize(replaceEnvVars(input), filename)
|
|
}
|
|
|
|
// replaceEnvVars replaces all occurrences of environment variables.
|
|
// It mutates the underlying array and returns the updated slice.
|
|
func replaceEnvVars(input []byte) []byte {
|
|
var offset int
|
|
for {
|
|
begin := bytes.Index(input[offset:], spanOpen)
|
|
if begin < 0 {
|
|
break
|
|
}
|
|
begin += offset // make beginning relative to input, not offset
|
|
end := bytes.Index(input[begin+len(spanOpen):], spanClose)
|
|
if end < 0 {
|
|
break
|
|
}
|
|
end += begin + len(spanOpen) // make end relative to input, not begin
|
|
|
|
// get the name; if there is no name, skip it
|
|
envString := input[begin+len(spanOpen) : end]
|
|
if len(envString) == 0 {
|
|
offset = end + len(spanClose)
|
|
continue
|
|
}
|
|
|
|
// split the string into a key and an optional default
|
|
envParts := strings.SplitN(string(envString), envVarDefaultDelimiter, 2)
|
|
|
|
// do a lookup for the env var, replace with the default if not found
|
|
envVarValue, found := os.LookupEnv(envParts[0])
|
|
if !found && len(envParts) == 2 {
|
|
envVarValue = envParts[1]
|
|
}
|
|
|
|
// get the value of the environment variable
|
|
// note that this causes one-level deep chaining
|
|
envVarBytes := []byte(envVarValue)
|
|
|
|
// splice in the value
|
|
input = append(input[:begin],
|
|
append(envVarBytes, input[end+len(spanClose):]...)...)
|
|
|
|
// continue at the end of the replacement
|
|
offset = begin + len(envVarBytes)
|
|
}
|
|
return input
|
|
}
|
|
|
|
// importAncestor tracks active physical files during formatting import discovery.
|
|
// Nodes are immutable and shared by all tokens from one import expansion.
|
|
type importAncestor struct {
|
|
info os.FileInfo
|
|
parent *importAncestor
|
|
}
|
|
|
|
type parser struct {
|
|
*Dispenser
|
|
block ServerBlock // current server block being parsed
|
|
eof bool // if we encounter a valid EOF in a hard place
|
|
definedSnippets map[string][]Token
|
|
nesting int
|
|
importGraph importGraph
|
|
// importObserver, when set, runs after an imported file is opened and read but
|
|
// before its tokens are inserted. Returning true skips token insertion. This
|
|
// internal hook lets FormatImports observe the parser's actual import execution
|
|
// and suppress physical-file cycles without duplicating parser semantics.
|
|
importObserver func(string, os.FileInfo, []byte, *importAncestor) (bool, error)
|
|
}
|
|
|
|
func (p *parser) parseAll() ([]ServerBlock, error) {
|
|
var blocks []ServerBlock
|
|
|
|
for p.Next() {
|
|
err := p.parseOne()
|
|
if err != nil {
|
|
return blocks, err
|
|
}
|
|
if len(p.block.Keys) > 0 || len(p.block.Segments) > 0 {
|
|
blocks = append(blocks, p.block)
|
|
}
|
|
if p.nesting > 0 {
|
|
return blocks, p.EOFErr()
|
|
}
|
|
}
|
|
|
|
return blocks, nil
|
|
}
|
|
|
|
func (p *parser) parseOne() error {
|
|
p.block = ServerBlock{}
|
|
return p.begin()
|
|
}
|
|
|
|
func (p *parser) begin() error {
|
|
if len(p.tokens) == 0 {
|
|
return nil
|
|
}
|
|
|
|
err := p.addresses()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
if p.eof {
|
|
// this happens if the Caddyfile consists of only
|
|
// a line of addresses and nothing else
|
|
return nil
|
|
}
|
|
|
|
if ok, name := p.isNamedRoute(); ok {
|
|
// we just need a dummy leading token to ease parsing later
|
|
nameToken := p.Token()
|
|
nameToken.Text = name
|
|
|
|
// named routes only have one key, the route name
|
|
p.block.Keys = []Token{nameToken}
|
|
p.block.IsNamedRoute = true
|
|
|
|
// get all the tokens from the block, including the braces
|
|
tokens, err := p.blockTokens(true)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// expand any import directives inside the named route block
|
|
expandedTokens, err := p.expandImportsInBlock(tokens)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
expandedTokens = append([]Token{nameToken}, expandedTokens...)
|
|
p.block.Segments = []Segment{expandedTokens}
|
|
return nil
|
|
}
|
|
|
|
if ok, name := p.isSnippet(); ok {
|
|
if p.definedSnippets == nil {
|
|
p.definedSnippets = map[string][]Token{}
|
|
}
|
|
if _, found := p.definedSnippets[name]; found {
|
|
return p.Errf("redeclaration of previously declared snippet %s", name)
|
|
}
|
|
// consume all tokens til matched close brace
|
|
tokens, err := p.blockTokens(false)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
// Just as we need to track which file the token comes from, we need to
|
|
// keep track of which snippet the token comes from. This is helpful
|
|
// in tracking import cycles across files/snippets by namespacing them.
|
|
// Without this, we end up with false-positives in cycle-detection.
|
|
for k, v := range tokens {
|
|
v.snippetName = name
|
|
tokens[k] = v
|
|
}
|
|
p.definedSnippets[name] = tokens
|
|
// empty block keys so we don't save this block as a real server.
|
|
p.block.Keys = nil
|
|
return nil
|
|
}
|
|
|
|
return p.blockContents()
|
|
}
|
|
|
|
func (p *parser) addresses() error {
|
|
var expectingAnother bool
|
|
|
|
for {
|
|
value := p.Val()
|
|
token := p.Token()
|
|
|
|
// Reject request matchers if trying to define them globally
|
|
if strings.HasPrefix(value, "@") {
|
|
return p.Errf("request matchers may not be defined globally, they must be in a site block; found %s", value)
|
|
}
|
|
|
|
// Special case: import directive replaces tokens during parse-time
|
|
if value == "import" && p.isNewLine() {
|
|
err := p.doImport(0)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
continue
|
|
}
|
|
|
|
// Open brace definitely indicates end of addresses
|
|
if isOpenCurlyBrace(token) {
|
|
if expectingAnother {
|
|
return p.Errf("Expected another address but had '%s' - check for extra comma", value)
|
|
}
|
|
// Mark this server block as being defined with braces.
|
|
// This is used to provide a better error message when
|
|
// the user may have tried to define two server blocks
|
|
// without having used braces, which are required in
|
|
// that case.
|
|
p.block.HasBraces = true
|
|
break
|
|
}
|
|
|
|
// Users commonly forget to place a space between the address and the '{'
|
|
if strings.HasSuffix(value, "{") && token.wasQuoted == 0 {
|
|
return p.Errf("Site addresses cannot end with a curly brace: '%s' - put a space between the token and the brace", value)
|
|
}
|
|
|
|
if value != "" { // empty token possible if user typed ""
|
|
// Trailing comma indicates another address will follow, which
|
|
// may possibly be on the next line
|
|
if value[len(value)-1] == ',' {
|
|
value = value[:len(value)-1]
|
|
expectingAnother = true
|
|
} else {
|
|
expectingAnother = false // but we may still see another one on this line
|
|
}
|
|
|
|
// If there's a comma here, it's probably because they didn't use a space
|
|
// between their two domains, e.g. "foo.com,bar.com", which would not be
|
|
// parsed as two separate site addresses.
|
|
if strings.Contains(value, ",") {
|
|
return p.Errf("Site addresses cannot contain a comma ',': '%s' - put a space after the comma to separate site addresses", value)
|
|
}
|
|
|
|
// After the above, a comma surrounded by spaces would result
|
|
// in an empty token which we should ignore
|
|
if value != "" {
|
|
// Add the token as a site address
|
|
token.Text = value
|
|
p.block.Keys = append(p.block.Keys, token)
|
|
}
|
|
}
|
|
|
|
// Advance token and possibly break out of loop or return error
|
|
hasNext := p.Next()
|
|
if expectingAnother && !hasNext {
|
|
return p.EOFErr()
|
|
}
|
|
if !hasNext {
|
|
p.eof = true
|
|
break // EOF
|
|
}
|
|
if !expectingAnother && p.isNewLine() {
|
|
break
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
func (p *parser) blockContents() error {
|
|
errOpenCurlyBrace := p.openCurlyBrace()
|
|
if errOpenCurlyBrace != nil {
|
|
// single-server configs don't need curly braces
|
|
p.cursor--
|
|
}
|
|
|
|
err := p.directives()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// only look for close curly brace if there was an opening
|
|
if errOpenCurlyBrace == nil {
|
|
err = p.closeCurlyBrace()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// directives parses through all the lines for directives
|
|
// and it expects the next token to be the first
|
|
// directive. It goes until EOF or closing curly brace
|
|
// which ends the server block.
|
|
func (p *parser) directives() error {
|
|
for p.Next() {
|
|
// end of server block
|
|
if isCloseCurlyBrace(p.Token()) {
|
|
// p.nesting has already been decremented
|
|
break
|
|
}
|
|
|
|
// special case: import directive replaces tokens during parse-time
|
|
if p.Val() == "import" {
|
|
err := p.doImport(1)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
p.cursor-- // cursor is advanced when we continue, so roll back one more
|
|
continue
|
|
}
|
|
|
|
// normal case: parse a directive as a new segment
|
|
// (a "segment" is a line which starts with a directive
|
|
// and which ends at the end of the line or at the end of
|
|
// the block that is opened at the end of the line)
|
|
if err := p.directive(); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// doImport swaps out the import directive and its argument
|
|
// (a total of 2 tokens) with the tokens in the specified file
|
|
// or globbing pattern. When the function returns, the cursor
|
|
// is on the token before where the import directive was. In
|
|
// other words, call Next() to access the first token that was
|
|
// imported.
|
|
func (p *parser) doImport(nesting int) error {
|
|
// syntax checks
|
|
if !p.NextArg() {
|
|
return p.ArgErr()
|
|
}
|
|
importPattern := p.Val()
|
|
if importPattern == "" {
|
|
return p.Err("Import requires a non-empty filepath")
|
|
}
|
|
|
|
// grab remaining args as placeholder replacements
|
|
args := p.RemainingArgs()
|
|
|
|
// set up a replacer for non-variadic args replacement
|
|
repl := makeArgsReplacer(args)
|
|
|
|
// grab all the tokens (if it exists) from within a block that follows the import
|
|
var blockTokens []Token
|
|
for currentNesting := p.Nesting(); p.NextBlock(currentNesting); {
|
|
blockTokens = append(blockTokens, p.Token())
|
|
}
|
|
// initialize with size 1
|
|
blockMapping := make(map[string][]Token, 1)
|
|
if len(blockTokens) > 0 {
|
|
// use such tokens to create a new dispenser, and then use it to parse each block
|
|
bd := NewDispenser(blockTokens)
|
|
|
|
// one iteration processes one sub-block inside the import
|
|
for bd.Next() {
|
|
currentMappingKey := bd.Val()
|
|
|
|
if isOpenCurlyBrace(bd.Token()) {
|
|
return p.Err("anonymous blocks are not supported")
|
|
}
|
|
|
|
// load up all arguments (if there even are any)
|
|
currentMappingTokens := bd.RemainingArgsAsTokens()
|
|
|
|
// load up the entire block
|
|
for mappingNesting := bd.Nesting(); bd.NextBlock(mappingNesting); {
|
|
currentMappingTokens = append(currentMappingTokens, bd.Token())
|
|
}
|
|
|
|
blockMapping[currentMappingKey] = currentMappingTokens
|
|
}
|
|
}
|
|
|
|
// splice out the import directive and its arguments
|
|
// (2 tokens, plus the length of args)
|
|
tokensBefore := p.tokens[:p.cursor-1-len(args)-len(blockTokens)]
|
|
tokensAfter := p.tokens[p.cursor+1:]
|
|
var importedTokens []Token
|
|
var nodes []string
|
|
|
|
// first check snippets. That is a simple, non-recursive replacement
|
|
if p.definedSnippets != nil && p.definedSnippets[importPattern] != nil {
|
|
importedTokens = p.definedSnippets[importPattern]
|
|
if len(importedTokens) > 0 {
|
|
// just grab the first one
|
|
nodes = append(nodes, fmt.Sprintf("%s:%s", importedTokens[0].File, importedTokens[0].snippetName))
|
|
}
|
|
} else {
|
|
// make path relative to the file of the _token_ being processed rather
|
|
// than current working directory (issue #867) and then use glob to get
|
|
// list of matching filenames
|
|
matches, globPattern, err := resolveImportGlob(p.Dispenser.File(), importPattern)
|
|
if err != nil {
|
|
return p.WrapErr(err)
|
|
}
|
|
if len(matches) == 0 {
|
|
if strings.ContainsAny(globPattern, "*?[]") {
|
|
caddy.Log().Warn("No files matching import glob pattern", zap.String("pattern", importPattern))
|
|
} else {
|
|
// filepath.Glob ignores file system errors (it only
|
|
// returns ErrBadPattern), so a file that exists but
|
|
// cannot be accessed - e.g. permission denied - looks
|
|
// the same as a missing file. Stat the path directly
|
|
// so the real error is reported instead (issue #8161).
|
|
if _, err := os.Stat(globPattern); err != nil && !os.IsNotExist(err) {
|
|
return p.Errf("Failed to import file %s: %v", importPattern, err)
|
|
}
|
|
return p.Errf("File to import not found: %s", importPattern)
|
|
}
|
|
}
|
|
|
|
// collect all the imported tokens
|
|
for _, importFile := range matches {
|
|
newTokens, err := p.doSingleImport(importFile)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
importedTokens = append(importedTokens, newTokens...)
|
|
}
|
|
nodes = matches
|
|
}
|
|
|
|
nodeName := p.File()
|
|
if p.Token().snippetName != "" {
|
|
nodeName += fmt.Sprintf(":%s", p.Token().snippetName)
|
|
}
|
|
p.importGraph.addNode(nodeName)
|
|
p.importGraph.addNodes(nodes)
|
|
if err := p.importGraph.addEdges(nodeName, nodes); err != nil {
|
|
p.importGraph.removeNodes(nodes)
|
|
return err
|
|
}
|
|
|
|
// copy the tokens so we don't overwrite p.definedSnippets
|
|
tokensCopy := make([]Token, 0, len(importedTokens))
|
|
|
|
var (
|
|
maybeSnippet bool
|
|
maybeSnippetId bool
|
|
index int
|
|
)
|
|
|
|
// run the argument replacer on the tokens
|
|
// golang for range slice return a copy of value
|
|
// similarly, append also copy value
|
|
for i, token := range importedTokens {
|
|
if p.importObserver != nil && p.definedSnippets[importPattern] != nil {
|
|
// A snippet executes in its caller's ancestry, not the ancestry
|
|
// of the file expansion where it was originally declared.
|
|
token.importAncestry = p.Token().importAncestry
|
|
}
|
|
// update the token's imports to refer to import directive filename, line number and snippet name if there is one
|
|
if token.snippetName != "" {
|
|
token.imports = append(token.imports, fmt.Sprintf("%s:%d (import %s)", p.File(), p.Line(), token.snippetName))
|
|
} else {
|
|
token.imports = append(token.imports, fmt.Sprintf("%s:%d (import)", p.File(), p.Line()))
|
|
}
|
|
|
|
// naive way of determine snippets, as snippets definition can only follow name + block
|
|
// format, won't check for nesting correctness or any other error, that's what parser does.
|
|
if !maybeSnippet && nesting == 0 {
|
|
// first of the line
|
|
if i == 0 || isNextOnNewLine(tokensCopy[len(tokensCopy)-1], token) {
|
|
index = 0
|
|
} else {
|
|
index++
|
|
}
|
|
|
|
if index == 0 && len(token.Text) >= 3 && strings.HasPrefix(token.Text, "(") && strings.HasSuffix(token.Text, ")") {
|
|
maybeSnippetId = true
|
|
}
|
|
}
|
|
|
|
switch {
|
|
case isOpenCurlyBrace(token):
|
|
nesting++
|
|
if index == 1 && maybeSnippetId && nesting == 1 {
|
|
maybeSnippet = true
|
|
maybeSnippetId = false
|
|
}
|
|
case isCloseCurlyBrace(token):
|
|
nesting--
|
|
if nesting == 0 && maybeSnippet {
|
|
maybeSnippet = false
|
|
}
|
|
}
|
|
// if it is {block}, we substitute with all tokens in the block
|
|
// if it is {blocks.*}, we substitute with the tokens in the mapping for the *
|
|
var tokensToAdd []Token
|
|
foundBlockDirective := false
|
|
switch {
|
|
case token.Text == "{block}":
|
|
foundBlockDirective = true
|
|
tokensToAdd = blockTokens
|
|
case strings.HasPrefix(token.Text, "{blocks.") && strings.HasSuffix(token.Text, "}"):
|
|
foundBlockDirective = true
|
|
// {blocks.foo.bar} will be extracted to key `foo.bar`
|
|
blockKey := strings.TrimPrefix(strings.TrimSuffix(token.Text, "}"), "{blocks.")
|
|
val, ok := blockMapping[blockKey]
|
|
if ok {
|
|
tokensToAdd = val
|
|
}
|
|
}
|
|
|
|
if foundBlockDirective {
|
|
if maybeSnippet {
|
|
tokensCopy = append(tokensCopy, token)
|
|
} else {
|
|
start := len(tokensCopy)
|
|
tokensCopy = append(tokensCopy, tokensToAdd...)
|
|
if p.importObserver != nil {
|
|
for i := range tokensToAdd {
|
|
tokensCopy[start+i].importAncestry = token.importAncestry
|
|
}
|
|
}
|
|
}
|
|
continue
|
|
}
|
|
|
|
if maybeSnippet {
|
|
tokensCopy = append(tokensCopy, token)
|
|
continue
|
|
}
|
|
|
|
foundVariadic, startIndex, endIndex := parseVariadic(token, len(args))
|
|
if foundVariadic {
|
|
for _, arg := range args[startIndex:endIndex] {
|
|
token.Text = arg
|
|
tokensCopy = append(tokensCopy, token)
|
|
}
|
|
} else {
|
|
token.Text = repl.ReplaceKnown(token.Text, "")
|
|
tokensCopy = append(tokensCopy, token)
|
|
}
|
|
}
|
|
|
|
// splice the imported tokens in the place of the import statement
|
|
// and rewind cursor so Next() will land on first imported token
|
|
p.tokens = append(tokensBefore, append(tokensCopy, tokensAfter...)...)
|
|
p.cursor -= len(args) + len(blockTokens) + 1
|
|
|
|
return nil
|
|
}
|
|
|
|
// expandImportsInBlock takes a slice of tokens (typically the contents of a
|
|
// named route block including its outer curly braces) and expands any import
|
|
// directives found at the beginning of a line. The expansion is done by
|
|
// creating a temporary parser that shares the same snippets and import graph,
|
|
// then looping through the tokens and calling doImport whenever an import
|
|
// directive is encountered. All other tokens are left untouched.
|
|
func (p *parser) expandImportsInBlock(tokens []Token) ([]Token, error) {
|
|
// Create a temporary parser that operates on the provided tokens.
|
|
// The Dispenser is initialized with the token slice; snippets and the
|
|
// import graph are shared so that imports and cycle detection work
|
|
// consistently across the whole Caddyfile.
|
|
tempParser := &parser{
|
|
Dispenser: NewDispenser(tokens),
|
|
definedSnippets: p.definedSnippets,
|
|
importGraph: p.importGraph,
|
|
// Carry the observer so callers watching import execution (such as
|
|
// FormatImports) also see files imported from inside a named route.
|
|
importObserver: p.importObserver,
|
|
}
|
|
|
|
// Loop through the tokens. We only care about import directives that
|
|
// appear at the start of a line (same logic as directives()).
|
|
for tempParser.Next() {
|
|
if tempParser.Val() == "import" && tempParser.isNewLine() {
|
|
if err := tempParser.doImport(1); err != nil {
|
|
return nil, err
|
|
}
|
|
// Roll back the cursor so the next iteration sees the first
|
|
// token of the imported content (or the next token after it).
|
|
tempParser.cursor--
|
|
}
|
|
}
|
|
|
|
// The temporary parser's token slice has been modified in place by
|
|
// doImport, so we return it directly.
|
|
return tempParser.tokens, nil
|
|
}
|
|
|
|
// doSingleImport lexes the individual file at importFile and returns
|
|
// its tokens or an error, if any.
|
|
func (p *parser) doSingleImport(importFile string) ([]Token, error) {
|
|
file, err := os.Open(importFile)
|
|
if err != nil {
|
|
return nil, p.Errf("Could not import %s: %v", importFile, err)
|
|
}
|
|
defer file.Close()
|
|
|
|
info, err := file.Stat()
|
|
if err != nil {
|
|
return nil, p.Errf("Could not import %s: %v", importFile, err)
|
|
}
|
|
if info.IsDir() {
|
|
return nil, p.Errf("Could not import %s: is a directory", importFile)
|
|
}
|
|
|
|
input, err := io.ReadAll(file)
|
|
if err != nil {
|
|
return nil, p.Errf("Could not read imported file %s: %v", importFile, err)
|
|
}
|
|
// Notify only after a successful read so observers receive bytes and metadata
|
|
// from the same open descriptor. A skipped cyclic import contributes no
|
|
// tokens at this occurrence.
|
|
if p.importObserver != nil {
|
|
skip, err := p.importObserver(importFile, info, input, p.Token().importAncestry)
|
|
if err != nil {
|
|
return nil, p.Errf("Could not import %s: %v", importFile, err)
|
|
}
|
|
if skip {
|
|
return nil, nil
|
|
}
|
|
}
|
|
|
|
// only warning in case of empty files
|
|
if len(input) == 0 || len(strings.TrimSpace(string(input))) == 0 {
|
|
caddy.Log().Warn("Import file is empty", zap.String("file", importFile))
|
|
return []Token{}, nil
|
|
}
|
|
|
|
importedTokens, err := allTokens(importFile, input)
|
|
if err != nil {
|
|
return nil, p.Errf("Could not read tokens while importing %s: %v", importFile, err)
|
|
}
|
|
|
|
// Tack the file path onto these tokens so errors show the imported file's name
|
|
// (we use full, absolute path to avoid bugs: issue #1892)
|
|
filename, err := caddy.FastAbs(importFile)
|
|
if err != nil {
|
|
return nil, p.Errf("Failed to get absolute path of file: %s: %v", importFile, err)
|
|
}
|
|
var ancestry *importAncestor
|
|
if p.importObserver != nil {
|
|
ancestry = &importAncestor{info: info, parent: p.Token().importAncestry}
|
|
}
|
|
for i := range importedTokens {
|
|
importedTokens[i].File = filename
|
|
importedTokens[i].importAncestry = ancestry
|
|
}
|
|
|
|
return importedTokens, nil
|
|
}
|
|
|
|
// directive collects tokens until the directive's scope
|
|
// closes (either end of line or end of curly brace block).
|
|
// It expects the currently-loaded token to be a directive
|
|
// (or } that ends a server block). The collected tokens
|
|
// are loaded into the current server block for later use
|
|
// by directive setup functions.
|
|
func (p *parser) directive() error {
|
|
// a segment is a list of tokens associated with this directive
|
|
var segment Segment
|
|
|
|
// the directive itself is appended as a relevant token
|
|
segment = append(segment, p.Token())
|
|
|
|
for p.Next() {
|
|
if isOpenCurlyBrace(p.Token()) {
|
|
p.nesting++
|
|
if !p.isNextOnNewLine() {
|
|
return p.Err("Unexpected next token after '{' on same line")
|
|
}
|
|
if p.isNewLine() {
|
|
return p.Err("Unexpected '{' on a new line; did you mean to place the '{' on the previous line?")
|
|
}
|
|
} else if p.Val() == "{}" && p.Token().wasQuoted == 0 {
|
|
if p.isNextOnNewLine() {
|
|
return p.Err("Unexpected '{}' at end of line")
|
|
}
|
|
} else if p.isNewLine() && p.nesting == 0 {
|
|
p.cursor-- // read too far
|
|
break
|
|
} else if isCloseCurlyBrace(p.Token()) && p.nesting > 0 {
|
|
p.nesting--
|
|
} else if isCloseCurlyBrace(p.Token()) && p.nesting == 0 {
|
|
return p.Err("Unexpected '}' because no matching opening brace")
|
|
} else if p.Val() == "import" && p.isNewLine() {
|
|
if err := p.doImport(1); err != nil {
|
|
return err
|
|
}
|
|
p.cursor-- // cursor is advanced when we continue, so roll back one more
|
|
continue
|
|
}
|
|
|
|
segment = append(segment, p.Token())
|
|
}
|
|
|
|
p.block.Segments = append(p.block.Segments, segment)
|
|
|
|
if p.nesting > 0 {
|
|
return p.EOFErr()
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// openCurlyBrace expects the current token to be an
|
|
// opening curly brace. This acts like an assertion
|
|
// because it returns an error if the token is not
|
|
// an opening curly brace. It does NOT advance the token.
|
|
func (p *parser) openCurlyBrace() error {
|
|
if !isOpenCurlyBrace(p.Token()) {
|
|
if p.valLooksLikeGlobalOptionsAfterImportedSnippets() {
|
|
return p.Err("global options block must appear before import directives; move the global options block to the top of the Caddyfile")
|
|
}
|
|
return p.SyntaxErr("{")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (p *parser) valLooksLikeGlobalOptionsAfterImportedSnippets() bool {
|
|
if p.Val() != "import" || len(p.block.Keys) == 0 {
|
|
return false
|
|
}
|
|
|
|
for _, key := range p.block.Keys {
|
|
if !strings.HasPrefix(key.Text, "(") || !strings.HasSuffix(key.Text, ")") {
|
|
return false
|
|
}
|
|
}
|
|
|
|
return true
|
|
}
|
|
|
|
// closeCurlyBrace expects the current token to be
|
|
// a closing curly brace. This acts like an assertion
|
|
// because it returns an error if the token is not
|
|
// a closing curly brace. It does NOT advance the token.
|
|
func (p *parser) closeCurlyBrace() error {
|
|
if !isCloseCurlyBrace(p.Token()) {
|
|
return p.SyntaxErr("}")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (p *parser) isNamedRoute() (bool, string) {
|
|
keys := p.block.Keys
|
|
// A named route block is a single key with parens, prefixed with &.
|
|
if len(keys) == 1 && strings.HasPrefix(keys[0].Text, "&(") && strings.HasSuffix(keys[0].Text, ")") {
|
|
return true, strings.TrimSuffix(keys[0].Text[2:], ")")
|
|
}
|
|
return false, ""
|
|
}
|
|
|
|
func (p *parser) isSnippet() (bool, string) {
|
|
keys := p.block.Keys
|
|
// A snippet block is a single key with parens. Nothing else qualifies.
|
|
if len(keys) == 1 && strings.HasPrefix(keys[0].Text, "(") && strings.HasSuffix(keys[0].Text, ")") {
|
|
return true, strings.TrimSuffix(keys[0].Text[1:], ")")
|
|
}
|
|
return false, ""
|
|
}
|
|
|
|
// read and store everything in a block for later replay.
|
|
func (p *parser) blockTokens(retainCurlies bool) ([]Token, error) {
|
|
// block must have curlies.
|
|
err := p.openCurlyBrace()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
nesting := 1 // count our own nesting
|
|
tokens := []Token{}
|
|
if retainCurlies {
|
|
tokens = append(tokens, p.Token())
|
|
}
|
|
for p.Next() {
|
|
if isCloseCurlyBrace(p.Token()) {
|
|
nesting--
|
|
if nesting == 0 {
|
|
if retainCurlies {
|
|
tokens = append(tokens, p.Token())
|
|
}
|
|
break
|
|
}
|
|
}
|
|
if isOpenCurlyBrace(p.Token()) {
|
|
nesting++
|
|
}
|
|
tokens = append(tokens, p.tokens[p.cursor])
|
|
}
|
|
// make sure we're matched up
|
|
if nesting != 0 {
|
|
return nil, p.SyntaxErr("}")
|
|
}
|
|
return tokens, nil
|
|
}
|
|
|
|
// ServerBlock associates any number of keys from the
|
|
// head of the server block with tokens, which are
|
|
// grouped by segments.
|
|
type ServerBlock struct {
|
|
HasBraces bool
|
|
Keys []Token
|
|
Segments []Segment
|
|
IsNamedRoute bool
|
|
}
|
|
|
|
func (sb ServerBlock) GetKeysText() []string {
|
|
res := make([]string, 0, len(sb.Keys))
|
|
for _, k := range sb.Keys {
|
|
res = append(res, k.Text)
|
|
}
|
|
return res
|
|
}
|
|
|
|
// DispenseDirective returns a dispenser that contains
|
|
// all the tokens in the server block.
|
|
func (sb ServerBlock) DispenseDirective(dir string) *Dispenser {
|
|
var tokens []Token
|
|
for _, seg := range sb.Segments {
|
|
if len(seg) > 0 && seg[0].Text == dir {
|
|
tokens = append(tokens, seg...)
|
|
}
|
|
}
|
|
return NewDispenser(tokens)
|
|
}
|
|
|
|
// Segment is a list of tokens which begins with a directive
|
|
// and ends at the end of the directive (either at the end of
|
|
// the line, or at the end of a block it opens).
|
|
type Segment []Token
|
|
|
|
// Directive returns the directive name for the segment.
|
|
// The directive name is the text of the first token.
|
|
func (s Segment) Directive() string {
|
|
if len(s) > 0 {
|
|
return s[0].Text
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// resolveImportGlob resolves importPattern relative to importerFile into a
|
|
// list of matching file paths. It handles making the pattern absolute (issue
|
|
// #867), enforces the single-wildcard limit (issue #2096), runs filepath.Glob,
|
|
// and skips dot-files for glob matches (issue #5295). It returns the matched
|
|
// paths, the resolved glob pattern (so callers can distinguish glob vs literal
|
|
// for warning/error purposes), and any hard error. Empty-match handling (the
|
|
// Warn vs "not found" decision) is left to the caller.
|
|
func resolveImportGlob(importerFile, importPattern string) (matches []string, globPattern string, err error) {
|
|
absFile, err := caddy.FastAbs(importerFile)
|
|
if err != nil {
|
|
return nil, "", fmt.Errorf("failed to get absolute path of file: %s: %v", importerFile, err)
|
|
}
|
|
|
|
if !filepath.IsAbs(importPattern) {
|
|
globPattern = filepath.Join(filepath.Dir(absFile), importPattern)
|
|
} else {
|
|
globPattern = importPattern
|
|
}
|
|
|
|
if strings.Count(globPattern, "*") > 1 || strings.Count(globPattern, "?") > 1 ||
|
|
(strings.Contains(globPattern, "[") && strings.Contains(globPattern, "]")) {
|
|
// See issue #2096 - a pattern with many glob expansions can hang for too long
|
|
return nil, globPattern, fmt.Errorf("glob pattern may only contain one wildcard (*), but has others: %s", globPattern)
|
|
}
|
|
|
|
matches, err = filepath.Glob(globPattern)
|
|
if err != nil {
|
|
return nil, globPattern, fmt.Errorf("failed to use import pattern %s: %v", importPattern, err)
|
|
}
|
|
|
|
// See issue #5295 - skip files that start with a . when the last path
|
|
// segment of the glob pattern starts with *.
|
|
sep := string(filepath.Separator)
|
|
segGlobPattern := strings.Split(globPattern, sep)
|
|
if strings.HasPrefix(segGlobPattern[len(segGlobPattern)-1], "*") {
|
|
var filtered []string
|
|
for _, m := range matches {
|
|
seg := strings.Split(m, sep)
|
|
if !strings.HasPrefix(seg[len(seg)-1], ".") {
|
|
filtered = append(filtered, m)
|
|
}
|
|
}
|
|
matches = filtered
|
|
}
|
|
|
|
return matches, globPattern, nil
|
|
}
|
|
|
|
// spanOpen and spanClose are used to bound spans that
|
|
// contain the name of an environment variable.
|
|
var (
|
|
spanOpen, spanClose = []byte{'{', '$'}, []byte{'}'}
|
|
envVarDefaultDelimiter = ":"
|
|
)
|