[feature] Extensible caddyfile

[elee1766]

in this pr, i extend the caddyfile to be extensible with "server blocks". this extension should be completely backwards compatible with existing caddyfile, but allows plugins to declare their own blocks.

block parsers should take in a slice of server blocks, and parse them as a single unit, using the configbuilder to create/modify apps as needed.

as for syntax - httpcaddyfile syntax, which is normally like this:

...keys { 
   ...inner_block
}

is instead extended the syntax to

[blocktype] ...keys { 
   ...inner_block
}

we can maintain caddyfile compatibility through parsing the first block, if it has no keys, as a [global] block, and a [http.server] block otherwise.

for instance, the below xcaddyfile

[global] {
	admin unix//run/caddy/admin.socket
	log DEBUG
	grace_period 30s
	https_port 8443
	http_port 8080
}
[http.server] https://example.com {
	root * /var/www/example
	file_server
	encode gzip
}

[http.server] https://api.example.com {
	reverse_proxy localhost:9000
}

is equivalent to

{
	admin unix//run/caddy/admin.socket
	log DEBUG
	grace_period 30s
	https_port 8443
	http_port 8080
}
https://example.com {
	root * /var/www/example
	file_server
	encode gzip
}

https://api.example.com {
	reverse_proxy localhost:9000
}

both of these files would resolve to the same json.

here is the pr for caddy-l4, which would be a possible consumer of this change: mholt/caddy-l4#342

Assistance Disclosure

claude code is used to write and modify nearly all lines of code. some lines are modified by hand when having the LLM do the modification would be more difficult.

[francislavoie]

All looks pretty good to me.

I think we need to add a bit of adapt test coverage.

Also I think the caddy fmt should at least not explode if it finds a [], I'm not sure how that behaves now but should be checked.

[elee1766]

tests pass now :)

it might be appropriate to add some more advanced adapt test cases to mainline, and make sure we pass them here as well.

will also for sure have to add some adapt cases for [block] syntax

[mholt]

(The failing test is unrelated, as noted is being worked in another issue; seems to be a faulty linter.)

[mholt]

Thanks for this, looks neat. I have a few questions/concerns but probably things we can answer:

• Is the value inside the [ ] arbitrary, like http.server? Or is it correlated to the JSON config structure, or...? Like does the . mean anything, as in a namespace?
• The current draft exports a ton of identifiers and adds substantially to the package structure of the module. (Block types, a config builder, and such.) Many of the types seem like internal implementation details like the pile. Is it feasible and practical to keep most of it unexported and simplify the package structure?
• Why does [global] need to exist... as opposed to just leaving it untitled?

[elee1766]

@mholt

is the value inside the [ ] arbitrary, like http.server? Or is it correlated to the JSON config structure, or...? Like does the . mean anything, as in a namespace?

it's an arbitrary string. it can be anything, no direct relationship to the json.

my thought is perhaps plugins would unofficially "reserve" a top level key foo for their plugin global settings, and then foo.bar or foo.baz as different blocks, in case the plugin supports more than one block type. it mirrored the namespace logic you use for plugins so I felt it fit.

I wanted to stay as flexible as extensible as possible - we can always add restrictions or semantics on top of this more flexible system if needed.

The current draft exports a ton of identifiers and adds substantially to the package structure of the module. (Block types, a config builder, and such.) Many of the types seem like internal implementation details like the pile. Is it feasible and practical to keep most of it unexported and simplify the package structure?

the config builder and some block type code needs to be exported so that plugins are able to properly consume and use the types. we can probably go through what l4ddy needs.

for sure it's possible to go through and unexport things that may not be useful to be exposed

Why does [global] need to exist... as opposed to just leaving it untitled?

one of my ambitions with this change is to make it so the syntax doesn't rely on this untitled first block=global settings semantic. I think this is a really painful part of configuring caddyfile.

I want to

1. have my global block as not the first block, and be able to be in a global.Caddyfile file that I import
2. have the global block be properly labeled so it's obviously not a site

being able to optionally label the global block is two birds in one stone.

personally I think the global block is sort of a icky piece of design debt. http settings should probably always have been configured in an http settings block, caddy settings like logging in a caddy options block.

the idea of everything being thrown into the global block feels like a hack that was deemed acceptable, but I think the user experience of configuring caddy security and caddy l4 through giant blocks in the global block imo is proof that it's probably not sustainable to scale plugin config with global directives

(it would be great if I could have security.Caddyfile for instance to store all my security global configs and snippets, instead of 100 lines in my root caddyfile...)

[mholt]

my thought is perhaps plugins would unofficially "reserve" a top level key foo for their plugin global settings, and then foo.bar or foo.baz as different blocks, in case the plugin supports more than one block type. it mirrored the namespace logic you use for plugins so I felt it fit.

I wanted to stay as flexible as extensible as possible - we can always add restrictions or semantics on top of this more flexible system if needed.

Fair enough -- I might suggest that either we define some structure to that label, though, or we make it a freeform string. [http.server] feels too structured for it to be arbitrary, like it should be [app.namespace] or something if we do that, or even just [http] -- let the HTTP Caddyfile parse this block. That kind of thing.

the config builder and some block type code needs to be exported so that plugins are able to properly consume and use the types. we can probably go through what l4ddy needs.

for sure it's possible to go through and unexport things that may not be useful to be exposed

Along with simplifying the package structure, if we do have to export new things, maybe some should still be less footgun-proof, like instead of exporting Pile, have a method to get something from the pile. (Does anything outside of the block ever add to the pile?)

one of my ambitions with this change is to make it so the syntax doesn't rely on this untitled first block=global settings semantic. I think this is a really painful part of configuring caddyfile.

I want to

1. have my global block as not the first block, and be able to be in a global.Caddyfile file that I import
2. have the global block be properly labeled so it's obviously not a site

being able to optionally label the global block is two birds in one stone.

What is painful about the global options being first? I don't think I've heard that before. I think it makes a lot of sense for it to be first, and is confusing if it's at the middle or end.

personally I think the global block is sort of a icky piece of design debt. http settings should probably always have been configured in an http settings block, caddy settings like logging in a caddy options block.

I have some regrets with the Caddyfile, indeed -- but I think the global options block has a place given what we had to work with. Individual app settings can -- and do -- go inside sub-blocks, like you're saying. (Though most of the HTTP ones have surfaced to the top level of the global options block because HTTP is the most common use case.)

[elee1766]

What is painful about the global options being first? I don't think I've heard that before. I think it makes a lot of sense for it to be first, and is confusing if it's at the middle or end.

there only being able to be one global block, and that global block must implicitly be the first block means that I can't compose multiple global blocks in arbitrary places together with import.

the best I can do I think is put import globals.Caddyfile as the first line of my caddyfile. but this still doesn't let me properly segment and organize my options.

it was really frustrating that I couldn't put my rate limit global config next to my ratelimiter handler snippet. for my personal caddy instance, I couldn't put caddy-security config next to my authentication middleware/snippets.

ideally my config file could look something like

import binds.Caddyfile
import logging.Caddyfile
import admin.Caddyfile
import auth.Caddyfile
import ratelimiter.Caddyfile

import middleware.Caddyfile

import sites/a.Caddyfile
import sites/b.Caddyfile

with each import at the top both configuring the global options AND adding reusable functions/snippets

fwiw, it already sort of looks like this just with a single global block at the top instead of a bunch of imports.

we used to use caddy on a larger scale and have a repo with 50 ish caddyfiles like this imported together for different sites etc for an API gateway with 7-8 custom plugins, but managing the config of all these plugins in a single global block was really messy since we couldn't meaningfully separate it. (distributed ratelimiting, request logging to nats, native jsonrpc websocket protocol handling, cert storage, custom filesystems, just to start, all required top level configuration, that made the global block giant and untenable)

[mholt]

I see... so you want flexibility to group your config lines by topic, or group related config lines.

I'm open to this idea.

[elee1766]

I see... so you want flexibility to group your config lines by topic, or group related config lines.

I'm open to this idea.

exactly. this is what i currently have:

/etc/caddy/Caddyfile

{
        admin "unix//run/caddy/admin.socket"

        order authenticate before respond
        order authorize before basicauth
        log default {
                format json
                level DEBUG
        }
        security {
                oauth identity provider google {
                        realm google
                        driver google
                        client_id  {env.GOOGLE_CLIENT_ID}.apps.googleusercontent.com
                        client_secret {env.GOOGLE_CLIENT_SECRET}
                        scopes openid email profile
                }
                authentication portal myportal {
                #blah
                }

                authorization policy admin_only {
                #blah
                }

                authorization policy family_only {
                #blah
                }
        }
}

import /etc/caddy/include.d/*
import /etc/caddy/conf.d/*

then i have

/etc/caddy/include.d/oauth.site

# vi: syntax=caddyfile

(oauth_secure) {
  authorize with admin_only
}

(oauth_family) {
  authorize with family_only
}

oauth.put.gay {
        authenticate with myportal
}

oauth.elee.bike {
        authenticate with myportal
}

so then i use these imports in my sites!
/etc/caddy/include.d/family.site

# vi: syntax=caddyfile

family.elee.bike {
  import oauth_family
  respond "hello family" 200
}

ngx.elee.bike {
  import oauth_family
  reverse_proxy 127.0.0.1:8000
}

but in better world, my root Caddyfile could look like

/etc/caddy/Caddyfile

{
        admin "unix//run/caddy/admin.socket"

        log default {
                format json
                level DEBUG
        }
}

import /etc/caddy/include.d/*
import /etc/caddy/conf.d/*

and then my /etc/caddy/include.d/oauth.site is...

[global] {
        order authenticate before respond
        order authorize before basicauth
}
[caddysecurity] {
            oauth identity provider google {
                    realm google
                    driver google
                    client_id  {env.GOOGLE_CLIENT_ID}.apps.googleusercontent.com
                    client_secret {env.GOOGLE_CLIENT_SECRET}
                    scopes openid email profile
            }
            authentication portal myportal {
            #blah
            }

            authorization policy admin_only {
            #blah
            }

            authorization policy family_only {
            #blah
            }
}
(oauth_secure) {
  authorize with admin_only
}

(oauth_family) {
  authorize with family_only
}

oauth.put.gay {
        authenticate with myportal
}

oauth.elee.bike {
        authenticate with myportal
}

in this way, i can now import modules (full caddyfiles) which provide functions (snippets) and also export endpoints (servers).

i can organize these modules by file, making it a lot easier to organize and keep track of what is configuring what.

when we were trying to do this at gfx, what ended up happening is the main Caddyfile had a nearly thousand line long global block because of all the global plugin configs. we could split up the snippets to another file, but then the plugin that the snippet depended on would need to be configured in global, which was very unfortunate (and why we reengineered our api gateway to be traefik based)

[elee1766]

as an aside - what i really like about this feature is it opens plugins to be able to configure themselves globally with blocks, and blocks can be more than just a site/server configuration.
for instance, a rate limiter could do something like...

[ratelimit.backend] myredis {
   driver redis
   url redis://[::]:6379
}

[ratelimit.policy] foo {
   limit 100
   burst 1000
   duration 5s
   backend myredis
} 

[ratelimit.policy] bar {
   limit 100
   burst 1000
   duration 5s
   backend myredis
} 

https://example.com {
  ratelimit_apply_policies foo bar 
  respond 200
}

which i think is a lot more flexible, clear, and, extensible, than a gigantic

{
  ratelimit {
    backend myredis {
      driver redis
      url redis://[[::]:6379
    }
    policy foo {
      limit 100
      burst 1000
      duration 5s
      backend myredis
    }
    policy bar {
     limit 100
     burst 1000
     duration 5s
     backend myredis
    }
  }
}

and even if repeated blocks not preferred, plugin could do

[ratelimit.global] {
  backend myredis {
    driver redis
    url redis://[[::]:6379
  }
  policy foo {
    limit 100
    burst 1000
    duration 5s
    backend myredis
  }
  policy bar {
   limit 100
   burst 1000
   duration 5s
   backend myredis
  }
}

[mholt]

Thanks for explaining/demonstrating.

I have to admit, I do prefer this personally:

{
  ratelimit {
    backend myredis {
      driver redis
      url redis://[[::]:6379
    }
    policy foo {
      limit 100
      burst 1000
      duration 5s
      backend myredis
    }
    policy bar {
     limit 100
     burst 1000
     duration 5s
     backend myredis
    }
  }
}

Just feels simpler, easier to reason about, versus:

[ratelimit.backend] myredis {
   driver redis
   url redis://[::]:6379
}

[ratelimit.policy] foo {
   limit 100
   burst 1000
   duration 5s
   backend myredis
} 

[ratelimit.policy] bar {
   limit 100
   burst 1000
   duration 5s
   backend myredis
} 

https://example.com {
  ratelimit_apply_policies foo bar 
  respond 200
}

I realize that's 100% personal preference though.

What I DO like from this proposal is the ability to do:

[tcp] :1234 {
    ...
}

or something like that, without needing to use the global options block for layer4 routes...

[elee1766]

yeah my issue is more for sure about the global block becoming giant and untenable, and currently it can't be organized across multiple files.

[francislavoie]

I don't think I like sub-names like [ratelimit.backend], I feel like such an app should just do [ratelimit] and have its options within that. I don't want to lift layers to top level blocks that much. It gets messy IMO. Best to keep all related options for config of an app together in a block.

Nothing stops the users from having multiple [ratelimit] blocks which get merged via being passed through unmarshal more than once, the app should handle merging or rejecting as appropriate, like in your example you could have two separate blocks each defining a policy and since they have a named key they will both be added to the app's mapping without conflict, but repeating backend config should probably result in an error.

For layer4 I think having [tcp] and [udp] aren't a good idea because we use network addresses format so it would be awkward to pull that up to the block name layer. I rather we go with [layer4] or we could decide to call it something like [raw] when we pull it into Caddy's standard distribution.

Btw what do we call this [name] thing? "block name"? "block type"? We should settle on that, especially for documentation.

[elee1766]

layer4.tcp and layer4.udp are another option.

I like block type over block name because the word name is used in named blocks which are user defined structures, while these are plugin/app defined structures which are more akin to types.

to throw out some words to brainstorm, "block variant", "block mode", "block extension", "block class"

[mholt]

(Or like instead of [raw] what about [packets]?)

I like block type over block name because the word name is used in named blocks which are user defined structures, while these are plugin/app defined structures which are more akin to types.

Yeah my votes would be like you said (in no particular order):

• Block type
• Block class
• Block mode

[francislavoie]

Looks like "block type" is the common denominator. I'm good with that.