systemctl reload hangs forever if reading TLS cert/key fails

[micopiira]

Issue Details

It appears that if reading TLS certificate/key file fails for whatever reason (For example permission denied, invalid/corrupt file) when doing a systemctl reload caddy, the command will hang forever. Even worse is that even after fixing the certificate/key file the systemctl reload caddy command will still hang, forcing you to restart caddy incurring downtime.

I'm using the default caddy.service systemd unit file.

Journalctl will show this when the command is hanging:

Dec 04 19:35:59 docker-test systemd[1]: Reloading Caddy...
Dec 04 19:35:59 docker-test caddy[54557]: {"level":"info","ts":1764876959.1031754,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Dec 04 19:35:59 docker-test caddy[54557]: {"level":"info","ts":1764876959.1038148,"msg":"adapted config to JSON","adapter":"caddyfile"}
Dec 04 19:35:59 docker-test caddy[54345]: {"level":"info","ts":1764876959.1047776,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"44380","headers":{"Accept-Encoding":["gzip"],"Cache-Control":["must-revalidate"],"Content-Length":["480"],"Content-Type":["application/json"],"Origin":["http://localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Dec 04 19:35:59 docker-test caddy[54345]: {"level":"info","ts":1764876959.1054118,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Dec 04 19:35:59 docker-test caddy[54345]: {"level":"info","ts":1764876959.10555,"logger":"http","msg":"servers shutting down with eternal grace period"}
Dec 04 19:35:59 docker-test caddy[54345]: {"level":"error","ts":1764876959.105726,"logger":"admin.api","msg":"request error","error":"loading config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: loading certificates: tls: failed to find any PEM data in key input","status_code":400}
Dec 04 19:35:59 docker-test caddy[54557]: Error: sending configuration to instance: caddy responded with error: HTTP 400: {"error":"loading config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: loading certificates: tls: failed to find any PEM data in key input"}
Dec 04 19:35:59 docker-test caddy[54345]: {"level":"info","ts":1764876959.1067598,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Dec 04 19:35:59 docker-test systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
Dec 04 19:37:29 docker-test systemd[1]: caddy.service: Reload operation timed out. Killing reload process.
Dec 04 19:38:59 docker-test systemd[1]: caddy.service: Reload operation timed out. Killing reload process.

Caddyfile:

example.com {
        tls /etc/caddy/example.com.cer /etc/caddy/example.com.key
        respond "Hello world!"
}

Is there something that I or Caddy could do differently to avoid having to restart Caddy in such cases?

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[francislavoie]

It happens here

caddy/caddy.go

Line 125 in be5f49f

if notifyErr := notify.Error(err, 0); notifyErr != nil {

We notify status to the socket https://github.com/caddyserver/caddy/blob/master/notify/notify_linux.go

I'm not sure why it would timeout, it's supposed to notify of an error 🤔 I don't understand systemd enough to know why this isn't working correctly.

[micopiira]

@francislavoie I suppose the READY=1 signal should also be sent to the socket on errors during reload? However after looking at the commits it seems that Caddy has done that in the past but it has been explicitly changed to not send that anymore in #5003, but this issue seems to be related to startup, not reloading 🤔

Quote from systemd.service docs:

When initiating the reload process the service is expected to reply with a notification message via sd_notify(3) that contains the "RELOADING=1" field in combination with "MONOTONIC_USEC=" set to the current monotonic time (i.e. CLOCK_MONOTONIC in clock_gettime(2)) in μs, formatted as decimal string. Once reloading is complete another notification message must be sent, containing "READY=1". Using this service type and implementing this reload protocol is an efficient alternative to providing an ExecReload= command for reloading of the service's configuration.

And also sd_notify docs:

RELOADING=1
Tells the service manager that the service is beginning to reload its configuration. This is useful to allow the service manager to track the service's internal state, and present it to the user. Note that a service that sends this notification must also send a "READY=1" notification when it completed reloading its configuration. Reloads the service manager is notified about with this mechanisms are propagated in the same way as they are when originally initiated through the service manager. This message is particularly relevant for Type=notify-reload services, to inform the service manager that the request to reload the service has been received and is now being processed.

and note: I'm not that familiar with systemd either, not sure if this is relevant at all or not