Caddy+Smallstep integration (trusted certs for localhost and internal sites)

[mholt]

Thanks to a recent agreement, I'll soon be working on integrating Smallstep into Caddy. 👍

This should allow us to serve HTTPS for localhost and internal sites automatically, much like we already do for public sites via Let's Encrypt. Essentially, we want https://localhost or https://example.local to be able to be the default using trusted certificates and properly managed local/internal PKI.

In Caddy 1, tls self_signed simply generated a private key in memory and used it to self-sign a certificate that lasted for a week, and that's all: it wasn't trusted, the key wasn't reused, and there was no proper PKI. With this integration in v2, Caddy will be able to serve your local dev and internal sites over HTTPS using trusted certificates with a proper, long-lived CA. This brings the ACME protocol into local and internal environments, rather than hacking private keys together in memory.

In the future, we can probably extend this to properly-managed mTLS that is (near-)fully automatic in a cluster.

This feature is not a release blocker for Caddy 2.0. It may get into 2.1.

This issue is for tracking discussion of the proposal and getting feedback from interested users and companies. Please get involved!

Here are some questions to help bootstrap development discussion:

• What do you want/need most from this feature?
• Should HTTPS be the default for all sites, including local and internal ones?
• What kind of configuration should be exposed?
• What should the cert and CA default lifetimes be?
• (Anything else that comes to your mind)

/cc Smallstep: @mmalone @maraino @mikemaxey

[mmalone]

Hi everyone. I'm Mike, from smallstep.

We're super interested in uncovering use cases for this integration. To help get a conversation going I wanted to drop a couple ideas to gauge interest, and share a few links for anyone who wants to learn more about this stuff.

Use cases

Integrating Caddy with smallstep's CA (called step-ca) will make it easy for users to run their own internal PKI -- like your own internal Let's Encrypt. In general, a certificate is a sort of credential that you can use to authenticate a server or a client (e.g., for clients it's a replacement for a username/password or API access token). Running your own CA means you can issue these credentials yourself, however you want, whenever you want.

Some use cases that we think may be interesting to the Caddy community are:

• Issuing certificates so you can use HTTPS during local development / staging (e.g., for foo.localhost)
• Serving internal websites (e.g., on an intranet) using HTTPS
• Securing (encrypting & mutually authenticating) communication between microservices using mutual TLS
• Securing connections between internal systems that need to communicate across the public internet or other untrusted networks

More info

• The smallstep CA is open source at https://github.com/smallstep/certificates
• I wrote a (long) blog post about PKI at https://smallstep.com/blog/everything-pki/
• I made the case for using TLS everywhere at https://smallstep.com/blog/use-tls/
• For more details about ACME support in step-ca see https://smallstep.com/blog/private-acme-server/

[rmhrisk]

@mholt I am not totally familiar with all of Caddys capabilities so this may not be totally relevant but in general, I see private PKI as being super important for API Key like use cases (mTLS) and service to service authentication (again mTLS).

As such, the idea of using SmallStep, over ACME, with external account binding, to acquire mTLS certificates makes a lot of sense to me.

You can see Caddy (as an FE) as the API authentication endpoint where mTLS is in use instead of the typical pre-shared key style API authentication, and having clients keeping their mTLS certificates fresh via an exposed smallstep ACME instance.

You can also see the upstream server being authenticated to via mTLS, where a different smallstep instance, again via external account binding or maybe via the traditional ACME challenges.

There is also the case where the upstream is another caddy instance and it is doing the certificate validation, and of course, getting its own certificate via ACME.

In models like this, you can have point-to-point TLS used for all communications via Private or Public TLS.

[jaredfolkins]

In the EDU space, we've talked about a binary/service that looks like our provider. A proxy to do DNS challenges to CloudFlare or something so that we can get back valid LE certs. And then on our internal servers, we'd setup certbot to call this proxy and vioala. Magic.

Smallstep looks cool but many SMBs do not have the resources to ship root CAs every where. Remove that friction and we will buy.

[rmhrisk]

I also like @jahands idea; the ability to use Caddy in the DMZ to do ACME challenge-response on behalf of hosts on the inside of a network is interesting (split horizon).

Though I think its a bad practice to use public SSL certs within an enterprise where machines are managed as it exposes the enterprise to associated operational risk and puts a weight on the public PKI (as we have seen with card readers doing the same thing) it's not clear to me UNIs have much of an option. I am sure there are other similar orgs as well as such something like this would be valuable and probably not a big extension to what you already have in Caddy.

[jaredfolkins]

Though I think its a bad practice to use public SSL certs within an enterprise where machines are managed as it exposes the enterprise to associated operational risk and puts a weight on the public PKI (as we have seen with card readers doing the same thing) it's not clear to me UNIs have much of an option. I am sure there are other similar orgs as well as such something like this would be valuable and probably not a big extension to what you already have in Caddy.

If we threat model this, we generally have a flow chart numbered below.

1. no certs
2. self signed certs
3. external LE certs
4. external certs + managed enterprise PKI

#1 is where most orgs are at #2 is a nice step but open to MiTM #3 Does put the burden on public PKI infra but I think that could be solved too #4 Would be ideal but many can't afford a FTE to support that infra.

If we are trying to make it harder for attackers, we want solutions that scale without a huge FTE and dollar investment. #3 is what I believe is best based on my experience and those who are underfunded who share their perspective at opsecedu.com. I'm open to other ideas that meet the criteria of super simple to install and scale without FTE or large dollar investments.

[nathanmcnulty]

Going to dump some relevant and some non-relevant info here that @mholt had asked about in a discussion at Opsecedu - sorry if it's a bit scattered. To preface, the discussion was around internal CA's at school districts and possible gaps that smallstep and/or caddy may have that we are currently handling in other ways.

We utilize ADCS internally to issue certificates for things like EAP-TLS, ConfigMgr which requires special purposes and EKU's on the certificates, and other things that are non-HTTPS focused. I don't believe you can request some of these things from Let's Encrypt, and I'm not familiar enough with smallstep to now if this is supported. It would be interesting if an ACME client were able to request these parameters, but that may help move off ADCS.

Another point was that a lot of MDM solutions and other equipment support SCEP but not ACME or maybe the ability to run the ACME client. I hear the frustration from the development side of things is the tooling operations uses is different than theirs, and I don't believe either of these solutions aim to solve that problem. For example, dev may spin up a box and grab a LE cert, but in production, ops may issue an internal cert via ADCS or SCEP where the dev does not have access/control.

To describe the proxy concept @jaredfolkins mentioned, I can explain what we are currently doing. We built what is basically a centralized certbot that grabs LE certificates for most of our servers with a dns-challenge. It then runs install scripts on remote hosts (think scp cert, relaunch daemon, then validate new cert) and has health check components built into the service as well. This maintains all of our LE stuff, but it would be arguably much easier if smallstep were able to act as a certbot server here.

Some mechanism, whether native smallstep or 3rd part, would get LE certificates onto smallstep, and then local ACME clients such as certbot would be able to request those publicly trusted LE certificates from smallstep. This achieves Jared's wishes that everything use the same protocols for requesting certificates, a unified CA for handling all of your certificates, and dev/ops are using the same tooling.

The part I'm unclear of is how we currently use ADCS to issue specific certificates to users and computers based on templates, disallow certain devices from requesting certificates, etc. I'm assuming that would be part of smallstep, but again, I'm just not familiar with it. Ultimately, we have to keep in mind that much of this is more than just HTTPS (we put certs on AD communications, RDP, ConfigMgr agents, SIEM agents, etc), so I'm not sure how much is really applicable to Caddy vs Smallstep vs anything else.

[rmhrisk]

Though I think its a bad practice to use public SSL certs within an enterprise where machines are managed as it exposes the enterprise to associated operational risk and puts a weight on the public PKI (as we have seen with card readers doing the same thing) it's not clear to me UNIs have much of an option. I am sure there are other similar orgs as well as such something like this would be valuable and probably not a big extension to what you already have in Caddy.

If we threat model this, we generally have a flow chart numbered below.

1. no certs
2. self signed certs
3. external LE certs
4. external certs + managed enterprise PKI

#1 is where most orgs are at #2 is a nice step but open to MiTM #3 Does put the burden on public PKI infra but I think that could be solved too #4 Would be ideal but many can't afford a FTE to support that infra.

If we are trying to make it harder for attackers, we want solutions that scale without a huge FTE and dollar investment. #3 is what I believe is best based on my experience and those who are underfunded who share their perspective at opsecedu.com. I'm open to other ideas that meet the criteria of super simple to install and scale without FTE or large dollar investments.

SmallStep is about lowering those costs and complexities so no full time FTE or other cost is required :)

[mmalone]

Gonna hop in here and try to summarize & synthesize a little bit, based on what I'm reading above.

Mutual TLS use cases

@rmhrisk it sounds like you're onboard with the two big mTLS use cases, which would be:

• The API Gateway use case, where Caddy would be validating client certificates sent by third party clients. This would be a replacement for ad-hoc API keys or something like two-legged OAuth. We'd need a way to issue certificates to third parties for this to work. I think OAuth actually has some spec extensions that do this. ACME probably does too. Either way, this would be a big project.
• The service-to-service use case, where Caddy would be deployed as a forward and/or reverse proxy in a "service mesh" style. For this, the existing certificate enrollment mechanisms are probably sufficient.

For either of these use cases, I think Caddy would have to grow better client certificate / mTLS support. I'm not sure what the current capabilities are in Caddy v2 but, minimally, it'd need to be able to:

• Configure an ingress to require client authentication, validating client certs using your internal CA's root certificate. If proxying, we'd probably want to send some/all of the cert details upstream so the app being proxied to can implement authorization.
• Configure an egress to send a client certificate when one is requested by an upstream (i.e. Caddy needs to be able to send a client certificate when it's proxying to an HTTPS server)

"Web PKI" inside the DMZ

I think this is really going to the bread-and-butter / low-hanging-fruit for an initial integration between Caddy & smallstep. Basically, the idea is to bring the standard PKI used by web browsers for HTTPS inside the DMZ. I think the big use cases for this are:

• Development & staging environments, where you want an environment that's as similar as possible to production. Having ACME in step-ca means we can simulate LE and you can delete non-HTTPS code paths from your code bases.
• Internal / intranet websites, where you want certificates that work in web browsers issued for internal domain names. Again, this will be easy to support with step-ca + Caddy.

I'm really interested in learning how common / how important these two use cases are, and whether there are additional use cases in this category.

@jaredfolkins your perspective here re: the cost of running an internal PKI is super valuable. Like @rmhrisk said, our mission is to make PKI more accessible. We want to bring that cost down. If you haven't checked out our CA it is pretty easy to get started, and I'd be very interested to hear feedback on how we can make things even easier.

You mentioned that many SMBs don't have the resources to ship root CAs. Is MDM too expensive (I honestly don't know what MDM solutions cost)? If so, you can actually use step to do root distribution if your audience is technical enough to run a couple commands:

# Download the root certificate
$ step ca root root.crt --ca-url <ca-url> --fingerprint <fingerprint>

# Install the root certificate in your system trust store(s)
$ step certificate install root.crt

Am I understanding the problem correctly? Any thoughts on how we can improve?

Other PKI use cases

@nathanmcnulty I think you're right that these non-HTTPS use cases aren't really relevant for the Caddy+smallstep integration. But we're absolutely interested in addressing these use cases with step-ca.

Certificate templating to support other purposes & EKUs is on our short-term open source roadmap. So is SCEP support. While it is possible for step-ca to work together with AD CS we've heard from a few folks that they'd rather not run more than one CA. We'd love to get to a place where we can replace AD CS! Is there anything else we need?

I don't think these use cases are really relevant to the Caddy integration, but they are important insofar as they make adopting step-ca more palatable. So I suppose it's worth cataloging use cases that might block adoption, too. Besides SCEP, the big ones that are on our radar already are:

• OpenVPN certificates
• Code signing certificates
• EAP-TLS certificates

We're also working to support a generic templating mechanism so you can totally customize certificate attributes. But I think that'll be for pretty advanced use cases. See smallstep/cli#110 for a discussion. The big blocker on that, at the moment, is figuring out what format to use for the cert template (not sure if there's a standard for that).

Links to commonly used tools, specs / standards, blog posts, etc. that are relevant to any of these use cases would also be super helpful.

[nathanmcnulty]

@mmalone That all looks great! I do think Microsoft has (unintentionally) made it difficult to fully replace AD CS because of how closely some features are integrated, such as Windows Hello for Business. These are pretty niche to a heavy Windows shop though, so I feel like those folks (like me) would tend to stick with is most documented and are most familiar with anyway.

Having said that, I don't foresee Microsoft adding ACME to AD CS, so there's still a lot to like about smallstep here, especially from a developer standpoint. Microsoft is also doing a better job of supporting 3rd party CA's in Azure/InTune (SCEP currently), so smallstep may be able to serve up those EAP-TLS certificates to InTune devices once you get further down the road :)

[rmhrisk]

Replacing AD/CS is straight forward. I built this previously and that’s exactly what it does: https://www.globalsign.com/en/auto-enrollment-gateway

Microsoft Intune SCEP really is a Microsoft proprietary protocol; they’ve added other stuff to it that’s not standard, the “standard” parts are not necessarily conformant with the SCEP draft either (there is no final RFC) and the CA must ask InTune to validate some proprietary blobs (a mix of XML and JSON as I recall) in the request prior to issuance.

[JalonSolov]

Just as a bit of extra information... I have successfully used certstrap to set up a CA, and generate site certificates for pods in a kubernetes cluster to talk to each other via TLS.

Once you have the CA, you can generate whatever certs you want, for localhost or wherever else, whenever you need them. And since you're generating them yourself, you control the expiration, etc.