NATS generates incorrect signature for ServerHello handshake in TLS when using rsa_pss_rsae_sha384/rsa_pss_rsae_sha512 with certificate from Windows Certificate Store

[bkaestner]

Observed behavior

When NATS gets used with a certificate from the Windows Certificate Store, the signature generation for both rsa_pss_rsae_sha512 and rsa_pss_rsae_sha384 is apparently broken. We observed this in nats-io/nats.rs#1436, but we can also observe this with openssl.

The same certificate works fine if used from a file instead of the Windows Cert Store.

Expected behavior

NATS should work with rsa_pss_rsae_sha512 and rsa_pss_rsae_sha384, regardless of whether the certificate is loaded from a file or from the Windows Certificate store.

Server and client version

nats-server: v2.11.9

Host environment

Windows 11

Steps to reproduce

1. Create and install a self signed certificate

   .\mkcert -install
   .\mkcert --cert-file cert.pem -key-file key.pem localhost 127.0.0.1 ::1

2. Create a combined certificate for import into windows certificate store.

   openssl pkcs12 -export -out ./combinded.pfx -inkey ./key.pem -in ./cert.pem --password pass:

3. Install the combined pfx file as exported certificate into the Windows certificate store

4. Create a minimal configuration, e.g.

   listen: 0.0.0.0:4222
   
   tls {
     cert_store: "WindowsCurrentUser"
     cert_match_by: "Thumbprint"
     cert_match: "335351297748b8e8d285e3d20fa1a3ec2ef55646" # replace with your certificate thumbprint
     handshake_first: true # required for TLS connection tests via openssl
   }
   

   and start the server via nats-server.exe -c path/to/your.config.

5. Run openssl against the server:

   for sigalg in "rsa_pss_rsae_sha256" "rsa_pss_rsae_sha384" "rsa_pss_rsae_sha512"; do
       openssl s_client -connect localhost:4222 -sigalgs "${sigalg}"  >&/dev/null ; echo "${sigalg} $?";
   done

   sigalg	$?
   rsa_pss_rsae_sha256	0
   rsa_pss_rsae_sha384	1
   rsa_pss_rsae_sha512	1

6. Change the configuration to use the file instead:

   listen: 0.0.0.0:4222
   
   tls {
     cert_file: "./cert.pem"
     key_file: "./key.pem"
     handshake_first: true
   }
   

   and start the server via nats-server.exe -c path/to/your.other.config.

7. Run openssl against the server:

   for sigalg in "rsa_pss_rsae_sha256" "rsa_pss_rsae_sha384" "rsa_pss_rsae_sha512"; do
       openssl s_client -connect localhost:4222 -sigalgs "${sigalg}"  >&/dev/null ; echo "${sigalg} $?";
   done

   sigalg	$?
   rsa_pss_rsae_sha256	0
   rsa_pss_rsae_sha384	0
   rsa_pss_rsae_sha512	0

This affects async_nats, as it supports all three variants via ring. It might be related to #6467.

We used openssl to verify that it's not an issue of async_nats. nats-cli does not seem affected because it lists rsa_pss_rsae_sha256 first in its HELLO message. It's reproducible with both openssl s_client as is, and with openssl s_client -tls1_2 ... (both TLS 1.2 and TLS 1.3).

[bkaestner]

A coworker has managed to fix the issue by changing

nats-server/server/certstore/certstore_windows.go

Line 67 in 4a2d68f

winBCryptPadPSSSalt uint32 = 32 // default 20, 32 optimal for typical SHA256 hash

to 64, e.g.

	winBCryptPadPSSSalt uint32  = 64  // default 20, 64 optimal for maximum SHA512 hash 

It seems like the argument for NCryptSignHash might not be correct?

[bkaestner]

I should probably have added the error log to the openssl output, my bad, although grepping for salt is enough:

for sigalg in "rsa_pss_rsae_sha256" "rsa_pss_rsae_sha384" "rsa_pss_rsae_sha512"; do
    echo -n "${sigalg}: "
    openssl s_client -connect localhost:4222 -sigalgs "${sigalg}" 2>&1 1>/dev/null | grep "salt" || echo "OK";
done

rsa_pss_rsae_sha256: OK
rsa_pss_rsae_sha384: 100000000A000000:error:02000088:rsa routines:RSA_verify_PKCS1_PSS_mgf1:salt length check failed:crypto/rsa/rsa_pss.c:117:expected: 48 retrieved: 32
rsa_pss_rsae_sha512: 100000000A000000:error:02000088:rsa routines:RSA_verify_PKCS1_PSS_mgf1:salt length check failed:crypto/rsa/rsa_pss.c:117:expected: 64 retrieved: 32

This indeed indicates that the winBCryptPadPSSSalt parameter isn't correctly set for NCryptSignHash, see rsa_pss.c:117.