Possible for Uncloud to use SSH configuration? (or provide an specific identity?)

[luislavena]

I've machines provisioned with SSH keys that are stored in 1Password SSH agent:

https://developer.1password.com/docs/ssh/agent/

The agent is running and is the default SSH_AUTH_SOCK, so doing ssh-add -L list all the SSH keys available in all the vaults I have access to.

$ ssh-add -L | wc -l
      73

Due that, when connecting to SSH, I normally receive the following error:

Received disconnect from REDACTED-IPV6::1 port 22:2: Too many authentication failures
Disconnected from REDACTED-IPV6::1 port 22

To workaround that, I've explicitly defined a SSH configuration in my ~/.ssh/config file for that server:

Host my-machine
    HostName my-machine
    User core
    IdentityAgent ~/.1password/agent.sock
    IdentitiesOnly yes
    IdentityFile ~/.ssh/keys/my-user-ssh-key.pub

This works great for ssh:

$ ssh my-machine "whoami"
core

$ ssh core@my-machine "whoami"
core

And also work for docker:

$ DOCKER_HOST=ssh://core@my-machine docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
2e677f8e8266   bridge    bridge    local
bc4366252ae3   host      host      local
872afa4e11ba   none      null      local

However, the same approach fails for uncloud (~/bin/uncloud is the latest build from main):

$ UNCLOUD_CONNECT=ssh://core@my-machine ~/bin/uncloud machine ls
Error: connect to cluster: connect to machine: SSH login to core@my-machine:22: connect using SSH agent: ssh: handshake failed: ssh: disconnect, reason 2: Too many authentication failures

It seems Docker relies on standard ssh command and has a hidden dial-stdio plugin under their system subcommand:

https://github.com/docker/cli/blob/75f3c08257ea0f04ddb57ec09afb7af182c36134/cli/connhelper/connhelper.go#L43-L66

Not submitting this as a feature request or a bug, but putting out there to gather feedback if I'm the only one on this scenario in order to determine if is worth to invest time on replace current SSH implementation with something similar.

Right now I've a workaround locally by commenting out all the other vaults and leave only the one that has the SSH keys for the active cluster. Also this is not a general issue as SSH to those machines is managed via Tailscale SSH.

Thank you.
❤️ ❤️ ❤️

[psviderski]

Thank you for providing a detailed break down of your ssh setup! 🙏

I have never checked how Docker implemented the handling of the ssh:// host target, and I'm surprised that they rely on the standard openssh CLI (I guess this works even on windows?). I avoided such a dependency to make the uncloud CLI platform-agnostic to easily support windows if needed.

We had an old discussion that is relevant to this issue: #22

Overall, I like the idea of offloading everything to the ssh call and let the user configure their ssh config however they like. Another advantage is we can rely on it to handle all the heavy lifting for making uc commands performant discussed in #119 (comment)

Some optiomisations:
Maintain a persistent connection in the background that can be reused for subsequent commands. In a similar way docker-pussh plugin reuses an ssh connection through a control unix socket.

Technically, I think the current connections architecture is well designed to allow adding another connection based on the ssh dialer. Or maybe even extending the current SSH one to work differently depending on whether ssh command is available or not. Could also be explicitly instructed in config.yaml.

Not sure if using the ssh CLI should be the default though 🤔. I guess my main concern is that this is an external dependency that we don't control, don't know how the behaviour may change depending on the version, and that we can avoid this dependency. However, it seems Docker somewhat proved that it's safe to go down that path?

Also this is not a general issue as SSH to those machines is managed via Tailscale SSH.

I'm a bit confused by this statement. I thought that Tailscale SSH doesn't use private ssh keys or does it?

[luislavena]

I avoided such a dependency to make the uncloud CLI platform-agnostic to easily support windows if needed.

It should work on Windows too. Windows 10 ships with ssh.exe and a SSH agent service that allow you to ssh-add keys from the credential store and keep them in memory.

This is of course those using Windows natively, but for those using WSL/WSL2, they have access to regular ssh agent, so it should continue to work.

I was not able to find any major bug report associated with this behavior that is worth considering as possible workaround.

I thought that Tailscale SSH doesn't use private ssh keys or does it?

Apologies for the confusion in the language. That is correct, when Tailscale SSH is enabled, it does not use the SSH agent for the SSH keys, so there is no Too many authentication failures like when ssh is trying one by one the keys from the SSH agent.

However, not sure if that works with current SSH implementation of Uncloud. I will give it a test and report back in the next few days.

Thank you for your patience and for looking at this! 😊
❤️ ❤️ ❤️

[luislavena]

Hola!

Coming back to this point:

However, not sure if that works with current SSH implementation of Uncloud. I will give it a test and report back in the next few days.

My setup was bit weird as these instances had a bootstrap user with an SSH key, which was making me think Tailscale SSH access was working, but actually does not work with current Uncloud setup.

To confirm, on my Tailnet I have another node (cero) I can access using ssh.

(stripped ssh -v for Tailscale elements):

$ unset SSH_AUTH_SOCK

$ ssh-add -L
Could not open a connection to your authentication agent.

$ ssh -v core@cero
OpenSSH_9.9p2, LibreSSL 3.3.6
debug1: Reading configuration data /Users/luis/.ssh/config
...
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to cero port 22.
debug1: Connection established.
...
debug1: Local version string SSH-2.0-OpenSSH_9.9
debug1: Remote protocol version 2.0, remote software version Tailscale
debug1: compat_banner: no match: Tailscale
debug1: Authenticating to cero:22 as 'core'
...
Authenticated to cero ([100.64.0.6]:22) using "none".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: filesystem
Last login: Sun Oct  5 17:40:46 UTC 2025 from 100.64.0.3 on pts/4
Flatcar Container Linux by Kinvolk stable 4230.2.3
core@cero ~ $

However, in my Uncloud configuration:

current_context: production
contexts:
  production:
    connections:
      - ssh: core@cero.tn.REDACTED

$ ~/bin/uncloud --uncloud-config ./config/production/production.yaml service ls
Error: connect to cluster: failed to connect to cluster context 'production': all connections (1) in the Uncloud config (./config/production/production.yaml) failed; last error: connect to machine: SSH login to core@cero.tn.REDACTED:22: connect using SSH agent: connect to SSH agent: dial unix: missing address

When I re-enable the SSH agent (1Password), the error is different, but still fails:

$ echo $SSH_AUTH_SOCK
/Users/luis/Library/Group Containers/2BUA8C4S2C.com.1password/t/agent.sock

$ ~/bin/uncloud --uncloud-config ./config/production/production.yaml service ls
Error: list services: list machines: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: connect to machine API socket '/run/uncloud/uncloud.sock' through SSH tunnel (is uncloud.service running on the remote machine and does the SSH user 'core' have permissions to access the socket?): ssh: rejected: unknown channel type (unsupported channel type)"

However, Docker works without issue with both SSH_AUTH_SOCK set or not:

$ DOCKER_HOST=ssh://core@cero.tn.REDACTED docker image ls
REPOSITORY                                   TAG        IMAGE ID       CREATED        SIZE
ghcr.io/luislavena/homelab-pocket-id         v0.3.0     08767f65d550   19 hours ago   113MB
ghcr.io/luislavena/homelab-headscale         v0.4.0     005d423e69e2   2 days ago     130MB
registry.docker.com/cloudflare/cloudflared   2025.9.1   4604b477520d   13 days ago    101MB
caddy                                        2.10.2     87aa104ed6c6   6 weeks ago    71.9MB

Considering that Docker remote SSH has been working for quite some time, I would assume they ironed out most of these external SSH issues (but I cannot fully confirm as my google-fu has been impacted the the new useless AI-results)

Thank you!
❤️ ❤️ ❤️

[psviderski]

The first error is expected as we currently support SSH connections only using either an SSH agent or a path to a private key.

connect using SSH agent: connect to SSH agent: dial unix: missing address

We get an empty address from SSH_AUTH_SOCK env var. Can probably provide a better error message if it's empty though:

uncloud/internal/sshexec/ssh.go

Lines 57 to 60 in 4590a51

	conn, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK"))
	if err != nil {
	return nil, func() {}, fmt.Errorf("connect to SSH agent: %w", err)
	}

The second error is more cryptic for me, but Claude suggests

Tailscale SSH does NOT support direct-tcpip or direct-streamlocal channel types, which are what client.Dial() and client.DialContext() use under the hood.

This is what the unsupported channel type error is about. We use client.DialContext(ctx, "unix", addr) to dial to the uncloud unix socket over the established ssh connection. Apparently, dialing over the ssh is an advanced feature of SSH protocol which isn't supported by Tailscale's SSH server.

https://claude.ai/share/75327c90-50ba-4d95-a0b6-23c648f45700

Quick googling didn't help find what Tailscale SSH server supports and what not. Looking at the code, it seems it only supports local port forwarding, but not remote one or other channel types: https://github.com/tailscale/tailscale/blob/main/ssh/tailssh/tailssh.go#L473-L482. So I believe Claude is right.

It's also clear why Docker works because it simply runs docker system dial-stdio process on the remote machine that basically streams the connection to the Docker unix socket as stdin/stdout over SSH. Which is actually clever tbf as this approach significantly reduces the requirements on the supported SSH features. This is another advantage of using the ssh binary instead of the native ssh client in Go.

[luislavena]

Hello @psviderski, took a stab at this and pushed a small experiment in #152. 😊