Understanding effective connection attempt timeout with heartbeat and idle settings

[scwhittle]

I am trying to reduce latency in detecting failed channel connections and reattempting connections sooner.

I am using the cloud spanner client library code which uses google api gax library code to connect to the spanner service. There was a period where rpcs sent to a newly created channel all failed after their deadline with errors like:
DEADLINE_EXCEEDED: CallOptions deadline exceeded after 29.996032979s. Name resolution delay 0.002361332 seconds. [closed=[], open=[[buffered_nanos=29996563395, waiting_for_connection]]]. The waiting_for_connection indicates that the connection had not yet completed connecting and that rpcs were instead buffered waiting for the connection before failing due to the deadline. This state lasted for 20 minutes until the process was restarted.

The following options are used when creating the channel:

keepAliveTime=2m
keepAliveTimeout=default
keepAliveWithoutCalls=false
idleTimeout=default (30m)

I see that keepAliveWithoutCalls being disabled means that keepalives are not sent when the transport is started.
KeepAliveManager.
This seems like the keepAliveTime thus won't help with detecting slow/stuck connection attempts. It seems then it may take up to the idleTimeout of 30 minutes to transfer the channel from CONNECTING->IDLE to detect the stuck CONNECTING state and reattempt connecting on the next call.

• Are there other mechanisms that would detect and exit long CONNECTING state sooner with this configuration? I see that Netty has ChannelOptions.CONNECT_TIMEOUT_MILLIS but this doesn't appear to be set by grpc itself (or gax).
• Though setting keepAliveWithoutCalls=true may help reduce the timeout to the keep-alive time it changes the behavior when the channel is not connecting. As a user since there are attempted calls it seems like keep-alives should be sent in this case. Is this a bug that should be addressed?

[AgraVator]

Hi,

To give you the short answer first: This is not a bug. The behavior you are observing (stuck in CONNECTING with waiting_for_connection) happens before a transport exists, so gRPC KeepAlives (which are HTTP/2 frames) cannot physically be sent yet.

To solve the "stuck" connecting state, you need to tune the Transport Layer (TCP) Connect Timeout as rightly pointed out by you, not the HTTP/2 KeepAlive.

1. Why keepAliveWithoutCalls Doesn't Help Here
   The waiting_for_connection state means your client is still trying to perform the TCP Handshake or TLS Handshake with the Spanner backend.

KeepAlive (HTTP/2 PING): This is a frame sent inside an established encrypted tunnel. It asks the server, "Are you still there?"

Connecting State: The tunnel does not exist yet. There is no path on which to send the PING.

Therefore, setting keepAliveWithoutCalls=true would not change anything for this specific error. It only helps if the connection successfully establishes, stays idle, and then the network silently drops it (e.g., a firewall killing a quiet connection).

2. The Real Culprit: TCP "Black Holes"
   The error DEADLINE_EXCEEDED after 30s with waiting_for_connection indicates your RPC deadline fired while the channel was still trying to connect.

The fact that this persisted for 20 minutes suggests something amiss.

[scwhittle]

Thanks for your response.

There is still something that doesn't make sense to me. When keepAliveWithoutCalls=true, the KeepAliveManager is started earlier, during onTransportStart. AFAICT this may also be before TCP Handshake or TLS handshake has completed as it is done after connect is called here but that is async. In that case there won't be a way to actually send the keepalive but the scheduled ping could perhaps detect the stuck connection here. This is why I was wondering if the keepAliveWithoutCalls would make a difference here.

I agree that there is likely something wrong with the networking but I am trying to make the grpc configuration more resistent to such issues. It is unfortunate if the netty specific timeout is needed as keep-alives seem like the general grpc concept that should help here.