WritePendingException over HTTP/2 tunnel

[sbordet]

Jetty version(s)
10+

Description
HttpClient communicating to a https server through a forward proxy via HTTP/2 tunnel.

Caused by: java.nio.channels.WritePendingException
	at org.eclipse.jetty.http2.HTTP2StreamEndPoint.write(HTTP2StreamEndPoint.java:404)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:1291)
	at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:53)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:283)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386)
	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:292)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.flush(HTTP2Session.java:2332)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.createLocalStream(HTTP2Session.java:2230)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.newLocalStream(HTTP2Session.java:2134)
	at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:603)
	at org.eclipse.jetty.http2.client.http.HttpSenderOverHTTP2.sendHeaders(HttpSenderOverHTTP2.java:127)
	at org.eclipse.jetty.client.HttpSender$ContentConsumer.onContent(HttpSender.java:496)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.notifyContent(AbstractRequestContent.java:207)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.processContent(AbstractRequestContent.java:181)
	at org.eclipse.jetty.client.util.BytesRequestContent$SubscriptionImpl.produceContent(BytesRequestContent.java:83)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.produce(AbstractRequestContent.java:117)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.demand(AbstractRequestContent.java:93)
	at org.eclipse.jetty.client.HttpSender.demand(HttpSender.java:237)
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:83)
	at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:102)
	at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:122)
	at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:111)
	at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:104)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:440)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:416)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:371)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:354)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:348)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:325)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:304)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:591)
	at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:819)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:807)

[sbordet]

This is ultimately caused by the HTTP2StreamEndPoint.flush() implementation.

The semantic of EndPoint.flush() is that it should attempt to send some bytes from the buffers, and report back whether it was able to fully send them or not, but when flush() returns there should be no pending operations trying to send those bytes.

In particular, SslConnection.DecryptedEndPoint.onIncompleteFlush() is called when the flush is incomplete, for example because the HTTP/2 stream is flow-control congested, and -- given the flush was incomplete -- tries to write() the buffers.
This causes the WritePendingException because flush() switched to PENDING state after issuing a stream.data(), and the subsequent write() finds the PENDING state.