Repository navigation
xds: avoid IllegalStateException on sendMessage/halfClose after internal cancellation - #13158
kannanjgithub wants to merge 2 commits into
Conversation
…nal cancellation
When RawMessageClientInterceptor or ExternalProcessorClientInterceptor
encounters an internal error (such as an IOException while reading the
request InputStream in sendMessage) and cancels the underlying call, the
application is not synchronously aware that the call was cancelled.
Callers such as ClientCalls.blockingUnaryCall immediately invoke
halfClose() after sendMessage(), which previously delegated to the
already-cancelled downstream call and threw
IllegalStateException("call was cancelled"), masking the underlying
CANCELLED status and cause.
Fix this by tracking internal cancellation (`aborted`) separately from
explicit application-initiated cancellation (`cancelCalled`):
1. RawMessageClientInterceptor (PartialForwardingClientCall):
- Move `sendSerializedMessage` into `PartialForwardingClientCall` so
`sendMessage`, `halfClose`, and `cancel` share per-call state.
- Set `aborted = true` when `sendSerializedMessage` catches an
`IOException` or `RuntimeException` and cancels `rawCall`, making
subsequent `sendMessage()` and `halfClose()` calls no-ops (matching
`ClientCallImpl.sendMessageInternal` when it cancels the underlying
stream internally).
- Set `cancelCalled = true` (and `aborted = false`) in `cancel()` and
delegate `rawCall.sendMessage(null)` before stream serialization
when `cancelCalled` is true, ensuring `sendMessage()` and
`halfClose()` after explicit application cancellation still throw
`IllegalStateException("call was cancelled")`.
2. ExternalProcessorClientInterceptor (DataPlaneClientCall):
- Set `aborted = true` in `cancelDownstream()` when triggered
internally (`!cancelCalled.get()`), turning subsequent
`sendMessage()`, `halfClose()`, and `proceedWithHalfClose()` calls
into no-ops.
- Set `cancelCalled = true` (and `aborted = false`) in `cancel()` and
check `!cancelCalled.get()` in `sendMessage()` and `halfClose()` so
explicit application cancellation enforces `IllegalStateException`
even when requests or half-close would otherwise be buffered or
sent to the ext_proc sidecar without synchronously reaching
`delayedCall`.
TAG=agy
CONV=56c87717-243d-4f12-af9e-c532e509892a
| @@ -1159,11 +1173,16 @@ public void halfClose() { | |||
| } | |||
|
|
|||
| void cancelDownstream(@Nullable String message, @Nullable Throwable cause) { | |||
There was a problem hiding this comment.
Is there a potential race between cancelDownstream and cancel for the value of aborted? cancelDownstream could set aborted to true after it's ben set to false by cancel.
There was a problem hiding this comment.
I have changed it now so that:
cancel()only setscancelCalled.set(true)and callsdelayedCall.cancel(message, cause)directly, whilecancelDownstream()only setsaborted.set(true)and callsdelayedCall.cancel(message, cause)(so neither flag ever overwrites the other).sendMessage()andhalfClose()checkcheckState(!cancelCalled.get(), "call was cancelled")beforeif (aborted.get()) { return; }, so if bothcancelDownstream()andcancel()occur, explicit application cancellation always takes precedence without needingcancel()to resetaborted.
| if (aborted.get()) { | ||
| return; | ||
| } | ||
| checkState(!cancelCalled.get(), "call was cancelled"); |
There was a problem hiding this comment.
This doesn't hold true.
We have cases where our methods get invoked after cancellation. In particular this is again a race between cancel from application and execution of buffered halfclose from the sidecar.
While this instance of race can probably be solved moving the cancel updation in streamlock and adding checks at the callsite, we might want to audit our cancellation behavior for other such cases.
There was a problem hiding this comment.
We should have a separate internalHalfClose method separate from the public halfClose. Only the latter should throw if application had cancelled the call already.
| } | ||
|
|
||
| private void proceedWithHalfClose() { | ||
| if (aborted.get()) { |
There was a problem hiding this comment.
Shouldn't both checks include cancellation as well as abort?
Additionally, should most top level public calls be gated via cancellation and abort as well?
maybe we should start by defining our cancellation semantic i.e. "after cancellation and its atomic cleanup"
- extproc impl absolutely cannot call any method
- extproc impl may call other methods but all of them should behave like no-op.
There was a problem hiding this comment.
Added check for cancellation as well.
It should just be a no-op and short-circuit ext-proc calls super.sendMessage or any public method if either aborted || cancelCalled is true.
| return; | ||
| } | ||
| if (cancelCalled) { | ||
| rawCall.sendMessage(null); |
There was a problem hiding this comment.
what's this about? What does sendMessage(null) do? Is this publicly documented API or is it something specific to ext proc interceptor?
I am worried about deeply coupling this to the ext proc implementation details and also running the risk of some middle chain interceptor before proc not handling null very well.
This will tie in to my comment about defining the semantics we want after cancellation first. In this case, my thoughts would be to do an explicit failure or exception if this was called after cancellation (i.e. handle at the first point of violation) instead of delegating it.
There was a problem hiding this comment.
Calling sendMessage(null) was simply a way to cause IllegalStateException("call was cancelled") to be thrown if the application had already cancelled.
However, doing checkState(!cancelCalled, "call was cancelled"); directly (just like in DataPlaneClientCall.sendMessage) is much cleaner.
1. Check !cancelCalled before aborted in sendMessage and halfClose, and have cancel() call delayedCall.cancel() directly (setting cancelCalled under streamLock) so cancelCalled and aborted never overwrite each other. 2. Replace rawCall.sendMessage(null) in RawMessageClientInterceptor with checkState(!cancelCalled, "call was cancelled"). 3. Separate public halfClose() from internalHalfClose() and guard downstream sendMessage/halfClose calls via proceedWithSendMessage() and proceedWithHalfClose() using isCancelledOrAborted() so sidecar and drain callbacks act as no-ops after cancellation or abort. TAG=agy CONV=56c87717-243d-4f12-af9e-c532e509892a
When RawMessageClientInterceptor or ExternalProcessorClientInterceptor encounters an internal error (such as an IOException while reading the request InputStream in sendMessage) and cancels the underlying call, the application is not synchronously aware that the call was cancelled. Callers such as ClientCalls.blockingUnaryCall immediately invoke halfClose() after sendMessage(), which previously delegated to the already-cancelled downstream call and threw
IllegalStateException("call was cancelled"), masking the underlying CANCELLED status and cause.
Fix this by tracking internal cancellation (
aborted) separately from explicit application-initiated cancellation (cancelCalled):RawMessageClientInterceptor (PartialForwardingClientCall):
sendSerializedMessageintoPartialForwardingClientCallsosendMessage,halfClose, andcancelshare per-call state.aborted = truewhensendSerializedMessagecatches anIOExceptionorRuntimeExceptionand cancelsrawCall, making subsequentsendMessage()andhalfClose()calls no-ops (matchingClientCallImpl.sendMessageInternalwhen it cancels the underlying stream internally).cancelCalled = true(andaborted = false) incancel()and delegaterawCall.sendMessage(null)before stream serialization whencancelCalledis true, ensuringsendMessage()andhalfClose()after explicit application cancellation still throwIllegalStateException("call was cancelled").ExternalProcessorClientInterceptor (DataPlaneClientCall):
aborted = trueincancelDownstream()when triggered internally (!cancelCalled.get()), turning subsequentsendMessage(),halfClose(), andproceedWithHalfClose()calls into no-ops.cancelCalled = true(andaborted = false) incancel()and check!cancelCalled.get()insendMessage()andhalfClose()so explicit application cancellation enforcesIllegalStateExceptioneven when requests or half-close would otherwise be buffered or sent to the ext_proc sidecar without synchronously reachingdelayedCall.Fixes the issue mentioned in an earlier comment in a PR.