Skip to content

fix(bigtable): session pool — re-drive creation after a never-activated close so the consecutive-failure breaker is reachable - #20355

Merged
mutianf merged 1 commit into
googleapis:mainfrom
mutianf:08-11-fix_bigtable_session_pool
Aug 12, 2026
Merged

mutianf merged 1 commit into
googleapis:mainfrom
mutianf:08-11-fix_bigtable_session_pool

Conversation

@mutianf

@mutianf mutianf commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When the vRPC OpenSession API is not served (server returns Unimplemented),
a session dies while still in StateStarting and never activates. onClose's
starting branch bumped the consecutive-failure counter but nothing re-drove
creation
: onClosing early-returns for starting handles, and no other of the
sizer's event-driven wake sites fires on a never-activated close.

A cold-start MinSessionCount (5) cohort therefore froze the counter at 5,
never reaching ConsecutiveSessionFailureThreshold (10), so the circuit breaker
never tripped. With nothing re-driving creation and nothing tripping the breaker,
a CheckoutSession caller parked on a deadline-less context hung forever (the
43-min stuck goroutine in the prod round-trip test).

Fix

In onClose's starting branch, after noteAbnormalCloseIfAny, re-drive creation
via spawnTickOnce (gated on sl.ReadyCount() < MaxSessions, mirroring
onClosing's replace-on-death kick). Each re-driven cohort fails again, the
counter climbs to the threshold, the existing breaker trips, and parked
waiters are drained with ErrConsecutiveFailures carrying the underlying
Unimplemented code — only after X consecutive failures, not on the first.
Paced by the throttler's failure penalty, so this is not a tight retry loop.

This replaces the earlier failWaitersOnCreateFailure approach from this PR,
which wrongly failed waiters on the first create failure instead of after a
sustained run. That mechanism (ErrSessionCreateFailed, sessionCreateError,
the Tick call, and the session_pool_waiters_failed_on_create tag) is reverted.

Also narrows the accelerator round-trip test's skipIfSessionUnavailable to skip
on Unimplemented only — the pool now surfaces that code deterministically
instead of hanging, so transient Unavailable / DeadlineExceeded are left to
retryTransient.

Testing

  • go test ./internal/transport/ -race — full suite green.
  • New tests: TestConsecutiveFailures_StartingDeathsReDriveUntilBreakerTrips
    (end-to-end: starting-deaths re-drive → breaker trips → caller wakes with
    ErrConsecutiveFailures + status.Code == Unimplemented) and
    TestConsecutiveFailures_SingleStartingCloseDoesNotWakeWaiters (a single
    starting-close increments but does not wake waiters).
  • Verified against prod cloud-bigtable-spacewalk / mattiefu-test (which
    does not serve OpenSession): the round-trip test now fails fast in 4.72s
    with the breaker trip carrying Unimplemented: Method is not available... and
    cleanly skips — previously it hung for 43 minutes.

@mutianf
mutianf requested review from a team as code owners August 11, 2026 02:32
@product-auto-label product-auto-label Bot added the api: bigtable Issues related to the Bigtable API. label Aug 11, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to prevent parked CheckoutSession callers from hanging indefinitely when a session-creation attempt fails and the pool has no remaining path to a ready session. It adds a new ErrSessionCreateFailed error, implements failWaitersOnCreateFailure to wake up stranded waiters, and includes comprehensive unit tests. The reviewer identified a potential data race in failWaitersOnCreateFailure where p.sl.ReadyCount() is called outside of the p.mu lock, and suggested a thread-safe implementation that reads the ready count while holding the lock.

@mutianf
mutianf force-pushed the 08-11-fix_bigtable_session_pool branch from d3caad7 to f89bb4c Compare August 11, 2026 03:02
…ed close so the consecutive-failure breaker is reachable

When the vRPC OpenSession API is not served (server returns Unimplemented),
a session dies while still in Starting and never activates. onClose's
starting branch bumped the consecutive-failure counter but nothing re-drove
creation: onClosing early-returns for starting handles, and no other of the
sizer's event-driven wake sites fires on a never-activated close. A
cold-start MinSessionCount cohort therefore froze the counter below
ConsecutiveSessionFailureThreshold, the circuit breaker never tripped, and a
CheckoutSession caller parked on a deadline-less context hung forever (the
43-min stuck goroutine in the prod round-trip test).

Fix: in onClose's starting branch, after noteAbnormalCloseIfAny, re-drive
creation via spawnTickOnce (gated on sl.ReadyCount() < MaxSessions, mirroring
onClosing's replace-on-death kick). Each re-driven cohort fails again, the
counter climbs to the threshold, the existing breaker trips, and parked
waiters are drained with ErrConsecutiveFailures carrying the underlying
Unimplemented code — only after X consecutive failures, not on the first.
Paced by the throttler's failure penalty so this is not a tight retry loop.

This replaces the earlier failWaitersOnCreateFailure approach, which wrongly
failed waiters on the first create failure instead of after a sustained run;
that mechanism (ErrSessionCreateFailed, sessionCreateError, the Tick call,
and the session_pool_waiters_failed_on_create tag) is reverted.

Also narrows the accelerator round-trip test's skipIfSessionUnavailable to
skip on Unimplemented only — the pool now surfaces that code deterministically
instead of hanging, so transient Unavailable/DeadlineExceeded are left to
retryTransient.

Change-Id: I4621616f7e81b4ccaca282b5758149612239041b
@mutianf
mutianf force-pushed the 08-11-fix_bigtable_session_pool branch from f89bb4c to 3ae233d Compare August 11, 2026 14:27
@mutianf mutianf changed the title fix(bigtable): session pool — wake parked waiters when a create failure strands the pool empty fix(bigtable): session pool — re-drive creation after a never-activated close so the consecutive-failure breaker is reachable Aug 11, 2026
@mutianf
mutianf merged commit ee74949 into googleapis:main Aug 12, 2026
17 of 19 checks passed
@mutianf
mutianf deleted the 08-11-fix_bigtable_session_pool branch August 12, 2026 14:32
sushanb pushed a commit that referenced this pull request Aug 28, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.53.0](bigtable/v1.52.0...bigtable/v1.53.0)
(2026-08-27)


### Features

* **bigtable/accelerator:** Add --user-agent flag to daemon
([#20341](#20341))
([6b33c06](6b33c06))
* **bigtable/accelerator:** Add daemon binary with identity verification
([#20330](#20330))
([2f9cf89](2f9cf89))
* **bigtable/accelerator:** Add proxy channel scaffold owning a sessi…
([#20310](#20310))
([d65ca7d](d65ca7d))
* **bigtable/accelerator:** Add stdin-EOF and parent-PID lifecycle wa…
([#20328](#20328))
([dc1c6ae](dc1c6ae))
* **bigtable/accelerator:** Add UDS server with authenticated proxy i…
([#20320](#20320))
([a837020](a837020))
* **bigtable/accelerator:** Add V2-to-session request adapters
([#20271](#20271))
([4cb6f16](4cb6f16))
* **bigtable/accelerator:** Authenticate UDS callers with a stdin tok…
([#20324](#20324))
([7847ab7](7847ab7))
* **bigtable/accelerator:** Dispatch MutateRow and single-row ReadRow…
([#20312](#20312))
([c89f336](c89f336))
* Update API sources and regenerate
([#20369](#20369))
([bfe3aaf](bfe3aaf))
* Update sources and regenerate
([#20399](#20399))
([a6b5cdb](a6b5cdb))


### Bug Fixes

* **bigtable/internal/transport:** Drop redundant defer p.Close() in AFE
picker tests
([#20311](#20311))
([c05b767](c05b767))
* **bigtable:** Register c2p resolver + RLS balancer for accelerator
daemon DirectPath
([#20362](#20362))
([95f372f](95f372f))
* **bigtable:** Session client — set AllowNonDefaultServiceAccount on
DirectPath dial
([#20343](#20343))
([b9c76ec](b9c76ec))
* **bigtable:** Session client — single ownership + single-flight pool
loading
([#20368](#20368))
([f4ad950](f4ad950))
* **bigtable:** Session pool — re-drive creation after a never-activated
close so the consecutive-failure breaker is reachable
([#20355](#20355))
([ee74949](ee74949))
* **bigtable:** Use composed user agent as CSM client_name
([#20383](#20383))
([cb02335](cb02335))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable Issues related to the Bigtable API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants