Repository navigation
fix(bigtable): session pool — re-drive creation after a never-activated close so the consecutive-failure breaker is reachable - #20355
Merged
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to prevent parked CheckoutSession callers from hanging indefinitely when a session-creation attempt fails and the pool has no remaining path to a ready session. It adds a new ErrSessionCreateFailed error, implements failWaitersOnCreateFailure to wake up stranded waiters, and includes comprehensive unit tests. The reviewer identified a potential data race in failWaitersOnCreateFailure where p.sl.ReadyCount() is called outside of the p.mu lock, and suggested a thread-safe implementation that reads the ready count while holding the lock.
mutianf
force-pushed
the
08-11-fix_bigtable_session_pool
branch
from
August 11, 2026 03:02
d3caad7 to
f89bb4c
Compare
…ed close so the consecutive-failure breaker is reachable When the vRPC OpenSession API is not served (server returns Unimplemented), a session dies while still in Starting and never activates. onClose's starting branch bumped the consecutive-failure counter but nothing re-drove creation: onClosing early-returns for starting handles, and no other of the sizer's event-driven wake sites fires on a never-activated close. A cold-start MinSessionCount cohort therefore froze the counter below ConsecutiveSessionFailureThreshold, the circuit breaker never tripped, and a CheckoutSession caller parked on a deadline-less context hung forever (the 43-min stuck goroutine in the prod round-trip test). Fix: in onClose's starting branch, after noteAbnormalCloseIfAny, re-drive creation via spawnTickOnce (gated on sl.ReadyCount() < MaxSessions, mirroring onClosing's replace-on-death kick). Each re-driven cohort fails again, the counter climbs to the threshold, the existing breaker trips, and parked waiters are drained with ErrConsecutiveFailures carrying the underlying Unimplemented code — only after X consecutive failures, not on the first. Paced by the throttler's failure penalty so this is not a tight retry loop. This replaces the earlier failWaitersOnCreateFailure approach, which wrongly failed waiters on the first create failure instead of after a sustained run; that mechanism (ErrSessionCreateFailed, sessionCreateError, the Tick call, and the session_pool_waiters_failed_on_create tag) is reverted. Also narrows the accelerator round-trip test's skipIfSessionUnavailable to skip on Unimplemented only — the pool now surfaces that code deterministically instead of hanging, so transient Unavailable/DeadlineExceeded are left to retryTransient. Change-Id: I4621616f7e81b4ccaca282b5758149612239041b
mutianf
force-pushed
the
08-11-fix_bigtable_session_pool
branch
from
August 11, 2026 14:27
f89bb4c to
3ae233d
Compare
sushanb
approved these changes
Aug 12, 2026
sushanb
pushed a commit
that referenced
this pull request
Aug 28, 2026
🤖 I have created a release *beep* *boop* --- ## [1.53.0](bigtable/v1.52.0...bigtable/v1.53.0) (2026-08-27) ### Features * **bigtable/accelerator:** Add --user-agent flag to daemon ([#20341](#20341)) ([6b33c06](6b33c06)) * **bigtable/accelerator:** Add daemon binary with identity verification ([#20330](#20330)) ([2f9cf89](2f9cf89)) * **bigtable/accelerator:** Add proxy channel scaffold owning a sessi… ([#20310](#20310)) ([d65ca7d](d65ca7d)) * **bigtable/accelerator:** Add stdin-EOF and parent-PID lifecycle wa… ([#20328](#20328)) ([dc1c6ae](dc1c6ae)) * **bigtable/accelerator:** Add UDS server with authenticated proxy i… ([#20320](#20320)) ([a837020](a837020)) * **bigtable/accelerator:** Add V2-to-session request adapters ([#20271](#20271)) ([4cb6f16](4cb6f16)) * **bigtable/accelerator:** Authenticate UDS callers with a stdin tok… ([#20324](#20324)) ([7847ab7](7847ab7)) * **bigtable/accelerator:** Dispatch MutateRow and single-row ReadRow… ([#20312](#20312)) ([c89f336](c89f336)) * Update API sources and regenerate ([#20369](#20369)) ([bfe3aaf](bfe3aaf)) * Update sources and regenerate ([#20399](#20399)) ([a6b5cdb](a6b5cdb)) ### Bug Fixes * **bigtable/internal/transport:** Drop redundant defer p.Close() in AFE picker tests ([#20311](#20311)) ([c05b767](c05b767)) * **bigtable:** Register c2p resolver + RLS balancer for accelerator daemon DirectPath ([#20362](#20362)) ([95f372f](95f372f)) * **bigtable:** Session client — set AllowNonDefaultServiceAccount on DirectPath dial ([#20343](#20343)) ([b9c76ec](b9c76ec)) * **bigtable:** Session client — single ownership + single-flight pool loading ([#20368](#20368)) ([f4ad950](f4ad950)) * **bigtable:** Session pool — re-drive creation after a never-activated close so the consecutive-failure breaker is reachable ([#20355](#20355)) ([ee74949](ee74949)) * **bigtable:** Use composed user agent as CSM client_name ([#20383](#20383)) ([cb02335](cb02335)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the vRPC
OpenSessionAPI is not served (server returnsUnimplemented),a session dies while still in
StateStartingand never activates.onClose'sstarting branch bumped the consecutive-failure counter but nothing re-drove
creation:
onClosingearly-returns for starting handles, and no other of thesizer's event-driven wake sites fires on a never-activated close.
A cold-start
MinSessionCount(5) cohort therefore froze the counter at 5,never reaching
ConsecutiveSessionFailureThreshold(10), so the circuit breakernever tripped. With nothing re-driving creation and nothing tripping the breaker,
a
CheckoutSessioncaller parked on a deadline-less context hung forever (the43-min stuck goroutine in the prod round-trip test).
Fix
In
onClose's starting branch, afternoteAbnormalCloseIfAny, re-drive creationvia
spawnTickOnce(gated onsl.ReadyCount() < MaxSessions, mirroringonClosing's replace-on-death kick). Each re-driven cohort fails again, thecounter climbs to the threshold, the existing breaker trips, and parked
waiters are drained with
ErrConsecutiveFailurescarrying the underlyingUnimplementedcode — only after X consecutive failures, not on the first.Paced by the throttler's failure penalty, so this is not a tight retry loop.
This replaces the earlier
failWaitersOnCreateFailureapproach from this PR,which wrongly failed waiters on the first create failure instead of after a
sustained run. That mechanism (
ErrSessionCreateFailed,sessionCreateError,the
Tickcall, and thesession_pool_waiters_failed_on_createtag) is reverted.Also narrows the accelerator round-trip test's
skipIfSessionUnavailableto skipon
Unimplementedonly — the pool now surfaces that code deterministicallyinstead of hanging, so transient
Unavailable/DeadlineExceededare left toretryTransient.Testing
go test ./internal/transport/ -race— full suite green.TestConsecutiveFailures_StartingDeathsReDriveUntilBreakerTrips(end-to-end: starting-deaths re-drive → breaker trips → caller wakes with
ErrConsecutiveFailures+status.Code == Unimplemented) andTestConsecutiveFailures_SingleStartingCloseDoesNotWakeWaiters(a singlestarting-close increments but does not wake waiters).
cloud-bigtable-spacewalk/mattiefu-test(whichdoes not serve
OpenSession): the round-trip test now fails fast in 4.72swith the breaker trip carrying
Unimplemented: Method is not available...andcleanly skips — previously it hung for 43 minutes.