Repository navigation
Description
What happened (please include outputs or screenshots):
InClusterConfigLoader schedules the projected service-account token re-read using timezone-naive datetime.datetime.now() (local wall-clock time). In kubernetes/config/incluster_config.py (v37.0.1):
# line ~47
self._token_refresh_period = datetime.timedelta(minutes=1)
# line ~96
if self.token_expires_at <= datetime.datetime.now():
# lines ~108-109
self.token_expires_at = datetime.datetime.now() + self._token_refresh_periodWhen the local wall clock jumps backwards — e.g. a DST fall-back while the container runs with a non-UTC TZ (such as TZ=Europe/Stockholm), an NTP step, or a manual clock change — the stored token_expires_at (computed before the jump) stays in the perceived future. The refresh condition token_expires_at <= now() never becomes true for up to the duration of the backward jump, so _read_token_file() is not called. Meanwhile the kubelet rotates the projected token on disk, the client keeps sending the stale cached bearer token, and the API server responds with 401 Unauthorized until wall-clock time catches up.
Observed at a CEST→CET fall-back: a ~46-minute burst of 401 Unauthorized responses, self-recovering once local time passed the stale token_expires_at.
What you expected to happen:
Token refresh timing should be immune to wall-clock discontinuities. Using a monotonic clock (time.monotonic()) for the refresh deadline — or a timezone-aware UTC datetime — would make the refresh fire on the intended cadence regardless of DST/NTP/manual wall-clock changes.
How to reproduce it (as minimally and precisely as possible):
- Run a pod using in-cluster config with a non-UTC
TZthat observes DST (e.g.TZ=Europe/Stockholm). - Let the client cache a token shortly before a DST fall-back (clock about to move back 1h).
- At the transition, the wall clock jumps back 1h;
token_expires_atnow sits ~1h in the "future". - kubelet rotates the SA token on disk; the client does not re-read it.
- Requests to the API server return
401 Unauthorizeduntil the wall clock reaches the pre-jumptoken_expires_at.
Equivalent without waiting for a real DST event: step the host/container clock backward after a token read and observe that _read_token_file() is not invoked by refresh_api_key_hook until the wall clock catches up.
Anything else we need to know?:
Suggested fix: compute the refresh deadline from time.monotonic() instead of datetime.datetime.now() (monotonic never goes backwards), or use timezone-aware UTC. The change is confined to _read_token_file() (store deadline) and the _refresh_api_key hook (compare against the same clock). This is a latent defect affecting any client using in-cluster config with a non-UTC TZ during a backward wall-clock change; it is not specific to any one service.
Environment:
- Kubernetes version (
kubectl version): N/A — client-side timing defect, reproducible against any API server - OS (e.g., MacOS 10.13.6): Linux container with
TZ=Europe/Stockholm(any non-UTC, DST-observing zone) - Python version (
python --version): Python 3.13 - Python client version (
pip list | grep kubernetes): kubernetes==37.0.1 (latest release; code path unchanged in currentmaster)
Activity
OKAN-SOFTWARE commentedon Nov 3, 2026
Hi,
I've reproduced the issue with in-cluster token refresh when the system clock moves backward.
I've prepared a focused fix that uses
time.monotonic()to schedule token refreshes, along with a regression test that reproduces the original failure.Validation:
Before opening a pull request, I'd appreciate confirmation that this approach aligns with the project's expectations.
I'm also happy to complete the CNCF Contributor License Agreement through EasyCLA when prompted.
Thanks for your time!
tusharmunavalli commentedon Nov 5, 2026
Hi, I’d like to work on this. I’m planning to switch the token expiry logic in InClusterConfigLoader to timezone-aware UTC datetimes and add a test for a backward clock jump.