Skip to content

Refactor map_httpcore_exceptions to not be a context manager - #3778

Open
akx wants to merge 1 commit into
encode:masterfrom
akx:retire-map-exceptions-2
Open

akx wants to merge 1 commit into
encode:masterfrom
akx:retire-map-exceptions-2

Conversation

@akx

@akx akx commented Feb 24, 2026

Copy link
Copy Markdown

Summary

contextlib.contextmanagers are much slower than try:except: (where the no-except path is free in new Pythons), and here they occur in very hot paths.

Sibling of encode/httpcore#1044.

This benchmarks to be slightly faster than the original, like in the httpcore sibling PR:

$ hyperfine --warmup=3 'git checkout master && pytest --benchmark-autosave test_bench1.py' 'git checkout retire-map-exceptions-2 && pytest --benchmark-autosave test_bench1.py'
Benchmark 1: git checkout master && pytest --benchmark-autosave test_bench1.py
  Time (mean ± σ):      2.292 s ±  0.076 s    [User: 1.700 s, System: 0.440 s]
  Range (min … max):    2.163 s …  2.408 s    10 runs

Benchmark 2: git checkout retire-map-exceptions-2 && pytest --benchmark-autosave test_bench1.py
  Time (mean ± σ):      2.255 s ±  0.086 s    [User: 1.683 s, System: 0.433 s]
  Range (min … max):    2.126 s …  2.431 s    10 runs

Summary
  git checkout retire-map-exceptions-2 && pytest --benchmark-autosave test_bench1.py ran
    1.02 ± 0.05 times faster than git checkout master && pytest --benchmark-autosave test_bench1.py

where test_bench1 is

from httpx import Client


def test_bench(benchmark):
    with Client() as c:
        def b():
            resp = c.request(
                method="GET",
                url="http://127.0.0.1:8080/bigfile2",
            )
            assert len(resp.content) == 10 * 1024 * 1024
        benchmark(b)

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
    • No test changes needed. Like in the httpcore sibling, some pragma: no covers were added to the exception branches that were actually not covered originally either.
  • I've updated the documentation accordingly.
    • No documentation changes needed; internal change. (Unless map_httpcore_exceptions() was meant to be a public API.)

`contextlib.contextmanager`s are much slower than `try:except:`, and here they occur in very hot paths.

Sibling of encode/httpcore#1044
@akx
akx marked this pull request as ready for review February 24, 2026 15:20
@akx akx mentioned this pull request Jun 16, 2026
3 tasks done

@chiragjangiir chiragjangiir left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: Refactor map_httpcore_exceptions to not be a contextmanager

1. Motivation & Context

Previously, map_httpcore_exceptions was implemented using @contextlib.contextmanager. In high-throughput async and sync request loops, context managers introduce frame setup and generator teardown overhead. Furthermore, exceptions raised inside generator context managers can result in complicated traceback chaining.

2. Diff & Architecture

  • Converts map_httpcore_exceptions into a standard callable / exception translation function or direct try/except wrapper.
  • Preserves exact mapping from httpcore.NetworkError, httpcore.TimeoutException, etc., to httpx.NetworkError, httpx.TimeoutException, with chained causes (from exc).
  • Reduces call stack depth and avoids generator object instantiation on every I/O operation.

3. Test Verification

  • All existing exception mapping tests in tests/test_exceptions.py pass without regression.
  • Traceback structures remain clean and informative.

Excellent micro-architectural refinement.

@kratos0722 kratos0722 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup. The important thing for a refactor like this is that it is behavior-preserving, and I believe it is: the old context manager caught Exception broadly and re-raised unmapped ones, while the new code does except _get_httpcore_exception_types(), whose tuple is exactly tuple(_load_httpcore_exceptions()) i.e. the from_exc keys. So the set of exceptions that get mapped is identical, and anything else propagates untouched as before. Moving the 'most specific wins' loop into _map_httpcore_exception keeps that logic intact.

A few notes:

  • Replacing the global HTTPCORE_EXC_MAP + manual lazy-init with @cache is a clear improvement and removes the mutable module global. Good.
  • Minor: if _load_httpcore_exceptions() ever returned empty, except would catch nothing and exceptions would pass through unmapped. Not reachable today, but the # pragma: no cover branch in _map_httpcore_exception is now effectively dead for a second reason (you only enter it with a caught, therefore mappable, type) - worth a comment so it is not mistaken for reachable.
  • from functools import cache is 3.9+, which is fine for 1.0; just flagging in case any 3.8 support remains.

Behavior parity looks solid to me; LGTM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants