Skip to content

ci: pr-test-summary job shows red when the GitHub API returns persistent 5xx errors #2256

Open

Description

@ramakrishnap-nv

Problem

The PR test summary (non-blocking) job in pr.yaml sometimes fails with a red check on PRs, for example:

Traceback (most recent call last):
  File "/home/runner/work/cuopt/cuopt/ci/utils/pr_test_summary.py", line 325, in <module>
    main()
...
urllib.error.HTTPError: HTTP Error 502: Bad Gateway

This was noticed in the discussion on #2241 (run https://github.com/NVIDIA/cuopt/actions/runs/41135602235). It was first mistaken for leftover RAPIDS CI telemetry, but it comes from this repo's own ci/utils/pr_test_summary.py.

Why it fails

pr_test_summary.py makes several GitHub API calls: it lists the run's jobs, downloads logs of failed jobs, and creates or updates the PR comment.

  • Transient errors (429, 5xx, network errors) are retried up to 4 attempts with 2**attempt second backoff (ci: retry transient GitHub API errors in pr-test-summary #2204). That is roughly 14 seconds in total. If the API keeps returning errors for longer, the last exception is re-raised.
  • The script then exits non-zero, and the run: python3 ci/utils/pr_test_summary.py step has no continue-on-error. ci: make pr-test-summary and NVSkills CI checks non-blocking #1696 removed it and instead excluded the job from the pr-builder merge gate (and checks has ignored_pr_jobs: "pr-test-summary").
  • The job is therefore not a merge gate, but it still shows as a failed check on the PR. That is confusing for contributors and reviewers, who may take it for a real test failure, and the summary comment is not posted.

Proposal

Make the summary best-effort so it cannot show red because of an API error:

  • Catch persistent API errors at the end of main(), print a warning to the job log, and exit 0.
  • Or set continue-on-error: true on the step.

Real script bugs (for example parsing errors) should still fail the job, so only the network and HTTP error classes would be swallowed.

Related: #2204, #1696, #2241.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions