[Experimental] Allow delivery retry backoff intervals to be capped #9278

Description

Resource owners can configure the number of delivery retries, the backoff policy, and the initial delay, but they cannot set a maximum interval between normal delivery attempts.

This makes a large retry count impractical with exponential backoff. For example, with backoffDelay: PT1S, the calculated interval is about 12 days after 20 retries and about 34 years after 30 retries. If a downstream service recovers during one of these waits, the event is not retried within a useful time. Resource owners therefore have to choose between stopping retries too soon during a long outage and allowing retry intervals to grow beyond an operationally useful range. Very large retry counts can also overflow the duration used by the delivery implementation.

Retry implementations commonly let users cap an increasing backoff interval. For example, Kubernetes exposes Backoff.Cap, and cenkalti/backoff exposes ExponentialBackOff.MaxInterval. Knative Eventing should offer the same control for delivery retries.

Design Overview

Add an optional backoffMax field to DeliverySpec. While the feature is experimental, cluster operators enable the field through the delivery-backoff-max flag in the knative-eventing/config-features ConfigMap. The field accepts a positive ISO 8601 duration and caps intervals calculated from backoffDelay and backoffPolicy for both linear and exponential backoff.

backoffMax has the same meaning for Subscription and Trigger delivery. A Subscription's delivery overrides its Channel's delivery defaults, while a Trigger's delivery overrides its Broker's delivery defaults.

This Subscription starts with a one-second exponential backoff and prevents later delivery attempts from waiting longer than ten minutes:

apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
  name: orders-to-processor
  namespace: default
spec:
  channel:
    apiVersion: messaging.knative.dev/v1
    kind: InMemoryChannel
    name: orders
  subscriber:
    ref:
      apiVersion: v1
      kind: Service
      name: order-processor
  delivery:
    retry: 100
    backoffPolicy: exponential
    backoffDelay: PT1S
    backoffMax: PT10M

The same delivery configuration applies when the service receives events through a Trigger:

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: orders-to-processor
  namespace: default
spec:
  broker: default
  subscriber:
    ref:
      apiVersion: v1
      kind: Service
      name: order-processor
  delivery:
    retry: 100
    backoffPolicy: exponential
    backoffDelay: PT1S
    backoffMax: PT10M

backoffMax applies only to intervals calculated from backoffDelay and backoffPolicy. A delay requested by a subscriber through a Retry-After response header remains controlled by the existing retryAfterMax field. When backoffMax is omitted, existing retry behavior remains unchanged for representable durations. Calculations that exceed time.Duration should saturate instead of overflowing to an invalid duration. Channel delivery defaults must preserve backoffMax when applied to a Subscription, and Broker delivery defaults must preserve it when used for a Trigger.

Exit Criteria

• With the feature flag enabled, resource owners can set a valid, positive DeliverySpec.backoffMax value for Subscription and Trigger delivery.
• Linear and exponential intervals calculated from backoffDelay and backoffPolicy never exceed the configured maximum.
• Invalid, zero, and negative values are rejected, and setting the field while the feature is disabled is rejected.
• Channel delivery defaults preserve backoffMax when applied to a Subscription, and Broker delivery defaults preserve it when used for a Trigger.
• Eventing implementations that accept backoffMax apply the same cap instead of silently ignoring the field.
• Retry-After remains independently controlled by retryAfterMax.
• Omitting backoffMax preserves existing retry behavior, and large durations or retry counts do not overflow.

Experimental Feature Flag Name

delivery-backoff-max

Experimental Feature Stages

The following is the proposed plan for moving through the stages of the experimental features process:

• Alpha: (Target v1.24)
  • Add DeliverySpec.backoffMax, the feature flag and validation, capped retry calculations, and support for Channel-to-Subscription and Broker-to-Trigger delivery. Upstream implementation PR: #9279.
  • Add unit tests for linear and exponential caps, invalid values, disabled feature behavior, large durations and retry counts, independence from Retry-After, and Broker-to-Trigger delivery propagation. #9279
  • Add E2E tests for the documented Subscription workflow and the Trigger workflow using the default Broker implementation. #9279
  • Update the Eventing API reference and delivery documentation. #9279
  • Add user documentation to knative/docs. Docs PR: knative/docs#6686.
• Beta: (Minimum of one release after Alpha)
  • Incorporate feedback from Alpha users and implementations.
  • Before enabling the feature by default, add support to the maintained implementations that expose DeliverySpec, including Kafka Broker, KafkaChannel, NATS Broker, and NatsJetStreamChannel. NATS implementation PR: knative-extensions/eventing-natss#802. Track Kafka support in a follow-up issue or PR.
  • Enable the feature by default.
  • Maintain more than 80% unit test coverage for the feature.
  • Expand E2E coverage across Subscription and Trigger delivery and the main Eventing implementations.
  • Stabilize and improve user documentation.
• Stable: (Minimum of two releases after Beta)
  • Remove the delivery-backoff-max feature flag and experimental validation.
  • Document DeliverySpec.backoffMax in knative/specs, including whether support is normative or non-normative.
  • Add conformance tests if required by the stable specification.

Affected WG

• Eventing WG
• Eventing Kafka WG

Additional Context

• Resources
  • Experimental Features Process
  • DeliverySpec RetryAfter feature track
  • DeliverySpec RetryAfter implementation
• Prior art
  • Kubernetes Backoff.Cap
  • cenkalti/backoff ExponentialBackOff.MaxInterval