Skip to content

Fix Airbyte deferrable execution timeout deadline evaluation and cleanup#67816

Open
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:AirbyteTriggerSyncOperator-Misc-Fixes
Open

Fix Airbyte deferrable execution timeout deadline evaluation and cleanup#67816
SameerMesiah97 wants to merge 1 commit into
apache:mainfrom
SameerMesiah97:AirbyteTriggerSyncOperator-Misc-Fixes

Conversation

@SameerMesiah97
Copy link
Copy Markdown
Contributor

Description

This change is a follow-up to PR #67382 and addresses the same execution timeout handling gap that was subsequently fixed for the dbt Cloud provider in PR #67626, but for the Airbyte provider.

It also adopts the deferred timeout buffering approach introduced in PR #67229 for the Kubernetes provider to ensure timeout cleanup logic can execute before Airflow's framework-level deferred timeout handling terminates the task.

The implementation now:

  • Uses wall-clock timestamps (time.time()) for serialized deadline calculations instead of monotonic time.
  • Enforces execution timeout checks before Airbyte job state evaluation within the trigger.
  • Adds a defer timeout buffer to allow timeout events to be processed and Airbyte job cancellation to occur before framework-level deferred timeout handling terminates the task.

Rationale

In deferrable mode, Airbyte job cancellation occurs in execute_complete() when the trigger emits a timeout event. However, Airflow's framework-level deferred timeout handling bypasses execute_complete(), which can leave Airbyte jobs running after task timeout.

Additionally, serialized deadlines must use wall-clock time rather than monotonic time, as they are persisted across defer/resume boundaries. Timeout evaluation must also take precedence over job state evaluation to ensure consistent handling once a deadline has elapsed.

This change keeps timeout handling within the trigger/operator flow and provides sufficient time for Airbyte job cancellation before framework-level task termination occurs.

Tests

Added/modified unit tests verifying that:

  • Wall-clock time is used for serialized deadline handling.
  • Execution timeout checks take precedence over Airbyte job state evaluation.
  • Deferred timeout buffering is applied when execution_timeout is configured.

Backwards Compatibility

This change does not modify public APIs or method signatures.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [GPT 5.5] following the guidelines

Use wall-clock deadlines instead of monotonic time, enforce timeout checks
before job state evaluation, and add a defer timeout buffer to allow
execute_complete() to cancel timed-out jobs before task termination.
@SameerMesiah97 SameerMesiah97 force-pushed the AirbyteTriggerSyncOperator-Misc-Fixes branch from 084e02e to 8849425 Compare May 31, 2026 21:09
@SameerMesiah97 SameerMesiah97 marked this pull request as ready for review June 1, 2026 08:45
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants