Fix Airbyte deferrable execution timeout deadline evaluation and cleanup#67816
Open
SameerMesiah97 wants to merge 1 commit into
Open
Fix Airbyte deferrable execution timeout deadline evaluation and cleanup#67816SameerMesiah97 wants to merge 1 commit into
SameerMesiah97 wants to merge 1 commit into
Conversation
Use wall-clock deadlines instead of monotonic time, enforce timeout checks before job state evaluation, and add a defer timeout buffer to allow execute_complete() to cancel timed-out jobs before task termination.
084e02e to
8849425
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change is a follow-up to PR #67382 and addresses the same execution timeout handling gap that was subsequently fixed for the dbt Cloud provider in PR #67626, but for the Airbyte provider.
It also adopts the deferred timeout buffering approach introduced in PR #67229 for the Kubernetes provider to ensure timeout cleanup logic can execute before Airflow's framework-level deferred timeout handling terminates the task.
The implementation now:
time.time()) for serialized deadline calculations instead of monotonic time.Rationale
In deferrable mode, Airbyte job cancellation occurs in
execute_complete()when the trigger emits a timeout event. However, Airflow's framework-level deferred timeout handling bypassesexecute_complete(), which can leave Airbyte jobs running after task timeout.Additionally, serialized deadlines must use wall-clock time rather than monotonic time, as they are persisted across defer/resume boundaries. Timeout evaluation must also take precedence over job state evaluation to ensure consistent handling once a deadline has elapsed.
This change keeps timeout handling within the trigger/operator flow and provides sufficient time for Airbyte job cancellation before framework-level task termination occurs.
Tests
Added/modified unit tests verifying that:
execution_timeoutis configured.Backwards Compatibility
This change does not modify public APIs or method signatures.
Was generative AI tooling used to co-author this PR?
Generated-by: [GPT 5.5] following the guidelines