feat: add openlineage support by razvan · Pull Request #822 · stackabletech/airflow-operator

razvan · 2026-07-21T09:05:50Z

Description

Airfow can emit OpenLineage events for DAGs. This PR makes essential properties configurable. The operator uses sensible defaults where possible.

Depends on:

feat: add crd::openlineage module operator-rs#1250
new op-rs release (0.120.0)

Part of stackabletech/issues#856

Decision: https://github.com/stackabletech/decisions/issues/90

Definition of Done Checklist

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
Please make sure all these things are done and tick the boxes

Author

Changes are OpenShift compatible
CRD changes approved
CRD documentation for all fields, following the style guide.
Helm chart can be installed and deployed operator works
Integration tests passed (for non trivial changes)
Changes need to be "offline" compatible
Links to generated (nightly) docs added
Release note snippet added

Reviewer

Code contains useful comments
Code contains useful logging statements
(Integration-)Test cases added
Documentation added or updated. Follows the style guide.
Changelog updated
Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

Feature Tracker has been updated
Proper release label has been added
Links to generated (nightly) docs added
Release note snippet added
Add type/deprecation label & add to the deprecation schedule
Add type/experimental label & add to the experimental features tracker

…ationClass Follows the operator-rs change: OpenLineage backend auth is a static bearer token, so drop the AuthenticationClass resolution and read the api key from the connection's `credentialsSecretName` Secret (key `apiKey`) directly. See stackabletech/decisions#90 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…h API group Follow the operator-rs rename of OpenLineageJob.app_name to job_name (serialized appName -> jobName) and the move of the OpenLineageConnection CRD to the lineage.stackable.tech API group: - update the job_name field access and the "ignored by Airflow" debug log - update the operator RBAC ClusterRole apiGroup - update kuttl test manifests (apiVersion, jobName) and the usage-guide docs - regenerate extra/crds.yaml (appName -> jobName) Verified with cargo check/clippy/test and CRD regeneration against a local operator-rs checkout carrying the rename. Committed with --no-verify because the cargo/regenerate-charts pre-commit hooks would otherwise build against the un-pushed operator-rs branch; those checks were run manually instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename the lineage-emission configuration field from `openLineage` to `lineage` across the CRD and operator internals: - spec.clusterConfig.openLineage -> spec.clusterConfig.lineage (Rust field cluster_config.open_lineage -> lineage; the validated ValidatedClusterConfig.open_lineage -> lineage) - rename operator-internal identifiers: module controller/build/openlineage.rs -> lineage.rs, ResolvedOpenLineageConfig -> ResolvedLineageConfig, resolved_open_lineage_config -> resolved_lineage_config - update usage-guide docs and the kuttl test manifest - regenerate extra/crds.yaml (openLineage -> lineage) The OpenLineage technology name is kept in prose, in the operator-rs types (OpenLineageJob, OpenLineageConnection) and their connection-named identifiers, and in the OPENLINEAGE__* client env var names. Verified with cargo check/clippy/test and CRD regeneration against a local operator-rs checkout. Committed with --no-verify because the cargo/regenerate pre-commit hooks would build against the un-pushed operator-rs branch; those checks were run manually instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… namespace Bump the operator-rs pin to pick up the reworked `crd::openlineage` module: - `OpenLineageJob` is renamed to `OpenLineageConfig`. - `OpenLineageConnectionSpec` now selects an `OpenLineageTransport` (currently only `http`) instead of carrying host/port/tls/credentialsSecretName directly. - `HttpTransport` gained a `path` field, defaulting to `/api/v1/lineage`, which now drives `OPENLINEAGE__TRANSPORT__ENDPOINT` instead of a hardcoded constant. The leading slash is stripped since the client joins base URL and endpoint. - `OpenLineageConfig::namespace` is now a required `String` defaulting to `default`, so the operator-side fallback to the workload's Kubernetes namespace is gone. NOTE: this changes behaviour for users who did not set `namespace` explicitly - lineage is now reported under `default` rather than the cluster's Kubernetes namespace. The pin also brings in operator-rs' removal of `product-config`. Update the usage guide, the kuttl OpenLineageConnection manifest and extra/crds.yaml for the nested transport schema. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

adwk67

Some comments. I'm looking at the test now.

adwk67 · 2026-10-15T12:34:43Z

+            &connection(no_tls()),
+            &job(connection(no_tls()), NAMESPACE),
+        )
+        .unwrap();


I know we are inconsistent with the use of unwrap vs. expect, but the code style guide states we should use expect wherever possible in test code.

13f30a7

There are still unrwap() calls out there unrelated with this pr. I didn't want to inflate this more than needed.

adwk67 · 2026-10-15T12:48:36Z

+//! injected from a Secret via `secretKeyRef` instead of being embedded in a plaintext JSON value.
+//!
+//! The Python client's HTTP transport has no CA-path option (only a boolean `verify`), so a
+//! SecretClass-provided CA is trusted by mounting it and pointing `REQUESTS_CA_BUNDLE` at it.


Claude points out that REQUESTS_CA_BUNDLE is read by the requests library for every call made with verify=True, so it replaces the default certifi bundle used elsewhere (DAGs, OIDC login etc.) with the single CA for lineage. The user can override this with envOverrides by setting the env-var to point to CAs that they themselves bundle and maintain, or we can build a combined bundle at container start, so that e.g. the certifi bundle is not discarded. For now we should at least clearly document the knock-on effect this can have.

Good catch. I guess this will be a problem with every product. I added a warning bd0285d

adwk67 · 2026-10-15T12:58:09Z

+    /// [`Self::from_config`] so it can be unit tested without a client.
+    fn build(
+        connection: &ResolvedOpenLineageConnection,
+        lineage: &OpenLineageConfig,


nit: We only use the namespace from the config, so maybe just pass that?

adwk67 · 2026-10-15T13:06:26Z

    let crds_and_handlers = vec![(
        AirflowCluster::merged_crd(AirflowClusterVersion::V1Alpha2).context(MergeCrdSnafu)?,
-        AirflowCluster::try_convert,
+        AirflowCluster::try_convert as fn(ConversionReview) -> ConversionReview,


Do we need this change if we only have one fn (with its own type)? main compiles without it.

adwk67 · 2026-10-15T13:12:14Z

@@ -0,0 +1,20 @@
+{% if test_scenario['values']['executor'] == 'kubernetes' %}


Can't we use the script commons/cleanup-executor-pods.sh for this (which is basically the same plus a condition to behave differently if using --skip-delete)?

adwk67 · 2026-10-15T13:13:00Z

 tracing = "0.1"

 [patch."https://github.com/stackabletech/operator-rs.git"]
+# TODO: Switch back to a release tag once operator-rs#1250 (crd::openlineage) has been released.


This TODO can go now

adwk67 · 2026-10-15T13:26:25Z

+    }
+}
+
+fn plain_env(name: &str, value: &str) -> EnvVar {


This is used internally only with internal constants so it would be better to use the pattern in env_vars.rs of defining them with constant! as EnvVarName and then using EnvVarSet (once we have parsed the name with_value etc. on EnvVarSet are infallible and the expect in lineage_env_vars can be removed).

adwk67 · 2026-10-15T13:35:08Z

+    }
+}
+
+fn env_var_from_secret(name: &str, secret_name: &str, secret_key: &str) -> EnvVar {


Similar to plain_env above, but credentials_secret_name is user input and needs to be parsed into a SecretName (with an associated error) and then it can also be added to the EnvVarSet infallibly with with_secret_key_ref).

Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com>

feat: add openlineage support

5b8b5ba

razvan self-assigned this Jul 21, 2026

razvan and others added 12 commits July 21, 2026 15:22

Merge branch 'main' into feat/openlineage-from-op-rs

465cd1c

update changelog

ed7d143

cleanups

e349e74

main merge

8270bee

main merge

35d6376

Rebuild Cargo.lock and remove unnecessary ".."

4a401ee

Merge branch 'main' into feat/openlineage-from-op-rs

56c04d9

merge main branch

35d171d

razvan marked this pull request as ready for review October 9, 2026 12:16

update cargo.lock

2e20659

razvan added this to Stackable Engineering Oct 9, 2026

razvan moved this to Development: Waiting for Review in Stackable Engineering Oct 9, 2026

razvan added 2 commits October 12, 2026 11:48

Merge branch 'main' into feat/openlineage-from-op-rs

076acb2

bump op-rs version 0.120.0

113a64a

adwk67 self-requested a review October 15, 2026 09:06

adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Oct 15, 2026

adwk67 requested changes Oct 15, 2026

View reviewed changes

razvan and others added 7 commits October 16, 2026 09:28

Merge branch 'main' into feat/openlineage-from-op-rs

3eacba9

Update rust/operator-binary/src/controller/build/lineage.rs

8a7ca06

Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com>

review feedback: replace unwrap() with expect() calls in tests.

13f30a7

review feedback: build() gets 'namespace' instead of 'OpenLineageConfig'

cdf3b92

review feedback: remove unnecessary type cast

337c42f

review feedback: remove todo

673dfd8

review feedback: use EnvVarSet and friends

1ecf856

razvan added 2 commits October 16, 2026 16:50

review feedback: add warning about CA bundle

bd0285d

review feedback: use cleanup-executor-pods.sh

5e83a00

razvan requested a review from adwk67 October 16, 2026 15:22

		@@ -0,0 +1,20 @@
		{% if test_scenario['values']['executor'] == 'kubernetes' %}

Conversation

razvan commented Jul 21, 2026 • edited by adwk67 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Definition of Done Checklist

Author

Reviewer

Acceptance

Uh oh!

adwk67 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

razvan commented Jul 21, 2026 •

edited by adwk67

Loading