Allow OpenLineage job name customization #25704
Praveen211221 days agoInstead of creating Optional here, can we directly use DEFAULT_JOB_NAME_FORMAT as a default value.
dolfinus21 days ago (edited 21 days ago)I thought about case like:
openLineageListenerConfig.setJobNameFormat(null)which resets value to its default
| 119 | @ConfigDescription("Set format for job name facet") | ||
| 120 | public OpenLineageListenerConfig setJobNameFormat(String jobNameFormat) | ||
| 121 | { | ||
| 122 | this.jobNameFormat = Optional.ofNullable(jobNameFormat); | ||
| 123 | if (jobNameFormat == null) { | ||
| 124 | return this; | ||
| 125 | } |
Praveen211221 days agoAirlift would invoke pass only non nullable String.
dolfinus21 days ago (edited 21 days ago)But class method could be called with null passed directly. Other methods above use Optional as well
| 129 | resolvedJobName = resolvedJobName.replace(substitution, ""); | ||
| 130 | } | ||
| 131 | |||
| 132 | if (!resolvedJobName.contains("$")) { | ||
| 133 | return this; | ||
| 134 | } |
Praveen211221 days agoCan we use a regexp based validator here ?
Praveen211221 days agoWe also use a similar validation here - FormatBasedRemoteQueryModifierConfig
dolfinus21 days ago (edited 21 days ago)I don't see there this method is called:
Is this some kind of airlift magic?
Regarding using SessionInterpolatedValues as an example for interpolation:
https://github.com/trinodb/trino/blob/master/lib/trino-plugin-toolkit/src/main/java/io/trino/plugin/base/logging/SessionInterpolatedValues.java#L31
I see an issue here - some fields are fetched from QueryContext, and others from QueryMetadata, but InterpolatedValue<T> generic accepts only one class, and Java doesn't have Union type...
Praveen211216 days agoThis is kind of Airlift magic i.e method with Assert annotation and which starts with is would be invoked as a part of validation
dolfinus16 days ago (edited 16 days ago)Ok, used the same approach as for SessionInterpolatedValues. Added a custom class OpenLineageJobContext used only to pass QueryContext and QueryMetadata around.
| 68 | Optional.empty(), // clientInfo | ||
| 69 | new HashSet<>(), // clientTags | ||
| 70 | new HashSet<>(), // clientCapabilities | ||
| 62 | Set.of("enabledRoles"), | ||
| 63 | Set.of("groups"), | ||
| 64 | Optional.of("traceToken"), | ||
| 65 | Optional.of("remoteClientAddress"), | ||
| 66 | Optional.of("userAgent"), | ||
| 67 | Optional.of("clientInfo"), | ||
| 68 | Set.of("clientTags"), | ||
| 69 | Set.of("clientCapabilities"), |
Praveen211221 days agoDo we need to change all the default values ?
dolfinus16 days agoReverted, but I don't see any issues with this change
| 82 | void testUnsupportedJobNameFormatSubstitutions() | ||
| 83 | throws Exception | ||
| 84 | { | ||
| 85 | assertThatThrownBy(() -> new OpenLineageListenerConfig().setJobNameFormat("${unknown}")) |
chenjian266416 days agoPlease add more invalid cases, for example ${xxx, ${${queryId}}, $-${source}.
Also does the fixed format i,e abc123 support ?
dolfinus16 days agoAlso does the fixed format i,e abc123 support ?
Yes
dolfinus16 days agoOutdated as FormatInterpolator is now responsible for all validation
chenjian266416 days agoWill the jobNameFormat be null?
dolfinus16 days agochenjian266416 days agoThen let's add requireNonNull check on this
dolfinus15 days agoI don't understand why this is needed
Remove leading docs from OpenLineageListenerConfig description
Make FormatInterpolator.InterpolatedValue interface public
Allow OpenLineage job name customization
Login to write a write a comment.
Description
Using
queryIdas OpenLineage's jobnamefield lead to registering all unique queries as ETL jobs. This is hardly useful in terms of lineage.Now integration allows to customize
namefield. For example, it may useX-Trino-SourceorX-Trino-Uservalues instead, or a combination or both.Additional context and related issues
As user can override only system or catalog properties via
SET SESSIONorX-Trino-Session, but OpenLineage integration is event listener, user cannot override job name using this mechanism. To avoid using static job names for all OpenLIneage events, config optionopenlineage-event-listener.job.nameFormatsupports substitution variables:$QUERY_ID- default for backward compatibility.$USER$SOURCEIn theory, user can connect to Trino and pass job name as
X-Trino-Source, and settingopenlineage-event-listener.job.nameFormat=$SOURCEwill pass this value directly to OpenLineage jobname.But all clients have some default value for this field (e.g.
trino-cli,trino-python-client,trino-jdbc). If user haven't set custom source name, then all OpenLineage events produced by the same client will got the same job name (e.g.trino-jdbc). To avoid this, it is possible to use multiple substitutions, like$SOURCE:$USERto produce jobs with names liketrino-jdbc:dolfinusortrino-jdbc:myawesomeuser.Closes: #25535
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(X) Release notes are required, with the following suggested text: