Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add callable where clause in union_relations #892

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

tnightengale
Copy link

@tnightengale tnightengale commented May 20, 2024

resolves #

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

I noticed that the where parameter in the union_relations macro does not allow one to create unique where clauses for each relation being unioned. In the case of an incremental model, it would be very helpful to be able to configure an incremental where clause for each model separately.

In the simplest form, a project could implement a macro like the following:

{% macro where_clause(relation) %}

{% if execute %}
{% set where_clause %}
{% if is_incremental() %}
occurred_at >= coalesce((select max(occurred_at) from {{ relation }}), '1900-01-01')
{% else %}
    true
{% endif %}
{% endset %}
{% endif %}

{{ where_clause }}

{% endmacro %}

And the pass it to the `union_relations macro like so:

{{
    config(
        materialized="incremental"
    )
}}


{{ dbt_utils.union_relations(
    relations=[
        ref("model_a"),
        ref("model_b")
    ],
    where=where_clause
) }}

Checklist

  • This code is associated with an Issue which has been triaged and accepted for development.
  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
  • I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • using the limit_zero() macro in place of the literal string: limit 0
    • using dbt.type_* macros instead of explicit datatypes (e.g. dbt.type_timestamp() instead of TIMESTAMP
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md

@tnightengale
Copy link
Author

I would welcome some discussion on this feature before I endeavor to add tests and the relevant changes to the README and CHANGELOG files.

Cheers!

@dbeatty10 dbeatty10 added the enhancement New feature or request label Jun 4, 2024
@tnightengale
Copy link
Author

@dbeatty10 Would you mind reviewing this draft? And if you agree, I can finish off the PR with tests and update the docs?

@tnightengale tnightengale marked this pull request as draft September 21, 2024 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants