Skip to content

Conversation

@FeatherAnalytics
Copy link

resolves #1037

Problem

Users working with datasets that have dynamically named columns (like year-based columns: 2024_revenue, 2023_revenue, or similarly named fields: current_item, last_item) need a way to programmatically select columns matching a pattern. While this can be done with a for loop in Jinja, there's no idiomatic dbt-utils macro for this common use case.

Solution

This PR adds a new get_columns_by_pattern macro that returns columns from a relation matching a SQL LIKE pattern.

Features:

  • Supports SQL LIKE pattern syntax (% for wildcard, _ for single character)
  • Optional exclude parameter to filter out unwanted columns
  • Case-insensitive matching (consistent with SQL standards)
  • Returns a simple list of column names for easy composition with other macros

Implementation follows existing dbt-utils patterns:

  • Uses adapter.dispatch() for cross-database compatibility
  • Includes input validation with _is_relation and _is_ephemeral
  • Guards against parsing mode with if not execute
  • Mirrors the design of get_filtered_columns_in_relation

Example usage:

-- Get all columns ending with '_item'
{% set item_columns = dbt_utils.get_columns_by_pattern(
    from=ref('sales'),
    pattern='%_item'
) %}

-- Get year columns excluding archived ones  
{% set year_columns = dbt_utils.get_columns_by_pattern(
    from=ref('revenue'),
    pattern='20%',
    exclude='%_archived'
) %}

Checklist

  • This code is associated with an issue which has been triaged and accepted for development
  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests (integration test with seed data and equality assertion)
  • I have updated the README.md (added documentation under "Introspective macros" section)

This macro returns columns from a relation that match a SQL LIKE pattern.
It supports pattern matching with % and _ wildcards and an optional exclude
parameter to filter out unwanted columns. Useful for working with dynamic
column sets like year-based columns (2024_*, 2023_*) or similarly named
fields (*_item, *_revenue, etc.).

Implementation follows existing dbt-utils patterns:
- Uses adapter.dispatch() for cross-database compatibility
- Includes input validation with _is_relation and _is_ephemeral
- Case-insensitive matching consistent with SQL standards
- Returns simple list of column names for easy composition

Includes full integration test suite with seed data and equality assertions.

Resolves dbt-labs#1037

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@FeatherAnalytics FeatherAnalytics requested a review from a team as a code owner December 31, 2025 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a new macro - get_columns_by_pattern

2 participants