BUG - Freshness check fails for organization_tag
Closed this issue ยท 5 comments
Are you a current Fivetran customer?
Jon Wu, Architect, Paper Culture
Describe the bug
zendesk.organization_tag
fails freshness checks. It appears that it shouldn't be checked.
Steps to reproduce
- Install this package and sync zendesk data
- Run
dbt source snapshot-freshness
- You'll see an error like
ERROR STALE freshness of zendesk.organization_tag
Expected behavior
No freshness error should occur if data sync is working. In my case, it appears that this data just doesn't get updated very often. I noticed that freshness: null
is configured for zendesk.organization
. Should that just be copied to organization_tag
to ignore freshness here too?
Project variables configuration
name: [project-name]
version: 1.0.0
config-version: 2
profile: default
source-paths: [models]
analysis-paths: [analysis]
test-paths: [tests]
data-paths: [data]
macro-paths: [macros]
snapshot-paths: [snapshots]
target-path: target
clean-targets:
- target
- dbt_modules
vars:
zendesk_source:
zendesk_database: [project]
# Disable models for Zendesk features we aren't using or syncing
using_ticket_form_history: False
using_schedules: False
Package Version
packages:
- package: fishtown-analytics/dbt_utils
version: 0.6.6
- package: fivetran/ad_reporting
version: 0.2.2
- package: fivetran/facebook_ads
version: 0.2.1
- package: fivetran/google_ads
version: 0.2.1
- package: fivetran/marketo
version: 0.4.0
- package: fivetran/microsoft_ads
version: 0.2.1
- package: fivetran/pinterest
version: 0.3.2
- package: fivetran/zendesk
version: 0.5.1
Warehouse
- BigQuery
- Redshift
- Snowflake
- Postgres
- Databricks
- Other (provide details below)
Additional context
Freshness is ignored for organization
: https://github.com/fivetran/dbt_zendesk_source/blob/master/models/src_zendesk.yml#L130
Screenshots
Please indicate the level of urgency
Testing for the first time, but hopefully this is a simple fix.
Are you interested in contributing to this package?
- Yes, I can do this and open a PR for your review.
- Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
- No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.
Hi @wuservices thanks for opening this Issue as well as Issue #18!
This is actually a request we have seen come through on our dbt_hubspot package and have been able to institute a relatively low impact fix for these disabled tables. You can see the fix we did here. I will be able to easily add this feature to the package. However, I will focus first on upgrading our dbt_zendesk package for dbt 20 compatibility first so there are no compatibility issues between this source package and the transform package.
hey @wuservices our own data is in line with the pre-existing freshness tests, so i am thinking that the ideal solution here may be for you to override the zendesk source and set the organization_tag
freshness to null
however, worth noting that dbt 0.21.0 will introduce --exclude
for freshness tests ๐ (see PR and pre-release notes)
hi @fivetran-jamie thanks for the tips! @fivetran-joemarkiewicz pointed out the same freshness override in office hours, and that was a huge help! We were able to get our freshness tuned up nicely with that, so we're OK.
However, we did also discuss that good defaults are always nice when possible. For the benefit of others, I'd be curious why you'd have freshness by default on organization_tag
, but no freshness for organization
? For many companies using Zendesk, I could imagine that the clients they work with organization
may not change often (or never at all for B2C). This seems to be reflected in your default disabled freshness for organization
. However, if organization_tag
has freshness, does that mean you'd assume that people would be changing tags on a regular basis? Although we don't use those, I'd imagine that most people would have stable tags.
that's a good point that joe and i were honestly unsure about. in our own zendesk data, organization
doesn't change too often, but it does appear that we have fresh tags on them. i'll look into this more and talk to our internal customer success folks who are using zendesk (and maybe do a quick poll in dbt slack or something) to see if this manual tagging is actually happening often.
definitely aiming for the nicest defaults possible and really appreciate your input! i think it'd be good to keep this issue open and mark it as won't fix
for now in case other customers are coming across the same staleness issues with this table. glad to hear the override works for you for the time being ๐
Closing this issue as it is a wontfix.