mozilla/mozanalysis

mozanalysis can't handle `submission_date_column`s that are SQL snippets

Opened this issue · 1 comments

The problem is occurring here:

AND ds.{submission_date} BETWEEN

As long as a data sources submission_date_column is only set to a field name everything is working fine, but as soon as SQL is used the generated query is invalid due to the alias. ds.{submission_date} cannot be transformed to ds.DATE(submission_timestamp), for example

The code below is useful to recreating this bug (assuming I understand the issue)

from mozanalysis.metrics import DataSource

data_source_definition = config_collection.get_data_source_definition(
    data_source_slug, app_name
)
if data_source_definition is None:
    raise Exception(
        f"Could not find definition for data source {data_source_slug}"
    )

clients_daily = DataSource(
    name=data_source_definition.name,
    from_expr=data_source_definition.from_expression,
    client_id_column=data_source_definition.client_id_column,
    submission_date_column="Date(submission_timestamp_min)",
    experiments_column_type=None
    if data_source_definition.experiments_column_type == "none"
    else data_source_definition.experiments_column_type,
    default_dataset=data_source_definition.default_dataset,
)

from mozanalysis.metrics import DataSource, Metric, agg_any, agg_sum

qcdou = Metric(
    name="qcdou",
    data_source=clients_daily,
    select_expr="""COUNTIF(
    active_hours_sum > 0 AND
    scalar_parent_browser_engagement_total_uri_count_normal_and_private_mode_sum > 0
)""",
)

df = test_target.get_single_window_data(
    bq_context=bq_context,
    metric_list=[qcdou],
    target_list=[weekday_regular_us]
)