ADF pipeline config_ExportData failing: property doesn't exist error
Opened this issue · 6 comments
🐛 Problem
Getting this error while running the ADF pipeline config_ExportData
Error code
ActionFailed
Failure type
User configuration issue
Details
Activity failed because an inner activity failed; Inner activity name: Run exports for scope, Error: Operation on target ForEach export scope failed: Activity failed because an inner activity failed; Inner activity name: If scheduled, Error: The expression 'and(equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)),startswith(toLower(item().name), toLower(variables('hubName'))))' cannot be evaluated because property 'recurrence' doesn't exist, available properties are 'status'.
Activity ID
faa571d8-a5dc-41d2-90c8-a532bd10ea3f
Error source
Pipeline
config_ExportData
📷 Screenshots
ℹ️ Additional context
TODO: Add any other context about the problem here. Remove if not applicable.
🙋♀️ Ask for the community
We could use your help:
- Please vote this issue up (👍) to prioritize it.
- Leave comments to help us solidify the vision.
@flanakin Is this really for September or was that a mistake?
@arthurclares I added it to September to make sure we look into it.
@santoshblearner Looking at this, I don't see an issue with the default configuration. Do you think the pipeline might have been updated accidentally? Can you confirm the config_RunExports pipeline has the Recurrence parameter?
- Open the hub resource group
- Select the Data Factory instance
- Select the Launch studio button
- Select the Author (pencil) icon on the left
- Select the config_RunExports pipeline
- Confirm the parameters in the Parameters tab:
hi @flanakin Thanks for your comments.
I debugged this issue over the weekend and found the fix.
Root Cause
The issue arises in the config_RunExports pipeline during the evaluation of an expression in the IfCondition activity. The pipeline tries to evaluate the following expression:
and(equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)),startswith(toLower(item().name), toLower(variables('hubName'))))
The error occurs because not all export items returned by the Get exports for scope activity have a recurrence property in the schedule object. I had a few one-time exports configured earlier and they don't have the recurrence property.
Since the function expects both conditions to be evaluated, the pipeline fails when the recurrence property is missing for one-time or unscheduled exports.
Proposed Fix
The pipeline processes a set of exports where only a subset is linked to the finOpsHub. However, the pipeline is currently attempting to evaluate all exports, even those not linked to the hub, and do not have a recurrence property. As a result, the pipeline fails when encountering unscheduled exports that lack the recurrence field.
The solution is to update the IfCondition activity’s expression so that it first filters exports based on whether they are linked to finOpsHub by checking if their name starts with the hubName variable. Only after filtering the exports should the pipeline check for the presence of the recurrence property. This will ensure that irrelevant or unscheduled exports are skipped early, preventing the error.
Updated Expression:
and(startswith(toLower(item().name), toLower(variables('hubName'))), has(item().properties.schedule, 'recurrence'), equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)))
Explanation:
startswith(toLower(item().name), toLower(variables('hubName'))): This filters the exports to include only those whose name starts with the hubName (i.e., exports relevant to finOpsHub).
has(item().properties.schedule, 'recurrence'): After filtering by hubName, this checks if the recurrence property exists in the export’s schedule object.
equals(toLower(item().properties.schedule.recurrence), toLower(pipeline().parameters.Recurrence)): Finally, this compares the recurrence value of the export with the Recurrence parameter in the pipeline, but only if the recurrence property is present.
This change ensures that:
- Only applicable exports (those linked to finOpsHub) are evaluated further.
- The pipeline gracefully handles exports that do not have a recurrence property by skipping them.
- The expression only runs the comparison of the recurrence property when it exists, preventing the pipeline from failing on unscheduled exports.
Hi @flanakin: I am unable to assign this issue back to you. I posted the root cause and the possible fix for this issue. Please check and let me know.