Afsarsoft/ADF-Samples

ADF Samples

ADF-Samples

ADF Samples

Pipeline: 01_plCopyOneTextFile

Description: Copy one Text/CSV file from one folder to another folder in BLOB
Components: Copy data

Pipeline: 02_plCopyAllTextFiles

Description: Copy all Text/CSV files from one folder to another folder in BLOB
Components: Get Metadata, ForEach, Copy data
Background: https://www.youtube.com/watch?v=duyr9tPh_Yk&list=PLm7Nm9btqfe3QNbrLY-Vyu8-wh3EanG6t

Pipeline: 03_plGameImportData

Description: Load data from ADLS Gen2 for Game project
Components: Stored Procedure & Copy data
Note: For more info about Game project, please see https://github.com/Afsarsoft/SQL-Game

Pipeline: 04_plGameExportData

Description: Export data from Azure SQL DB to ADLS Gen2
Components: Copy data
Note: For more info about Game project, please see https://github.com/Afsarsoft/SQL-Game

Pipeline: 05_plGameExportDataDynamic

Description: Dynamically Export data from Azure SQL DB to ADLS Gen2
Components: Lookup, ForEach, Copy data
Process flow:

• Get few tables, using Lookup
  SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA = 'game' AND TABLE_NAME IN ('Discount', 'Game', 'GameTeam', 'Order', 'Partner', 'PartnerInfo', 'Retailer', 'Team', 'Type');
  
• For each table, bring tables
  
• Copy data to ADLS Gen2 using Copy data
  
  Note: For more info about Game project, please see https://github.com/Afsarsoft/SQL-Game
  Background: https://www.youtube.com/watch?v=KsO2FHQdILs It is documented in https://microsoft-bitools.blogspot.com/2019/06/staging-with-azure-data-factory-foreach.html
  

Pipeline: 06_plGameLookupCondition

Description: Dynamically Export data from Azure SQL DB to ADLS Gen2
Components: Stored procedure, Lookup, ForEach, if Condition, Copy data
Process flow:

• Truncate Table History, using Stored procedure
  
• Get few tables, using Lookup
  SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' AND TABLE_SCHEMA = 'game' AND TABLE_NAME IN ('Discount', 'Game', 'GameTeam', '[Order]', 'Partner', 'PartnerInfo', 'Retailer', 'Team', 'Type', 'History');
  
• For each table, bring tables which are not empty by using Lookup and If Condition
  @greater(activity('LookupTable').output.firstRow.count,0)
  
• Copy data to ADLS Gen2 using Copy data
  
  Note: For more info about Game project, please see https://github.com/Afsarsoft/SQL-Game
  

Pipeline: 07_plGameDeleteFiles

Description: Delete all files from ADLS Gen2 folder.
Components: Get Metadata, ForEach, Delete