datafold/data-diff

ValueError: Cannot apply [String_VaryingAlphanum(_notes=[])] to ...

lokofoko opened this issue · 5 comments

lokofoko commented

Describe the bug
Your code doesn't include special characters in alphanum string. It worked fine if compare between tables in snowflake. But now I have this error when comparing mssql vs snowflake. Datatypes are the same, but full traceback lead me to the alphanums thing.
image

As you can see, it doesn't include special characters. Can you advise how I can compare data? I used concat on three of my fields, to have key column. Maybe I can use them without concationation? As composite primary key? I didn't find examples of how to use it in your documentation.

Make sure to include the following (minus sensitive information):

  • The command or code you used
    data-diff --conf /c/Users/Vitalii/WorkRepos/data_diff_configs/datadiff.toml --run compare_unuseddata_sqlserver_snowflake -k 'unique_id' -c % -l 1000 -m test_results_%t --no-tracking --table-write-limit 50000000
  • The run output + error you're getting. (including tracestack)
    DEBUG Applied run configuration:
    INFO [MsSQL] Starting a threadpool, size=1. base.py:1197
    DEBUG Database 'MsSQL(default_schema='dbo', _interactive=False, is_closed=False, _dialect=Dialect(_prevent_overflow_when_concat=False), thread_count=1, _init_error=None, _queue=<concurrent.futures.thread.ThreadPoolExecutor object at 0x00000238D5BCF640>, thread_local=<_thread._local object at 0x00000238D5CBBB00>, _connect.py:300
    15:41:12 DEBUG Running SQL (Snowflake): base.py:962
    ALTER SESSION SET TIMEZONE = 'UTC'
    15:41:13 DEBUG Running SQL (MsSQL): ('UNUSEDDATA',) base.py:960
    SELECT column_name, data_type, datetime_precision, numeric_precision, numeric_scale FROM [MonsterDataGeneration].information_schema.columns WHERE table_name = 'UNUSEDDATA' AND table_schema = 'dbo'
    DEBUG Running SQL (Snowflake): ('COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE',) base.py:960
    SELECT column_name, data_type, datetime_precision, numeric_precision, numeric_scale FROM information_schema.columns WHERE table_name = 'COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE' AND table_schema = 'DATA_INTEGRATIONS'
    15:41:15 INFO [MsSQL] Schema = {'CorelogicDataID': ('CorelogicDataID', 'int', None, 10, 0), 'Appreciation': ('Appreciation', 'decimal', None, 18, 5), 'AVM': ('AVM', 'decimal', None, 18, 5), 'CountyId': ('CountyId', 'int', None, 10, 0), 'Estimated CLTV': ('Estimated CLTV', 'decimal', None, 18, 5), 'First MTG Amount': ('First MTG Amount', 'money', schema.py:12
    None, 19, 4), 'First MTG Assigned Lender': ('First MTG Assigned Lender', 'varchar', None, None, None), 'First MTG Fixed Rate Loan Ind': ('First MTG Fixed Rate Loan Ind', 'bit', None, None, None), 'First MTG Lender Name': ('First MTG Lender Name', 'varchar', None, None, None), 'First MTG Loan Type': ('First MTG Loan Type', 'varchar',
    None, None, None), 'First MTG LTV': ('First MTG LTV', 'decimal', None, 18, 5), 'First MTG Origination Date': ('First MTG Origination Date', 'date', 0, None, None), 'First MTG Refinance Loan Ind': ('First MTG Refinance Loan Ind', 'bit', None, None, None), 'First MTG Variable Rate Loan Ind': ('First MTG Variable Rate Loan Ind', 'bit',
    None, None, None), 'Junior MTG Amount': ('Junior MTG Amount', 'money', None, 19, 4), 'Junior MTG Assigned Lender': ('Junior MTG Assigned Lender', 'varchar', None, None, None), 'Junior MTG Lender Name': ('Junior MTG Lender Name', 'varchar', None, None, None), 'Junior MTG Origination Date': ('Junior MTG Origination Date', 'date', 0,
    None, None), 'Junior MTG Subordinate Type': ('Junior MTG Subordinate Type', 'varchar', None, None, None), 'LandUseDescriptionID': ('LandUseDescriptionID', 'int', None, 10, 0), 'LastUsed': ('LastUsed', 'date', 0, None, None), 'Mailing Address Full': ('Mailing Address Full', 'varchar', None, None, None), 'Mailing City': ('Mailing City',
    'varchar', None, None, None), 'Mailing State': ('Mailing State', 'varchar', None, None, None), 'Mailing Zip Code': ('Mailing Zip Code', 'varchar', None, None, None), 'MailingZipCodeId': ('MailingZipCodeId', 'int', None, 10, 0), 'Owner 1 First Name': ('Owner 1 First Name', 'varchar', None, None, None), 'Owner 1 Last Name': ('Owner 1
    Last Name', 'varchar', None, None, None), 'Owner 2 First Name': ('Owner 2 First Name', 'varchar', None, None, None), 'Owner 2 Last Name': ('Owner 2 Last Name', 'varchar', None, None, None), 'Property Address Full': ('Property Address Full', 'varchar', None, None, None), 'Property City': ('Property City', 'varchar', None, None, None),
    'Property State': ('Property State', 'varchar', None, None, None), 'Property Zip Code': ('Property Zip Code', 'varchar', None, None, None), 'PropertyStateId': ('PropertyStateId', 'int', None, 10, 0), 'PropertyZipCodeId': ('PropertyZipCodeId', 'int', None, 10, 0), 'ReferenceNr': ('ReferenceNr', 'varchar', None, None, None), 'Tax
    Amount': ('Tax Amount', 'varchar', None, None, None), 'Estimated Equity': ('Estimated Equity', 'varchar', None, None, None), 'First MTG ARM Next Reset Date': ('First MTG ARM Next Reset Date', 'date', 0, None, None), 'First MTG ARM Maximum Interest Rate': ('First MTG ARM Maximum Interest Rate', 'decimal', None, 18, 2), 'Junior MTG
    Equity Loan Ind': ('Junior MTG Equity Loan Ind', 'bit', None, None, None), 'FormattedCountyName': ('FormattedCountyName', 'varchar', None, None, None), 'MailingStateId': ('MailingStateId', 'int', None, 10, 0), 'County Name': ('County Name', 'varchar', None, None, None), 'First MTG ARM Change Percent Limit': ('First MTG ARM Change
    Percent Limit', 'decimal', None, 18, 2), 'PrevailingInterestRate': ('PrevailingInterestRate', 'decimal', None, 18, 2), 'First MTG ARM Initial Reset Date': ('First MTG ARM Initial Reset Date', 'date', 0, None, None), 'First MTG ARM Change Interval': ('First MTG ARM Change Interval', 'int', None, 10, 0), 'First MTG ARM Change Freq':
    ('First MTG ARM Change Freq', 'varchar', None, None, None), 'First MTG ARM First Change Max Pct': ('First MTG ARM First Change Max Pct', 'decimal', None, 8, 4), 'First MTG ARM Calculation Change': ('First MTG ARM Calculation Change', 'decimal', None, 8, 4), 'First MTG Interest Rate': ('First MTG Interest Rate', 'decimal', None, 8, 4),
    'First MTG Term': ('First MTG Term', 'int', None, 10, 0), 'First MTG Interest Rate Type': ('First MTG Interest Rate Type', 'varchar', None, None, None), 'APN Unformatted': ('APN Unformatted', 'varchar', None, None, None), 'APN Sequence Number': ('APN Sequence Number', 'int', None, 10, 0), 'FIPS': ('FIPS', 'int', None, 10, 0), 'First
    MTG Modified Ind': ('First MTG Modified Ind', 'bit', None, None, None), 'Owner Occupied Indicator': ('Owner Occupied Indicator', 'bit', None, None, None), 'First MTG ARM Index Type': ('First MTG ARM Index Type', 'varchar', None, None, None), 'DownPmtAssistFGrant_Ind': ('DownPmtAssistFGrant_Ind', 'bit', None, None, None), 'First MTG
    Loan Purpose': ('First MTG Loan Purpose', 'varchar', None, None, None), 'LoanPurposeID': ('LoanPurposeID', 'int', None, 10, 0), 'MonthlySavingQuoteRate': ('MonthlySavingQuoteRate', 'decimal', None, 18, 2), 'ConsolidationMonthlySavingsAmt': ('ConsolidationMonthlySavingsAmt', 'decimal', None, 18, 2), 'ConsolidationMonthlySavingsPercent':
    ('ConsolidationMonthlySavingsPercent', 'decimal', None, 18, 2), 'NewConfLoanLimit': ('NewConfLoanLimit', 'decimal', None, 18, 2), 'DataProviderID': ('DataProviderID', 'int', None, 10, 0), 'MailingAddressIsLenderAddress': ('MailingAddressIsLenderAddress', 'bit', None, None, None), 'CurrentPrevailingInterestRate':
    ('CurrentPrevailingInterestRate', 'decimal', None, 18, 2), 'DifferencePrevailingInterestRate': ('DifferencePrevailingInterestRate', 'decimal', None, 18, 2), 'MonthlySavingPrevailingInterestRate': ('MonthlySavingPrevailingInterestRate', 'decimal', None, 18, 2), 'IsNyBurrough': ('IsNyBurrough', 'bit', None, None, None),
    'FirstMortgageLoanTypeId': ('FirstMortgageLoanTypeId', 'int', None, 10, 0), 'MlgSalePrice': ('MlgSalePrice', 'decimal', None, 18, 2), 'IsHistoricDirectMailResponder': ('IsHistoricDirectMailResponder', 'bit', None, None, None), 'CashOutFlag': ('CashOutFlag', 'bit', None, None, None), 'RateAndTermReductionFlag':
    ('RateAndTermReductionFlag', 'bit', None, None, None), 'ConsolidationFlag': ('ConsolidationFlag', 'bit', None, None, None), 'PropensityToTransact': ('PropensityToTransact', 'decimal', None, 18, 2), 'MonthsSinceMortgageDefaultNotice': ('MonthsSinceMortgageDefaultNotice', 'int', None, 10, 0), 'MonthsSinceMortgageDefaultRelease':
    ('MonthsSinceMortgageDefaultRelease', 'int', None, 10, 0), 'MonthsSinceMortgageModification': ('MonthsSinceMortgageModification', 'int', None, 10, 0), 'MonthsSinceMortgageBankruptcyDischarge': ('MonthsSinceMortgageBankruptcyDischarge', 'int', None, 10, 0), 'IsReverseMortgage': ('IsReverseMortgage', 'bit', None, None, None),
    'IsHistoricPurlResponder': ('IsHistoricPurlResponder', 'bit', None, None, None), 'IsHistoricQrResponder': ('IsHistoricQrResponder', 'bit', None, None, None), 'IsHistoricPhoneResponder': ('IsHistoricPhoneResponder', 'bit', None, None, None), 'MlgEstimatedUnpaidBalance': ('MlgEstimatedUnpaidBalance', 'decimal', None, 18, 5), 'IsSenior':
    ('IsSenior', 'bit', None, None, None), 'HasHistoricFha': ('HasHistoricFha', 'bit', None, None, None), 'HasRepeatedRefinance': ('HasRepeatedRefinance', 'bit', None, None, None), 'HasVetInHousehold': ('HasVetInHousehold', 'bit', None, None, None), 'MailingOptOutIndicator': ('MailingOptOutIndicator', 'bit', None, None, None),
    'MidAvmMlgCltv': ('MidAvmMlgCltv', 'decimal', None, 18, 2), 'unique_id': ('unique_id', 'varchar', None, None, None)}
    INFO [Snowflake] Schema = {'MONTHLYSAVINGPREVAILINGINTERESTRATE': ('MONTHLYSAVINGPREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'JUNIOR_MTG_LENDER_NAME': ('JUNIOR_MTG_LENDER_NAME', 'TEXT', None, None, None), 'FIRST_MTG_FIXED_RATE_LOAN_IND': ('FIRST_MTG_FIXED_RATE_LOAN_IND', 'BOOLEAN', None, None, None), 'FIRST_MTG_ARM_INITIAL_RESET_DATE': schema.py:12
    ('FIRST_MTG_ARM_INITIAL_RESET_DATE', 'DATE', None, None, None), 'MAILINGADDRESSISLENDERADDRESS': ('MAILINGADDRESSISLENDERADDRESS', 'NUMBER', None, 1, 0), 'FIRST_MTG_ORIGINATION_DATE': ('FIRST_MTG_ORIGINATION_DATE', 'DATE', None, None, None), 'JUNIOR_MTG_ASSIGNED_LENDER': ('JUNIOR_MTG_ASSIGNED_LENDER', 'TEXT', None, None, None),
    'MAILING_STATE': ('MAILING_STATE', 'TEXT', None, None, None), 'MONTHLYSAVINGQUOTERATE': ('MONTHLYSAVINGQUOTERATE', 'NUMBER', None, 18, 2), 'PROPERTY_CITY': ('PROPERTY_CITY', 'TEXT', None, None, None), 'FIRST_MTG_AMOUNT': ('FIRST_MTG_AMOUNT', 'NUMBER', None, 38, 4), 'HASHISTORICFHA': ('HASHISTORICFHA', 'NUMBER', None, 1, 0),
    'FIRST_MTG_LTV': ('FIRST_MTG_LTV', 'NUMBER', None, 18, 5), 'CURRENTPREVAILINGINTERESTRATE': ('CURRENTPREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'FIRST_MTG_MODIFIED_IND': ('FIRST_MTG_MODIFIED_IND', 'BOOLEAN', None, None, None), 'FIPS': ('FIPS', 'NUMBER', None, 38, 0), 'MAILING_ZIP_CODE': ('MAILING_ZIP_CODE', 'TEXT', None, None,
    None), 'AVM': ('AVM', 'NUMBER', None, 18, 5), 'REFERENCENR': ('REFERENCENR', 'TEXT', None, None, None), 'FIRST_MTG_INTEREST_RATE': ('FIRST_MTG_INTEREST_RATE', 'NUMBER', None, 8, 4), 'MLG_SALE_PRICE': ('MLG_SALE_PRICE', 'NUMBER', None, 18, 2), 'ESTIMATED_EQUITY': ('ESTIMATED_EQUITY', 'TEXT', None, None, None), 'PROPERTYZIPCODEID':
    ('PROPERTYZIPCODEID', 'NUMBER', None, 38, 0), 'JUNIOR_MTG_SUBORDINATE_TYPE': ('JUNIOR_MTG_SUBORDINATE_TYPE', 'TEXT', None, None, None), 'FIRST_MTG_ASSIGNED_LENDER': ('FIRST_MTG_ASSIGNED_LENDER', 'TEXT', None, None, None), 'OWNER_2_LAST_NAME': ('OWNER_2_LAST_NAME', 'TEXT', None, None, None), 'PROPENSITYTOTRANSACT':
    ('PROPENSITYTOTRANSACT', 'NUMBER', None, 18, 2), 'MONTHSSINCEMORTGAGEDEFAULTNOTICE': ('MONTHSSINCEMORTGAGEDEFAULTNOTICE', 'NUMBER', None, 38, 0), 'APN_SEQUENCE_NUMBER': ('APN_SEQUENCE_NUMBER', 'NUMBER', None, 38, 0), 'DOWNPMTASSISTFGRANT_IND': ('DOWNPMTASSISTFGRANT_IND', 'NUMBER', None, 1, 0), 'NEWCONFLOANLIMIT': ('NEWCONFLOANLIMIT',
    'NUMBER', None, 18, 2), 'COUNTYID': ('COUNTYID', 'NUMBER', None, 38, 0), 'RATEANDTERMREDUCTIONFLAG': ('RATEANDTERMREDUCTIONFLAG', 'NUMBER', None, 1, 0), 'MAILINGSTATEID': ('MAILINGSTATEID', 'NUMBER', None, 38, 0), 'HASREPEATEDREFINANCE': ('HASREPEATEDREFINANCE', 'NUMBER', None, 1, 0), 'OWNER_1_LAST_NAME': ('OWNER_1_LAST_NAME', 'TEXT',
    None, None, None), 'FIRST_MTG_ARM_CALCULATION_CHANGE': ('FIRST_MTG_ARM_CALCULATION_CHANGE', 'NUMBER', None, 8, 4), 'PREVAILINGINTERESTRATE': ('PREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'MAILINGOPTOUTINDICATOR': ('MAILINGOPTOUTINDICATOR', 'NUMBER', None, 1, 0), 'PROPERTY_ZIP_CODE': ('PROPERTY_ZIP_CODE', 'TEXT', None, None, None),
    'APN_UNFORMATTED': ('APN_UNFORMATTED', 'TEXT', None, None, None), 'ISREVERSEMORTGAGE': ('ISREVERSEMORTGAGE', 'NUMBER', None, 1, 0), 'PROPERTY_STATE': ('PROPERTY_STATE', 'TEXT', None, None, None), 'JUNIOR_MTG_ORIGINATION_DATE': ('JUNIOR_MTG_ORIGINATION_DATE', 'DATE', None, None, None), 'LANDUSEDESCRIPTIONID': ('LANDUSEDESCRIPTIONID',
    'NUMBER', None, 38, 0), 'FIRST_MTG_ARM_INDEX_TYPE': ('FIRST_MTG_ARM_INDEX_TYPE', 'TEXT', None, None, None), 'PROPERTYSTATEID': ('PROPERTYSTATEID', 'NUMBER', None, 38, 0), 'OWNER_1_FIRST_NAME': ('OWNER_1_FIRST_NAME', 'TEXT', None, None, None), 'DIFFERENCEPREVAILINGINTERESTRATE': ('DIFFERENCEPREVAILINGINTERESTRATE', 'NUMBER', None, 18,
    2), 'HASVETINHOUSEHOLD': ('HASVETINHOUSEHOLD', 'NUMBER', None, 1, 0), 'LOANPURPOSEID': ('LOANPURPOSEID', 'NUMBER', None, 38, 0), 'APPRECIATION': ('APPRECIATION', 'NUMBER', None, 18, 5), 'ISHISTORICDIRECTMAILRESPONDER': ('ISHISTORICDIRECTMAILRESPONDER', 'NUMBER', None, 1, 0), 'ISHISTORICPURLRESPONDER': ('ISHISTORICPURLRESPONDER',
    'NUMBER', None, 1, 0), 'FIRST_MTG_INTEREST_RATE_TYPE': ('FIRST_MTG_INTEREST_RATE_TYPE', 'TEXT', None, None, None), 'ISSENIOR': ('ISSENIOR', 'NUMBER', None, 1, 0), 'JUNIOR_MTG_EQUITY_LOAN_IND': ('JUNIOR_MTG_EQUITY_LOAN_IND', 'BOOLEAN', None, None, None), 'UNIQUE_ID': ('UNIQUE_ID', 'TEXT', None, None, None), 'ISHISTORICPHONERESPONDER':
    ('ISHISTORICPHONERESPONDER', 'NUMBER', None, 1, 0), 'MIDAVMMLGCLTV': ('MIDAVMMLGCLTV', 'NUMBER', None, 18, 2), 'MAILING_ADDRESS_FULL': ('MAILING_ADDRESS_FULL', 'TEXT', None, None, None), 'TAX_AMOUNT': ('TAX_AMOUNT', 'TEXT', None, None, None), 'ISHISTORICQRRESPONDER': ('ISHISTORICQRRESPONDER', 'NUMBER', None, 1, 0),
    'FIRST_MTG_ARM_CHANGE_PERCENT_LIMIT': ('FIRST_MTG_ARM_CHANGE_PERCENT_LIMIT', 'NUMBER', None, 18, 2), 'MLGESTIMATEDUNPAIDBALANCE': ('MLGESTIMATEDUNPAIDBALANCE', 'NUMBER', None, 18, 5), 'DATAPROVIDERID': ('DATAPROVIDERID', 'NUMBER', None, 38, 0), 'ESTIMATED_CLTV': ('ESTIMATED_CLTV', 'NUMBER', None, 18, 5), 'FIRST_MTG_TERM':
    ('FIRST_MTG_TERM', 'NUMBER', None, 38, 0), 'CONSOLIDATIONFLAG': ('CONSOLIDATIONFLAG', 'NUMBER', None, 1, 0), 'OWNER_2_FIRST_NAME': ('OWNER_2_FIRST_NAME', 'TEXT', None, None, None), 'MAILINGZIPCODEID': ('MAILINGZIPCODEID', 'NUMBER', None, 38, 0), 'FIRST_MTG_REFINANCE_LOAN_IND': ('FIRST_MTG_REFINANCE_LOAN_IND', 'BOOLEAN', None, None,
    None), 'FIRST_MTG_ARM_FIRST_CHANGE_MAX_PCT': ('FIRST_MTG_ARM_FIRST_CHANGE_MAX_PCT', 'NUMBER', None, 8, 4), 'FIRST_MTG_LOAN_PURPOSE': ('FIRST_MTG_LOAN_PURPOSE', 'TEXT', None, None, None), 'FIRST_MTG_VARIABLE_RATE_LOAN_IND': ('FIRST_MTG_VARIABLE_RATE_LOAN_IND', 'BOOLEAN', None, None, None), 'CONSOLIDATIONMONTHLYSAVINGSPERCENT':
    ('CONSOLIDATIONMONTHLYSAVINGSPERCENT', 'NUMBER', None, 18, 2), 'FORMATTEDCOUNTYNAME': ('FORMATTEDCOUNTYNAME', 'TEXT', None, None, None), 'FIRST_MTG_LOAN_TYPE': ('FIRST_MTG_LOAN_TYPE', 'TEXT', None, None, None), 'MONTHSSINCEMORTGAGEDEFAULTRELEASE': ('MONTHSSINCEMORTGAGEDEFAULTRELEASE', 'NUMBER', None, 38, 0),
    'CONSOLIDATIONMONTHLYSAVINGSAMT': ('CONSOLIDATIONMONTHLYSAVINGSAMT', 'NUMBER', None, 18, 2), 'FIRST_MTG_ARM_MAXIMUM_INTEREST_RATE': ('FIRST_MTG_ARM_MAXIMUM_INTEREST_RATE', 'NUMBER', None, 18, 2), 'MAILING_CITY': ('MAILING_CITY', 'TEXT', None, None, None), 'LASTUSED': ('LASTUSED', 'DATE', None, None, None), 'CASHOUTFLAG':
    ('CASHOUTFLAG', 'NUMBER', None, 1, 0), 'MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE': ('MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE', 'NUMBER', None, 38, 0), 'OWNER_OCCUPIED_INDICATOR': ('OWNER_OCCUPIED_INDICATOR', 'NUMBER', None, 1, 0), 'FIRST_MTG_ARM_NEXT_RESET_DATE': ('FIRST_MTG_ARM_NEXT_RESET_DATE', 'DATE', None, None, None),
    'CORELOGICDATAID': ('CORELOGICDATAID', 'NUMBER', None, 38, 0), 'FIRST_MTG_ARM_CHANGE_FREQ': ('FIRST_MTG_ARM_CHANGE_FREQ', 'TEXT', None, None, None), 'FIRST_MTG_ARM_CHANGE_INTERVAL': ('FIRST_MTG_ARM_CHANGE_INTERVAL', 'NUMBER', None, 38, 0), 'JUNIOR_MTG_AMOUNT': ('JUNIOR_MTG_AMOUNT', 'NUMBER', None, 38, 4), 'ISNYBURROUGH':
    ('ISNYBURROUGH', 'NUMBER', None, 1, 0), 'PROPERTY_ADDRESS_FULL': ('PROPERTY_ADDRESS_FULL', 'TEXT', None, None, None), 'FIRST_MTG_LENDER_NAME': ('FIRST_MTG_LENDER_NAME', 'TEXT', None, None, None), 'COUNTY_NAME': ('COUNTY_NAME', 'TEXT', None, None, None), 'MONTHSSINCEMORTGAGEMODIFICATION': ('MONTHSSINCEMORTGAGEMODIFICATION', 'NUMBER',
    None, 38, 0)}
    DEBUG Available mutual columns: {'rateandtermreductionflag', 'monthssincemortgagebankruptcydischarge', 'monthlysavingquoterate', 'issenior', 'unique_id', 'monthlysavingprevailinginterestrate', 'mailingoptoutindicator', 'consolidationmonthlysavingsamt', 'referencenr', 'hashistoricfha', 'landusedescriptionid', main.py:489
    'monthssincemortgagemodification', 'currentprevailinginterestrate', 'hasrepeatedrefinance', 'countyid', 'ishistoricphoneresponder', 'avm', 'propertyzipcodeid', 'ishistoricqrresponder', 'ishistoricpurlresponder', 'consolidationflag', 'cashoutflag', 'propertystateid', 'newconfloanlimit', 'dataproviderid', 'corelogicdataid',
    'mailingstateid', 'fips', 'prevailinginterestrate', 'isreversemortgage', 'mlgestimatedunpaidbalance', 'differenceprevailinginterestrate', 'appreciation', 'loanpurposeid', 'formattedcountyname', 'monthssincemortgagedefaultnotice', 'lastused', 'ishistoricdirectmailresponder', 'mailingaddressislenderaddress', 'mailingzipcodeid',
    'consolidationmonthlysavingspercent', 'midavmmlgcltv', 'monthssincemortgagedefaultrelease', 'propensitytotransact', 'hasvetinhousehold', 'downpmtassistfgrant_ind', 'isnyburrough'}
    INFO Diffing using columns: key=('unique_id',) update=None extra=('rateandtermreductionflag', 'monthssincemortgagebankruptcydischarge', 'monthlysavingquoterate', 'issenior', 'monthlysavingprevailinginterestrate', 'mailingoptoutindicator', 'consolidationmonthlysavingsamt', 'referencenr', 'hashistoricfha', 'landusedescriptionid', main.py:518
    'monthssincemortgagemodification', 'currentprevailinginterestrate', 'hasrepeatedrefinance', 'ishistoricphoneresponder', 'downpmtassistfgrant_ind', 'avm', 'propertyzipcodeid', 'ishistoricqrresponder', 'ishistoricpurlresponder', 'consolidationflag', 'cashoutflag', 'propertystateid', 'newconfloanlimit', 'dataproviderid',
    'corelogicdataid', 'mailingstateid', 'fips', 'prevailinginterestrate', 'isreversemortgage', 'mlgestimatedunpaidbalance', 'differenceprevailinginterestrate', 'appreciation', 'loanpurposeid', 'formattedcountyname', 'monthssincemortgagedefaultnotice', 'lastused', 'ishistoricdirectmailresponder', 'mailingaddressislenderaddress',
    'mailingzipcodeid', 'consolidationmonthlysavingspercent', 'midavmmlgcltv', 'monthssincemortgagedefaultrelease', 'propensitytotransact', 'hasvetinhousehold', 'countyid', 'isnyburrough').
    INFO Using algorithm 'hashdiff'. main.py:519
    DEBUG Running SQL (MsSQL): ('UNUSEDDATA',) base.py:960
    SELECT TRIM([ReferenceNr]), TRIM([FormattedCountyName]), TRIM([unique_id]) FROM [UNUSEDDATA] ORDER BY 1 OFFSET 0 ROWS FETCH NEXT 64 ROWS ONLY
    15:41:25 INFO [MsSQL] Schema = {'CorelogicDataID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'Appreciation': Decimal(_notes=[], precision=5), 'AVM': Decimal(_notes=[], precision=5), 'CountyId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'LandUseDescriptionID': Integer(_notes=[], precision=0, python_type=<class schema.py:12
    'int'>), 'LastUsed': Timestamp(_notes=[], precision=0, rounds=True), 'MailingZipCodeId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PropertyStateId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PropertyZipCodeId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'ReferenceNr':
    Text(_notes=[]), 'FormattedCountyName': String_VaryingAlphanum(_notes=[]), 'MailingStateId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PrevailingInterestRate': Decimal(_notes=[], precision=2), 'FIPS': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'DownPmtAssistFGrant_Ind': Boolean(_notes=[]),
    'LoanPurposeID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthlySavingQuoteRate': Decimal(_notes=[], precision=2), 'ConsolidationMonthlySavingsAmt': Decimal(_notes=[], precision=2), 'ConsolidationMonthlySavingsPercent': Decimal(_notes=[], precision=2), 'NewConfLoanLimit': Decimal(_notes=[], precision=2),
    'DataProviderID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MailingAddressIsLenderAddress': Boolean(_notes=[]), 'CurrentPrevailingInterestRate': Decimal(_notes=[], precision=2), 'DifferencePrevailingInterestRate': Decimal(_notes=[], precision=2), 'MonthlySavingPrevailingInterestRate': Decimal(_notes=[], precision=2),
    'IsNyBurrough': Boolean(_notes=[]), 'IsHistoricDirectMailResponder': Boolean(_notes=[]), 'CashOutFlag': Boolean(_notes=[]), 'RateAndTermReductionFlag': Boolean(_notes=[]), 'ConsolidationFlag': Boolean(_notes=[]), 'PropensityToTransact': Decimal(_notes=[], precision=2), 'MonthsSinceMortgageDefaultNotice': Integer(_notes=[], precision=0,
    python_type=<class 'int'>), 'MonthsSinceMortgageDefaultRelease': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthsSinceMortgageModification': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthsSinceMortgageBankruptcyDischarge': Integer(_notes=[], precision=0, python_type=<class 'int'>),
    'IsReverseMortgage': Boolean(_notes=[]), 'IsHistoricPurlResponder': Boolean(_notes=[]), 'IsHistoricQrResponder': Boolean(_notes=[]), 'IsHistoricPhoneResponder': Boolean(_notes=[]), 'MlgEstimatedUnpaidBalance': Decimal(_notes=[], precision=5), 'IsSenior': Boolean(_notes=[]), 'HasHistoricFha': Boolean(_notes=[]), 'HasRepeatedRefinance':
    Boolean(_notes=[]), 'HasVetInHousehold': Boolean(_notes=[]), 'MailingOptOutIndicator': Boolean(_notes=[]), 'MidAvmMlgCltv': Decimal(_notes=[], precision=2), 'unique_id': String_VaryingAlphanum(_notes=[])}
    DEBUG Running SQL (Snowflake): ('COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE',) base.py:960
    SELECT * FROM (SELECT TRIM("REFERENCENR"), TRIM("UNIQUE_ID"), TRIM("FORMATTEDCOUNTYNAME") FROM "COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE") AS LIMITED_SELECT LIMIT 64
    WARNING Mixed UUID/Non-UUID values detected in column COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE.UNIQUE_ID, disabling UUID support. base.py:1102
    INFO [Snowflake] Schema = {'MONTHLYSAVINGPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2), 'MAILINGADDRESSISLENDERADDRESS': Decimal(_notes=[], precision=0), 'MONTHLYSAVINGQUOTERATE': Decimal(_notes=[], precision=2), 'HASHISTORICFHA': Decimal(_notes=[], precision=0), 'CURRENTPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2), schema.py:12
    'FIPS': Decimal(_notes=[], precision=0), 'AVM': Decimal(_notes=[], precision=5), 'REFERENCENR': Text(_notes=[]), 'PROPERTYZIPCODEID': Decimal(_notes=[], precision=0), 'PROPENSITYTOTRANSACT': Decimal(_notes=[], precision=2), 'MONTHSSINCEMORTGAGEDEFAULTNOTICE': Decimal(_notes=[], precision=0), 'DOWNPMTASSISTFGRANT_IND':
    Decimal(_notes=[], precision=0), 'NEWCONFLOANLIMIT': Decimal(_notes=[], precision=2), 'COUNTYID': Decimal(_notes=[], precision=0), 'RATEANDTERMREDUCTIONFLAG': Decimal(_notes=[], precision=0), 'MAILINGSTATEID': Decimal(_notes=[], precision=0), 'HASREPEATEDREFINANCE': Decimal(_notes=[], precision=0), 'PREVAILINGINTERESTRATE':
    Decimal(_notes=[], precision=2), 'MAILINGOPTOUTINDICATOR': Decimal(_notes=[], precision=0), 'ISREVERSEMORTGAGE': Decimal(_notes=[], precision=0), 'LANDUSEDESCRIPTIONID': Decimal(_notes=[], precision=0), 'PROPERTYSTATEID': Decimal(_notes=[], precision=0), 'DIFFERENCEPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2),
    'HASVETINHOUSEHOLD': Decimal(_notes=[], precision=0), 'LOANPURPOSEID': Decimal(_notes=[], precision=0), 'APPRECIATION': Decimal(_notes=[], precision=5), 'ISHISTORICDIRECTMAILRESPONDER': Decimal(_notes=[], precision=0), 'ISHISTORICPURLRESPONDER': Decimal(_notes=[], precision=0), 'ISSENIOR': Decimal(_notes=[], precision=0), 'UNIQUE_ID':
    String_VaryingAlphanum(_notes=[]), 'ISHISTORICPHONERESPONDER': Decimal(_notes=[], precision=0), 'MIDAVMMLGCLTV': Decimal(_notes=[], precision=2), 'ISHISTORICQRRESPONDER': Decimal(_notes=[], precision=0), 'MLGESTIMATEDUNPAIDBALANCE': Decimal(_notes=[], precision=5), 'DATAPROVIDERID': Decimal(_notes=[], precision=0), 'CONSOLIDATIONFLAG':
    Decimal(_notes=[], precision=0), 'MAILINGZIPCODEID': Decimal(_notes=[], precision=0), 'CONSOLIDATIONMONTHLYSAVINGSPERCENT': Decimal(_notes=[], precision=2), 'FORMATTEDCOUNTYNAME': String_VaryingAlphanum(_notes=[]), 'MONTHSSINCEMORTGAGEDEFAULTRELEASE': Decimal(_notes=[], precision=0), 'CONSOLIDATIONMONTHLYSAVINGSAMT': Decimal(_notes=[],
    precision=2), 'LASTUSED': Date(_notes=[], precision=6, rounds=False), 'CASHOUTFLAG': Decimal(_notes=[], precision=0), 'MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE': Decimal(_notes=[], precision=0), 'CORELOGICDATAID': Decimal(_notes=[], precision=0), 'ISNYBURROUGH': Decimal(_notes=[], precision=0), 'MONTHSSINCEMORTGAGEMODIFICATION':
    Decimal(_notes=[], precision=0)}
    WARNING Using reduced precision Timestamp(_notes=[], precision=0, rounds=True) for column 'lastused'. Types=Timestamp(_notes=[], precision=0, rounds=True), Date(_notes=[], precision=6, rounds=False) hashdiff_tables.py:102
    WARNING [MsSQL] Column 'referencenr' of type 'Text(_notes=[])' has no compatibility handling. If encoding/formatting differs between databases, it may result in false positives. hashdiff_tables.py:125
    WARNING [Snowflake] Column 'referencenr' of type 'Text(_notes=[])' has no compatibility handling. If encoding/formatting differs between databases, it may result in false positives. hashdiff_tables.py:125
    DEBUG Running SQL (MsSQL): base.py:962
    SELECT CONVERT(varchar, min([unique_id])), CONVERT(varchar, max([unique_id])) FROM [UNUSEDDATA]
    DEBUG Running SQL (Snowflake): base.py:962
    SELECT cast(min("UNIQUE_ID") as string), cast(max("UNIQUE_ID") as string) FROM "COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE"
    ERROR Cannot apply [String_VaryingAlphanum(_notes=[])] to '('#A374300001155127',)', '('ZZ14489900036138057',)'. main.py:348
    Traceback (most recent call last):
    File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 334, in _parse_key_range_result
    min_key = Vector(key_type.make_value(mn) for key_type, mn in safezip(key_types, min_key_values))
    File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 334, in
    min_key = Vector(key_type.make_value(mn) for key_type, mn in safezip(key_types, min_key_values))
    File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\abcs\database_types.py", line 148, in make_value
    return self.python_type(value)
    File "", line 5, in init
    File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\utils.py", line 193, in attrs_post_init
    raise ValueError(f"Unexpected character {ch} in alphanum string")
    ValueError: Unexpected character # in alphanum string

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts\data-diff.exe_main.py", line 7, in
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 783, in invoke
return _callback(*args, **kwargs)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff_main.py", line 344, in main
return data_diff(
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff_main.py", line 539, in _data_diff
for op, values in diff_iter:
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 93, in iter
for i in self.diff:
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 249, in _diff_tables_wrapper
raise error
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 222, in _diff_tables_wrapper
yield from self._diff_tables_root(table1, table2, info_tree)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 255, in _diff_tables_root
return self._bisect_and_diff_tables(table1, table2, info_tree)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 290, in _bisect_and_diff_tables
min_key1, max_key1 = self._parse_key_range_result(key_types1, next(key_ranges))
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 337, in _parse_key_range_result
raise type(e)(f"Cannot apply {key_types} to '{min_key_values}', '{max_key_values}'.") from e
ValueError: Cannot apply [String_VaryingAlphanum(_notes=[])] to '('#A374300001155127',)', '('ZZ14489900036138057',)'.

  • Run data-diff with the -d switch for extra debug information.

If possible, please paste these as text, and not a screenshot.

Describe the environment
Windows 11, version 23H2
data-diff v0.10.1
Describe which OS you're using, which data-diff version, and any other information that might be relevant to this bug.

lokofoko commented

If I try and manually add code to include # symbol in alphanum I still got an error, but different one
image

data-diff --conf /c/Users/Vitalii/WorkRepos/data_diff_configs/datadiff.toml --run compare_unuseddata_sqlserver_snowflake -k 'unique_id' -c % -l 1000 -m test_results_%t -d --no-tracking --table-write-limit 50000000
INFO [MsSQL] Starting a threadpool, size=1. base.py:1197
DEBUG Database 'MsSQL(default_schema='dbo', _interactive=False, is_closed=False, _dialect=Dialect(_prevent_overflow_when_concat=False), thread_count=1, _init_error=None, _queue=<concurrent.futures.thread.ThreadPoolExecutor object at 0x00000264EADC0580>, thread_local=<_thread._local object at 0x00000264EAEB7880>, _connect.py:300
15:38:41 DEBUG Attempting to acquire lock 2632470653696 on C:\Users\Vitalii\AppData\Local\Snowflake\Caches\ocsp_response_validation_cache.lock _api.py:254
DEBUG Lock 2632470653696 acquired on C:\Users\Vitalii\AppData\Local\Snowflake\Caches\ocsp_response_validation_cache.lock _api.py:257
DEBUG Attempting to release lock 2632470653696 on C:\Users\Vitalii\AppData\Local\Snowflake\Caches\ocsp_response_validation_cache.lock _api.py:286
DEBUG Lock 2632470653696 released on C:\Users\Vitalii\AppData\Local\Snowflake\Caches\ocsp_response_validation_cache.lock _api.py:289
DEBUG Running SQL (Snowflake): base.py:962
ALTER SESSION SET TIMEZONE = 'UTC'
DEBUG Running SQL (MsSQL): ('UNUSEDDATA',) base.py:960
SELECT column_name, data_type, datetime_precision, numeric_precision, numeric_scale FROM [MonsterDataGeneration].information_schema.columns WHERE table_name = 'UNUSEDDATA' AND table_schema = 'dbo'
DEBUG Running SQL (Snowflake): ('COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE',) base.py:960
SELECT column_name, data_type, datetime_precision, numeric_precision, numeric_scale FROM information_schema.columns WHERE table_name = 'COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE' AND table_schema = 'DATA_INTEGRATIONS'
15:38:43 INFO [MsSQL] Schema = {'CorelogicDataID': ('CorelogicDataID', 'int', None, 10, 0), 'Appreciation': ('Appreciation', 'decimal', None, 18, 5), 'AVM': ('AVM', 'decimal', None, 18, 5), 'CountyId': ('CountyId', 'int', None, 10, 0), 'Estimated CLTV': ('Estimated CLTV', 'decimal', None, 18, 5), 'First MTG Amount': ('First MTG Amount', 'money', schema.py:12
None, 19, 4), 'First MTG Assigned Lender': ('First MTG Assigned Lender', 'varchar', None, None, None), 'First MTG Fixed Rate Loan Ind': ('First MTG Fixed Rate Loan Ind', 'bit', None, None, None), 'First MTG Lender Name': ('First MTG Lender Name', 'varchar', None, None, None), 'First MTG Loan Type': ('First MTG Loan Type', 'varchar',
None, None, None), 'First MTG LTV': ('First MTG LTV', 'decimal', None, 18, 5), 'First MTG Origination Date': ('First MTG Origination Date', 'date', 0, None, None), 'First MTG Refinance Loan Ind': ('First MTG Refinance Loan Ind', 'bit', None, None, None), 'First MTG Variable Rate Loan Ind': ('First MTG Variable Rate Loan Ind', 'bit',
None, None, None), 'Junior MTG Amount': ('Junior MTG Amount', 'money', None, 19, 4), 'Junior MTG Assigned Lender': ('Junior MTG Assigned Lender', 'varchar', None, None, None), 'Junior MTG Lender Name': ('Junior MTG Lender Name', 'varchar', None, None, None), 'Junior MTG Origination Date': ('Junior MTG Origination Date', 'date', 0,
None, None), 'Junior MTG Subordinate Type': ('Junior MTG Subordinate Type', 'varchar', None, None, None), 'LandUseDescriptionID': ('LandUseDescriptionID', 'int', None, 10, 0), 'LastUsed': ('LastUsed', 'date', 0, None, None), 'Mailing Address Full': ('Mailing Address Full', 'varchar', None, None, None), 'Mailing City': ('Mailing City',
'varchar', None, None, None), 'Mailing State': ('Mailing State', 'varchar', None, None, None), 'Mailing Zip Code': ('Mailing Zip Code', 'varchar', None, None, None), 'MailingZipCodeId': ('MailingZipCodeId', 'int', None, 10, 0), 'Owner 1 First Name': ('Owner 1 First Name', 'varchar', None, None, None), 'Owner 1 Last Name': ('Owner 1
Last Name', 'varchar', None, None, None), 'Owner 2 First Name': ('Owner 2 First Name', 'varchar', None, None, None), 'Owner 2 Last Name': ('Owner 2 Last Name', 'varchar', None, None, None), 'Property Address Full': ('Property Address Full', 'varchar', None, None, None), 'Property City': ('Property City', 'varchar', None, None, None),
'Property State': ('Property State', 'varchar', None, None, None), 'Property Zip Code': ('Property Zip Code', 'varchar', None, None, None), 'PropertyStateId': ('PropertyStateId', 'int', None, 10, 0), 'PropertyZipCodeId': ('PropertyZipCodeId', 'int', None, 10, 0), 'ReferenceNr': ('ReferenceNr', 'varchar', None, None, None), 'Tax
Amount': ('Tax Amount', 'varchar', None, None, None), 'Estimated Equity': ('Estimated Equity', 'varchar', None, None, None), 'First MTG ARM Next Reset Date': ('First MTG ARM Next Reset Date', 'date', 0, None, None), 'First MTG ARM Maximum Interest Rate': ('First MTG ARM Maximum Interest Rate', 'decimal', None, 18, 2), 'Junior MTG
Equity Loan Ind': ('Junior MTG Equity Loan Ind', 'bit', None, None, None), 'FormattedCountyName': ('FormattedCountyName', 'varchar', None, None, None), 'MailingStateId': ('MailingStateId', 'int', None, 10, 0), 'County Name': ('County Name', 'varchar', None, None, None), 'First MTG ARM Change Percent Limit': ('First MTG ARM Change
Percent Limit', 'decimal', None, 18, 2), 'PrevailingInterestRate': ('PrevailingInterestRate', 'decimal', None, 18, 2), 'First MTG ARM Initial Reset Date': ('First MTG ARM Initial Reset Date', 'date', 0, None, None), 'First MTG ARM Change Interval': ('First MTG ARM Change Interval', 'int', None, 10, 0), 'First MTG ARM Change Freq':
('First MTG ARM Change Freq', 'varchar', None, None, None), 'First MTG ARM First Change Max Pct': ('First MTG ARM First Change Max Pct', 'decimal', None, 8, 4), 'First MTG ARM Calculation Change': ('First MTG ARM Calculation Change', 'decimal', None, 8, 4), 'First MTG Interest Rate': ('First MTG Interest Rate', 'decimal', None, 8, 4),
'First MTG Term': ('First MTG Term', 'int', None, 10, 0), 'First MTG Interest Rate Type': ('First MTG Interest Rate Type', 'varchar', None, None, None), 'APN Unformatted': ('APN Unformatted', 'varchar', None, None, None), 'APN Sequence Number': ('APN Sequence Number', 'int', None, 10, 0), 'FIPS': ('FIPS', 'int', None, 10, 0), 'First
MTG Modified Ind': ('First MTG Modified Ind', 'bit', None, None, None), 'Owner Occupied Indicator': ('Owner Occupied Indicator', 'bit', None, None, None), 'First MTG ARM Index Type': ('First MTG ARM Index Type', 'varchar', None, None, None), 'DownPmtAssistFGrant_Ind': ('DownPmtAssistFGrant_Ind', 'bit', None, None, None), 'First MTG
Loan Purpose': ('First MTG Loan Purpose', 'varchar', None, None, None), 'LoanPurposeID': ('LoanPurposeID', 'int', None, 10, 0), 'MonthlySavingQuoteRate': ('MonthlySavingQuoteRate', 'decimal', None, 18, 2), 'ConsolidationMonthlySavingsAmt': ('ConsolidationMonthlySavingsAmt', 'decimal', None, 18, 2), 'ConsolidationMonthlySavingsPercent':
('ConsolidationMonthlySavingsPercent', 'decimal', None, 18, 2), 'NewConfLoanLimit': ('NewConfLoanLimit', 'decimal', None, 18, 2), 'DataProviderID': ('DataProviderID', 'int', None, 10, 0), 'MailingAddressIsLenderAddress': ('MailingAddressIsLenderAddress', 'bit', None, None, None), 'CurrentPrevailingInterestRate':
('CurrentPrevailingInterestRate', 'decimal', None, 18, 2), 'DifferencePrevailingInterestRate': ('DifferencePrevailingInterestRate', 'decimal', None, 18, 2), 'MonthlySavingPrevailingInterestRate': ('MonthlySavingPrevailingInterestRate', 'decimal', None, 18, 2), 'IsNyBurrough': ('IsNyBurrough', 'bit', None, None, None),
'FirstMortgageLoanTypeId': ('FirstMortgageLoanTypeId', 'int', None, 10, 0), 'MlgSalePrice': ('MlgSalePrice', 'decimal', None, 18, 2), 'IsHistoricDirectMailResponder': ('IsHistoricDirectMailResponder', 'bit', None, None, None), 'CashOutFlag': ('CashOutFlag', 'bit', None, None, None), 'RateAndTermReductionFlag':
('RateAndTermReductionFlag', 'bit', None, None, None), 'ConsolidationFlag': ('ConsolidationFlag', 'bit', None, None, None), 'PropensityToTransact': ('PropensityToTransact', 'decimal', None, 18, 2), 'MonthsSinceMortgageDefaultNotice': ('MonthsSinceMortgageDefaultNotice', 'int', None, 10, 0), 'MonthsSinceMortgageDefaultRelease':
('MonthsSinceMortgageDefaultRelease', 'int', None, 10, 0), 'MonthsSinceMortgageModification': ('MonthsSinceMortgageModification', 'int', None, 10, 0), 'MonthsSinceMortgageBankruptcyDischarge': ('MonthsSinceMortgageBankruptcyDischarge', 'int', None, 10, 0), 'IsReverseMortgage': ('IsReverseMortgage', 'bit', None, None, None),
'IsHistoricPurlResponder': ('IsHistoricPurlResponder', 'bit', None, None, None), 'IsHistoricQrResponder': ('IsHistoricQrResponder', 'bit', None, None, None), 'IsHistoricPhoneResponder': ('IsHistoricPhoneResponder', 'bit', None, None, None), 'MlgEstimatedUnpaidBalance': ('MlgEstimatedUnpaidBalance', 'decimal', None, 18, 5), 'IsSenior':
('IsSenior', 'bit', None, None, None), 'HasHistoricFha': ('HasHistoricFha', 'bit', None, None, None), 'HasRepeatedRefinance': ('HasRepeatedRefinance', 'bit', None, None, None), 'HasVetInHousehold': ('HasVetInHousehold', 'bit', None, None, None), 'MailingOptOutIndicator': ('MailingOptOutIndicator', 'bit', None, None, None),
'MidAvmMlgCltv': ('MidAvmMlgCltv', 'decimal', None, 18, 2), 'unique_id': ('unique_id', 'varchar', None, None, None)}
INFO [Snowflake] Schema = {'MONTHLYSAVINGPREVAILINGINTERESTRATE': ('MONTHLYSAVINGPREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'FIRST_MTG_FIXED_RATE_LOAN_IND': ('FIRST_MTG_FIXED_RATE_LOAN_IND', 'BOOLEAN', None, None, None), 'JUNIOR_MTG_LENDER_NAME': ('JUNIOR_MTG_LENDER_NAME', 'TEXT', None, None, None), 'FIRST_MTG_ARM_INITIAL_RESET_DATE': schema.py:12
('FIRST_MTG_ARM_INITIAL_RESET_DATE', 'DATE', None, None, None), 'FIRST_MTG_ORIGINATION_DATE': ('FIRST_MTG_ORIGINATION_DATE', 'DATE', None, None, None), 'MAILINGADDRESSISLENDERADDRESS': ('MAILINGADDRESSISLENDERADDRESS', 'NUMBER', None, 1, 0), 'JUNIOR_MTG_ASSIGNED_LENDER': ('JUNIOR_MTG_ASSIGNED_LENDER', 'TEXT', None, None, None),
'MAILING_STATE': ('MAILING_STATE', 'TEXT', None, None, None), 'PROPERTY_CITY': ('PROPERTY_CITY', 'TEXT', None, None, None), 'MONTHLYSAVINGQUOTERATE': ('MONTHLYSAVINGQUOTERATE', 'NUMBER', None, 18, 2), 'FIRST_MTG_AMOUNT': ('FIRST_MTG_AMOUNT', 'NUMBER', None, 38, 4), 'HASHISTORICFHA': ('HASHISTORICFHA', 'NUMBER', None, 1, 0),
'FIRST_MTG_LTV': ('FIRST_MTG_LTV', 'NUMBER', None, 18, 5), 'CURRENTPREVAILINGINTERESTRATE': ('CURRENTPREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'FIRST_MTG_MODIFIED_IND': ('FIRST_MTG_MODIFIED_IND', 'BOOLEAN', None, None, None), 'FIPS': ('FIPS', 'NUMBER', None, 38, 0), 'MAILING_ZIP_CODE': ('MAILING_ZIP_CODE', 'TEXT', None, None,
None), 'AVM': ('AVM', 'NUMBER', None, 18, 5), 'REFERENCENR': ('REFERENCENR', 'TEXT', None, None, None), 'FIRST_MTG_INTEREST_RATE': ('FIRST_MTG_INTEREST_RATE', 'NUMBER', None, 8, 4), 'MLG_SALE_PRICE': ('MLG_SALE_PRICE', 'NUMBER', None, 18, 2), 'ESTIMATED_EQUITY': ('ESTIMATED_EQUITY', 'TEXT', None, None, None),
'JUNIOR_MTG_SUBORDINATE_TYPE': ('JUNIOR_MTG_SUBORDINATE_TYPE', 'TEXT', None, None, None), 'PROPERTYZIPCODEID': ('PROPERTYZIPCODEID', 'NUMBER', None, 38, 0), 'FIRST_MTG_ASSIGNED_LENDER': ('FIRST_MTG_ASSIGNED_LENDER', 'TEXT', None, None, None), 'OWNER_2_LAST_NAME': ('OWNER_2_LAST_NAME', 'TEXT', None, None, None), 'PROPENSITYTOTRANSACT':
('PROPENSITYTOTRANSACT', 'NUMBER', None, 18, 2), 'DOWNPMTASSISTFGRANT_IND': ('DOWNPMTASSISTFGRANT_IND', 'NUMBER', None, 1, 0), 'MONTHSSINCEMORTGAGEDEFAULTNOTICE': ('MONTHSSINCEMORTGAGEDEFAULTNOTICE', 'NUMBER', None, 38, 0), 'APN_SEQUENCE_NUMBER': ('APN_SEQUENCE_NUMBER', 'NUMBER', None, 38, 0), 'NEWCONFLOANLIMIT': ('NEWCONFLOANLIMIT',
'NUMBER', None, 18, 2), 'COUNTYID': ('COUNTYID', 'NUMBER', None, 38, 0), 'RATEANDTERMREDUCTIONFLAG': ('RATEANDTERMREDUCTIONFLAG', 'NUMBER', None, 1, 0), 'MAILINGSTATEID': ('MAILINGSTATEID', 'NUMBER', None, 38, 0), 'HASREPEATEDREFINANCE': ('HASREPEATEDREFINANCE', 'NUMBER', None, 1, 0), 'OWNER_1_LAST_NAME': ('OWNER_1_LAST_NAME', 'TEXT',
None, None, None), 'FIRST_MTG_ARM_CALCULATION_CHANGE': ('FIRST_MTG_ARM_CALCULATION_CHANGE', 'NUMBER', None, 8, 4), 'PREVAILINGINTERESTRATE': ('PREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'MAILINGOPTOUTINDICATOR': ('MAILINGOPTOUTINDICATOR', 'NUMBER', None, 1, 0), 'PROPERTY_ZIP_CODE': ('PROPERTY_ZIP_CODE', 'TEXT', None, None, None),
'APN_UNFORMATTED': ('APN_UNFORMATTED', 'TEXT', None, None, None), 'ISREVERSEMORTGAGE': ('ISREVERSEMORTGAGE', 'NUMBER', None, 1, 0), 'PROPERTY_STATE': ('PROPERTY_STATE', 'TEXT', None, None, None), 'JUNIOR_MTG_ORIGINATION_DATE': ('JUNIOR_MTG_ORIGINATION_DATE', 'DATE', None, None, None), 'LANDUSEDESCRIPTIONID': ('LANDUSEDESCRIPTIONID',
'NUMBER', None, 38, 0), 'FIRST_MTG_ARM_INDEX_TYPE': ('FIRST_MTG_ARM_INDEX_TYPE', 'TEXT', None, None, None), 'PROPERTYSTATEID': ('PROPERTYSTATEID', 'NUMBER', None, 38, 0), 'OWNER_1_FIRST_NAME': ('OWNER_1_FIRST_NAME', 'TEXT', None, None, None), 'HASVETINHOUSEHOLD': ('HASVETINHOUSEHOLD', 'NUMBER', None, 1, 0), 'LOANPURPOSEID':
('LOANPURPOSEID', 'NUMBER', None, 38, 0), 'DIFFERENCEPREVAILINGINTERESTRATE': ('DIFFERENCEPREVAILINGINTERESTRATE', 'NUMBER', None, 18, 2), 'APPRECIATION': ('APPRECIATION', 'NUMBER', None, 18, 5), 'ISHISTORICDIRECTMAILRESPONDER': ('ISHISTORICDIRECTMAILRESPONDER', 'NUMBER', None, 1, 0), 'FIRST_MTG_INTEREST_RATE_TYPE':
('FIRST_MTG_INTEREST_RATE_TYPE', 'TEXT', None, None, None), 'ISHISTORICPURLRESPONDER': ('ISHISTORICPURLRESPONDER', 'NUMBER', None, 1, 0), 'ISSENIOR': ('ISSENIOR', 'NUMBER', None, 1, 0), 'JUNIOR_MTG_EQUITY_LOAN_IND': ('JUNIOR_MTG_EQUITY_LOAN_IND', 'BOOLEAN', None, None, None), 'UNIQUE_ID': ('UNIQUE_ID', 'TEXT', None, None, None),
'ISHISTORICPHONERESPONDER': ('ISHISTORICPHONERESPONDER', 'NUMBER', None, 1, 0), 'MAILING_ADDRESS_FULL': ('MAILING_ADDRESS_FULL', 'TEXT', None, None, None), 'TAX_AMOUNT': ('TAX_AMOUNT', 'TEXT', None, None, None), 'MIDAVMMLGCLTV': ('MIDAVMMLGCLTV', 'NUMBER', None, 18, 2), 'ISHISTORICQRRESPONDER': ('ISHISTORICQRRESPONDER', 'NUMBER', None,
1, 0), 'FIRST_MTG_ARM_CHANGE_PERCENT_LIMIT': ('FIRST_MTG_ARM_CHANGE_PERCENT_LIMIT', 'NUMBER', None, 18, 2), 'MLGESTIMATEDUNPAIDBALANCE': ('MLGESTIMATEDUNPAIDBALANCE', 'NUMBER', None, 18, 5), 'DATAPROVIDERID': ('DATAPROVIDERID', 'NUMBER', None, 38, 0), 'FIRST_MTG_TERM': ('FIRST_MTG_TERM', 'NUMBER', None, 38, 0), 'ESTIMATED_CLTV':
('ESTIMATED_CLTV', 'NUMBER', None, 18, 5), 'CONSOLIDATIONFLAG': ('CONSOLIDATIONFLAG', 'NUMBER', None, 1, 0), 'OWNER_2_FIRST_NAME': ('OWNER_2_FIRST_NAME', 'TEXT', None, None, None), 'MAILINGZIPCODEID': ('MAILINGZIPCODEID', 'NUMBER', None, 38, 0), 'FIRST_MTG_REFINANCE_LOAN_IND': ('FIRST_MTG_REFINANCE_LOAN_IND', 'BOOLEAN', None, None,
None), 'FIRST_MTG_ARM_FIRST_CHANGE_MAX_PCT': ('FIRST_MTG_ARM_FIRST_CHANGE_MAX_PCT', 'NUMBER', None, 8, 4), 'CONSOLIDATIONMONTHLYSAVINGSPERCENT': ('CONSOLIDATIONMONTHLYSAVINGSPERCENT', 'NUMBER', None, 18, 2), 'FIRST_MTG_LOAN_PURPOSE': ('FIRST_MTG_LOAN_PURPOSE', 'TEXT', None, None, None), 'FIRST_MTG_VARIABLE_RATE_LOAN_IND':
('FIRST_MTG_VARIABLE_RATE_LOAN_IND', 'BOOLEAN', None, None, None), 'FORMATTEDCOUNTYNAME': ('FORMATTEDCOUNTYNAME', 'TEXT', None, None, None), 'FIRST_MTG_LOAN_TYPE': ('FIRST_MTG_LOAN_TYPE', 'TEXT', None, None, None), 'CONSOLIDATIONMONTHLYSAVINGSAMT': ('CONSOLIDATIONMONTHLYSAVINGSAMT', 'NUMBER', None, 18, 2),
'MONTHSSINCEMORTGAGEDEFAULTRELEASE': ('MONTHSSINCEMORTGAGEDEFAULTRELEASE', 'NUMBER', None, 38, 0), 'MAILING_CITY': ('MAILING_CITY', 'TEXT', None, None, None), 'FIRST_MTG_ARM_MAXIMUM_INTEREST_RATE': ('FIRST_MTG_ARM_MAXIMUM_INTEREST_RATE', 'NUMBER', None, 18, 2), 'LASTUSED': ('LASTUSED', 'DATE', None, None, None), 'CASHOUTFLAG':
('CASHOUTFLAG', 'NUMBER', None, 1, 0), 'MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE': ('MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE', 'NUMBER', None, 38, 0), 'OWNER_OCCUPIED_INDICATOR': ('OWNER_OCCUPIED_INDICATOR', 'NUMBER', None, 1, 0), 'FIRST_MTG_ARM_NEXT_RESET_DATE': ('FIRST_MTG_ARM_NEXT_RESET_DATE', 'DATE', None, None, None),
'FIRST_MTG_ARM_CHANGE_INTERVAL': ('FIRST_MTG_ARM_CHANGE_INTERVAL', 'NUMBER', None, 38, 0), 'JUNIOR_MTG_AMOUNT': ('JUNIOR_MTG_AMOUNT', 'NUMBER', None, 38, 4), 'CORELOGICDATAID': ('CORELOGICDATAID', 'NUMBER', None, 38, 0), 'FIRST_MTG_ARM_CHANGE_FREQ': ('FIRST_MTG_ARM_CHANGE_FREQ', 'TEXT', None, None, None), 'ISNYBURROUGH':
('ISNYBURROUGH', 'NUMBER', None, 1, 0), 'PROPERTY_ADDRESS_FULL': ('PROPERTY_ADDRESS_FULL', 'TEXT', None, None, None), 'FIRST_MTG_LENDER_NAME': ('FIRST_MTG_LENDER_NAME', 'TEXT', None, None, None), 'COUNTY_NAME': ('COUNTY_NAME', 'TEXT', None, None, None), 'MONTHSSINCEMORTGAGEMODIFICATION': ('MONTHSSINCEMORTGAGEMODIFICATION', 'NUMBER',
None, 38, 0)}
DEBUG Available mutual columns: {'mailingzipcodeid', 'avm', 'currentprevailinginterestrate', 'ishistoricpurlresponder', 'appreciation', 'monthlysavingquoterate', 'mailingaddressislenderaddress', 'newconfloanlimit', 'cashoutflag', 'hasvetinhousehold', 'isreversemortgage', 'lastused', 'ishistoricphoneresponder', 'hashistoricfha', main.py:489
'mailingoptoutindicator', 'monthssincemortgagedefaultrelease', 'monthssincemortgagemodification', 'isnyburrough', 'formattedcountyname', 'differenceprevailinginterestrate', 'monthlysavingprevailinginterestrate', 'loanpurposeid', 'corelogicdataid', 'propensitytotransact', 'dataproviderid', 'mailingstateid', 'consolidationflag',
'rateandtermreductionflag', 'prevailinginterestrate', 'referencenr', 'consolidationmonthlysavingspercent', 'countyid', 'consolidationmonthlysavingsamt', 'propertyzipcodeid', 'monthssincemortgagedefaultnotice', 'ishistoricdirectmailresponder', 'ishistoricqrresponder', 'issenior', 'unique_id', 'midavmmlgcltv',
'downpmtassistfgrant_ind', 'mlgestimatedunpaidbalance', 'landusedescriptionid', 'monthssincemortgagebankruptcydischarge', 'fips', 'hasrepeatedrefinance', 'propertystateid'}
INFO Diffing using columns: key=('unique_id',) update=None extra=('mailingzipcodeid', 'avm', 'currentprevailinginterestrate', 'ishistoricpurlresponder', 'appreciation', 'monthlysavingquoterate', 'mailingaddressislenderaddress', 'newconfloanlimit', 'cashoutflag', 'hasvetinhousehold', 'isreversemortgage', 'lastused', main.py:518
'ishistoricphoneresponder', 'hashistoricfha', 'mailingoptoutindicator', 'monthssincemortgagedefaultrelease', 'monthssincemortgagemodification', 'isnyburrough', 'formattedcountyname', 'differenceprevailinginterestrate', 'monthlysavingprevailinginterestrate', 'loanpurposeid', 'corelogicdataid', 'propensitytotransact',
'dataproviderid', 'mailingstateid', 'consolidationflag', 'rateandtermreductionflag', 'prevailinginterestrate', 'referencenr', 'consolidationmonthlysavingspercent', 'countyid', 'consolidationmonthlysavingsamt', 'propertyzipcodeid', 'monthssincemortgagedefaultnotice', 'ishistoricdirectmailresponder', 'ishistoricqrresponder',
'issenior', 'midavmmlgcltv', 'downpmtassistfgrant_ind', 'mlgestimatedunpaidbalance', 'landusedescriptionid', 'monthssincemortgagebankruptcydischarge', 'fips', 'hasrepeatedrefinance', 'propertystateid').
INFO Using algorithm 'hashdiff'. main.py:519
DEBUG Running SQL (MsSQL): ('UNUSEDDATA',) base.py:960
SELECT TRIM([ReferenceNr]), TRIM([FormattedCountyName]), TRIM([unique_id]) FROM [UNUSEDDATA] ORDER BY 1 OFFSET 0 ROWS FETCH NEXT 64 ROWS ONLY
15:40:29 INFO [MsSQL] Schema = {'CorelogicDataID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'Appreciation': Decimal(_notes=[], precision=5), 'AVM': Decimal(_notes=[], precision=5), 'CountyId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'LandUseDescriptionID': Integer(_notes=[], precision=0, python_type=<class schema.py:12
'int'>), 'LastUsed': Timestamp(_notes=[], precision=0, rounds=True), 'MailingZipCodeId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PropertyStateId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PropertyZipCodeId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'ReferenceNr':
Text(_notes=[]), 'FormattedCountyName': String_VaryingAlphanum(_notes=[]), 'MailingStateId': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'PrevailingInterestRate': Decimal(_notes=[], precision=2), 'FIPS': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'DownPmtAssistFGrant_Ind': Boolean(_notes=[]),
'LoanPurposeID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthlySavingQuoteRate': Decimal(_notes=[], precision=2), 'ConsolidationMonthlySavingsAmt': Decimal(_notes=[], precision=2), 'ConsolidationMonthlySavingsPercent': Decimal(_notes=[], precision=2), 'NewConfLoanLimit': Decimal(_notes=[], precision=2),
'DataProviderID': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MailingAddressIsLenderAddress': Boolean(_notes=[]), 'CurrentPrevailingInterestRate': Decimal(_notes=[], precision=2), 'DifferencePrevailingInterestRate': Decimal(_notes=[], precision=2), 'MonthlySavingPrevailingInterestRate': Decimal(_notes=[], precision=2),
'IsNyBurrough': Boolean(_notes=[]), 'IsHistoricDirectMailResponder': Boolean(_notes=[]), 'CashOutFlag': Boolean(_notes=[]), 'RateAndTermReductionFlag': Boolean(_notes=[]), 'ConsolidationFlag': Boolean(_notes=[]), 'PropensityToTransact': Decimal(_notes=[], precision=2), 'MonthsSinceMortgageDefaultNotice': Integer(_notes=[], precision=0,
python_type=<class 'int'>), 'MonthsSinceMortgageDefaultRelease': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthsSinceMortgageModification': Integer(_notes=[], precision=0, python_type=<class 'int'>), 'MonthsSinceMortgageBankruptcyDischarge': Integer(_notes=[], precision=0, python_type=<class 'int'>),
'IsReverseMortgage': Boolean(_notes=[]), 'IsHistoricPurlResponder': Boolean(_notes=[]), 'IsHistoricQrResponder': Boolean(_notes=[]), 'IsHistoricPhoneResponder': Boolean(_notes=[]), 'MlgEstimatedUnpaidBalance': Decimal(_notes=[], precision=5), 'IsSenior': Boolean(_notes=[]), 'HasHistoricFha': Boolean(_notes=[]), 'HasRepeatedRefinance':
Boolean(_notes=[]), 'HasVetInHousehold': Boolean(_notes=[]), 'MailingOptOutIndicator': Boolean(_notes=[]), 'MidAvmMlgCltv': Decimal(_notes=[], precision=2), 'unique_id': String_VaryingAlphanum(_notes=[])}
DEBUG Running SQL (Snowflake): ('COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE',) base.py:960
SELECT * FROM (SELECT TRIM("REFERENCENR"), TRIM("UNIQUE_ID"), TRIM("FORMATTEDCOUNTYNAME") FROM "COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE") AS LIMITED_SELECT LIMIT 64
15:40:57 WARNING Mixed UUID/Non-UUID values detected in column COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE.UNIQUE_ID, disabling UUID support. base.py:1102
INFO [Snowflake] Schema = {'MONTHLYSAVINGPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2), 'MAILINGADDRESSISLENDERADDRESS': Decimal(_notes=[], precision=0), 'MONTHLYSAVINGQUOTERATE': Decimal(_notes=[], precision=2), 'HASHISTORICFHA': Decimal(_notes=[], precision=0), 'CURRENTPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2), schema.py:12
'FIPS': Decimal(_notes=[], precision=0), 'AVM': Decimal(_notes=[], precision=5), 'REFERENCENR': Text(_notes=[]), 'PROPERTYZIPCODEID': Decimal(_notes=[], precision=0), 'PROPENSITYTOTRANSACT': Decimal(_notes=[], precision=2), 'DOWNPMTASSISTFGRANT_IND': Decimal(_notes=[], precision=0), 'MONTHSSINCEMORTGAGEDEFAULTNOTICE':
Decimal(_notes=[], precision=0), 'NEWCONFLOANLIMIT': Decimal(_notes=[], precision=2), 'COUNTYID': Decimal(_notes=[], precision=0), 'RATEANDTERMREDUCTIONFLAG': Decimal(_notes=[], precision=0), 'MAILINGSTATEID': Decimal(_notes=[], precision=0), 'HASREPEATEDREFINANCE': Decimal(_notes=[], precision=0), 'PREVAILINGINTERESTRATE':
Decimal(_notes=[], precision=2), 'MAILINGOPTOUTINDICATOR': Decimal(_notes=[], precision=0), 'ISREVERSEMORTGAGE': Decimal(_notes=[], precision=0), 'LANDUSEDESCRIPTIONID': Decimal(_notes=[], precision=0), 'PROPERTYSTATEID': Decimal(_notes=[], precision=0), 'HASVETINHOUSEHOLD': Decimal(_notes=[], precision=0), 'LOANPURPOSEID':
Decimal(_notes=[], precision=0), 'DIFFERENCEPREVAILINGINTERESTRATE': Decimal(_notes=[], precision=2), 'APPRECIATION': Decimal(_notes=[], precision=5), 'ISHISTORICDIRECTMAILRESPONDER': Decimal(_notes=[], precision=0), 'ISHISTORICPURLRESPONDER': Decimal(_notes=[], precision=0), 'ISSENIOR': Decimal(_notes=[], precision=0), 'UNIQUE_ID':
String_VaryingAlphanum(_notes=[]), 'ISHISTORICPHONERESPONDER': Decimal(_notes=[], precision=0), 'MIDAVMMLGCLTV': Decimal(_notes=[], precision=2), 'ISHISTORICQRRESPONDER': Decimal(_notes=[], precision=0), 'MLGESTIMATEDUNPAIDBALANCE': Decimal(_notes=[], precision=5), 'DATAPROVIDERID': Decimal(_notes=[], precision=0), 'CONSOLIDATIONFLAG':
Decimal(_notes=[], precision=0), 'MAILINGZIPCODEID': Decimal(_notes=[], precision=0), 'CONSOLIDATIONMONTHLYSAVINGSPERCENT': Decimal(_notes=[], precision=2), 'FORMATTEDCOUNTYNAME': String_VaryingAlphanum(_notes=[]), 'CONSOLIDATIONMONTHLYSAVINGSAMT': Decimal(_notes=[], precision=2), 'MONTHSSINCEMORTGAGEDEFAULTRELEASE': Decimal(_notes=[],
precision=0), 'LASTUSED': Date(_notes=[], precision=6, rounds=False), 'CASHOUTFLAG': Decimal(_notes=[], precision=0), 'MONTHSSINCEMORTGAGEBANKRUPTCYDISCHARGE': Decimal(_notes=[], precision=0), 'CORELOGICDATAID': Decimal(_notes=[], precision=0), 'ISNYBURROUGH': Decimal(_notes=[], precision=0), 'MONTHSSINCEMORTGAGEMODIFICATION':
Decimal(_notes=[], precision=0)}
WARNING Using reduced precision Timestamp(_notes=[], precision=0, rounds=True) for column 'lastused'. Types=Timestamp(_notes=[], precision=0, rounds=True), Date(_notes=[], precision=6, rounds=False) hashdiff_tables.py:102
WARNING [MsSQL] Column 'referencenr' of type 'Text(_notes=[])' has no compatibility handling. If encoding/formatting differs between databases, it may result in false positives. hashdiff_tables.py:125
WARNING [Snowflake] Column 'referencenr' of type 'Text(_notes=[])' has no compatibility handling. If encoding/formatting differs between databases, it may result in false positives. hashdiff_tables.py:125
DEBUG Running SQL (MsSQL): base.py:962
SELECT CONVERT(varchar, min([unique_id])), CONVERT(varchar, max([unique_id])) FROM [UNUSEDDATA]
DEBUG Running SQL (Snowflake): base.py:962
SELECT cast(min("UNIQUE_ID") as string), cast(max("UNIQUE_ID") as string) FROM "COMPARING_UNUSEDDATA_SNOWFLAKE_TABLE"
ERROR main.py:348
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts\data-diff.exe_main.py", line 7, in
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1157, in call
return self.main(*args, **kwargs)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1078, in main
rv = self.invoke(ctx)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\click\core.py", line 783, in invoke
return _callback(*args, **kwargs)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff_main.py", line 344, in main
return data_diff(
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff_main.py", line 539, in _data_diff
for op, values in diff_iter:
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 93, in iter
for i in self.diff:
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 249, in _diff_tables_wrapper
raise error
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 222, in _diff_tables_wrapper
yield from self._diff_tables_root(table1, table2, info_tree)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 255, in _diff_tables_root
return self._bisect_and_diff_tables(table1, table2, info_tree)
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\diff_tables.py", line 296, in _bisect_and_diff_tables
f"size: table1 <= {btable1.approximate_size()}, table2 <= {btable2.approximate_size()}"
File "C:\Users\Vitalii\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\data_diff\table_segment.py", line 274, in approximate_size
assert all(d > 0 for d in diff)
AssertionError

lokofoko commented

@sungchun12 hi there. Could you advise on this issue?

sungchun12 commented

@lokofoko Just a heads up that my team and I don't have capacity to address open issues for now, but if you're able to find a solution, please feel free to open a PR and we can review it!

github-actions commented

This issue has been marked as stale because it has been open for 60 days with no activity. If you would like the issue to remain open, please comment on the issue and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

github-actions commented

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment and it will be reopened for triage.