CSV harvester fails creating table with primary key

2021-07-26 06:08:42,117 SYSINFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] get_harvester_runner -> CsvHarvesterRunner()
2021-07-26 06:08:42,473 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Compare schema not yet implemented...
2021-07-26 06:08:42,473 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Using process sequence `replace`
2021-07-26 06:08:42,475 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Executing steps for bgc_trip
2021-07-26 06:08:42,475 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Dropping table bgc_trip
2021-07-26 06:08:42,561 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Creating table bgc_trip
2021-07-26 06:08:42,607 INFO tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] Rolling back changes
2021-07-26 06:08:42,635 ERROR tasks.BGC[8ed9b2ed-9aa8-4837-bd6a-ccc1ccec4947] InvalidSQLTransactionError: column "t" named in key does not exist
LINE 1: ..._MGM3 numeric,SECCHI_M numeric,SAMPLETYPE varchar,PRIMARY KE...
                                                             ^
Traceback (most recent call last):
  File "/vagrant/src/python-aodncore/aodncore/pipeline/db.py", line 72, in __exec
    self._cur.execute(sql.SQL(statement))
psycopg2.errors.UndefinedColumn: column "t" named in key does not exist
LINE 1: ..._MGM3 numeric,SECCHI_M numeric,SAMPLETYPE varchar,PRIMARY KE...
                                                             ^


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/vagrant/src/python-aodncore/aodncore/pipeline/handlerbase.py", line 1054, in run
    self.trigger(transition['trigger'])
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 859, in _get_trigger
    return event.trigger(model, *args, **kwargs)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 393, in trigger
    return self.machine._process(func)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 1148, in _process
    return trigger()
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 411, in _trigger
    return self._process(event_data)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 420, in _process
    if trans.execute(event_data):
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 268, in execute
    event_data.machine.callbacks(itertools.chain(event_data.machine.before_state_change, self.before), event_data)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 1083, in callbacks
    self.callback(func, event_data)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 1104, in callback
    func(*event_data.args, **event_data.kwargs)
  File "/vagrant/src/python-aodncore/aodncore/pipeline/handlerbase.py", line 783, in _publish
    self._harvest()
  File "/vagrant/src/python-aodncore/aodncore/pipeline/handlerbase.py", line 747, in _harvest
    harvest_runner.run(files_to_harvest)
  File "/vagrant/src/python-aodncore/aodncore/pipeline/steps/harvest.py", line 449, in run
    getattr(conn, task)(step)
  File "/vagrant/src/python-aodncore/aodncore/pipeline/db.py", line 189, in create_table_from_yaml_file
    self.__exec('CREATE TABLE {} ({})'.format(step['name'], ','.join(columns)))
  File "/vagrant/src/python-aodncore/aodncore/pipeline/db.py", line 74, in __exec
    raise InvalidSQLTransactionError(error)
aodncore.pipeline.exceptions.InvalidSQLTransactionError: column "t" named in key does not exist
LINE 1: ..._MGM3 numeric,SECCHI_M numeric,SAMPLETYPE varchar,PRIMARY KE...
                                                             ^

The primary key for this table should be "TRIP_CODE", so maybe it's only getting the first letter for some reason?

@ndhogan FYI this is what I was thinking that might be helpful:

python-aodncore/aodncore/util/misc.py

Line 170 in b002530

def is_nonstring_iterable(sequence):

To be addressed as part of https://github.com/aodn/backlog/issues/3284

Resolved? @mhidas

Yes!