entity_import fails if last field in any row is empty

[dheiman]

On Wed, Jan 31, 2018 at 12:54 PM, @eddiebroad wrote:

I ran fissfc (v 0.16.7) using the command shown here:

wm8b1-75c:no_question esalinas$ /usr/local/bin/fissfc entity_import -w MiniMutationCalling_V1_Tutorial_copy_for_line_count -p broad-firecloud-eddie -f CLL_NeoVax_GAF3_Individuals.tsv.fc.tsv
2018-01-31 12:32:35::INFO Starting new HTTPS connection (1): accounts.google.com
2018-01-31 12:32:35::INFO Starting new HTTPS connection (1): api.firecloud.org
Error 400 (FireCloud): TSV parsing error in line 0: wrong number of fields
wm8b1-75c:no_question esalinas$ /usr/local/bin/fissfc --version
0.16.7
wm8b1-75c:no_question esalinas$

However, when I upload the same TSV to firecloud via the UI, I do not get this error.

I haven't done much investigation of this.

There seems to be a contradiction because the fissfc upload reports an error from FC
whereas the upload via the UI does not report the error. Do you know if I'm the only one
to have observed this seemingly contradictory behavior? I'd kind of be surprised if yes,
but I don't know.








awk (running in ubuntu:16.04) shows 28 columns everywhere. Line counts match too (2011) and also the column count on the first line.




root@8073c665e938:/mnt/Users/esalinas/bits/fh_to_fc/no_question# cat CLL_NeoVax_GAF3_Individuals.tsv.fc.tsv |awk -F '\t' '{print NF}'|sort|uniq -c
2011 28
root@8073c665e938:/mnt/Users/esalinas/bits/fh_to_fc/no_question# head -1 CLL_NeoVax_GAF3_Individuals.tsv.fc.tsv |tr "\t" "\n"|wc -l
28
root@8073c665e938:/mnt/Users/esalinas/bits/fh_to_fc/no_question# wc -l CLL_NeoVax_GAF3_Individuals.tsv.fc.tsv
2011 CLL_NeoVax_GAF3_Individuals.tsv.fc.tsv

I observed the same similarly contradictory behaviour with another file (for samples) in the same workspace.
The sample file gave the same error message with fissfc, but uploaded with a success message via the UI.

---

On Wed, Jan 31, 2018 at 1:02 PM, @dheiman wrote:

Hi Eddie,

Is the last field empty in some rows of your file? I'm looking at the code for entity_import, and that could definitely cause an issue. We'll put together a hotfix for this ASAP

-David

---

On Wed, Jan 31, 2018 at 1:04 PM, @eddiebroad wrote:

Hi David,

Yes, it appears that the last column is empty yes...

wm8b1-75c:no_question esalinas$ cut -f 28 CLL_NeoVax_GAF3_Individuals.tsv|head
tumor_ABSOLUTE_file


















wm8b1-75c:no_question esalinas$ cut -f 1,2 CLL_NeoVax_GAF3_Individuals.tsv|head
individual_id alias
13240-001
13240-002
13240-003
13240-004
13240-005
13240-006
13240-007
13240-007_FFPE
13240-008
wm8b1-75c:no_question esalinas$ cut -f 29 CLL_NeoVax_GAF3_Individuals.tsv|head
























-eddie