format of the .ind file for vcf2eigenstrat

[Carol-Symbiomics]

Hi!
I'm trying to use your vcf2eigenstrat.py script using the .ind file, which I assumed was a two column file with the individual's name and the pop name associated to it (pop_map-like), but it didn't work

python /home/buitracn/RADseq/tools/gdc/vcf2eigenstrat.py -v spis.296.all.filtered.markers.recode.vcf -i pop_map_spis296.ind -o spis296.all.loci.eigenstrat
[('-v', 'spis.296.all.filtered.markers.recode.vcf'), ('-i', 'pop_map_spis296.ind'), ('-o', 'spis296.all.loci.eigenstrat')] []
-v spis.296.all.filtered.markers.recode.vcf
-i pop_map_spis296.ind
-o spis296.all.loci.eigenstrat
found options:
{'indmap': 'pop_map_spis296.ind', 'indAsPop': False, 'ref': None, 'vcf': 'spis.296.all.filtered.markers.recode.vcf', 'out': 'spis296.all.loci.eigenstrat'}
Traceback (most recent call last):
File "/home/buitracn/RADseq/tools/gdc/vcf2eigenstrat.py", line 138, in
main(options)
File "/home/buitracn/RADseq/tools/gdc/vcf2eigenstrat.py", line 56, in main
pop_map[bits[0]]=bits[2]
IndexError: list index out of range

could you please define how should I format such file in order to work?

Thanks in advance,
Carol

[mathii]

The .ind file should be in eigenstrat format. It has three columns: Individual_Name, Sex, Population. Sex is usually coded as M/F/U. That won't necessarily matter for this conversion, but it may be used in downstream analysis.

[Carol-Symbiomics]

Hi Ian, Thanks for the quick answer! Fortunately, I realized it myself by looking at the format of the .ind file (silly me) Just to let you know I had to manually reformat the .snp file cause the output was in the wrong order. Your script outputs SNP.ID, CHROM, genetic.pos in Morgans (absent from a VCF file), physical.position (in bp=POS in vcf file), REF and ALT alleles But according to EIGENSTRAT it should be: The snp file contains 1 line per SNP. There are 6 columns (last 2 optional): 1st column is chromosome. Use X for X chromosome. Note: SNPs with illegal chromosome values, such as 0, will be removed 2nd column is SNP name 3rd column is genetic position (in Morgans) 4th column is physical position (in bases) Optional 5th and 6th columns are reference and variant alleles. cheers, *Carol Buitrago, Ph.D. Candidate* Red Sea Research Center KAUST - King Abdullah University of Science and Technology Thuwal 23955-6900 , Kingdom of Saudi Arabia Building 2, Level 2, Desk 2246-WS12 Tel: +966 55 4704523 [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> 02/10/18 8:28:08 p. m.

…

On Tue, Oct 2, 2018 at 8:05 PM Iain Mathieson ***@***.***> wrote: The .ind file should be in eigenstrat format. It has three columns: Individual_Name, Sex, Population. Sex is usually coded as M/F/U. That won't necessarily matter for this conversion, but it may be used in downstream analysis. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVS0PPG09gche4l15yokdkF5HyO77lwQks5ug5y5gaJpZM4XEKj8> .

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[mathii]

Hi Carol,

Hm. Where do you find that eigenstrat format definition? I don't think that is correct. See line 45 onwards here: https://github.com/DReichLab/AdmixTools/blob/master/convertf/README.

The snp file contains 1 line per SNP. There are 6 columns (last 2 optional):
1st column is SNP name
2nd column is chromosome. X chromosome is encoded as 23.
...

• Iain

[Carol-Symbiomics]

mmm that is strange.... this is where I found the ordering https://github.com/argriffing/eigensoft/tree/master/CONVERTF But something must be wrong with my file cause the smartpca.perl of eigenstrat is not running properly. sh: line 1: 21829 Segmentation fault (core dumped) smartpca -p spis296.all.loci.eigenstrat.pca.par > spis296.all.loci.eigenstrat.log Cheers *Carol Buitrago, Ph.D. Candidate* Red Sea Research Center KAUST - King Abdullah University of Science and Technology Thuwal 23955-6900 , Kingdom of Saudi Arabia Building 2, Level 2, Desk 2246-WS12 Tel: +966 55 4704523 [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> 02/10/18 8:45:47 p. m.

…

On Tue, Oct 2, 2018 at 8:42 PM Iain Mathieson ***@***.***> wrote: Hi Carol, Hm. Where do you find that eigenstrat format definition? I don't think that is correct. See line 45 onwards here: https://github.com/DReichLab/AdmixTools/blob/master/convertf/README. The snp file contains 1 line per SNP. There are 6 columns (last 2 optional): 1st column is SNP name 2nd column is chromosome. X chromosome is encoded as 23. ... - Iain — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVS0PCbmQeZufLLUqCCV_NHzi6mWgKOhks5ug6VvgaJpZM4XEKj8> .

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[mathii]

The text you quoted refers to ped format, not eigenstrat/ancestrymap. If you use the original format you should be ok.

Note that my script output unpacked (i.e. plain text) eigenstrat/ancestrymap format. If you have a large dataset, it is worth converting it to PACKEDANCESTRYMAP binary format using convertf, which will take up much less space and be more efficient. I should add an option to do this automatically.

[Carol-Symbiomics]

You are right! Thanks a lot for addressing the issue so fast. Unfortunately, I still don't get the smartpca to run, and it is because of the .snp file :( Cheers *Carol Buitrago, Ph.D. Candidate* Red Sea Research Center KAUST - King Abdullah University of Science and Technology Thuwal 23955-6900 , Kingdom of Saudi Arabia Building 2, Level 2, Desk 2246-WS12 Tel: +966 55 4704523 [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality5&> 02/10/18 9:04:32 p. m.

…

On Tue, Oct 2, 2018 at 8:53 PM Iain Mathieson ***@***.***> wrote: The text you quoted refers to ped format, not eigenstrat/ancestrymap. If you use the original format you should be ok. Note that my script output unpacked (i.e. plain text) eigenstrat/ancestrymap format. If you have a large dataset, it is worth converting it to PACKEDANCESTRYMAP binary format using convertf, which will take up much less space and be more efficient. I should add an option to do this automatically. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVS0PK2pwtTbIwhbSzWvlLw_w6GtixTvks5ug6f6gaJpZM4XEKj8> .

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[mathii]

What is the error message in spis296.all.loci.eigenstrat.log?

[Carol-Symbiomics]

This is my command perl /home/buitracn/RADseq/tools/EIG/bin/smartpca.perl -i spis296.all.loci.eigenstrat.geno -a spis296.all.loci.eigenstrat.snp -b spis296.all.loci .eigenstrat.ind -o spis296.all.loci.eigenstrat.pca -p spis296.all.loci.eigenstrat.plot -e spis296.all.loci.eigenstrat.eigval -l spis296.all.loci.eigenstrat.log -k 15 this is the error that gets printed in the terminal smartpca -p spis296.all.loci.eigenstrat.pca.par

spis296.all.loci.eigenstrat.log

sh: line 1: 23916 Segmentation fault (core dumped) smartpca -p spis296.all.loci.eigenstrat.pca.par > spis296.all.loci.eigenstrat.log *this is the content of the .log file* cat spis296.all.loci.eigenstrat.log parameter file: spis296.all.loci.eigenstrat.pca.par ### THE INPUT PARAMETERS ##PARAMETER NAME: VALUE genotypename: spis296.all.loci.eigenstrat.geno snpname: spis296.all.loci.eigenstrat.snp indivname: spis296.all.loci.eigenstrat.ind evecoutname: spis296.all.loci.eigenstrat.pca.evec evaloutname: spis296.all.loci.eigenstrat.eigval altnormstyle: NO numoutevec: 15 numoutlieriter: 5 numoutlierevec: 10 outliersigmathresh: 6 qtmode: 0 ## smartpca version: 16000 norm used *Carol Buitrago, Ph.D. Candidate* Red Sea Research Center KAUST - King Abdullah University of Science and Technology Thuwal 23955-6900 , Kingdom of Saudi Arabia Building 2, Level 2, Desk 2246-WS12 Tel: +966 55 4704523 [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality6&> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality6&> 02/10/18 9:12:31 p. m.

…

On Tue, Oct 2, 2018 at 9:09 PM Iain Mathieson ***@***.***> wrote: What is the error message in spis296.all.loci.eigenstrat.log? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVS0PIeOaRf3Cswmr0yg5_WJB_3hZ1_Lks5ug6u3gaJpZM4XEKj8> .

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[mathii]

Can't really see what is going on from that. If you are able to share your .snp .ind and .geno files, I could take a look and see if I can figure it out - you can email them to me. Or, if you can post the first few lines of each file, it might be clear what's going on.

[Carol-Symbiomics]

Hi Ian, What email address can I use to send you the files? It seems to be a problem of the .SNP file. I work with corals, so the reference genome I used isn't chromosome arranged but instead I have a large number of scaffolds. Apparently EIGENSTRAT has a threshold of 22 Chrom. Cheers El mar., 2 de oct. de 2018 9:27 PM, Iain Mathieson <notifications@github.com> escribió:

…

Can't really see what is going on from that. If you are able to share your .snp .ind and .geno files, I could take a look and see if I can figure it out - you can email them to me. Or, if you can post the first few lines of each file, it might be clear what's going on. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVS0PA2Jp54jwezJB6Bn4HOid3EANP-sks5ug7AJgaJpZM4XEKj8> .

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[mathii]

You can email me at mathi@pennmedicine.upenn.edu. Don't send the files if they are too big though.

Try just renumbering all the chromosomes to "1". it shouldn't matter for the purposes of running PCA.