These are a bunch of tools for dealing with the ISI Web of Science. You can use them to extract, clean, and process article records in the ISI Flat File format.
Some examples of the sorts of work that can be done with this data:
- Kieran Healy: Gender and Citation
- Neal Caren: A Sociology Citation Network
The tools are very alpha at this stage and have a heavy Unix bias. Please submit bug reports and feature requests. I would love to be useful to the wider world.
[kousu@galleon isi]$ ./isi_scrape.py [user name] [library barcode] SU=Sociology PY="2006-2015"
In using this to download records from the Web of Science, you should be aware of the terms of service:
Thomson Reuters determines reasonable of data to download by comparing your download activity
against the average annual download rates for all Thomson Reuters clients using the product in question.
Thomson Reuters determines insubstantial of downloaded data to mean an amount of data taken
from the product which (1) would not have significant commercial value of its own; and (2) would not act
as a substitute for access to a Thomson Reuters product for someone who does not have access to the product.
The authors of this software take no responsibility for your use of it. Don't get b&.
Started new UW Library Proxy session 'oUeCh7QJ89pR15H'
Logged into ISI as UW:[user name].
Got 69220 results
Ripping results.
Exporting records [1,501) to 2AiZ7oSbJ2Y7a2MctLA_0001.isi
Exporting records [501,1001) to 2AiZ7oSbJ2Y7a2MctLA_0501.isi
Exporting records [1001,1501) to 2AiZ7oSbJ2Y7a2MctLA_1001.isi
Exporting records [1501,2001) to 2AiZ7oSbJ2Y7a2MctLA_1501.isi
Exporting records [2001,2501) to 2AiZ7oSbJ2Y7a2MctLA_2001.isi
Exporting records [2501,3001) to 2AiZ7oSbJ2Y7a2MctLA_2501.isi
[...]
[kousu@galleon isi]$ ls PY\=2006-2015_SU\=Sociology/
2AiZ7oSbJ2Y7a2MctLA_0001.isi 2AiZ7oSbJ2Y7a2MctLA_1001.isi 2AiZ7oSbJ2Y7a2MctLA_2001.isi parameters.txt
2AiZ7oSbJ2Y7a2MctLA_0501.isi 2AiZ7oSbJ2Y7a2MctLA_1501.isi 2AiZ7oSbJ2Y7a2MctLA_2501.isi [...]
[kousu@galleon isi]$ cat PY\=2006-2015_SU\=Sociology/parameters.txt
ISI scrape
==========
Query: PY=2006-2015 SU=Sociology
Records: 69220
ISI Session: 2AiZ7oSbJ2Y7a2MctLA
Date: 2015-03-12 13:20:38.785762
[kousu@galleon isi]$
A tip:
once you are comfortable with the tool, write yourself a script isi_scrape.logined.sh like this
#!/bin/sh
HERE=$(dirname $0)
$HERE/isi_scrape.py -q <username> <password> "$@"
Verifies the integrity of a scraped corpus.
TODO (not written yet)
Combine separate .isi files into a single file. This is needed for processing with sci^2.
TODO
Counts the number of records in a set of ISI files.
TODO