spaCy: Industrial-strength NLP
spaCy is a library for advanced natural language processing in Python and Cython. spaCy is built on the very latest research, but it isn't researchware. It was designed from day one to be used in real products. spaCy currently supports English and German, as well as tokenization for Chinese, Spanish, Italian, French, Portuguese, Dutch, Swedish and Hungarian. It's commercial open-source software, released under the MIT license.
📖 Documentation
Usage Workflows | How to use spaCy and its features. |
API Reference | The detailed reference for spaCy's API. |
Tutorials | End-to-end examples, with code you can modify and run. |
Showcase & Demos | Demos, libraries and products from the spaCy community. |
Contribute | How to contribute to the spaCy project and code base. |
💬 Where to ask questions
Bug reports | GitHub Issue tracker |
Usage questions | StackOverflow, Reddit usergroup, Gitter chat |
General discussion | Reddit usergroup, Gitter chat |
Commercial support | contact@explosion.ai |
Features
- Non-destructive tokenization
- Syntax-driven sentence segmentation
- Pre-trained word vectors
- Part-of-speech tagging
- Named entity recognition
- Labelled dependency parsing
- Convenient string-to-int mapping
- Export to numpy data arrays
- GIL-free multi-threading
- Efficient binary serialization
- Easy deep learning integration
- Statistical models for English and German
- State-of-the-art speed
- Robust, rigorously evaluated accuracy
See facts, figures and benchmarks.
Top Performance
- Fastest in the world: <50ms per document. No faster system has ever been announced.
- Accuracy within 1% of the current state of the art on all tasks performed (parsing, named entity recognition, part-of-speech tagging). The only more accurate systems are an order of magnitude slower or more.
Supports
- CPython 2.6, 2.7, 3.3, 3.4, 3.5 (only 64 bit)
- macOS / OS X
- Linux
- Windows (Cygwin, MinGW, Visual Studio)
Install spaCy
spaCy is compatible with 64-bit CPython 2.6+/3.3+ and runs on Unix/Linux, OS X and Windows. Source packages are available via pip. Please make sure that you have a working build enviroment set up. See notes on Ubuntu, macOS/OS X and Windows for details.
pip
When using pip it is generally recommended to install packages in a virtualenv to avoid modifying system state:
pip install spacy
Python packaging is awkward at the best of times, and it's particularly tricky with C extensions, built via Cython, requiring large data files. So, please report issues as you encounter them.
Install model
After installation you need to download a language model. Currently only models for
English and German, named en
and de
, are available.
python -m spacy.en.download all
python -m spacy.de.download all
The download command fetches about 1 GB of data which it installs
within the spacy
package directory.
Upgrading spaCy
To upgrade spaCy to the latest release:
pip
pip install -U spacy
Sometimes new releases require a new language model. Then you will have to upgrade to a new model, too. You can also force re-downloading and installing a new language model:
python -m spacy.en.download --force
Compile from source
The other way to install spaCy is to clone its GitHub repository and build it from source. That is the common way if you want to make changes to the code base.
You'll need to make sure that you have a development enviroment consisting of a Python distribution including header files, a compiler, pip, virtualenv and git installed. The compiler part is the trickiest. How to do that depends on your system. See notes on Ubuntu, OS X and Windows for details.
# make sure you are using recent pip/virtualenv versions
python -m pip install -U pip virtualenv
# find git install instructions at https://git-scm.com/downloads
git clone https://github.com/explosion/spaCy.git
cd spaCy
virtualenv .env && source .env/bin/activate
pip install -r requirements.txt
pip install -e .
Compared to regular install via pip requirements.txt additionally installs developer dependencies such as cython.
Ubuntu
Install system-level dependencies via apt-get
:
sudo apt-get install build-essential python-dev git
macOS / OS X
Install a recent version of XCode, including the so-called "Command Line Tools". macOS and OS X ship with Python and git preinstalled.
Windows
Install a version of Visual Studio Express or higher that matches the version that was used to compile your Python interpreter. For official distributions these are VS 2008 (Python 2.7), VS 2010 (Python 3.4) and VS 2015 (Python 3.5).
Run tests
spaCy comes with an extensive test suite. First, find out where spaCy is installed:
python -c "import os; import spacy; print(os.path.dirname(spacy.__file__))"
Then run pytest
on that directory. The flags --vectors
, --slow
and --model
are optional and enable additional tests:
# make sure you are using recent pytest version
python -m pip install -U pytest
python -m pytest <spacy-directory> --vectors --model --slow
Download model to custom location
You can specify where spacy.en.download
and spacy.de.download
download the language model
to using the --data-path
or -d
argument:
python -m spacy.en.download all --data-path /some/dir
If you choose to download to a custom location, you will need to tell spaCy where to load the model
from in order to use it. You can do this either by calling spacy.util.set_data_path()
before
calling spacy.load()
, or by passing a path
argument to the spacy.en.English
or
spacy.de.German
constructors.
Changelog
v1.5.0: Alpha support for Swedish and Hungarian
2016-12-27- NEW: Alpha support for Swedish tokenization.
- NEW: Alpha support for Hungarian tokenization.
- Update language data for Spanish tokenization.
- Speed up tokenization when no data is preloaded by caching the first 10,000 vocabulary items seen.
- List the
language_data
package in thesetup.py
. - Fix missing
vec_path
declaration that was failing ifadd_vectors
was set. - Allow
Vocab
to load withoutserializer_freqs
.
- NEW: spaCy Jupyter notebooks repo: ongoing collection of easy-to-run spaCy examples and tutorials.
- Fix issue #657: Generalise dependency parsing annotation specs beyond English.
- Fix various typos and inconsistencies.
Thanks to @oroszgy, @magnusburton, @jmizgajski, @aikramer2, @fnorf and @bhargavvader for the pull requests!
v1.4.0: Improved language data and alpha Dutch support
2016-12-18- NEW: Alpha support for Dutch tokenization.
- Reorganise and improve format for language data.
- Add shared tag map, entity rules, emoticons and punctuation to language data.
- Convert entity rules, morphological rules and lemmatization rules from JSON to Python.
- Update language data for English, German, Spanish, French, Italian and Portuguese.
- Fix issue #649: Update and reorganise stop lists.
- Fix issue #672: Make
token.ent_iob_
return unicode. - Fix issue #674: Add missing lemmas for contracted forms of "be" to
TOKENIZER_EXCEPTIONS
. - Fix issue #683
Morphology
class now supplies tag map value for the special space tag if it's missing. - Fix issue #684: Ensure
spacy.en.English()
loads the Glove vector data if available. Previously was inconsistent with behaviour ofspacy.load('en')
. - Fix issue #685: Expand
TOKENIZER_EXCEPTIONS
with unicode apostrophe (’
). - Fix issue #689: Correct typo in
STOP_WORDS
. - Fix issue #691: Add tokenizer exceptions for "gonna" and "Gonna".
No changes to the public, documented API, but the previously undocumented language data and model initialisation processes have been refactored and reorganised. If you were relying on the bin/init_model.py
script, see the new spaCy Developer Resources repo. Code that references internals of the spacy.en
or spacy.de
packages should also be reviewed before updating to this version.
- NEW: "Adding languages" workflow.
- NEW: "Part-of-speech tagging" workflow.
- NEW: spaCy Developer Resources repo – scripts, tools and resources for developing spaCy.
- Fix various typos and inconsistencies.
Thanks to @dafnevk, @jvdzwaan, @RvanNieuwpoort, @wrvhage, @jaspb, @savvopoulos and @davedwards for the pull requests!
v1.3.0: Improve API consistency
2016-12-03- Add
Span.sentiment
attribute. - #658: Add
Span.noun_chunks
iterator (thanks @pokey). - #642: Let
--data-path
be specified when running download.py scripts (thanks @ExplodingCabbage). - #638: Add German stopwords (thanks @souravsingh).
- #614: Fix
PhraseMatcher
to work with newMatcher
(thanks @sadovnychyi).
- Fix issue #605:
accept
argument toMatcher
now rejects matches as expected. - Fix issue #617:
Vocab.load()
now works with string paths, as well asPath
objects. - Fix issue #639: Stop words in
Language
class now used as expected. - Fix issues #656, #624:
Tokenizer
special-case rules now support arbitrary token attributes.
- Add "Customizing the tokenizer" workflow.
- Add "Training the tagger, parser and entity recognizer" workflow.
- Add "Entity recognition" workflow.
- Fix various typos and inconsistencies.
Thanks to @pokey, @ExplodingCabbage, @souravsingh, @sadovnychyi, @manojsakhwar, @TiagoMRodrigues, @savkov, @pspiegelhalter, @chenb67, @kylepjohnson, @YanhaoYang, @tjrileywisc, @dechov, @wjt, @jsmootiv and @blarghmatey for the pull requests!
v1.2.0: Alpha tokenizers for Chinese, French, Spanish, Italian and Portuguese
2016-11-04- NEW: Support Chinese tokenization, via Jieba.
- NEW: Alpha support for French, Spanish, Italian and Portuguese tokenization.
- Fix issue #376: POS tags for "and/or" are now correct.
- Fix issue #578:
--force
argument on download command now operates correctly. - Fix issue #595: Lemmatization corrected for some base forms.
- Fix issue #588: Matcher now rejects empty patterns.
- Fix issue #592: Added exception rule for tokenization of "Ph.D."
- Fix issue #599: Empty documents now considered tagged and parsed.
- Fix issue #600: Add missing
token.tag
andtoken.tag_
setters. - Fix issue #596: Added missing unicode import when compiling regexes that led to incorrect tokenization.
- Fix issue #587: Resolved bug that caused
Matcher
to sometimes segfault. - Fix issue #429: Ensure missing entity types are added to the entity recognizer.
v1.1.0: Bug fixes and adjustments
2016-10-23- Rename new
pipeline
keyword argument ofspacy.load()
tocreate_pipeline
. - Rename new
vectors
keyword argument ofspacy.load()
toadd_vectors
.
- Fix issue #544: Add
vocab.resize_vectors()
method, to support changing to vectors of different dimensionality. - Fix issue #536: Default probability was incorrect for OOV words.
- Fix issue #539: Unspecified encoding when opening some JSON files.
- Fix issue #541: GloVe vectors were being loaded incorrectly.
- Fix issue #522: Similarities and vector norms were calculated incorrectly.
- Fix issue #461:
ent_iob
attribute was incorrect after setting entities viadoc.ents
- Fix issue #459: Deserialiser failed on empty doc
- Fix issue #514: Serialization failed after adding a new entity label.
v1.0.0: Support for deep learning workflows and entity-aware rule matcher
2016-10-18- NEW: custom processing pipelines, to support deep learning workflows
- NEW: Rule matcher now supports entity IDs and attributes
- NEW: Official/documented training APIs and GoldParse class
- Download and use GloVe vectors by default
- Make it easier to load and unload word vectors
- Improved rule matching functionality
- Move basic data into the code, rather than the json files. This makes it simpler to use the tokenizer without the models installed, and makes adding new languages much easier.
- Replace file-system strings with
Path
objects. You can now load resources over your network, or do similar trickery, by passing any object that supports thePath
protocol.
- The data_dir keyword argument of
Language.__init__
(and its subclassesEnglish.__init__
andGerman.__init__
) has been renamed topath
. - Details of how the Language base-class and its sub-classes are loaded, and how defaults are accessed, have been heavily changed. If you have your own subclasses, you should review the changes.
- The deprecated
token.repvec
name has been removed. - The
.train()
method of Tagger and Parser has been renamed to.update()
- The previously undocumented
GoldParse
class has a new__init__()
method. The old method has been preserved inGoldParse.from_annot_tuples()
. - Previously undocumented details of the
Parser
class have changed. - The previously undocumented
get_package
andget_package_by_name
helper functions have been moved into a new module,spacy.deprecated
, in case you still need them while you update.
- Fix
get_lang_class
bug when GloVe vectors are used. - Fix Issue #411:
doc.sents
raised IndexError on empty string. - Fix Issue #455: Correct lemmatization logic
- Fix Issue #371: Make
Lexeme
objects hashable - Fix Issue #469: Make
noun_chunks
detect root NPs
Thanks to @daylen, @RahulKulhari, @stared, @adamhadani, @izeye and @crawfordcomeaux for the pull requests!
v0.101.0: Fixed German model
2016-05-10- Fixed bug that prevented German parses from being deprojectivised.
- Bug fixes to sentence boundary detection.
- Add rich comparison methods to the Lexeme class.
- Add missing
Doc.has_vector
andSpan.has_vector
properties. - Add missing
Span.sent
property.
v0.100.7: German!
2016-05-05spaCy finally supports another language, in addition to English. We're lucky
to have Wolfgang Seeker on the team, and the new German model is just the
beginning. Now that there are multiple languages, you should consider loading
spaCy via the load()
function. This function also makes it easier to load extra
word vector data for English:
import spacy
en_nlp = spacy.load('en', vectors='en_glove_cc_300_1m_vectors')
de_nlp = spacy.load('de')
To support use of the load function, there are also two new helper functions:
spacy.get_lang_class
and spacy.set_lang_class
. Once the German model is
loaded, you can use it just like the English model:
doc = nlp(u'''Wikipedia ist ein Projekt zum Aufbau einer Enzyklopädie aus freien Inhalten, zu dem du mit deinem Wissen beitragen kannst. Seit Mai 2001 sind 1.936.257 Artikel in deutscher Sprache entstanden.''')
for sent in doc.sents:
print(sent.root.text, sent.root.n_lefts, sent.root.n_rights)
# (u'ist', 1, 2)
# (u'sind', 1, 3)
The German model provides tokenization, POS tagging, sentence boundary detection, syntactic dependency parsing, recognition of organisation, location and person entities, and word vector representations trained on a mix of open subtitles and Wikipedia data. It doesn't yet provide lemmatisation or morphological analysis, and it doesn't yet recognise numeric entities such as numbers and dates.
Bugfixes
- spaCy < 0.100.7 had a bug in the semantics of the
Token.__str__
andToken.__unicode__
built-ins: they included a trailing space. - Improve handling of "infixed" hyphens. Previously the tokenizer struggled with multiple hyphens, such as "well-to-do".
- Improve handling of periods after mixed-case tokens
- Improve lemmatization for English special-case tokens
- Fix bug that allowed spaces to be treated as heads in the syntactic parse
- Fix bug that led to inconsistent sentence boundaries before and after serialisation.
- Fix bug from deserialising untagged documents.
v0.100.6: Add support for GloVe vectors
2016-03-08This release offers improved support for replacing the word vectors used by spaCy. To install Stanford's GloVe vectors, trained on the Common Crawl, just run:
sputnik --name spacy install en_glove_cc_300_1m_vectors
To reduce memory usage and loading time, we've trimmed the vocabulary down to 1m entries.
This release also integrates all the code necessary for German parsing. A German model
will be released shortly. To assist in multi-lingual processing, we've added a load()
function. To load the English model with the GloVe vectors:
spacy.load('en', vectors='en_glove_cc_300_1m_vectors')
v0.100.5
2016-02-07Fix incorrect use of header file, caused from problem with thinc
v0.100.4: Fix OSX problem introduced in 0.100.3
2016-02-07Small correction to right_edge calculation
v0.100.3
2016-02-06Support multi-threading, via the .pipe
method. spaCy now releases the GIL around the
parser and entity recognizer, so systems that support OpenMP should be able to do
shared memory parallelism at close to full efficiency.
We've also greatly reduced loading time, and fixed a number of bugs.
v0.100.2
2016-01-21Fix data version lock that affected v0.100.1
v0.100.1: Fix install for OSX
2016-01-21v0.100 included header files built on Linux that caused installation to fail on OSX. This should now be corrected. We also update the default data distribution, to include a small fix to the tokenizer.
v0.100: Revise setup.py, better model downloads, bug fixes
2016-01-19- Redo setup.py, and remove ugly headers_workaround hack. Should result in fewer install problems.
- Update data downloading and installation functionality, by migrating to the Sputnik data-package manager. This will allow us to offer finer grained control of data installation in future.
- Fix bug when using custom entity types in
Matcher
. This should work by default when using theEnglish.__call__
method of running the pipeline. If invokingParser.__call__
directly to do NER, you should call theParser.add_label()
method to register your entity type. - Fix head-finding rules in
Span
. - Fix problem that caused
doc.merge()
to sometimes hang - Fix problems in handling of whitespace
v0.99: Improve span merging, internal refactoring
2015-11-08- Merging multi-word tokens into one, via the
doc.merge()
andspan.merge()
methods, no longer invalidates existingSpan
objects. This makes it much easier to merge multiple spans, e.g. to merge all named entities, or all base noun phrases. Thanks to @andreasgrv for help on this patch. - Lots of internal refactoring, especially around the machine learning module, thinc. The thinc API has now been improved, and the spacy._ml wrapper module is no longer necessary.
- The lemmatizer now lower-cases non-noun, noun-verb and non-adjective words.
- A new attribute,
.rank
, is added to Token and Lexeme objects, giving the frequency rank of the word.
v0.98: Smaller package, bug fixes
2015-11-03- Remove binary data from PyPi package.
- Delete archive after downloading data
- Use updated cymem, preshed and thinc packages
- Fix information loss in deserialize
- Fix
__str__
methods for Python2
v0.97: Load the StringStore from a json list, instead of a text file
2015-10-23- Fix bugs in download.py
- Require
--force
to over-write the data directory in download.py - Fix bugs in
Matcher
anddoc.merge()
v0.96: Hotfix to .merge method
2015-10-19- Fix bug that caused text to be lost after
.merge
- Fix bug in Matcher when matched entities overlapped
v0.95: Bugfixes
2015-10-18- Reform encoding of symbols
- Fix bugs in
Matcher
- Fix bugs in
Span
- Add tokenizer rule to fix numeric range tokenization
- Add specific string-length cap in Tokenizer
- Fix
token.conjuncts
v0.94
2015-10-09- Fix memory error that caused crashes on 32bit platforms
- Fix parse errors caused by smart quotes and em-dashes
v0.93
2015-09-22Bug fixes to word vectors