/scrapybook

Scrapy Book Code

Primary LanguagePython

Learning Scrapy Book

This book covers the long awaited Scrapy v 1.0 that empowers you to extract useful data from virtually any source with very little effort. It starts off by explaining the fundamentals of Scrapy framework, followed by a thorough description of how to extract data from any source, clean it up, shape it as per your requirement using Python and 3rd party APIs. Next you will be familiarised with the process of storing the scrapped data in databases as well as search engines and performing real time analytics on them with Spark Streaming. By the end of this book, you will perfect the art of scraping data for your applications with ease.

This book is now available on Amazon and Packt.

What you will learn

  • Understand HTML pages and write XPath to extract the data you need
  • Write Scrapy spiders with simple Python and do web crawls
  • Push your data into any database, search engine or analytics system
  • Configure your spider to download files, images and use proxies
  • Create efficient pipelines that shape data in precisely the form you want
  • Use Twisted Asynchronous API to process hundreds of items concurrently
  • Make your crawler super-fast by learning how to tune Scrapy's performance
  • Perform large scale distributed crawls with scrapyd and scrapinghub

Tutorials

  • How to Setup Software and Run Examples On A Windows Machine

image

  • Chapter 4 - Create Appery.io mobile application - Updated process

image

  • Chapter 3 & 9 on a 32-bit VM (for computers limited memory/processing power)

image

To use Docker directly without installing Vagrant

A docker-compose.yml file is included, mainly for those who already have Docker installed. For completeness, here are the links to go about installing Docker.

Once you have Docker installed and started, change to the project directory and run:

  1. docker-compose pull - To check for updated images
  2. docker-compose up - Will scroll log messages as various containers (virtual machines) start up. To stop the containers, Ctrl-C in this window, or enter docker-compose down in another shell window.

docker system prune will delete the system-wide Docker images, containers, and volumes that are not in use when you want to recover space.

See also the official website