scrapy-with-selenium

Demo to use Scrapy with Selenium to build web-crawlers/Spiders for modern web applications

Scrapy is a great framework for web crawling/scraping, but its only suitable for crawling a static web pages, which is useless considering the fact that most of the modern web pages are dynamic and most of the content is loaded asynchronously via JS after the page is loaded. Therefore we have a use Selenium beacuse of the advanced DOM navigationg abilities it provides. Here is a simple demo of a spider using Scrapy & Selenium.

Usage

/Demo> scrapy crawl demo

More information

You can see how to use Scrapy & Selenium from their official documentation below:

• [http://doc.scrapy.org/en/latest/index.html] - Scrapy
• [http://selenium-python.readthedocs.org/] -Selenium bindings for python