Running Scrapy In Celery Tasks

A practical, production-ready solution for running scrapers as Celery tasks

Photo by Nicolas Picard on Unsplash

Some time ago, I accepted a project to build a portal for commercial users to scrape product details by submitting links to the product listings. The portal was a site built entirely with Python, and Scrapy.

As scraping pages typically takes time, it is often best to run them in the background and keep the…