GitHub - Decodo/python-scrapy-amazon: Scrape Amazon product listings utilising scrapy & residential proxies

Python Scrapy Amazon Scraper

Scrape Amazon product listings utilising Scrapy & residential proxies

Prerequisites

To get started with Scrapy, you will first need to install it using the methods provided in its documentation. Check here for more information

Authentication & Proxy setup

Once you have an active subscription, you can find your credentials & proxy addresses in Dashboard > Residential > Proxy Setup

Navigate to settings.py in /amazon/amazon/ folder and modify the following lines to authenticate.

DECODO_USER = 'username' ## Decodo Username (Sub-user)
DECODO_PASSWORD = 'password' ## Password for your user
DECODO_ENDPOINT = 'gate.decodo.com' ## Endpoint you'd like to use
DECODO_PORT = '7000' ## Port of the endpoint you are using.

Running the scraper

Navigate to the project folder and run the following command:

scrapy crawl amazon_search

Results

Amazon search results will be saved in /amazon/data folder in a .csv format