WSL Setup For Windows Users

The web scrapers for shopify and crystal commerce websites will run significantly faster if you run it in a windows subsystem for linux due to being able to adjust your file descriptors which can’t be done on Windows OS.

If you’re running this on MacOS or any other flavour of Linux then you can ignore steps 1-2.

Windows WSL Ubuntu Setup Steps

1. Install the Ubuntu app from the microsoft store and run it.

2. Mount your file system to where your snapcaster-backend\scraper\v2\shopify directory is. For example:

   cd /mnt/c/Users/Henry/Desktop/snapcaster/snapcaster-backend/scraper/v2/shopify

3. Update

   sudo apt update && sudo apt upgrade

4. Install your Python3 packages

   sudo apt install python3-pip

5. Download your package files for the PostgreSQL C client library (libpq)

   sudo apt-get install libpq-dev

6. Install your the packages needed to run the scrapers from requirements.txt. You shouldn’t need to re run this command in the crystal-commerce directory.

   pip install -r requirements.txt

7. Ensure your proxies.txt file is setup from the previous instructions: Proxies Setup