Skip to content
Snapcaster Docs

Proxy Setup

The web scrapers in our backend code requires a list of proxies in a proxies.txt file in order to prevent rate limitation. We recommend using premium ISP proxies from a known provider such as Ping Proxies. We recommend purchasing 5-6 which should cost around 10 USD/month.

Proxies Text File

You will need to create a proxies.txt file in the following directories:

  • ./SNAPCASTER-BACKEND/scraper/v2/face-to-face/config
  • ./SNAPCASTER-BACKEND/scraper/v2/crystal-commerce/config
  • ./SNAPCASTER-BACKEND/scraper/v2/shopify
proxies.txt
ip:port:username:password
ip:port:username:password
ip:port:username:password
ip:port:username:password
ip:port:username:password

Now when you run your python scraper scripts, they will rotate through the proxy list as requests get rate limited.