Crystal Commerce

Crystal Commerce is the second most common backend platform local games stores use to manage their trading card inventories. Since there isn’t any public API endpoint to extract in stock inventory from, we have to manually web scrape all in stock cards using the advanced search webpage and the Beautiful Soup library to parse this information.

Crystal Commerce Scraper File Structure

DirectorySNAPCASTER-BACKEND
- Directoryscraper
  - Directoryv2
    Directorycrystal-commerce
    Directoryconfig
    Directorymtg-advanced
    mtg_advanced_foil_map.json
    mtg_advanced_frame_map.json
    mtg_advanced_showcase_map.json
    Directoryyugioh-advanced
    yugioh_advanced_foil_map.json
    websites.json
    proxies.txt
    Directoryconsumers
    base_consumer.py
    lorcana_consumer.py
    mtg_consumer.py
    pokemon_consumer.py
    yugioh_consumer.py
    Directoryproducers
    producer.py
    Directoryscrapers
    base_scraper.py
    lorcana_scraper.py
    mtg_scraper_advanced.py
    pokemon_scraper.py
    yugioh_scraper.py
    Directoryutils
    utils.py
    requirements.txt
- .env
- …

How to Run the Crystal Commerce Producers & Consumers

Change to the the snapcaster-backend-main\snapcaster-backend\scraper\v2\crystal-commerce directory. If you’re on windows then do so using WSL and mount the file path into the linux enviornment set up previously.
Ensure your Rabbit MQ container is running at http://localhost:15672/
Run the producers/producer.py script until completion
```
  python -m producers.producer
```

Run each of your TCG specific consumers

  python -m consumers.mtg_consumer

  python -m consumers.pokemon_consumer

  python -m consumers.yugioh_consumer

  python -m consumers.lorcana_consumer