Skip to content
Snapcaster Docs

Shopify

Shopify Scraper File Structure


  • DirectorySNAPCASTER-BACKEND
    • Directoryscraper
      • Directoryv2
        • Directoryshopify
          • Directoryconfig
            • Directorymtg-advanced
              • mtg_advanced_foil_map.json
              • mtg_advanced_frame_map.json
              • mtg_advanced_showcase_map.json
            • Directoryyugioh-advanced
              • yugioh_advanced_foil_map.json
            • websites.json
          • Directoryconsumers
            • lorcana_consumer.py
            • mtg_consumer.py
            • one_piece_consumer.py
            • pokemon_consumer.py
            • tcg_consumer.py
            • yugioh_consumer.py
          • Directoryproducers
            • producer.py
          • Directoryscrapers
            • lorcana_scraper.py
            • mtg_scraper_advanced.py
            • one_piece_scraper.py
            • pokemon_scraper.py
            • tcg_scraper.py
            • website_scraper.py
            • yugioh_scraper.py
          • Directoryutils
            • utils.py
          • proxies.txt
          • requirements.txt
    • .env
    • …

Shopify products.json API Endpoint

Below is an example data set from Shopify’s products.json endpoint which is parsed through by product category and availibilty.

[shopify_url]/products.json?limit=250&page=1
"products": [
    {
        "title": "Geralf, the Fleshwright [Outlaws of Thunder Junction]",
        "handle": "geralf-the-fleshwright-outlaws-of-thunder-junction",
        "vendor": "Magic: The Gathering",
        "product_type": "MTG Single",
        "variants": [
            {
                "title": "Mint / Near Mint",
                "option1": "Mint / Near Mint",
                "available": false,
                "price": "34.10",
            },
            ...
        ]
        "images": [
            {
                "src": "https://cdn.shopify.com/s/files/1/0266/9081/4011/files/f57a5686-951c-5ea6-bb85-13a2190e1ab5.png?v=1712092595",
            },
            ...
        ]
    },
    ...
]

How to Run the Shopify Producers & Consumers

  1. Change to the the snapcaster-backend-main\snapcaster-backend\scraper\v2\shopify directory. If you’re on windows then do so using WSL and mount the file path into the linux enviornment set up previously.

  2. Ensure your Rabbit MQ container is running at http://localhost:15672/

  3. Run the producers/producer.py script until completion

        python -m producers.producer

  4. Run each of your TCG specific consumers

        python -m consumers.mtg_consumer
        python -m consumers.pokemon_consumer
        python -m consumers.yugioh_consumer
        python -m consumers.lorcana_consumer
        python -m consumers.one_piece_consumer