Skip to content
Snapcaster Docs

Populate Database Data

Now that we have our docker containers and enviornment variables setup we will now populate our MongoDB Collections and Postgres Tables with basic data. The scraper section will show you how to populate your singles inventory collections for each of the TCG’s if you chose not not to use pyimport to populate your collections (Ask an admin for the json files).

MongoDB Autofill Collections

Setup Enviornment Variables & Install Packages

  1. Ensure your ./snapcaster-backend/.env MONGO_URI and DISCORD_WEBHOOK_URL variables are setup

  2. Install the required packages.

    pip install -r ./scripts/cron/update_yugioh_cards_db/requirements.txt

pokemonDB.cards

python -m scripts.cron.update_pokemon_db.update_pokemon_db

lorcanaDB.cards

python -m scripts.cron.update_lorcana_cards_db.update_lorcana_db

yugiohDB.cards

python -m scripts.cron.update_yugioh_cards_db.update_yugioh_db

onepieceDB.cards

python -m scripts.cron.update_onepiece_cards_db.update_onepiece_cards_db

mtgDB.cards

python -m scripts.cron.update_mtg_cards_db.update_mtg_cards_db

You can use the MongoDB Compass GUI tool for connecting to your Mongo databases. In MongoDB Compass, click New Connection and enter your SRV URI connection string. The format of this connection string is mongodb://[user:password ]@[hostIP:port ]/ which you will find in the docker-compose.yml file.

Instructions for importing data:

You can use mongoimport to import large json files into your collections quickly. MongoDB won’t let you mass import the large files we’re working with. We reccomend you use this to import your crystal commerce inventory as that can take hours to process otherwise.

(An Admin can send you old exported data for all the collections for you to import into your local MongoDB instance)

mongoimport command format:

mongoimport --uri [uri database connection string] --collection [collection name you're writing to] --file [json file you want to import] --jsonArray --authenticationDatabase [authentication database name]

Example of importing shopify mtgSingles.json into the shopifyDB.mtgSingles collection:

mongoimport --uri mongodb://root:password@127.0.0.1:27017/shopifyDB --collection mtgSingles --file shopify_mtg_singles.json --jsonArray --authenticationDatabase admin

Postgres Populate Websites

  • Directorysnapcaster-backend
    • Directoryscripts
      • Directorypopulate_websites
        • populate_websites_db.py
        • requirements.txt
    • …

  1. Install the required packages.

    pip install -r ./scripts/populate_websites/requirements.txt

  2. Ensure your local postgres tables were setup from the docker compose. (e.g. vendor/vendor_backends table and tcg_enum/vendor_backend Types)

  3. (Temporary Step) Step 4 relies on having a snapcasterDB.websites collection in your mongo database in order to return. It doesn’t make sense to have this dependency and will be updated soon. Before you go to step 4 message an Admin for the json file for this websites collections.

  4. Run populate_websites_db.py (Run once only)

    python -m scripts.populate_websites.populate_websites_db

Setup User Accounts

We haven’t setup an init script yet so just manually create 2 accounts through your localhost client for now.

Below are the steps for setting an account to a premium user in order to test premium features.

  1. From your local host client, click the login button in the top right and create an account.

  2. Insert the user’s information in the stripe_customers table. Enter any string you want in the strip_id field.

  3. Insert the user’s information in the stripe_subscriptions table. The status field should be set to “active”. The customer_id field will be your id in the stripe_customers table.

  4. Login to the pro account on localhost and check if you have the pro status under you account settings.