Skip to content
Snapcaster Docs

Raspberry Pi SQS Setup (Depreciated)

Raspberry Pi Image Processing

This guides through deploying sqs_consumer.py on a Raspberry Pi. The SQS queue contains batches of image urls. The consumer is meant to:

  1. Pull batch of image urls from SQS

  2. Fetch and download the image

  3. Resize the image, and convert to jpeg

  4. Upload image to S3 bucket

  5. Update image map in MongoDB

The script currently is set to run forever, and restart in the event of an error. Another option, would be to remove the call to safe_main() in sqs_consumer.py and setup a cron task to run the script on a schedule.G

Image processing pipeline

Deploying

This will explain how to configure AWS and necessary dependencies on the Pi.

1. Requirements

First, install dependencies on the Pi

cd snapcaster-backend/scripts/image_cdn/process_img_handler/pi
pip3 install -r requirements.txt

OR

pip3 install boto3 pillow pymongo requests

2. Setup consumer service

First we need the environment variables. I have setup snapcaster-iam-user in AWS that has S3, SQS and SES permissions. Since we are setting this up as a systemd service, the environment variables must go in the service file.

There are two files we need to create:

  1. sqs_consumer.service
  2. sqs_consumer_env

In this directory, there are templates for each file.

2.1 Create systemd service

  1. First, copy the sqs_consumer.service template file to /etc/systemd/system/sqs_consumer.service

    Terminal window
    sudo cp sqs_consumer.service /etc/systemd/system/

  2. Next, we will do the same for the sqs_consumer_env file, which goes to /etc/

    Terminal window
    sudo cp sqs_consumer_env /etc/

  3. Now we need to adjust the files for your environment.

    Terminal window
    # edit service makefile
    sudo vim /etc/systemd/system/sqs_consumer.service
    sqs_consumer.service
    # Adjust the file for your environment
    # Here, the user is hydrogen, and the snapcaster repo is in the `/home` directory.
    [Unit]
    Description=SQS Consumer Service
    

    [Service]
    # Change user
    User=hydrogen
    # Change working directory where sqs_consumer.py is located
    WorkingDirectory=/home/hydrogen/snapcaster-backend/scripts/image_cdn/process_img_handler/pi
    # Path to python and sqs_consumer.py
    ExecStart=/usr/bin/python3 /home/hydrogen/snapcaster-backend/scripts/image_cdn/process_img_handler/pi/sqs_consumer.py
    StandardOutput=inherit
    StandardError=inherit
    Restart=always
    EnvironmentFile=/etc/sqs_consumer_env
    

    

    [Install]
    WantedBy=multi-user.target

  4. Now we have the service setup, we can configure the sqs_consumer_env file.

    Terminal window
    vim /etc/sqs_consumer_env
    sqs_consumer_env
    AWS_ACCESS_KEY_ID=your_access_key_id
    AWS_SECRET_ACCESS_KEY=your_secret_access_key
    AWS_DEFAULT_REGION=ca-central-1
    MONGO_URI=your_mongo_uri

3. Running the service

Now we can run the service:

Terminal window
# Enable and start service
sudo systemctl enable sqs_consumer.service
sudo systemctl start sqs_consumer.service

If you want to enable on boot, run:

Terminal window
# restart service and enable on boot
sudo systemctl daemon-reload
sudo systemctl restart sqs_consumer.service
sudo systemctl enable sqs_consumer.service

Check the status to verify it is running:

Terminal window
# check status of the service
sudo systemctl status sqs_consumer.service

Now you can use journalctl to view logs from the new service. Ensure there are no errors.

Terminal window
# check logs produced by service (oldest first)
journalctl -u sqs_consumer.service


# check logs produced by service (most recent first) (limit 20)
journalctl -u sqs_consumer.service -r -n 20


# check recent logs since yesterday
journalctl -u sqs_consumer.service --since yesterday -r


# follow along most recent logs in real-time
journalctl -u sqs_consumer.service -f