Production server setup¶

As mentioned in the introduction, the steps you need to setup a saltant production server are very sensitive to your particular environment and implementation needs. Hence, for this section of hosting instructions we will discuss a way to set up saltant in production, not the way (and not necessarily even the “best” way).

Also note that production settings are finicky, and the packages mentioned here may change their interface over time! Make you sure you adapt any changes as necessary.

This section continues directly from the Development server setup instructions.

Allowing incoming network traffic¶

For simplicity, let’s assume we are using AWS Route 53 to route traffic from our domain, www.fictionaljobrunner.com, to an AWS EC2 instance used to host our saltant project, PostgreSQL database server, and RabbitMQ server. These steps should translate over to most implementation methods with (hopefully) minimal modification. We will also assume that the PostgreSQL database server doesn’t need to expose itself to the network, and that the RabbitMQ server does need to expose itself to the network.

Start by routing traffic from your domain to your EC2 instance by following. Set up ALIAS DNS records for fictionaljobrunner.com and www.fictionaljobrunner.com. You may find these routing instructions helpful.

Now, make sure you have ports open for SSH (22), HTTP (80), HTTPS (443), and AMQP (5671) traffic. [1] Later, we will redirect HTTP requests to HTTPS in Let’s encrypt!, and secure incoming Redis traffic with SSL in Securing RabbitMQ with SSL.

Setting up production environment variables¶

We’ll need to fill in production values for a few of our environment variables, namely ALLOWED_HOSTS and DJANGO_BASE_URL. Assuming that we’ve set up ALIAS DNS records for fictionaljobrunner.com and www.fictionaljobrunner.com in the previous step, the relevant variables of our .env file might look like

ALLOWED_HOSTS='fictionaljobrunner.com','www.fictionaljobrunner.com','127.0.0.1'
DJANGO_BASE_URL='https://www.fictionaljobrunner.com'

Collecting static files¶

Before we host saltant, we need to collect all of our project’s static files so we can serve them efficiently. From the base of the project, simply run

$ ./manage.py collectstatic

Hosting saltant on a socket with uWSGI¶

This and the next section are adapted from these uWSGI and nginx instructions, which provide a much more detailed reference for the following instructions.

First we need to install uWSGI; if we’re on Ubuntu we can install it with

$ sudo pip3 install uwsgi

We’re going to daemonize saltant with the uWSGI Emperor and systemd. To do this we need to edit/create a few files:

/etc/uwsgi/emperor.ini

[uwsgi]
emperor = /etc/uwsgi/vassals
uid = www-data
gid = www-data
logto = /var/log/uwsgi-emperor.log

This file tells the uWSGI emperor to run the “vassals” in /etc/uwsgi/vassals, one of which we will define now:

/etc/uwsgi/vassals/saltant_uwsgi.ini

[uwsgi]
chdir = /home/ubuntu/saltant
module = saltant.wsgi
home = /home/ubuntu/saltant/venv
master = true
processes = 10
socket = /tmp/saltant.sock
vacuum = true

This file defines a “vassal” which hosts saltant’s WSGI module saltant.wsgi found at the root of the project /home/ubuntu/saltant using the project’s virtual environment located at /home/ubuntu/saltant/venv. It also defines a socket to connect to, /tmp/saltant.sock, and declares that it can handle 10 requests from that socket simultaneously. For more information, see these uWSGI Emperor vassal instructions.

Next we need to daemonize the uWSGI Emperor we’ve just configured using systemd:

/etc/systemd/system/emperor.uwsgi.service

[Unit]
Description=uWSGI Emperor for saltant
After=syslog.target

[Service]
ExecStart=/usr/local/bin/uwsgi --ini /etc/uwsgi/emperor.ini
RuntimeDirectory=uwsgi
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Make sure this file is executable:

$ sudo chmod +x /etc/systemd/system/emperor.uwsgi.service

Now you can enable the uWSGI-loaded saltant server with

$ sudo systemctl enable emperor.uwsgi.service

Serving the socket with nginx¶

We need to serve the socket with nginx so that the outside world can interface with it.

First install and start nginx:

$ sudo apt install nginx
$ sudo /etc/init.d/nginx start

Now we need to edit the following file:

/etc/nginx/sites-available/saltant_nginx.conf

upstream django {
    server unix:///tmp/saltant.sock;
}

server {
    listen 80;
    listen [::]:80;

    server_name fictionaljobrunner.com www.fictionaljobrunner.com;

    charset utf-8;
    client_max_body_size 10M;

    location /static {
        alias /home/ubuntu/saltant/static;
    }

    location / {
        uwsgi_pass django;
        include /etc/nginx/uwsgi_params;
    }
}

This will route HTTP traffic (which is not secure) to our saltant project.

To enable this site, we need create the following symlink so nginx knows to enable it:

$ cd /etc/nginx/sites-enabled
$ sudo ln -s ../sites-available/saltant_nginx.conf saltant_nginx.conf

Let’s encrypt!¶

Thanks to Let’s Encrypt and EFF Certbot, securing our traffic with SSL and redirecting all HTTP to HTTPS is ridiculously easy.

First install the Certbot for nginx with

$ sudo apt install python-certbot-nginx

Then run it and follow its instructions with

$ sudo certbot --nginx

Congrats to us! Now our site is secured with SSL with automatically renewed certificates!

Hosting RabbitMQ on a network¶

Now let’s focus on RabbitMQ. If all of your Celery workers will be running on the local machine, then you can safely ignore this section.

By default, RabbitMQ will bind to all interfaces, on IPv4 and IPv6 if available. Let’s suppose our IP is 192.168.1.100. The minimum amount of work required to host RabbitMQ on a network is to change the CELERY_BROKER_URL in our .env from

CELERY_BROKER_URL='pyamqp://'

to

CELERY_BROKER_URL='pyamqp://192.168.1.100:5671'

But suppose we want some basic authentication. Let’s include that now. RabbitMQ comes with a default user guest (with password guest) and a default virtual host /. Let’s remove those:

$ sudo rabbitmqctl delete_user guest
$ sudo rabbitmqctl delete_vhost /

Now let’s add our own admin user AzureDiamond (with password hunter2) and virtual host AzureDiamond_vhost:

$ sudo rabbitmqctl add_user AzureDiamond hunter2
$ sudo rabbitmqctl add_vhost AzureDiamond_vhost``
$ sudo rabbitmqctl set_user_tags AzureDiamond administrator
$ sudo rabbitmqctl set_permissions -p AzureDiamond_vhost AzureDiamond ".*" ".*" ".*"

Now that we’ve done this, we need to update the CELERY_BROKER_URL variable in our project’s .env:

CELERY_BROKER_URL='pyamqp://AzureDiamond:hunter2@192.168.1.100:5671/AzureDiamond_vhost'

Hosting the RabbitMQ management console with SSL¶

Our strategy here will be to host the RabbitMQ management console on localhost and create a reverse proxy with nginx to expose to the network. All we need to do is edit the nginx saltant configuration again, and add two new locations within the server block: [2]

/etc/nginx/sites-available/saltant_nginx.conf

server {

    ... # stuff we added before (and that Certbot added to!)

    location ~* /rabbitmq/api/(.*?)/(.*) {
        proxy_pass http://localhost:15672/api/$1/%2F/$2?$query_string;
        proxy_buffering                    off;
        proxy_set_header Host              $http_host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location ~* /rabbitmq/(.*) {
        rewrite ^/rabbitmq/(.*)$ /$1 break;
        proxy_pass http://localhost:15672;
        proxy_buffering                    off;
        proxy_set_header Host              $http_host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Now let’s change the RABBITMQ_MANAGEMENT_URL in your .env to

RABBITMQ_MANAGEMENT_URL='https://www.fictionaljobrunner.com/rabbitmq/'

Securing RabbitMQ with SSL¶

Even though we have secured the RabbitMQ management console with SSL, RabbitMQ is still insecure. If you’re hosting all of your workers on a secure network, then feel free to skip this section.

We’re going to make use of the certs we created in Let’s encrypt!. Noting where those files are, create or edit the following file, like so:

/etc/rabbitmq/rabbitmq.config

[
 {rabbit,
  [
   {tcp_listeners, []},
   {ssl_listeners, [5671]},
   {ssl_options, [{cacertfile,           "/etc/letsencrypt/live/fictionaljobrunner.com/fullchain.pem"},
                  {certfile,             "/etc/letsencrypt/live/fictionaljobrunner.com/cert.pem"},
                  {keyfile,              "/etc/letsencrypt/live/fictionaljobrunner.com/privkey.pem"},
                  {verify,               verify_peer},
                  {fail_if_no_peer_cert, false}]}
  ]
 }
].

Make sure that the rabbitmq user on your machine has read access to the above certs. (One way to do this is to let the ssl-cert group control /etc/letsencrypt and add rabbitmq to this group.)

Hosting Flower with SSL¶

Hosting Flower is simple with nginx. First let’s daemonize Flower with systemd (assuming our saltant virtual environment is located at /home/ubuntu/saltant/venv:

/etc/systemd/system/flower.service

[Unit]
Description=Flower
After=syslog.target

[Service]
WorkingDirectory=/home/ubuntu/saltant/
ExecStart=/home/ubuntu/saltant/venv/bin/flower -A saltant --url-prefix=flower --basic_auth=AzureDiamond:hunter2 --db=flower.db --persistent=True
Restart=always
KillSignal=SIGQUIT
Type=notify
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Just like in Hosting saltant on a socket with uWSGI, we need to make this service executable and enable it:

$ sudo chmod +x /etc/systemd/system/flower.service
$ sudo servicectl enable flower.service

Now we have Flower daemonized on our local machine with some basic authentication [3], but it’s still not exposed to the network. To do so we’ll take the reverse proxy tack taken in Hosting the RabbitMQ management console with SSL. First, get the directory path for Flower’s static files (let’s assume the path is /home/ubuntu/saltant/venv/lib/python3.6/site-packages/flower/static; your’s should be similar). Then let’s add the following two locations to the server block in our nginx configuration file:

/etc/nginx/sites-available/saltant_nginx.conf

server {

    ... # stuff we added before (and that Certbot added to!)

    location /flower/static {
        alias /home/ubuntu/saltant/venv/lib/python3.6/site-packages/flower/static;
    }

    location /flower {
        proxy_pass http://localhost:5555/;
        rewrite ^/flower/(.*)$ /$1 break;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Now let’s let saltant know about Flower. Change the FLOWER_URL variable in .env to

FLOWER_URL='https://www.fictionaljobrunner.com/flower/'

Setting up Rollbar error tracking¶

Rollbar provides a beautiful error-tracking solution for development teams. It also has a generous free tier (yay!). You can sign up here.

Once you, have, fill in the ROLLBAR_ACCESS_TOKEN and ROLLBAR_PROJECT_URL variables in your .env.

Final thoughts¶

This guide has demonstrated one way you can host salant in production. It covers basic security and it should work. However, there’s a whole world (well, industry) worth of extra security and optimization that can be added on top of this to make saltant run better. Be aware of that.

[1]	See here for instructions on opening EC2 instance ports.

[2]	Thanks to Dario Zadro for his post here.

[3]	See more authentication options here.