ravenscroftj
/
brainsteam.co.uk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
brainsteam.co.uk/brainsteam/content/posts/2023/12/29/Serving Django inside Docke...

378 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
categories:
- Software Development
date: '2023-12-29 14:22:27'
draft: false
tags:
- django
- docker
- python
title: Serving Django inside Docker the Right Way
type: posts
---
I've seen a number of tutorials that incorrectly configure django to run inside docker containers by leveraging it's built in dev server. In this post I explore the benefits of using django with gunicorn and nginx and how to set this up using Docker and docker-compose.
<!--more-->
<!-- wp:paragraph -->
<p>I'm working on a couple of <a href="https://brainsteam.co.uk/2023/11/13/gastronaut-fediverse-recipe-app/">side projects</a> that use django and I'm a big fan of docker for simplifying deployment.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>Django is a bit of a strange beast in that, it has a simple, single-threaded development server that you can start with <code>python manage.py runserver</code> but then it can also be run in prod mode using <a href="https://wsgi.readthedocs.io/en/latest/what.html">WSGI</a> but once in this mode, it doesn't serve static files any more. This can be a little off-putting for people who are used to packaging a single server that does everything (like a nodejs app). It is especially confusing to people used to packaging an app along with everything it needs inside docker. </p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading">Part 1: Why not just use runserver?</h3>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p><strong><em>If you already understand why it's better to use WSGI than runserver and just want to see the working config, skip down to <a href="#part-2">Part 2</a> below.</em></strong></p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>I've seen a few tutorials for packaging up django apps inside docker containers that just use the <code>runserver</code> mechanism. The problem with this is that you don't get any of the performance benefits of using a proper WSGI runner and in order to handle server load, you end up needing to run multiple copies of the docker container very quickly. </p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">A Rudimentary Performance Test</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>I did a quick performance test against the python runserver versus my WSGI + Nginx configuration (below) to illustrate the difference on my desktop machine. I used <a href="https://github.com/codesenberg/bombardier">bombardier</a> and asked it to make as many requests as it can for 10s with up to 200 concurrent connections:</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p><code>bombardier -c 200 -d 10s http://localhost:8000</code></p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>The thing at that address is the index <code>view</code> of my django app so we're interested in how quickly we can get the Python interpreter to run and return a response.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>The python runserver results:</p>
<!-- /wp:paragraph -->
<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Statistics Avg Stdev Max
Reqs/sec 1487.50 633.11 2988.81
Latency 133.84ms 259.97ms 7.12s
HTTP codes:
1xx - 0, 2xx - 15042, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 13.70MB/s</pre>
<!-- /wp:preformatted -->
<!-- wp:paragraph -->
<p>And the WSGI config results:</p>
<!-- /wp:paragraph -->
<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Statistics Avg Stdev Max
Reqs/sec 1754.20 666.40 16224.55
Latency 115.05ms 7.23ms 174.44ms
HTTP codes:
1xx - 0, 2xx - 17472, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 15.95MB/s</pre>
<!-- /wp:preformatted -->
<!-- wp:paragraph -->
<p>As you can see, using a proper deployment configuration, the average number of requests handled per second goes up by about 15% but also we get a much more consistent latency (115ms average with a deviation of about 7ms as opposed to in the first example where latency is all over the place and if you're really unlucky, you're the person waiting 7s for the index page to load).</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":5} -->
<h5 class="wp-block-heading">Testing Static File Service</h5>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>Now let's look at handling files. When we use <code>runserver</code> we are relying on the python script to serve up the files we care about. I ask bombardier to request the logo of my app as many times as it can for 10 seconds like before:</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p><code>bombardier -c 200 -d 10s http://localhost:8000/static/images/logo.png</code></p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>First we run this with django runserver:</p>
<!-- /wp:paragraph -->
<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Statistics Avg Stdev Max
Reqs/sec 731.51 252.55 1795.93
Latency 270.50ms 338.53ms 5.01s
HTTP codes:
1xx - 0, 2xx - 7504, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 255.27MB/s</pre>
<!-- /wp:preformatted -->
<!-- wp:paragraph -->
<p>And again with Nginx and WSGI. </p>
<!-- /wp:paragraph -->
<!-- wp:preformatted -->
<pre class="wp-block-preformatted">Statistics Avg Stdev Max
Reqs/sec 6612.33 705.07 9332.41
Latency 30.27ms 19.95ms 1.30s
HTTP codes:
1xx - 0, 2xx - 66156, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 2.25GB/s</pre>
<!-- /wp:preformatted -->
<!-- wp:paragraph -->
<p>And suddenly the counter-intuitive reason for Django splitting static file service from code execution makes a little bit more sense. Since we are just requesting static files, Python never actually gets called. Nginx, which is an efficient server that is written in battle-hardened C, is able to just directly serve up the static files. </p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>In the first example, Python is the bottleneck and using Nginx + WSGI just makes some of the shifting around of information a little bit smoother. In the second example, we can completely sidestep python.</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">If you still need convincing...</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>The devs literally tell you not to use runserver in prod in the official docs:</p>
<!-- /wp:paragraph -->
<!-- wp:quote -->
<blockquote class="wp-block-quote"><!-- wp:paragraph -->
<p>DO NOT USE THIS SERVER IN A PRODUCTION SETTING. It has not gone through security audits or performance tests. (And that’s how it’s gonna stay. We’re in the business of making web frameworks, not web servers, so improving this server to be able to handle a production environment is outside the scope of Django.)</p>
<!-- /wp:paragraph --><cite><a href="https://docs.djangoproject.com/en/5.0/ref/django-admin/#runserver">django-admin and manage.py - Django documentation</a></cite></blockquote>
<!-- /wp:quote -->
<!-- wp:separator -->
<hr class="wp-block-separator has-alpha-channel-opacity"/>
<!-- /wp:separator -->
<!-- wp:heading {"level":3} -->
<h3 class="wp-block-heading" id="part-2">Part 2: Packaging Django + WSGI in Docker</h3>
<!-- /wp:heading -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Getting Started</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>Ok so I'm going to assume that you have a django project that you want to deploy and it has a <code>requirements.txt</code> file containing the dependencies that you have installed. If you are using a <a href="https://brainsteam.co.uk/2021/04/01/opinionated-guide-to-virtualenvs/">python package manager</a>, I'll drop some hints but you'll have to infer what is needed in a couple of places.</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Install and Configure Gunicorn</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>Firstly, we need to add a WSGI server component that we can run inside the docker container. I will use <a href="https://gunicorn.org/">gunicorn</a>.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p><code>pip install gunicorn</code> (or you know, pdm add/poetry add etc)</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>We can test that it's installed and working by running:</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p> <code>gunicorn -b :8000 appname.wsgi</code> </p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>If you go to localhost:8000 you should see your app there but, wait a minute, there are no images or css or js. As I mentioned, django won't serve your static resources so we'll pair gunicorn up with nginx in order to do that. </p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Collect Static Resources</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>nginx needs a folder that it can serve static files from. Thankfully django's manage.py has a command to do this so we can simply run:</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p><code>python manage.py collectstatic --noinput</code></p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>The <code>--noinput</code> argument prevents the script from asking you questions in the terminal and it will simply dump the files into a <code>static</code> folder in the current directory.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>Try running the command in your django project to see how it works. We'll be using this in the next step</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Build a Dockerfile for the app</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>We can produce a docker file that builds our django app and packages any necessary files along with it. </p>
<!-- /wp:paragraph -->
<!-- wp:enlighter/codeblock {"language":"dockerfile"} -->
<pre class="EnlighterJSRAW" data-enlighter-language="dockerfile" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">FROM python:3
WORKDIR /app
ADD . /app
RUN python3 -m pip install -r requirements.txt
# nb if you are using poetry or pdm you might need to do something like:
# RUN python3 -m pip install pdm
# RUN pdm install
ENV STATIC_ROOT /static
CMD ["/app/entrypoint.sh"]</pre>
<!-- /wp:enlighter/codeblock -->
<!-- wp:paragraph -->
<p>NB: if you are using pdm or poetry or similar, you will want to install them </p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>We also need to create the entrypoint.sh file which docker will run when the container starts up. Save this file in the root of your project so that it can be picked up by Docker when it builds:</p>
<!-- /wp:paragraph -->
<!-- wp:enlighter/codeblock {"language":"bash"} -->
<pre class="EnlighterJSRAW" data-enlighter-language="bash" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">#!/usr/bin/env bash
pdm run manage.py collectstatic --noinput
pdm run manage.py migrate --noinput
pdm run gunicorn -b :8000 appname.wsgi</pre>
<!-- /wp:enlighter/codeblock -->
<!-- wp:paragraph -->
<p>This script runs the collectstatic command which, with a little bit of docker magic we will hook up to our nginx instance later. Then we run any necessary database migrations and then we use gunicorn to start the web app.</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Build the nginx.conf </h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>We need to configure nginx to serve static files when someone asks for <code>/static/something</code> and forward any other requests to the django app. Create a file called nginx.conf and copy the following:</p>
<!-- /wp:paragraph -->
<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">events {
worker_connections 1024; # Adjust this to your needs
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
# Server block
server {
listen 80;
server_name localhost;
# Static file serving
location /static/ {
alias /static/;
expires 30d;
}
# Proxy pass to WSGI server
location / {
proxy_pass http://frontend:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
}</pre>
<!-- /wp:enlighter/codeblock -->
<!-- wp:paragraph -->
<p>This configuration should be relatively self-explanatory but a couple of notes:</p>
<!-- /wp:paragraph -->
<!-- wp:list -->
<ul><!-- wp:list-item -->
<li>The alias command tells nginx to serve static files from the <code>/static/</code> folder on the filesystem. This is where we will mount the static files using docker later. The behaviour of alias is described <a href="https://stackoverflow.com/questions/10631933/nginx-static-file-serving-confusion-with-root-alias"><span style="text-decoration: underline;">here</span></a>.</li>
<!-- /wp:list-item -->
<!-- wp:list-item -->
<li>The expires 30d directive tells nginx that it can let browsers cache these static files for up to 30 days at a time - hopefully saving bandwidth and speeding things up.</li>
<!-- /wp:list-item -->
<!-- wp:list-item -->
<li>If the request does not start with <code>/static/</code> then nginx will assume it is a request for django and send it to <code>http://frontend:8000</code> - again we will configure docker so that the django gunicorn process is listening from there.</li>
<!-- /wp:list-item -->
<!-- wp:list-item -->
<li>Note that we use <code>/static/</code> but we could change this - the value needs to match whatever we set as <code>STATIC_ROOT</code> in the docker file.</li>
<!-- /wp:list-item --></ul>
<!-- /wp:list -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Glue it together with docker compose</h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>We will use docker compose to combine our nginx and django containers together.</p>
<!-- /wp:paragraph -->
<!-- wp:enlighter/codeblock {"language":"yaml"} -->
<pre class="EnlighterJSRAW" data-enlighter-language="yaml" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">services:
frontend:
build: .
restart: unless-stopped
volumes:
- ./static:/app/static
environment:
DJANGO_CSRF_TRUSTED_ORIGINS: 'http://localhost:8000'
frontend-proxy:
image: nginx:latest
ports:
- "8000:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./static:/static:ro
depends_on:
- frontend</pre>
<!-- /wp:enlighter/codeblock -->
<!-- wp:paragraph -->
<p>Ok so what we are doing here is using volume mounts to connect <code>/app/static</code> inside the django container where the results of <code>collectstatic</code> are dumped to <code>/static/</code> in our nginx container where the static files are served from.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>We also mount the <code>nginx.conf</code> file in the nginx container. You'll probably end up using docker compose to add database connections too or perhaps a volume mount for a sqlite database file.</p>
<!-- /wp:paragraph -->
<!-- wp:paragraph -->
<p>Finally we bind port <code>8000</code> on the host machine to port <code>80</code> in nginx so that when we go to <code>http://localhost:8000</code> we can see the running app.</p>
<!-- /wp:paragraph -->
<!-- wp:heading {"level":4} -->
<h4 class="wp-block-heading">Running it </h4>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>Now we need to build and run the solution. You can do this by running:</p>
<!-- /wp:paragraph -->
<!-- wp:preformatted -->
<pre class="wp-block-preformatted">docker compose build
docker compose up -d
</pre>
<!-- /wp:preformatted -->
<!-- wp:paragraph -->
<p>Now we can test it out by going to <a href="http://localhost:8000">http://localhost:8000</a>. Hopefully you will see your app running in all its glory. We can debug it by using <code>docker compose logs -f</code> if we need to.</p>
<!-- /wp:paragraph -->
<!-- wp:heading -->
<h2 class="wp-block-heading">Conclusion</h2>
<!-- /wp:heading -->
<!-- wp:paragraph -->
<p>Hopefully this post has shown you why it is important to set up Django properly rather than relying on <code>runserver</code> and how to do that using Docker, Nginx and Gunicorn. As you can see, it is a little bit more involved than your average npm application install but it isn't too complicated.</p>
<!-- /wp:paragraph -->
Reference in New Issue View Git Blame Copy Permalink