First-Hand Experience: Django Project Template by Steelkiwi

When you work individually you never think about what structure your project has, or how much time you’ll spend for initialization, or if everything is convenient or not. Your main concern is to finish the project and meet the deadline. When you work in the team though these question gain a good deal of importance. It’s all about little standardization. What would we like to do?

1. Initialization speed: project’s primary configuration should take no more than an hour;
2. Team development orientation: each team member should deploy the project easily on any stage of the development;
3. Deployment-friendly: your DevOps shouldn’t curse you when deploying a project, but do that in a glimpse of an eye;
4. Scalable: there shouldn’t be any problems if you add new apps or libraries;
5. Well-structured: your project should be fancy and have a neat structure;
6. Environment isolated: the project should work independently from your system. For the development of the given template we decided to compare popular templates and template engines.

Django template

The project is installed by a simple command:

$ django-admin.py startproject myproject
$ python manage.py startapp myapp

myproject/
    manage.py
    myproject/
        __init__.py
        settings.py
        urls.py
        wsgi.py
        myapp/

As a result we get a template of the simplest possible structure. What does it contain? Settings file contains the minimum parameters set BASE_DIR, SECRET_KEY, INSTALLED_APPS (django standard applications), MIDDLEWARE, TEMPLATES, DATABASES etc.

So we have a minimum settings set. However we can’t control different types of settings for production, staging and local development, and there’s no requirements.txt. We can surely set it up ourselves, though it would be a mile better if we could get all at once without wasting a couple of hours for configuration.

Django-two-scoops

Though it uses older versions of Django and it’s not actually supported yet, there are templates that use this structure, for example, https://github.com/imkevinxu/django-kevin.

Let’s setup this template from the repository:

$ django-admin.py startproject myproject2 --template=https://github.com/twoscoops/django-twoscoops-project/archive/develop.zip

We can see that now there appear packages set, more branches in settings, which by the way became more flexible due to environment variables, and both STATIC_ROOT and MEDIA_ROOT have been adjusted, etc. It sounds good, doesn’t it? But let’s look at the project’s structure:

myproject_2/
    docs/
    requirements/
    myproject_2/
        manage.py
        static/
        templates/
        myproject_2/
            __init__.py
            settings/
                __init__.py
                local.py
                base.py
                production.py
                test.py
            urls.py
            wsgi.py

To get to settings we have to dig deep, and this can get really annoying in the process of deploying or developing the project. Nevertheless, on the whole, the template is really interesting and it makes project’s setting up quite fast.

Сookiecutter-django

This is a template engine that uses a like-named library for template configuration and adjustment. And it’s quite easy to launch. First, set up a like-named library, then, initialize the project.

$ pip install "cookiecutter>=1.4.0"
$ cookiecutter https://github.com/pydanny/cookiecutter-django

After answering all the questions we get a pre-installed project of the following structure:

myproject_3/
    manage.py
    config/
        __init__.py
        settings/
            __init__.py
            local.py
            common.py
            production.py
            test.py
    urls.py
    wsgi.py
    requirements/
    myproject_3/
        static/
        templates/
        myapp/

All the settings are prefilled. Moreover, we can choose in the installation stage what exactly we’re going to use during the development of the project. It has quite straightforward structure, all the settings are stored in the config folder, and all the rest concurrent elements (like apps, statistics, and templates) are stored in the folder with the project’s name. However, we can trace a couple of disadvantages here as well. For example, the developers of this template state that they write it for their personal use, so there’s no guarantee that it’ll fit your project. The biggest drawback here is probably the abundance of unnecessary information, redundant libraries, and lots of excessive settings.

For the development of the template used in our company, we needed to find a happy medium and make sure that in the process of development the project could be easily opened on any computer of a frontend or backend developer.

We used Cookiecutter as a basis, used its library for project configuration, and removed everything we thought to be irrelevant and redundant adding what we actually needed.

We changed the original project configuration

We decided to remove setting for Heroku unnecessary for us, js_runner, choice of postgresql version, etc. Though we added the choice of additional redis and rabitmq backends of different types. Also, we made it possible to add allauth package as it is not always necessary, to choose between different DBs, and to integrate postgis. Also, we added a possibility to include supervisord, apach or nginx in the template.

Universal settings file

Many may claim this to be inconvenient, as it’s possible to move everything to local.py for local development, and to staging.py for staging server, or to move everything to production.py for a production server and use base.py for general settings. However, we have Django-environ, and reading .env files we can configure anything having general settings for all types of servers by only changing parameters in .env file. And to run it locally on any computer we use default value declaring it in the beginning of the file which lets you launch the project without tweaking, so to speak. Your DevOps will easily get which settings it has to fill in production or staging.

env = environ.Env(
    DJANGO_DEBUG=(bool, False),
    DJANGO_SECRET_KEY=(str, 'CHANGEME!!!8l=tkv+t@)ilij-w=hogt8qgs-zf(i3or-k7w_d(7_1=&l1##l'),
    DJANGO_ADMINS=(list, []),
    DJANGO_ALLOWED_HOSTS=(list, []),
    DJANGO_STATIC_ROOT=(str, str(APPS_DIR('staticfiles'))),
    DJANGO_MEDIA_ROOT=(str, str(APPS_DIR('media'))),
    DJANGO_DATABASE_URL=(str, 'postgres:///myproject4'),
    DJANGO_EMAIL_URL=(environ.Env.email_url_config, 'consolemail://'),
    DJANGO_DEFAULT_FROM_EMAIL=(str, 'admin@example.com'),
    DJANGO_EMAIL_BACKEND=(str, 'django.core.mail.backends.smtp.EmailBackend'),
    DJANGO_SERVER_EMAIL=(str, 'root@localhost.com'),
    DJANGO_CELERY_BROKER_URL=(str, 'redis://localhost:6379/0'),
    DJANGO_CELERY_BACKEND=(str, 'redis://localhost:6379/0'),
    DJANGO_CELERY_ALWAYS_EAGER=(bool, False),
    DJANGO_USE_DEBUG_TOOLBAR=(bool, False),
    DJANGO_TEST_RUN=(bool, False),
)

If looking at the default settings we’ll see a standard settings list for project configuration. Let’s see how django-environ works with these settings. Each setting starts with DJANGO prefix. It is essential for having the possibility while reading environment variables to separate the settings needed for DJANGO application only and for other services. For example, spotipy, the library for working with Spotify, also obtains keys via environment variables, they also have an identificator in the very beginning which points to the library.

$ export SPOTIPY_CLIENT_ID='your-spotify-client-id'
$ export SPOTIPY_CLIENT_SECRET='your-spotify-client-secret'
$ export SPOTIPY_REDIRECT_URI='your-app-redirect-url'

Each variable has its own type and default value. Only two variables from the list will provoke questions: DJANGO_DATABASE_URL and DJANGO_EMAIL_URL.

DJANGO_DATABASE_URL is formed according to the definite rules stated in the dj-database-url application. https://github.com/kennethreitz/dj-database-url

DJANGO_EMAIL_URL has some equivalent parameters, which will become clear after looking through the documentation https://github.com/migonzalvar/dj-email-url

smtp+tls://user:SuperSecretPassword@smtp_server.com:587

Also, there are other types of variables described in django-environ documentation https://github.com/joke2k/django-environ

And what about the parameters for local development, debug toolbar and test, you may ask. It’s all as easy as a pie here. We can preset everything using simple logical conditions in settings.py.

if env.bool('DJANGO_USE_DEBUG_TOOLBAR'):
    MIDDLEWARE_CLASSES += ('debug_toolbar.middleware.DebugToolbarMiddleware', )
    INSTALLED_APPS += ('debug_toolbar', )
    DEBUG_TOOLBAR_CONFIG = {
        'DISABLE_PANELS': [
            'debug_toolbar.panels.redirects.RedirectsPanel',
        ],
        'SHOW_TEMPLATE_CONTEXT': True,
    }

if env.bool('DJANGO_TEST_RUN'):
    pass

Virtual environment

What is it for? What’s that at all? Let’s say you have 3 projects with different python packages in each. We won’t install them in the system, right? Virtualenv creates isolated python environment and if activated it enables you to use packages only from this environment. You can create it using the following set of commands:

$ pip install virtualenv
$ cd my_project_folder
$ virtualenv venv

You can set up a particular python interpreter version:

$ virtualenv -p /usr/bin/python2.7 venv

After activating environment you can install packages set and use them in the context of your project.

$ source venv/bin/activate
$ pip install requests
$ pip install -r requirements.txt

Now we have an isolated environment for python packages and we can avoid worrying about conflicts in different projects and our system remains clean. And what about the rest of the utilities, like postgis, redis etc?

Vagrant

What Vagrant? Vagrant is cli for VirtalBox, VMWare, Amazon EC2, LXC. Why do we need it? Let’s still consider those 3 projects.

• The first project uses celery, postgresql-9.4, postgis, redis, solr search engine.
• The second project uses celery, postgresql-9.4, postgis, rabitMQ, elasticsearch.
• The third project uses celery, mysql, rabitMQ and it’s also bundled with mongodb.

Now imagine that you integrated everything mentioned into your system. Wicked, isn’t it? Of course, you can disable and enable them when you need, but is it comfortable? And now let's put them in an isolated environment. To do this, we may use or Vagrant Docker. Why Vagrant? Since docker is not virtual it is quicker. But let's make it clear why we need it.

Vagrant is designed to manage virtual machines, Docker is intended to create and deploy applications by packaging them into light containers. It seems that even this small list favors docker, but Vagrant has one huge advantage, it is supported by any operating system. The reason why we use Vagrant is simple: for the opportunity to create an isolated development environment, based on a virtual machine with all the necessary services, which would have to reproduce the real working environment.

How do we use it in our template?

Initially, we used one of the boxes in VagrantCloud Hashicorp ubuntu / trasty64.

All settings are defined in Vagrant file: - current box; - ports and folders mapping; - current resources; - isolated environment filling method.

#### Box
    develop.vm.box = "develop"
    develop.vm.box_url = "http://vagrant.steel.kiwi/vagrant/develop/"
#### Network
    develop.vm.network :forwarded_port, guest: 8000, host: 8000
    develop.vm.network :private_network, ip: '192.168.80.50'
    develop.vm.synced_folder ".", "/vagrant", type: "nfs"
#### ansible_provision
    develop.vm.provision "ansible" do |ansible|
    ansible.verbose = "v"
    ansible.playbook = "ansible/vagrant.yml"
    end
#### system resources
    develop.vm.hostname = "develop"
      develop.vm.provider "virtualbox" do |v|
      v.memory = 1024
      v.cpus = 1

How do we fill our isolated environment? The initial variant was quite simple: we used bash scripts and filling consisted in the sequential execution of *.sh files that have been installing corresponding components in turn. For example, here’s the installation of postgresql:

Though the adjustment of all the necessary roles and packages using bash scripts required nothing less but magic... We needed a more flexible way to do that, so the magic we used is called ansible.

Ansible

What is ansible? Ansible is a relatively young configuration management system. The filling is carried out by executing playbook vagrant.yml, or consistent implementation of a set of roles, to be more precise. For example, we need to install postgreqsl, create a user and create a database. We connect the corresponding role, and using YAML syntax create a simple structure like this: Ansible ubuntu/trusty64

- name: Configure the PostgreSQL APT key
  apt_key: url=https://www.postgresql.org/media/keys/ACCC4CF8.asc state=present

- name: Configure the PostgreSQL APT repositories
  apt_repository: repo="deb http://apt.postgresql.org/pub/repos/apt/ {{ ansible_distribution_release}}-pgdg main"
                  state=present

- name: Update PostgreSQL Cache
  apt: update_cache=yes

- name: Install PostgreSQL
  apt: name={{ item }} force=yes state=installed
  with_items:
    - postgresql-9.4
    - postgresql-contrib-9.4
    - python-psycopg2
  tags: packages

Now it looks more straightforward and vivid. But we decided to take it a step further and we made a box with already installed services, so their activation is as simple as incorporating the necessary service.

Ansible steelkiwi.box

- name: enable postgresql
  service: >
    name=postgresql
    enabled=yes
    runlevel=default
    state=started
  tags: postgresql

- name: Create database
  postgresql_db: >
    login_user={{ db_user }}
    name={{ db_name }}
    owner={{ db_user }}
    encoding='UTF-8'
    lc_collate='en_US.UTF-8'
    lc_ctype='en_US.UTF-8'
    state=present
  become: yes
  become_user: vagrant
  tags: create_db

So here are Ansible cons:

• Everything works through SSH;
• Written in Python;
• YAML;
• Well-structured and constantly updated documentation;
• Sequential nodes update.

And where is the relation to our project template? Let’s remember our primary goal: we wanted to make an easily deployable template which would speed up the start of a project and which would enable us to hand over a project from one developer to another and run it in any environment. And here’s what we've got.

The start of the project: thanks to cookiecutter we pull the template repository and predefine some configuration values. As a result, we have a django project and preset ansible configuration for vagrant. Let's have a look at vagrant.yml and vars-> vagrant.yml

Vagrant.yml

- name: Configure box
  hosts: all
  become: yes
  become_user: root
  remote_user: vagrant
  vars:
    - update_apt_cache: yes
  vars_files:
    - vars/vagrant.yml


  roles:
    - { role: mysql, when: "mysql == true" }
    - { role: apache, when: "apache == true" }
    - { role: nginx, when: "nginx == true" }
    - { role: postgresql, when: "postgresql == true" }
    - { role: rabbitmq, when: "rabbitmq == true" }
    - { role: redis, when: "redis == true" }
    - { role: supervisor, when: "supervisor == true" }
    - { role: app, when: "app == true" }

Vars - > vagrant.yml

project_name: myproject4
application_name: myproject4

# Database settings.
db_user: "vagrant"
db_name: "{{ application_name }}"
db_password: ""

# Application settings.
virtualenv_path: "/home/vagrant/venv"
project_path: "/vagrant"
requirements_file: "{{ project_path }}/requirements/local.txt"
django_settings_path: "{{ project_path }}/config"
django_settings: "config.settings"

#Python version - change if you needed
python_version: 3

# Services
mysql: no
apache: no
nginx: no
postgresql: true
rabbitmq: no
redis: true
supervisor: no
app: true

These examples visualize a set of options and a set of activated roles for filling our isolated environment. After this, we perform a simple command vagrant up and wait for about 5 minutes. During this period of time, all the required roles are being processed, packages are being incorporated, database and virtual environment are being created, python packages are being installed, needed migrations are being performed, etc. To shorten waiting time we created a box with the main preset packages, so packages installation simply consists in switching on a required service. When the installation is finished we get a box with a set up database and django application. We can connect to our isolated environment using ssh interface and run django application.

Handing the project over to another developer: everything is made even simpler here. A developer pulls the repository, execute vagrant up command, waits for another 3 minutes and he’s ready to go.

At this point our next aspiration is to add the ability to support and docker and docker-compose, so it wouldn’t restrict developers to use Vagrant.

Useful links

1. SteelKiwi@GitHub
2. Official page of Vagrant
3. Official page of Ansible
4. Kenneth Reitz@GitHub
5. Daniele Faraglia@GitHub