Tool Your Django Project: Pre-Commit Hooks

Last update: 07.03.2021.

In this article, I will explain how to set-up a pre-commit pipeline for a Django project.

What and why Pre-Commit?

Git supports running hooks at different stages of its workflow(s). For a detailed discussion see the official SCM book. One of these hooks is the “pre-commit” hook, which allows a user to validate and transform code before the commit process.

Pre-Commit, in turn, is a Python package that makes installing and running Git hooks quite straightforward and is a great solution for Python-based or multi-lingual repositories.

Setting-Up Pre-Commit

The simplest way of installing pre-commit in a Python project is by adding it to the project’s requirements.txt (or requirements-dev.txt) file and then installing it into the local virtual environment with:

pip install -r requirements.txt

If this is not optimal for you then check out the documentation that gives several different ways of installing it (e.g. using homebrew or curl).

After you have Pre-Commit installed, the next stage is to add a YAML config file to the root of your repository called “.pre-commit-config.yaml” with the following:

default_language_version:
  # default language version for each language
  python: python3.9
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.4.0
    hooks:
      # See https://pre-commit.com/hooks.html for more hooks
      - id: check-ast
      - id: check-case-conflict
      - id: check-executables-have-shebangs
      - id: check-merge-conflict
      - id: debug-statements
      - id: end-of-file-fixer
      - id: name-tests-test
        args: [ "--django" ]
      - id: trailing-whitespace

In the “default_language_version” section you can pre-define the default version for each supported language used in the repo. You should set this to the Python version used in your project, and if you set up hooks for other supported languages then add these as well.

In the “repos” section in turn we tell pre-commit where to locate or download the hooks from. The “-” is the YAML syntax for array, and this section will have multiple entries. For now, though, we are only referring to the official pre-commit-hooks package repo and each “id” entry under the “hooks” key refers to a specific hook from this package.

As you can see, we are also passing an argument to the last hook called “name-tests-test”, telling it that this is a Django project, and — as such — the test files should be named test_*.py rather than *_test.py (if you are not setting the pipeline for a Django project, adjust this as necessary of course). This nifty package has many other hooks and options, so take a look at its docs.

At this point you can install pre-commit itself into the Git hooks folder (located under .git/hooks in the repo root) by executing in the terminal:

pre-commit install

You will need to run this command only once (unless for some reason we delete the Git hooks folder) but any collaborator with whom you share the repo will also need to run this command because the .git folder is local.

At this point, the Pre-Commit installation is finished and the pre-commit hooks will run for every future commit. If for some reason you need to skip the hooks when committing, you can do so by using the Git “-n” or “ — no-verify” flag, for example:

git commit --no-verify …

One last important feature of Pre-Commit before we turn to other tools: it can be tedious to manually update hooks. Luckily Pre-Commit has a handy way of dealing with this called a hook auto-update. To use it type:

pre-commit autoupdate

Go ahead and run this command, it will make sure that the repo we installed is at its latest version.

Code Linters and Analysis Tools

The Python ecosystem has many different linters available — we will install four different tools — Bandit, Flake8, Pylint and MyPy.

Bandit

Bandit is a security linter that checks code and reports security issues. It's very fast and is simple to set up as a pre-commit hook. To do so, simply add the following to your “.pre-commit-config.yaml”:

- repo: https://github.com/pycqa/bandit
  rev: 1.6.2
  hooks:
    - id: bandit
      args: ['-iii', '-ll']

The “-iii” and “-ll” args are flags that tell bandit to only report issues with high confidence and of medium or above severity. You can modify these as you see fit and add any other option as specified in the docs.

Flake8

Flake8 is a wrapper around three different tools — pyflakes (code linter), McCabe (code complexity analyzer), and pydocstyle (docstring linter). It’s popular, fast, requires little configuration, has many cool plugins and has no additional dependencies which makes it an ideal pre-commit hook.

To setup flake8 you will need to add a config file to the project, which can either be a .flake8 (dedicated config) file or an entry in a setup.cfg (shared config) file. Assuming shared config files are preferable, add a setup.cfg file in the root folder of your repo with the following settings:

[flake8]
max-line-length = 88
max-complexity = 12
ignore = E501, W503, E203

“Max-complexity” is a setting for McCabe — again decide on a level of cyclomatic complexity that you think is acceptable, in my book 10–12 is pretty much the range to aim for.

The “ignore” clause turns off some flake8 formatting errors that cause it to clash with Black. If you do not use Black you should remove the ignore clause.

After adding the configs we can add the flake8 pre-commit hook to the “.pre-commit-config.yaml” file:

- repo: https://gitlab.com/pycqa/flake8
  rev: 3.8.4
  hooks:
    - id: flake8
      additional_dependencies: [
          "flake8-bugbear",
          "flake8-comprehensions",
          "flake8-mutable",
          "flake8-print",
          "flake8-simplify",
      ]

In additional_dependencies we specify an array of plugins. You can checkout and find more plugins here.

Pylint

Pylint is a powerful code analysis and linting tool that aims to cover a lot of different functionalities.

You can either go for a dedicated .pylintrc file or add the settings into a shared config file. I recommend using the “pyproject.toml” format. To use this format, create a file called “pyproject.toml” in the root of your repo and then add to it the following settings:

[tool.pylint.MESSAGE_CONTROL]
disable = """
    line-too-long,
    abstract-method,
    no-member,
"""
enable = "useless-suppression"

[tool.pylint.REPORTS]
reports="no"

[tool.pylint.FORMAT]
max-line-length="88"

[tool.pylint.BASIC]
good-names = "_,i,e,setUp,tearDown,maxDiff"

As you can see the configs are broken into sections.

The “reports=no” flag tells Pylint to only display messages rather than the full report output.

The “good-names” section whitelists some variable and function names which are useful for a Django project — adjust it as required for your project, of course.

As with flake8 — the “max-line-length” setting is required for working with Black, as well as the “bad-continuation” and “line-too-long” errors being disabled.

The other errors that I disabled above are disabled because they clash with Django.

Now we can add the actual hook:

- repo: https://github.com/pycqa/pylint
  rev: "pylint-2.7.2"
  hooks:
    - id: pylint
      exclude: "[a-zA-Z]*/(migrations)/(.)*"
      args: [ "--load-plugins=pylint_django", "--django-settings-module=app.settings" ]
      additional_dependencies: [
          # pylint dependency
          pylint_django,
          # runtime dependencies
          ...
      ]

In difference to the previous hooks we added, Pylint needs to have access to the runtime dependencies when it analyzes the code.

This means we need to specify the runtime dependencies of the project inside the “additional_dependencies” array, and pre-commit will install these in its virtual env.

The first dependency in the array is the Pylint-Django plugin, which makes Pylint better compatible with django.

Note that we are passing two args to Pylint when invoking it — the first telling Pylint to load the plugin, and the second one pointing it at the Django project’s settings.py. You should of course update the second argument to point at your project’s settings file.

Mypy

Mypy is a static type checker. You should definitely use mypy or a similar tool (there are some competitors) if you use type hints in your codebase. On the other hand, if you do not use type hints or have only very rudimentary typings, Mypy might be difficult to adopt because of the amount of changes it will require. So try it out and see how it works for you.

As a first step add a mypy.ini file with the following configs:

[mypy]
plugins = mypy_django_plugin.main, mypy_drf_plugin.main
ignore_missing_imports = True
warn_unused_ignores = True
warn_redundant_casts = True
strict_optional = False

[mypy.plugins.django-stubs]
django_settings_module = "app.settings"

[mypy_django_plugin]
ignore_missing_model_attributes = True

We specify two mypy plugins —one for Django and the other for the Django rest-framework. If you do not use the DRF, remove that plugin.

“ignore_missing_imports” is necessary because we do not want to explicitly stub each untyped project dependency or exclude it specifically in the configs. The two warning are useful to let us know when we have redundant type ignores or type casts in the code — these are optional.

The “strict_optional” option is meant to avoid issues Mypy has with resolving union types that include Optional[] types.

The other configs are for the django plugin — you should set “django_settings_module” to point at your project’s django settings.py file.

We can now add the Pre-Commit hook itself:

- repo: https://github.com/pre-commit/mirrors-mypy
  rev: 'v0.812'
  hooks:
    - id: mypy
      exclude: "[a-zA-Z]*/(migrations)/(.)*"
      additional_dependencies: [
          # mypy dependencies
          djangorestframework-stubs,
          django-stubs,
          # runtime dependencies
          ...
      ]

Mypy is similar to Pylint in that it requires access to the project’s runtime dependencies, so make sure to add these to the “additional_dependencies”.

Note the two stubs packages we are adding — if you do not use DRF remove the “djangorestframework-stubs” package.

Code Formatters

The Python ecosystem has many code formatters as well. For more information here you can consult this article, which offers a comparison of four of the most popular code formatters. I will focus on two tools which are becoming the de-facto standard, these are called Black and isort.

Black

Black is the “uncompromising code-formatter” for Python. It’s a great tool and very effective, but it takes an approach that some might find a bit difficult to digest as it is highly opinionated with very little configuration.

If this isn’t acceptable for you then I would suggest that you look into one of the other code formatters mentioned in the above-linked article, all of which can be used as pre-commit hooks instead.

Otherwise, to use Black add the following config to your “pyproject.toml” file:

[tool.black]
line-length = 88
include = '\.pyi?$'

This is pretty self-explanatory; the “line-length” setting is the length that Black will aim for (it’s a baseline value but Black will deviate from this as needed by its algorithms), with 88 being the recommended setting. If you’d like to use a different value then make sure to adjust the settings for flake8, Pylint, and isort (below) as well.

Finally, add the following to your “.pre-commit.config.yaml”:

- repo: https://github.com/psf/black
  rev: 20.8b1
  hooks:
    - id: black

isort

isort is a tool that sorts imports. This is great and makes project files way more consistent. Begin by adding the following configs to your pyproject.toml file:

[tool.isort]
profile = "black"
multi_line_output = 3
line_length = 88
default_section = "THIRDPARTY"
known_first_party = []
known_third_party = []

The “profile”, “multi_line_output” and “line_length” settings are required for making isort compatible with Black, so if you decide not to use Black you can either remove these or adjust them as you see fit.

The “default_section” tells isort where to group imports it doesn't recognize; in this instance, we are telling it to place them in the “THIRDPARTY” category by default.

The “known_first_party” entry is used to tell isort about the project’s local apps. For example, in a Django project that has an “API” and a “user” package, known_first_party should equal:

known_first_party = ["api", “user"]

Go ahead and update it with your own project’s packages.

The “known_third_party” is for third party packages — basically the entire project’s requirements. It would be very tedious to add these manually and luckily there is an excellent tool for auto-generating and updating this automatically called seed-isort-config. To install both this tool and isort, add the following to your “.pre-commit-config.yaml” (make sure to keep seed-isort-config above isort):

- repo: https://github.com/asottile/seed-isort-config
  rev: v2.2.0
  hooks:
    - id: seed-isort-config
- repo: https://github.com/pycqa/isort
  rev: 5.7.0
  hooks:
    - id: isort

That's it — now every-time you run pre-commit, the isort “known_third_party” entry will be updated based on the code-base before isort itself will be invoked.

Other Tools

Pyupgrade

Pyupgrade is a nifty pre-commit tool that upgrades older python syntax into newer python syntax. It supports different flags dependending on the python version upgrades that should be applied, so checkout it’s docs. The following config is for using python 3.9:

- repo: https://github.com/asottile/pyupgrade
  rev: v2.10.0
  hooks:
    - id: pyupgrade
      args: [ "--py36-plus", "--py37-plus", "--py38-plus", "--py39-plus" ]

Wrapping Up

If you followed the instructions above to the letter you should have the following configs in place:

pyproject.toml

[tool.black]
line-length = 88
include = '\.pyi?$'

[tool.isort]
profile = "black"
multi_line_output = 3
line_length = 88
default_section = "THIRDPARTY"
known_third_party = []
known_first_party = []

[tool.pylint.MESSAGE_CONTROL]
disable = """
    line-too-long,
    abstract-method,
    no-member,
"""
enable = "useless-suppression"

[tool.pylint.REPORTS]
reports="no"

[tool.pylint.FORMAT]
max-line-length="88"

[tool.pylint.BASIC]
good-names = "_,i,e,setUp,tearDown,maxDiff"

setup.cfg

[flake8]
max-line-length = 88
max-complexity = 12
ignore = E501, W503, E203

.pre-commit-config.yaml

# See https://pre-commit.com for more information
default_language_version:
  # default language version for each language used in the repository
  python: python3.9
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v3.4.0
    hooks:
      # See https://pre-commit.com/hooks.html for more hooks
      - id: check-ast
      - id: check-case-conflict
      - id: check-executables-have-shebangs
      - id: check-merge-conflict
      - id: debug-statements
      - id: end-of-file-fixer
      - id: name-tests-test
        args: [ "--django" ]
      - id: trailing-whitespace
  - repo: https://github.com/asottile/seed-isort-config
    rev: v2.2.0
    hooks:
      - id: seed-isort-config
  - repo: https://github.com/asottile/pyupgrade
    rev: v2.10.0
    hooks:
      - id: pyupgrade
        args: [ "--py36-plus", "--py37-plus", "--py38-plus", "--py39-plus" ]
  - repo: https://github.com/pycqa/isort
    rev: 5.7.0
    hooks:
      - id: isort
  - repo: https://github.com/psf/black
    rev: 20.8b1
    hooks:
      - id: black
  - repo: https://github.com/pycqa/bandit
    rev: 1.7.0
    hooks:
      - id: bandit
        args: [ "-iii", "-ll" ]
  - repo: https://gitlab.com/pycqa/flake8
    rev: 3.8.4
    hooks:
      - id: flake8
        additional_dependencies: [
            "flake8-bugbear",
            "flake8-comprehensions",
            "flake8-mutable",
            "flake8-print",
            "flake8-simplify",
        ]
  - repo: https://github.com/pycqa/pylint
    rev: "pylint-2.7.2"
    hooks:
      - id: pylint
        exclude: "[a-zA-Z]*/(migrations)/(.)*"
        args: [ "--load-plugins=pylint_django", "--django-settings-module=app.settings" ]
        additional_dependencies: [
            # pylint dependency
            pylint_django,
            # runtime dependencies
            # ...
        ]
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: 'v0.812'
    hooks:
      - id: mypy
        exclude: "[a-zA-Z]*/(migrations)/(.)*"
        additional_dependencies: [
            # mypy dependencies
            djangorestframework-stubs,
            django-stubs,
            # runtime dependencies
            # ...
        ]

If you are setting up a new project then you should be golden. The only thing left to do is to adjust the configs as you see fit and ensure that the hooks are all up-to-date by running:

pre-commit autoupdate

To make sure that the hooks work, just create a Git commit and when you see the hooks all installing and running you are done!

But how do you handle an existing code-base?

As a first stage, you can add code linters and static analysis tools immediately because these tools do not transform the code they inspect. As such they will not create unintentional diffs.

Black and sort, however, are a different ball game: The minute you add these tools to a project you are basically committing to creating one massive pull request that reformats the entire codebase in one go. If you don’t do this you will basically create a situation where every commit of a pre-existing file will be full of unintentional changes, making code review impossible and “polluting” the diff.

If you are apprehensive about the change to your project’s Git history that such a massive pull request will introduce, you can read the advice about how to handle this in Black’s GitHub page. For my money though, having one PR that formats the entire code-base even if it “pollutes the git blame” is not a bad tradeoff as well-formatted code is worth the price.

So how do we go about doing this? Luckily Pre-Commit has another great piece of functionality:

pre-commit run <hook> <target>

This allows you to run a specific hook against specific target files as a CLI tool, or when using the “ — all-files” flag, the entire repo. In our case though we are going for a more brute force solution which is to run all hooks against all files in the repo by executing:

pre-commit run --all-files

Before doing this, make sure to commit your configs because the diff will be massive! Then run the above command which will run the hooks against all python files in the repo. The process is basically this: run against all files and repeat until Black and isort both pass. You might need to manually fix some places where isort and Black disagree on to get there, (such as occasions where Black formats a piece of code and then isort reformats this code again and vice versa).

Once these two tools pass (which means that they no longer make changes to the project’s code-base) you can commit. Note that you will probably need to use the “--no-verify” flag because of failures in the linting hooks, but this should not be an issue (as I explained previously linters do not cause the same problem with git diffs that code formatters do, and — as such — this isn’t much of a blocker).

After committing the code reformat you might go about fixing some of the linting issues or bulk ignoring errors by updating each tool’s respective configs. You can also locally ignore errors in the code by settings inline comments. Each tool has different flags so you will need to consult the docs about doing this. In most cases though it’s not pragmatic to try and fix all linting issues in one go — even for a small code-base there can be dozens or even hundreds of issues that will require quite a while to resolve.

Conclusion

This article detailed how to set-up a pre-commit pipeline for a Django project. If you have any suggestions for improvement — feel post these in the comments!