Django: Adding a Custom Attribute to All Log Lines

Django provides logging out of the box, and gives us the tools to customize what gets logged how.

This is not an in-depth primer on logging with Django— I’ve linked to the relevant documentation for the different pieces discussed here, but this is more an end-to-end implementation than a deep-dive — we’re going to look at how to add a custom attribute to all of our log lines. For example, if we have a multi-tenant application, we may want to know which organization performed each action for which there is a log line. So let’s implement this, by adding an organization_id to each log line.

In order to add an attribute to our log lines, the attribute will first need to be accessible on the local thread — by the time we get to the point of altering our log record, we don’t have access to the request, but we do have access to the thread. Adding an attribute can be accomplished with middleware, in which we can add a custom attribute in process_request. We then remove it in process_response — by the time we get there the log lines have already been processed and it is no longer necessary. Here’s what a simple middleware class to add the organization_id to the thread local might look like:

(The below is based on Django v. 1.10 — The syntax of this has changed with Django 2.0, so you may need to adjust using the documentation if you’re running a newer version.)

import threading
local = threading.local()

class OrganizationIdMiddleware(object):
    def process_request(self, request):
        organization_id = # logic to get attribute off request
        setattr(local, 'organization_id', organization_id)

    def process_response(self, request, response):
        setattr(local, 'organization_id', None)
        return response

We’ll then add our new middleware class to settings.py so that it gets run for each request/response cycle:

MIDDLEWARE_CLASSES = [
    '<app>.logging.middleware.OrganizationIdMiddleware', 
    ...
]

We’ll now create a custom logging filter, to get the attribute off of the thread and set it on the log record, so that it can be used later. At the most basic, our new filter might look something like this:

import logging
class OrganizationIdFilter(logging.Filter):
    def filter(self, record):
        record.organization_id = getattr(local, 'organization_id', 'no_organization_id')
        return True

Note that our filter function needs to return True, or the record won’t be passed on to the handler (below).

We can then add this filter to our logging set up in settings.py:

LOGGING = {
    'filters': {
        'organization_id': {
            '()': 'everpath.logging.filters.OrganizationIdFilter'
        },
# More to come
    }

Now that we’ve added the filter in the handlers so that the attribute is on our log record, we can access it in our formatter, which we use to specify the information and format of each log line, using the name we gave the attribute above in our filter:

LOGGING = {
    'filters': {
        'organization_id': {
            '()': 'everpath.logging.filters.OrganizationIdFilter'
        },
    },
    'formatters': {
        'standard': {
            'format': 'organization_id=%(organization_id)s ...',
        },
    },

Python provides the base logging handlers, with a mail admins addition from Django — whether we’re using one of these or a custom one of our own, we’ll need to add our new filter to the handler, and use the formatter we already defined:

LOGGING = {
    ...    
    'filters': {        
        'organization_id': {            
            '()': 'everpath.logging.filters.OrganizationIdFilter'              },
        ...    
        'formatters': {        
            'standard': {            
                'format': 'organization_id=%(organization_id)s ...',
            },    
        },    
        'handlers': {        
            'console': {            
                'level': 'INFO',            
                'filters': ['organization_id', ...],            
                'formatter': 'standard',        
            },
        }

This will result in the organization_id attribute being present on all console log lines — the filter and formatter will need to be added to each handler on which we want the attribute, but each can be done differently if needs are different.

Happy logging!