No-downtime deployments is one of my favorite things to think about. It requires thinking about inputs and outputs and the different states of your application and database and other infrastructure during the phases of a deployment. It requires you to be intentional about how you make changes, not just what changes you make.
As business needs change and features evolve, it is common to change the signature of a function. Usually this is fine — you can control what the input of a function is, and adjust the logic within the function accordingly. Not so simple when your function is a background task though!
The problem here is best demonstrated by example. Lets say you have a task that currently looks like this (Syntax in the examples assumes you’re using celery with Django, but the concepts apply even if you’re not!):
@shared_task(ignore_result=True)def do_something(arg_1): ...
and you now need it to take an additional parameter, so that it looks like this:
@shared_task(ignore_result=True)def do_something(arg_1, arg_2): ...
The issue here is that background tasks are definitionally not evaluated at the time they are enqueued, so a task that was enqueued with only
arg_1 passed in will throw an error if you’ve deployed your change between when it was enqueued and when it will be processed — by the time it’s picked up, the function is expecting both
arg_2 to be passed in, and it only has
This is most likely only a problem for a minute or two, while your app finishes processing any tasks that were enqueued before the change, and it will then self-resolve, but some number of tasks will not be able to be processed successfully. This is obviously not ideal.
As is often the case with challenges like these, a change like this will need to be made in phases, and doing it without downtime will require multiple deployments. We’ll take advantage of the fact that this works in Python:
def something(arg_1, arg_2=None): if arg_2: print arg_1, arg_2 >>> something("foo", arg_2="bar") # => foo bar >>> something("foo", "bar") # => foo bar
This means that we can do the following:
None, and updates the function to handle the fact that it may or may not exist
Note: this works even if your function already has multiple args and kwargs. You would add your new argument as the first kwarg (
arg3 in this example):
def do_something(arg1, arg2, arg3=None, arg4=None): print arg1, arg2, arg3, arg4 >>> do_something(1, 2, arg3=3, arg4=4) # => 1 2 3 4 >>> do_something(1, 2, 3, arg4=4) # => 1 2 3 4