Django example: adding a new non-nullable foreign key

Perhaps a better title for this blog series would be “Django lessons learned the hard way”, since I seem to only have something to write about when I’ve really screwed something up and had to struggle to fix it. Usually that error came from not thinking out my data structures from the beginning, and having to back track.

Here’s where I screwed up this time. In my project I have a model called “Organization” which holds information about companies in my app. I’d built the model several months ago and went on my merry way creating objects and adding some related data when I realized, I had forgotten to add a user group to hold the administrators of the companies. This group is incredibly important because it will define who is allowed to view certain information about the company and edit details about it. Realizing my mistake I went back to add the group foreign key field, which would be fairly straight forward if it were a nullable field. Since it isn’t though, it takes a few steps to add it to the model.

In this mini tutorial we’ll add a new non-nullable One To One Field representing the administrator group for an organization. As always, I want to add the caveat here that I am still just a learner myself, so I may not know the best solution, but it’s one that has worked for me!

First, add the field to model as a nullable One To One Field:

class Organization(models.Model):
    # ...
    # Note: I'm using a one-to-one field here, but if your related object has 
    # a many-to-one relationship be sure to use models.ForeignKey() instead
    admin = models.OneToOneField(Group, related_name="administrators_of", null=True)

With the field added in code, make the migrations for the model using the standard terminal command. You can go ahead and migrate it at the same time:

$ python manage.py makemigrations
$ python manage.py migrate

If we tried to make the field non-nullable now the migration command would choke on those organizations not having anything in that field. So before we remove it we need to create an admin group for each existing organization. Start with an empty migration:

$ python manage.py makemigrations --empty organization

Inside the new migration we’ll write a couple Python methods and hook it into the migration code using a RunPython() operation. I went ahead and created a reverse function too, so that if I needed to back out the migration for any reason I could.

In this example, the forward function creates an administrator groups per organization, and the reverse function will delete the created groups if needed. I also added a bit of code to ensure that the group names are unique.

def create_groups(apps, schema_editor):
    Organization = apps.get_model('contact', 'Organization')
    Group = apps.get_model('auth', 'Group')
    for organization in Organization.objects.all():
        base_name = organization.name[:50]
        # if a group with the base name already exists, append the org id to make it unique
        if Group.objects.filter(name=base_name + " Admin").count() > 0:
            base_name = base_name + str(organization.id)
        admin_group = Group(name=base_name + " Admin")
        admin_group.save()
        organization.admin = admin_group
        organization.save()


def delete_groups(apps, schema_editor):
    Organization = apps.get_model('contact', 'Organization')
    for organization in Organization.objects.all():
        if organization.admin:
            organization.admin.delete()
            organization.admin = None
        organization.save()


class Migration(migrations.Migration):

    dependencies = [
        ('organization', '0008_auto_20171126_1736'),
    ]

    operations = [
        migrations.RunPython(create_groups, delete_groups),
    ]

It’s now safe to migrate this change. To test it out, try reversing and re-migrating the change, fixing errors along the way. It’s also a good idea to query the database to make sure the groups were added before moving on to the next step. I checked using this script:

$ python manage.py dbshell
mydb=> select * from auth_group;
 id | name 
----+-----------------------------------------------------
 1  | Widget Company Admin
...
(10 rows)
mydb=> select * from organization_organization;
 id |         name         | admin_id
----+----------------------+------------------------------
 1  | Widget Company Admin | 1

Great! That’s in order, so we can go ahead and make the admin field non-nullable by just removing the null=True attribute.

class Organization(models.Model):
    # ...
    admin = models.OneToOneField(Group, related_name="administrators_of")

Then we need to migrate one more time. When you make this migration the engine will warn you that you are making a nullable field non-nullable, and ask you to provide a default or indicate that you’ve already handled the existing rows yourself. Choose option 2.:

$ python manage.py makemigrations
$ python manage.py migrate
You are trying to change the nullable field 'admin' on organization to non-nullable without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
 1) Provide a one-off default now (will be set on all existing rows with a null value for this column)
 2) Ignore for now, and let me handle existing rows with NULL myself (e.g. because you added a RunPython or RunSQL operation to handle NULL values in a previous data migration)
 3) Quit, and let me add a default in models.py
Select an option: 2

Now that the database is all ready for the new group, let’s make sure the application is as well. If we tried to create a new organization now we would run into null reference errors because the admin group doesn’t exist. We’ll need to create the group before saving. There are a couple of ways to do this, but I like to use signals for this purpose. In the Organization model file, add this method using the pre_save signal:

@receiver(pre_save, sender=Organization)
def organization_pre_save(sender, instance, **kwargs):
    organization = instance
    # if the organization doesn't have admin and reviewer groups, add them now
    try:
        organization.admin
    except Group.DoesNotExist:
        index = 0
        group_base_name = organization.name[:50]
        group_constructed_name = group_base_name + " Admin"
        # if a group with the base name already exists, append the org id to make it unique
        while Group.objects.filter(name=group_constructed_name).count() > 0:
            index += 1
            group_constructed_name = "{0} {1} Admin".format(group_base_name, str(index))
        admin_group = Group(name=group_constructed_name)
        admin_group.save()
        organization.admin = admin_group

This method is called before any Organization object is saved, and if there is currently no admin group for the object it will create it and assign the group to the organization. You’ll notice that the method here looks similar to the one we wrote in the migration above, but there are a couple of differences. First, in the migration code we were able to ensure a unique group name by adding the organization ID into the name of the group if a naming conflict occurs. That’s not possible here because since the code is run before the organization is actually saved, the ID is not yet assigned. So in this method we can instead create a while loop that will find the next available sequential name for the admin group. The other major difference here is that we must be careful not to call organization.save() in this method because, recursion.

That’s it! You now should have a brand spankin’ new non-nullable, auto-generated foreign key reference field in your existing model!

Lessons learned

Along the way through this process I learned a few things that might be interesting to anyone who’s wanting to add a field like this. Here are some things I thought worth mentioning.

Lesson 1: Don’t try to create data in the same migration where you’re modifying schema.

Previously when I needed to modify some data right after making a schema change I would just plop a RunPython() operation right in the same migration as where I added the field. At first I thought that worked fine in this case, since migration was successful. The problem came when I tried to reverse the migration and got the following error:

cannot ALTER TABLE "organization_organization" because it has pending trigger events

The problem here was that the group deletions were not complete by the time it was trying to remove the new column from the database. I learned then that it’s best to create a separate migration where the only action was to create or delete the data. Which leads me to my next lesson…

Lesson 2: Don’t be afraid to fake a migration.

When I first was learning Django I was petrified of doing anything in migrations except for the standard makemigrations and migrate. I was so afraid of making a mistake! But recently I’ve learned that it’s not all that difficult to fix mistakes using different features of Django’s migration framework. One thing I’ve learned to use this time is the --fake flag for the migrate command. As mentioned above, I initially added the RunPython() operation straight into the migration where I had created the admin field. This was fine when I migrated, but when I tried to reverse the migration I got the error above. It was a bit of a catch-22 because in order to fix the error I actually needed to run the reverse method first.

Thankfully, I was able to fix the mistake by creating the empty migration, moving the Python code to the new migration, and running the migration using the –fake flag:

$ python manage.py migrate --fake organization 0009

Using that flag tells Django to acknowledge the 0009 migration as “migrated”, even though it doesn’t actually run any code from that migration – which is perfect for this case since the code had already run. From there I could reverse back to the previous 2 migrations without issue.

If you accidentally fake a migration and want to undo it you can easily do so by running the fake migration back to the pervious spot:

$ python manage.py migrate --fake organization 0008

Lesson 3: Write tests!

In the process of making these changes I didn’t really have a UI to test them since the UI is still a bit of a mess. So instead I wrote some automated tests and I am so glad I did! I caught several errors just by writing a really simple test that I would not have caught otherwise until a long way down the road. Here’s an example of a simple test I wrote:

def test__new_org__security_groups_created_with_name_conflict(self):
    new_org = Organization(name='Test Groups With Conflict Org')
    new_org.save()
    new_org2 = Organization(name='Test Groups With Conflict Org')
    new_org2.save()
    new_org3 = Organization(name='Test Groups With Conflict Org')
    new_org3.save()
    self.assertIsInstance(new_org2.admin, Group)
    self.assertEqual(new_org2.admin.name, "Test Groups With Conflict Org 1 Admin",
                     "Expected name conflict to be handled")
    self.assertEqual(new_org3.admin.name, "Test Groups With Conflict Org 2 Admin",
                     "Expected name conflict to be handled")

I hope this might help someone out there struggling with the same problem that I seem to keep running into. Do you know a better way to accomplish this issue? Please let me know in the comments – I’d love to hear about it!