New Adventures in Software by Dan Dyer

I recently upgraded a project from Django 2.2.24 to the latest 3.2.x release. This was mostly straightforward except for one problem that took a while to figure out.

I have a view that generates SVG and I have another view that generates HTML that embeds this SVG using an <Object> tag. For some reason I was getting “broken pipe” errors on the server when accessing the HTML view and the SVG would not load. Accessing the SVG directly was not a problem. This was the same whether using the development server or Apache with mod_wsgi.

I tried both Django 3.1 and 3.0 as well with similar results but different errors. With these versions the error was “Connection reset by peer”. For some reason the browser was closing the connection before reading the SVG.

I couldn’t find an explanation and it wasn’t urgent so I gave up for a while. Today I took another look and found the reason could be found in the Django 3.0 release notes:

X_FRAME_OPTIONS now defaults to 'DENY'. In older versions, the X_FRAME_OPTIONS setting defaults to 'SAMEORIGIN'. If your site uses frames of itself, you will need to explicitly set X_FRAME_OPTIONS = 'SAMEORIGIN' for them to continue working.

This is a change to prevent click-jacking attacks. Evidently browsers treat <object> elements the same way as an <iframe> in this respect. You can resolve the issue globally as described in the release notes above, or you can decorate the view in question to allow it to be embedded:

from django.views.decorators.clickjacking import xframe_options_sameorigin

@xframe_options_sameorigin
def svg_view(request):
    # Do something

I’ve been dabbling with Django over the last couple of months. I’ve played around with it a couple of times previously but this time I’ve actually built something reasonably substantial, which has meant that I’ve had to delve a bit deeper. One of the minor problems I solved today was how to group items in the HTML <select> element generated by a form’s ModelChoiceField. HTML has the <optgroup> tag for this purpose.

It’s not immediately obvious how you can get ModelChoiceField to use optgroups without over-riding the render method and re-implementing the HTML generation yourself, or bypassing ModelChoiceField completely and building something based on the basic ChoiceField (which does support optgroups). The only potential solutions I found from searching took the former approach (here and here). The reason I’m writing this post is because I think I’ve found a better, more concise solution that might be of use to future searchers.

ChoiceField accepts a choices parameter to its constructor. In the simple case this is just a list of items (each item is value/label pair). However, it can also accept a list of groups where each group is a tuple consisting of the group label and a list of items. The problem is that ModelChoiceField is different in that it has a queryset parameter instead, so there is no way to pass in the group information.

However, a comment in the source code says that we can set a property called choices after constructing the ModelChoiceField instance and the queryset will be ignored. The HTML <select> will instead be populated from this data structure with <optgroup> elements as required.

Assuming we want to group items by a field on the model, we can build the list of tuples from the queryset by sub-classing ModelChoiceField and over-riding the constructor. In this example I’m assuming that the field is a list of countries grouped by continent, where the continent is just a text field on the country model.

from django.forms import ModelChoiceField
from itertools import groupby
from operator import attrgetter
 
class CountryChoiceField(ModelChoiceField):
    def __init__(self, *args, **kwargs):
        super(CountryChoiceField, self).__init__(*args, **kwargs)
        groups = groupby(kwargs['queryset'], attrgetter('continent'))
        self.choices = [(continent, [(c.id, self.label_from_instance(c)) for c in countries])
                        for continent, countries in groups]

In order for this to work, the queryset must be sorted by the field that it is to be grouped by so that items in the same group are adjacent. You can do this before you pass it to the constructor or you can change the code above to call order_by on the queryset.

It’s called “Programming Collective Intelligence” and is presented as a book for building “Smart Web 2.0 Applications” but it is essentially an extremely accessible explanation of a wide array of machine learning and data-mining algorithms. How do sites like Amazon and Last.FM make recommendations? How do search engines work? How does Google News manage to categorise and present the most important news articles without human intervention? How do you build a useful spam filter?

All of these questions are answered and compelling example applications are built step-by-step to demonstrate the power of the ideas presented here. Decision trees, genetic algorithms, neural networks, support vector machines, genetic programming, Bayesian classifiers and non-negative matrix factorisation are some of the techniques covered and all without the dry, maths-heavy text that normally fills books on these topics.

The examples throughout are exclusively in Python, which may have put me off had I realised this when I ordered it. I have nothing against Python except for my complete lack of experience with it. However, the examples are easy enough to understand for anybody familiar with other high-level languages. As result of reading the book, I may actually try my hand at a bit of Python hacking now.

How well do these techniques work? Well I’d never have found out about this book but for Amazon’s automated recommendations system. I’d thoroughly recommend this book to anyone looking to learn about interesting AI techniques without wading through opaque academic papers.

(If you find the genetic algorithms and genetic programming topics interesting, check out the Watchmaker Framework for Evolutionary Computation and some of the books recommended there.)