New Adventures in Software by Dan Dyer

I recently upgraded a project from Django 2.2.24 to the latest 3.2.x release. This was mostly straightforward except for one problem that took a while to figure out.

I have a view that generates SVG and I have another view that generates HTML that embeds this SVG using an <Object> tag. For some reason I was getting “broken pipe” errors on the server when accessing the HTML view and the SVG would not load. Accessing the SVG directly was not a problem. This was the same whether using the development server or Apache with mod_wsgi.

I tried both Django 3.1 and 3.0 as well with similar results but different errors. With these versions the error was “Connection reset by peer”. For some reason the browser was closing the connection before reading the SVG.

I couldn’t find an explanation and it wasn’t urgent so I gave up for a while. Today I took another look and found the reason could be found in the Django 3.0 release notes:

X_FRAME_OPTIONS now defaults to 'DENY'. In older versions, the X_FRAME_OPTIONS setting defaults to 'SAMEORIGIN'. If your site uses frames of itself, you will need to explicitly set X_FRAME_OPTIONS = 'SAMEORIGIN' for them to continue working.

This is a change to prevent click-jacking attacks. Evidently browsers treat <object> elements the same way as an <iframe> in this respect. You can resolve the issue globally as described in the release notes above, or you can decorate the view in question to allow it to be embedded:

from django.views.decorators.clickjacking import xframe_options_sameorigin

@xframe_options_sameorigin
def svg_view(request):
    # Do something

About 8 years ago I made the decision to use TestNG for unit testing a Java project at work because it offered a much neater approach to the problem than JUnit 3.x. The only problem was that the HTML reports it generated were ugly. So I quickly hacked together a plug-in using Apache Velocity that would present the information in a way that I preferred and called it ReportNG. There was nothing elegant about the solution and the reports were functional but not particularly attractive, but it scratched an itch. It seemed to be an itch that irritated a lot of other developers because it attracted probably more users than any other, better written code that I’ve open-sourced before or since. I have made no substantial changes to it in years but there are still people using it.

I don’t do nearly as much Java as I used to and where I do I am either not using ReportNG or it is sufficient as it is. For these reasons, and because it ought to be replaced by something better (which probably already exists somewhere – I haven’t looked), I wanted to make it explicit to people still relying on it that there will be no further updates to ReportNG. You have three options:

1. Continue using version 1.1.4 and accept that it will never get any better.
2. Fork it on GitHub and make whatever changes you like. The Apache licence is permissive enough to allow most things.
3. Stop using ReportNG and find something better.

Some time ago I wrote about my issues with Google and its anti-privacy agenda. It was my intention to avoid using Google services where possible because I don’t like Larry Page’s vision and Google’s (and Facebook’s and others’) tracking of individuals’ activity across the web. I have very little to hide but that’s not the point.

Two of the biggest enablers of Google’s surveillance machinery are Adsense and Google Analytics. These services give Mountain View a foothold on millions of properties across the web, from which they can observe users activities and track them from site to site. Today I finally got around to removing these from my own sites. I never paid much attention to the analytics anyway and there are more privacy-friendly alternatives if required. Adsense never generated enough income to compensate for the cheapening of the message.

Other unrelated changes I’ve made to this blog include changing the canonical domain to blog.dandyer.co.uk instead of blog.uncommons.org (the old URLs will redirect). In addition, some time ago I disabled comments. There were occasionally a few worthwhile responses but over 99.6% of all comments submitted were spam. The bottom half of the web is generally a cesspit. If you feel the need to respond to something I’ve written, you can contact me on Twitter or e-mail dan at this domain.

Back in 1999, I was one of a group of Computer Science students invited to visit the research labs of a large consumer electronics company. I don’t remember a great deal about the prototype products we were shown but I do clearly remember being told that wearable computing was the next big thing. 15 years later wearable computers are still the next big thing.

This time around the devices have made it beyond the lab with a slew of underwhelming “smart” watches already on sale. In addition, Apple has been rumoured to be preparing its own for years now. Google on the other hand has taken a different approach by creating a new category of product with Google Glass.

It’s been over two years since Glass sky-dived into public view but it remains subject to a pretty exclusive public beta that you have to pay a hefty premium to join ($1500/£1000). Until a few weeks ago Glass headsets could only be purchased in the US. They are now also available in the UK. On Wednesday evening Google held its first European Glass developer meet-up at Skills Matter in London.

Since first hearing about Glass I’ve been deeply sceptical about it. It’s clearly capable of doing a few neat things that could be useful in a few niche areas but it just seems so inessential, it looks thoroughly ridiculous, and the current price tag is not destined to appeal to sensible individuals. However, I’ve often been too dismissive of new technologies in the past, so I was prepared to at least give it a go.

There was a full house at the Skills Matter Exchange including several Glass-adorned Googlers and a surprising number of other people who had presumably been parted from a grand of their own money. Interestingly, almost everybody had opted for the version with the plastic lenses. When the alternative is the bizarre lens-less titanium forehead band with nose perch it is entirely understandable, whether you need vision correction or not. The spectacle facade makes Google Glass look a lot less conspicuously weird. It’s still not a good look though, even without the beyond-parody third-party add-ons.

Google developer advocates Hoi Lam and Timothy Jordan delivered a couple of presentations suggesting how you should approach building apps for Glass (or Glassware as Google likes to call them). One major drawback for those who might be interested in building these apps is that at present there is no direct way for developers to make money from developing Glassware. Presumably that has to change at some point but for now apps can only be distributed free-of-charge (subject to Google’s approval), and in-app advertising is, mercifully, banned.

Following the presentations, those of us who hadn’t experienced Glass firsthand were given the opportunity to try out the headsets. Due to time constraints and the number of people who wanted to have a go, we didn’t get long enough to be able to get a feel for what it would be like to have this thing on your face all day but here are a few things I noted that might be of interest to those who haven’t tried one of the devices yet.

1. The early promo shots of Glass tended to avoid showing the battery pack that rests behind your right ear and those pictures that did show it made it look awkwardly bulky. In reality this battery is quite thin and the headset is not as heavy as it looks.
2. As a POV camera, Glass works well. It’s very easy to take snapshots or video (although I didn’t get an opportunity to check the quality of the results on a bigger screen). Unfortunately, according to Scoble, the battery is only good for 45 minutes of video and it will cook your face in the process. If all you care about is POV photography there are probably much better/cheaper options available.
3. In a room with a lot of background conversation, the voice recognition worked well. The microphone clearly does a good job of isolating the voice of the wearer.
4. The “screen” was underwhelming, albeit in a poorly-lit environment. The resolution wasn’t great and the focus didn’t feel entirely comfortable. It may well have been possible to adjust the focus but I didn’t have the time to find out.
5. The user interface doesn’t feel like it would scale well to having a large number of apps installed. At the moment there is a lot of swiping through cards in a linear fashion.
6. Unsurprisingly, Glass appears to be tightly integrated with Google+.

I was never going to be a person who would consider paying £1000 for one of these devices but having tried it very briefly I’m now certain that I wouldn’t buy one at a lower price either, even if I ignored the way it looks. There is, as yet, no compelling use case for the average person. One of the main features is that you can get your e-mails, SMS messages and other Android notifications beamed directly on to your retina without having to remove your phone from your pocket. I really don’t have any need for that kind of urgency. I’d rather ignore interruptions until I choose to deal with them. That’s not to say that there aren’t people who could find a use for Glass given better battery life and a more attractive price tag, but for most of us it’s a clever solution in search of a problem – the hardware equivalent of Google Wave.

Following my detour to Djangoland, the last week or so I have been back in the world of mobile app development. One of the things I’ve been working on is updating an Android app that has two versions built from the same codebase. Part of this update has been to re-skin both versions of the app and to make sure that everything looks reasonable on various devices running versions of Android from 2.2 up to the latest 4.4. When I build a new APK I want to check it on each of the three devices connected to my machine. If I had more spare USB ports I would conceivably be testing with even more devices. In some scenarios I might also have one or more emulators running to test configurations not represented among my devices. Installing the new APK on each of these from the command line is cumbersome. The adb tool will only install on one device/emulator at a time and, if you have more than one connected, you have to specify a long unique device ID to tell it which one to use. You could write a bash script to retrieve the IDs of connected devices and emulators and to run the adb install command for each, but since I’m using Ant I have written a custom task to do the job. For bonus points I’ve made it issue the adb commands in parallel so that the total install time is determined by the slowest device and not the sum of all the devices.

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
 
/**
 * Custom task that uses the ADB tool to install a specified APK on all
 * connected devices and emulators.
 * @author Daniel Dyer
 */
public class InstallAPK extends Task
{
    private File apkFile;
 
    public void setAPK(File apkFile)
    {
        this.apkFile = apkFile;
    }
 
    @Override
    public void execute() throws BuildException
    {
        if (apkFile == null)
        {
            throw new BuildException("APK file must be specified");
        }
        try
        {
            List<String> devices = getDeviceIdentifiers();
            System.out.printf("Installing %s on %d device(s)...%n", apkFile, devices.size());
            ExecutorService executor = Executors.newFixedThreadPool(devices.size());
            List<Future<Void>> futures = new ArrayList<Future<Void>>(devices.size());
            for (final String device : devices)
            {
                futures.add(executor.submit(new Callable<Void>()
                {
                    public Void call() throws IOException, InterruptedException
                    {
                        installOnDevice(device);
                        return null;
                    }
                }));
            }
            for (Future<Void> future : futures)
            {
                future.get();
            }
            executor.shutdown();
            executor.awaitTermination(60, TimeUnit.SECONDS);
        }
        catch (Exception ex)
        {
            throw new BuildException(ex);
        }
    }
 
    private void installOnDevice(String device) throws IOException, InterruptedException
    {
        String[] command = new String[]{"adb", "-s", device, "install", "-r", apkFile.toString()}
        Process process = Runtime.getRuntime().exec(command);
        consumeStream(process.getInputStream(), System.out, device);
        if (process.waitFor() != 0)
        {
            consumeStream(process.getErrorStream(), System.err, device);
            throw new BuildException(String.format("Installing APK on %s failed.", device));
        }
    }
 
    private void consumeStream(InputStream in, PrintStream out, String tag) throws IOException
    {
        BufferedReader reader = new BufferedReader(new InputStreamReader(in));
        try
        {
            for (String line = reader.readLine(); line != null; line = reader.readLine())
            {
                out.println(tag != null ? String.format("[%s] %s", tag, line.trim()) : line);
            }
        }
        finally
        {
            reader.close();
        }
    }
 
    private List<String> getDeviceIdentifiers() throws IOException, InterruptedException
    {
        Process process = Runtime.getRuntime().exec("adb devices");
        List devices = new ArrayList<String>(10);
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        try
        {
            for (String line = reader.readLine(); line != null; line = reader.readLine())
            {
                if (line.endsWith("device"))
                {
                    devices.add(line.split("s")[0]);
                }
            }
            if (process.waitFor() != 0)
            {
                consumeStream(process.getErrorStream(), System.err, null);
                throw new BuildException("Failed getting list of connected devices/emulators.");
            }
        }
        finally
        {
            reader.close();
        }
        return devices;
    }
}

Once compiled and on your classpath, using it is simple:

<taskdef name="installapk" classname="yourpackage.InstallAPK" />
<installapk apk="path/to/yourapp.apk" />

I’ve been dabbling with Django over the last couple of months. I’ve played around with it a couple of times previously but this time I’ve actually built something reasonably substantial, which has meant that I’ve had to delve a bit deeper. One of the minor problems I solved today was how to group items in the HTML <select> element generated by a form’s ModelChoiceField. HTML has the <optgroup> tag for this purpose.

It’s not immediately obvious how you can get ModelChoiceField to use optgroups without over-riding the render method and re-implementing the HTML generation yourself, or bypassing ModelChoiceField completely and building something based on the basic ChoiceField (which does support optgroups). The only potential solutions I found from searching took the former approach (here and here). The reason I’m writing this post is because I think I’ve found a better, more concise solution that might be of use to future searchers.

ChoiceField accepts a choices parameter to its constructor. In the simple case this is just a list of items (each item is value/label pair). However, it can also accept a list of groups where each group is a tuple consisting of the group label and a list of items. The problem is that ModelChoiceField is different in that it has a queryset parameter instead, so there is no way to pass in the group information.

However, a comment in the source code says that we can set a property called choices after constructing the ModelChoiceField instance and the queryset will be ignored. The HTML <select> will instead be populated from this data structure with <optgroup> elements as required.

Assuming we want to group items by a field on the model, we can build the list of tuples from the queryset by sub-classing ModelChoiceField and over-riding the constructor. In this example I’m assuming that the field is a list of countries grouped by continent, where the continent is just a text field on the country model.

from django.forms import ModelChoiceField
from itertools import groupby
from operator import attrgetter
 
class CountryChoiceField(ModelChoiceField):
    def __init__(self, *args, **kwargs):
        super(CountryChoiceField, self).__init__(*args, **kwargs)
        groups = groupby(kwargs['queryset'], attrgetter('continent'))
        self.choices = [(continent, [(c.id, self.label_from_instance(c)) for c in countries])
                        for continent, countries in groups]

In order for this to work, the queryset must be sorted by the field that it is to be grouped by so that items in the same group are adjacent. You can do this before you pass it to the constructor or you can change the code above to call order_by on the queryset.

It’s been a couple of years since the last release of ReportNG, my cobbled together alternative reporting plug-in for TestNG. In the time since I’ve done nothing but a few users have submitted useful improvements. So, after some prompting, I’ve released version 1.1.4 (download zip, tgz).

Thanks to Kayla Nimis, criccio, Arcadie, and Nalin Makar the new version has the following improvements:

• The report now shows the reason for skipping a test.
• You can specify the org.uncommons.reportng.failures-only property to generate a more minimal report.
• ReportNG will now create any missing parent directories of the report directory rather than failing if they are absent.

Google policy is to get right up to the creepy line and not cross it.

These are the words of former Google CEO Eric Schmidt, speaking to The Atlantic in October 2010.

We don’t need you to type at all. We know where you are. We know where you’ve been. We can more or less know what you’re thinking about.

This was two and a half years ago. Today Google is so far over the creepy line it can’t even see the line any more. The problem is not so much individual Google products but the way in which the vast data from all these disparate services is combined to build a very detailed profile of you. Not just what you do on Google sites but also any other site you visit that includes Google’s +1, Adsense or Analytics JavaScript (including this one). Google can read every e-mail you send and receive, it knows everything you search for online and knows pretty much every website you ever visit. It knows where you are, where you’ve been and probably with whom. Google knows more about you than your mother does and with Google+ it’s all neatly connected to your real identity.

I’ve so far avoided signing up for Google+ but it’s increasingly difficult to ignore, particularly as an Android developer, as it’s becoming more tightly integrated into everything that Google does. It’s the keystone of Google’s anti-privacy agenda.

The privacy implications are not the only issue. Most users perceive Google’s search results to represent some sort of objective truth, with links ranked only by their relevance to the search query, but in fact each user is served personalised results. If you and I both search for information on some contentious political topic, Google won’t necessarily give us the same response; it will show us each what it thinks we want to see based on its profiling of us.

In his Google I/O Q&A yesterday, Google CEO Larry Page dismissed concerns over the implications of this profiling:

[Audience member]: Most of my opinion, I can trace back to a Google search. As search becomes more and more personalized, and predictive, I worry that it informs my world view and rules out the possibility of some other serendipitous discovery. Any comment on that?

[Page]: People have a lot of concern about that – I’m totally not worried about that at all.

Personally, I don’t like Google’s all-encompassing vision. It’s not the only company that employs such methods but I can easily ignore the likes of Facebook and Bing as I don’t use them. Google is everywhere.

Spurred more by the closure of Google Reader than anything else, a couple of months ago I began to consider alternatives to relying on the benevolence of one omniscient company for the services I use every day.

The first thing to go was GMail. E-mail is important enough to be worth paying for so I signed-up with FastMail (now owned by Opera), which offers an ad-free service from $4.95 per year. It has a refined web interface, IMAP access, and a simple way to import your existing messages from GMail.

I eventually replaced Google Reader with The Old Reader and for search I’m now using DuckDuckGo, which promises not to track its users or filter search results. DuckDuckGo doesn’t quite match Google in terms of the freshness or depth of its results but it’s fast and has some useful features. On DuckDuckGo’s recommendation I also installed Ghostery for Opera to block Google and others’ attempts to track me on third-party sites. I’ve removed the Google +1 buttons from this blog and will be looking for more privacy-friendly alternatives to Google Analytics and Adsense (both of which would be useless anyway if everybody is driven to use the likes of Ghostery).

I’m not giving up on Google entirely, I just prefer to keep it at arm’s length. I’ll still be developing for Android (for which there were many very welcome developer announcements at I/O yesterday) and no doubt I’ll continue to use some of its other services.

A post with code in it. Because it’s been a long time since the last one.

If you work with ListViews in Android, as all Android developers will do at some point, you may notice that if you set your list items to be non-selectable the dividers that are drawn between each cell disappear. In the Holo Light theme you would normally get a thin light grey dividing line between cells. Depending on how you’ve implemented your adapter, you may find that these dividers become white/transparent when the cells are not selectable. According to Android framework engineer Romain Guy, this is the intended behaviour.

As Haythem Souissi points out in this Stack Overflow answer, you can work around this by ensuring that the areAllItemsEnabled method returns true, even though all items are not enabled (maybe none of them are). The isEnabled method will take care of actually disabling the cells and the dividers will be drawn between each of them.

All very straightforward so far but it all goes wrong again when you try to add a non-selectable header view to the list. The way that ListView deals with headers and footers is that it creates its own adapter to wrap/decorate yours and insert the header/footer views in the appropriate places. This is fine but it’s not delegating the areAllItemsEnabled method, so our above fix no longer works.

Fortunately, and unusually for Android, everything we need to resolve this issue is part of the public API. The adapter decorator class is android.widget.HeaderViewListAdapter. We just need to create our own instance, wrapping our own adapter, and override areAllItemsEnabled as above. There’s a slight complication in that we have to wrap the header view in an instance of ListView.FixedViewInfo and this is a non-static inner class of ListView, but we can reach into our bag of obscure Java tricks to create an instance from outside the enclosing class.

// I assume you know how get a reference to the ListView and create your own adapter.
ListView listView = (ListView) view.findViewById(android.R.id.list);
CustomAdapter adapter = new CustomAdapter(context, listOfItems);
 
// You can create any view you like for the header.
TextView listHeader = (TextView) inflater.inflate(R.layout.list_header, null);
listHeader.setText("My Header Text");
 
// This is how you create an instance of the non-static nested class, providing
// it with a reference to an instance of the containing class (ListView).
ListView.FixedViewInfo headerInfo = listView.new FixedViewInfo();
 
headerInfo.view = listHeader;
headerInfo.isSelectable = false;
 
// HeaderViewListAdapter insists on concrete ArrayLists.
ArrayList headers = new ArrayList(1);
headerInfoList.add(headerInfo);
ArrayList footers = new ArrayList(0);
 
HeaderViewListAdapter wrapper = new HeaderViewListAdapter(headers, footers, adapter)
{
    @Override
    public boolean areAllItemsEnabled()
    {
        return true;
    }
};
listView.setAdapter(wrapper);

On 12th October 2011 Apple unveiled iOS 5.0 to the world. 7 days later Google released Android 4.0 (Ice Cream Sandwich).

Apple rarely announces how many devices are running each version of its mobile operating system but, according to third-party stats^[1], today only a few percent of iPhone/iPad/iPod Touch owners are still running an earlier version of iOS.

In contrast, Google is much more up-front about how many people are running each version but it doesn’t compare favourably. Just 39.3% of users have access to the 15-month-old Android OS or one of its later revisions.

Of course, there are some mitigating factors. Android is a much more open ecosystem with dozens of manufacturers needing to test and approve updates for hundreds or even thousands of devices. Apple on the other hand retains a strict control over its (smaller) mobile empire and is able to push out updates directly to end users. But whatever the reasons, and despite Google’s recent efforts to rein in manufacturers and network operators, the fact remains that, unlike their iOS counterparts, Android app developers cannot target a recent version of the operating system and hope to have their app runnable by the majority of device owners.

Google has tried to address this issue with its support library, which brings some of the more recent API additions to earlier Android versions but this is only a partial solution that adds complexity and does nothing to resolve the vast visual differences between Android 4.x and its predecessors.

Today if you want to reach over 90% of iOS users you must support iOS 5.1, which was released in March 2012, and later versions. To reach the same proportion of Android users you would have to target Android 2.2 (Froyo), which dates back to May 2010.

The many comparisons of the relative merits of the latest iOS versus the latest Android version are largely irrelevant to Android app developers who are working from a baseline that is almost two years older than that of their iOS counterparts. An iOS developer can build an app that requires the absolute latest version of the operating system confident that the user base will soon be there (iOS 6 has reached 78.5% penetration in four months). Android developers will always need to be more conservative.

1. These stats may not be fully representative of all iOS users but they provide useful ballpark figures.