New Adventures in Software by Dan Dyer

Following my detour to Djangoland, the last week or so I have been back in the world of mobile app development. One of the things I’ve been working on is updating an Android app that has two versions built from the same codebase. Part of this update has been to re-skin both versions of the app and to make sure that everything looks reasonable on various devices running versions of Android from 2.2 up to the latest 4.4. When I build a new APK I want to check it on each of the three devices connected to my machine. If I had more spare USB ports I would conceivably be testing with even more devices. In some scenarios I might also have one or more emulators running to test configurations not represented among my devices. Installing the new APK on each of these from the command line is cumbersome. The adb tool will only install on one device/emulator at a time and, if you have more than one connected, you have to specify a long unique device ID to tell it which one to use. You could write a bash script to retrieve the IDs of connected devices and emulators and to run the adb install command for each, but since I’m using Ant I have written a custom task to do the job. For bonus points I’ve made it issue the adb commands in parallel so that the total install time is determined by the slowest device and not the sum of all the devices.

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PrintStream;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
 
/**
 * Custom task that uses the ADB tool to install a specified APK on all
 * connected devices and emulators.
 * @author Daniel Dyer
 */
public class InstallAPK extends Task
{
    private File apkFile;
 
    public void setAPK(File apkFile)
    {
        this.apkFile = apkFile;
    }
 
    @Override
    public void execute() throws BuildException
    {
        if (apkFile == null)
        {
            throw new BuildException("APK file must be specified");
        }
        try
        {
            List<String> devices = getDeviceIdentifiers();
            System.out.printf("Installing %s on %d device(s)...%n", apkFile, devices.size());
            ExecutorService executor = Executors.newFixedThreadPool(devices.size());
            List<Future<Void>> futures = new ArrayList<Future<Void>>(devices.size());
            for (final String device : devices)
            {
                futures.add(executor.submit(new Callable<Void>()
                {
                    public Void call() throws IOException, InterruptedException
                    {
                        installOnDevice(device);
                        return null;
                    }
                }));
            }
            for (Future<Void> future : futures)
            {
                future.get();
            }
            executor.shutdown();
            executor.awaitTermination(60, TimeUnit.SECONDS);
        }
        catch (Exception ex)
        {
            throw new BuildException(ex);
        }
    }
 
    private void installOnDevice(String device) throws IOException, InterruptedException
    {
        String[] command = new String[]{"adb", "-s", device, "install", "-r", apkFile.toString()}
        Process process = Runtime.getRuntime().exec(command);
        consumeStream(process.getInputStream(), System.out, device);
        if (process.waitFor() != 0)
        {
            consumeStream(process.getErrorStream(), System.err, device);
            throw new BuildException(String.format("Installing APK on %s failed.", device));
        }
    }
 
    private void consumeStream(InputStream in, PrintStream out, String tag) throws IOException
    {
        BufferedReader reader = new BufferedReader(new InputStreamReader(in));
        try
        {
            for (String line = reader.readLine(); line != null; line = reader.readLine())
            {
                out.println(tag != null ? String.format("[%s] %s", tag, line.trim()) : line);
            }
        }
        finally
        {
            reader.close();
        }
    }
 
    private List<String> getDeviceIdentifiers() throws IOException, InterruptedException
    {
        Process process = Runtime.getRuntime().exec("adb devices");
        List devices = new ArrayList<String>(10);
        BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        try
        {
            for (String line = reader.readLine(); line != null; line = reader.readLine())
            {
                if (line.endsWith("device"))
                {
                    devices.add(line.split("s")[0]);
                }
            }
            if (process.waitFor() != 0)
            {
                consumeStream(process.getErrorStream(), System.err, null);
                throw new BuildException("Failed getting list of connected devices/emulators.");
            }
        }
        finally
        {
            reader.close();
        }
        return devices;
    }
}

Once compiled and on your classpath, using it is simple:

<taskdef name="installapk" classname="yourpackage.InstallAPK" />
<installapk apk="path/to/yourapp.apk" />

There are good reasons to want to build two versions of the same Android application. The most common scenario is to produce a free demo/reduced-functionality version of a non-free app. You could achieve this by maintaining two separate source trees but the duplication would make most developers wince.  So the question is how to build two differing apps without duplicating the code and resources?

Building Android Apps with Custom Ant Scripts

The answer is Ant. Maybe it’s also possible with Eclipse. I doubt it but I don’t know and I don’t care to find out. As I mentioned previously, the Android Ant build system is fairly painless to use. If you want to customise it though you’ll have to do a fair bit of digging as it doesn’t appear to be documented anywhere. The first place to look is the <sdk_dir>/platforms/<target_dir>/templates/android_rules.xml file that is used by the build.xml that the Android tools generate for you. This will give you a good idea of the various stages of the build but if you want to experiment with the Android Ant tasks you’ll have to look at the source to see what attributes they support (making sure that the version you are looking at is the same as the version you are using). The AaptExecLoopTask is just a simple wrapper around the aapt tool but the ApkBuilderTask is non-trivial.

The outline of a minimal Ant build for an Android app looks something like this:

1. Generate the R.java file from the resources using the the exec task and the aapt tool with the -J option.
2. Compile all of the app source, including the class generated in step 1.
3. Use the dx tool to convert the Java bytecode into a .dex (Dalvik executable) file.
4. Use the aapt tool (either via the exec task or via the provided AaptExecLoopTask) to package the resources.
5. Build the .apk package using the ApkBuilderTask.
6. If you are building a release package rather than a debug package, sign the .apk file using the signjar task.
7. Use exec to run the zipalign tool on the .apk file.

More advanced applications will involve additional steps, such as running the aidl tool.

Once you have the outline of your build in place, you can start to customise it using your standard issue Ant prowess.

Different Resources for Different Versions

In my scenario I don’t mind shipping all of the code with both versions of the application. The difference is just the initial Activity that each uses. It doesn’t add much bloat to the package and the code is useless without the full set of resources so there is no danger of somebody hacking the demo version into a fully functioning version. In other words, I don’t need to worry about excluding certain classes from one version or the other.

I do however only want to include a subset of the resources in the demo version.  The way I have achieved this is to replace the default res directory with three directories: res-common, res-demo and res-full. I then use the Ant copy task to construct the res directory from the appropriate pair of directories prior to building.

Different Manifests

Because I want to invoke different activities, I need to define the AndroidManifest.xml differently for each version. My first idea was to just have two different files with different names but I discovered that the Android tools get upset if the file is not called AndroidManifest.xml, even if you explicitly specify which file to use. This just means that I have to copy the chosen manifest so that it has the correct name.

The biggest problem with building two versions from the same tree is resolving conflicts in the application package name. Each Android application must have a unique application package. This is specified in the manifest. You can just use the same package name for both versions but users will run into problems if they ever try to install both versions at the same time. They will likely end up not being able to use either. Despite the drawbacks, this method seems to be widely used judging by the number of apps in the Android Market that warn users to uninstall the demo version first.

So we’ll just change the package name in one of the manifests then? Yeah, that would be nice. The problem is that the package name specified in the manifest is the package into which the R class is generated. If the two versions of your application have their R classes in different packages then common classes will not compile for both versions.

Android 2.1 introduces the --custom-package option for the aapt tool which allows you to over-ride where R.java will be generated. So the idea is to set the application package differently in one of the manifests but then to use this option to make sure that R.java is still in the same place as in the other version. I tried this and it resolved my compile problems but there were resource problems at runtime. I didn’t fully investigate what was wrong because I found another approach that appears to work.

The aapt tool also has the --rename-manifest-package option. Leaving the application package the same in both manifests and then using this option at step 4 above, I was able to generate a working demo version and full version from the same source tree and have them both functional when installed at the same time.

The whole custom Ant build exercise was not nearly as straightforward as I would have liked, mostly due to lack of documentation, but it is at least a working solution to the problem. Another, probably more recommended, way of achieving something similar would be to use library projects.

1. Automate everything.

You’re in charge, make the computer do the work. Ant can insert configuration into properties files, run SQL scripts, change file permissions, upload to FTP sites, and lots more. RTFM and make sure you are aware of everything that Ant can do for you.

2. Make builds Atomic.

Once you’ve automated everything, make sure you can run it all with a single command.Most software engineering endeavours are part of the eternal struggle to limit the consequences of human incompetence. Unlike a computer, the human mind is not well suited to performing a long sequenece of instructions without error. People get distracted, they forget things. My own experience tells me that the longest sequence of actions that a human can reliably reproduce consists of at most one step. Anything more than that and there is a tendency to miss steps out or do things in the wrong order. A slow build is much more bearable if it is atomic because you don’t need to get involved. You can leave it to complete while you do other important stuff.

3. Make it right first, then make it fast.

A build that is fast but wrong is not fast at all; it’s just wrong. A target that does not correctly ensure that its dependencies are rebuilt (when appropriate) will save you 3 seconds on each of the 30 rebuilds that it takes to debug the problem.A useful tip for improving the runtime of slow builds is to use the ANT_OPTS environment variable to tweak the JVM settings used by Ant. For long-running builds you may find that the -server switch provides a noticeable improvement. Alternatively, increasing the heap size (via the -Xmx switch) may help.

4. If you use Ant, use only Ant.

The build.xml is the one true build script. Don’t disrespect the build.xml by using your IDE’s build process in its place. Even if you are really diligent about configuring your IDE, you are very likely to end up with a slightly different build process. Better still, when your colleague checks in a change to build.xml, he or she won’t tell you and your build will be broken. If you’re lucky this will become obvious sometime after you start complaining loudly about others checking in broken code. In the worst case your build will be subtly broken and you won’t notice. Avoid the contempt of your co-workers and make your IDE use Ant. Don’t let the old UNIX hacker in the corner use make just because he doesn’t like XML (he doesn’t like Java either, he’s just doing it because the company hasn’t had any C work in the last 6 months 10 years). If he uses make, you all have to use make (good luck), since there can be only one true build script.

5. Follow conventions.

Ant expects your build file to be called build.xml (unless you tell it otherwise). Developers expect the build file to be in the root of the project. Other than the understandable desire to confuse and irritate the rest of your team, including the lucky souls that get to maintain your code long after you get head-hunted by Google, there’s no reason not to follow these conventions. Another less well-known convention is to prefix all “internal” targets (those that should not be called directly) with a hyphen. This is suggested in the Ant manual. It has the advantage that it is not possible to invoke targets that follow this naming scheme from the command line.

6. Provide a clean-up target.

It is usual for makefiles and Ant scripts to provide a target called “clean” that removes all generated files and returns the project to its initial state. Without this functionality failed builds can be a real problem. Providing a clean target is made easier if source files (those that are under version control) and generated files (those that are derived from versioned files) are kept separate.

7. Compile test code at the same time as production code.

Test code, by definition, depends on production code (if it didn’t, your test coverage scores would not be very impressive). Therefore, changes to production code can potentially break test code. For this reason it is a good idea to enforce the rule that test classes are always compiled at the same time as production classes (you don’t have to run the tests at this point). Since you are firing up the compiler anyway, it won’t add too much overhead and it avoids breaking windows.

8. Make builds self-contained.

A build that has external dependencies is a build that is difficult to to configure. When your hard drive fails or a new developer joins the team, you want to be able to configure a new machine for building ASAP. If your build depends on tools like Checkstyle or TestNG, add these to the project repository. This makes your build simpler because you know exactly where to find the files and you don’t have to worry about version mis-matches. Other than Ant itself and the JDK, external dependencies should be eliminated wherever possible (Maven has a different approach to these kind of dependencies).

9. Parameterise essential configuration.

It’s not always practicable to eliminate all external dependencies. For example, you may need to configure a database connection as part of your build. Don’t hard-code this kind of configuration in the build.xml, use Ant’s support for properties files to allow individual developers to set configuration parameters. Don’t complicate matters by making everything configurable. Only parameterise those settings that really need to be configurable.

10. Provide sensible defaults.

Where parameters are required for building, you can make things much simpler by providing sensible default values rather than leaving everything to be configured by the individual developer. For example, if one of the parameters is a JDBC URL, a sensible default might be to point to localhost (most developers will probably be developing against their local database). If one of the required properties is the file system path to a particular tool, provide the default installation directory. By providing sensible defaults you can eliminate most of the configuration hassles.

11. Visualise dependencies.

The build script for a large project can have complex, often ad-hoc dependencies. Tools such as yWorks Ant Explorer can provide insight into dependencies by providing a graphical representation of the structure of the build. If the graph looks like one of these, consider refactoring.

12. Constrain classpaths.

Avoid the temptation to use a single global classpath for compilation. Always ensure that code is built with a classpath that contains only those classes that will be available to it at runtime. For example, if your build creates an EAR file and a Swing client, make sure that the Swing client is compiled with access only to those libraries that will be deployed with it. Do the same for the EAR file and any other modules. This approach minimises the potential for embarrassing NoClassDefFoundErrors at runtime and also helps to detect inappropriate dependencies.

13. Modularise your build.

Arrange your project into coherent, self-contained modules. For example, you may have a Swing GUI module and web module among others. Layout your project files to support this arrangement. The goal is to make modules as self-contained as possible and to minimise dependencies between them. The following example shows one possible way of arranging your project files:

myproject
|__modules
|  |__gui
|  |  |__lib
|  |  |__src
|  |     |__java
|  |        |__main
|  |        |__test
|  |__web
|     |__lib
|     |__src
|        |__html
|        |__java
|           |__main
|           |__test
|__build.xml

14. Favour convention over configuration.

If the source code for module1 is in module1/src/java, where would you expect to find the source code for module2? If files are in consistent predictable locations within each module, it’s not necessary to explicitly configure each location. By always following the same conventions about where things go in each module, you minimise complexity and make it easier to implement the next recommendation…

15. Eliminate duplication with macros.

If all of your modules are laid out identically, the common tasks (such as compiling source trees and creating JAR files) are essentially identical for each module. Ant 1.6 introduced macros to enable these operations to be defined just once and applied repeatedly with different parameters. Familiarise yourself with the Macrodef task and consider how it can be applied in your build files. The example macros below are for compiling generic modules. The first macro defines how a single source tree is compiled. Because we have followed the advice from the rest of this article, all we need to provide to run the macro is the name of the module (by convention modules are in directories of the same name, all Java source is in the src/java directory of the module and below this are the main and test source trees). The second macro simplifies things further by combining the two separate calls to the first macro. The main classes are built, then the test classes are built with the main classes added to the compiler’s classpath. Using this macro we can compile an entire module with a single line:

<compilemodule name="mymodule" />

<!-- This macro compiles one source tree (i.e. the main
     source tree or the unit test source tree) of a given
     module. -->
<macrodef name="compiletree">
  <attribute name="module"/>
  <attribute name="tree"/>
  <element name="treeclasspath" optional="true"/>
  <sequential>
    <mkdir dir="./@{module}/build/classes/@{tree}" />
    <javac destdir="./@{module}/build/classes/@{tree}"
                   debug="on"
                   deprecation="on"
                   optimize="on"
                   source="1.5"
                   target="1.5"
                   srcdir="./@{module}/src/java/@{tree}">
      <classpath>
        <treeclasspath/>
        <path refid="base.path"/>
      </classpath>
      <compilerarg value="-Xlint:unchecked" />
    </javac>
  </sequential>
</macrodef>
 
<!-- This macro compiles all source (including unit tests)
     for a single module.  -->
<macrodef name="compilemodule">
  <attribute name="name"/>
  <element name="moduleclasspath" optional="true"/>
  <sequential>
    <compiletree module="@{name}" tree="main">
      <treeclasspath>
        <moduleclasspath />
      </treeclasspath>
    </compiletree>
    <compiletree module="@{name}" tree="test">
      <treeclasspath>
        <!-- Add the main classes to the classpath for unit
                             test compilation. -->
        <path location="./@{name}/build/classes/main" />
        <moduleclasspath />
      </treeclasspath>
    </compiletree>
  </sequential>
</macrodef>

These macro definitions are somewhat verbose (due to Ant’s XML syntax), but the benefit is that they ensure that all modules are built in the same way. Rules for compilation only have to be defined once. Also, keep in mind that once you have written macros for one project, you can reuse them elsewhere. You may choose to put all of your macros in a separate file and import this wherever it is required.