Parallelize your builds with ANT

Today even most laptops have more than one cpu, but also most build processes do not utilize all cpu's thus waisting time. While my bnm tool allows to parellelized maven based builds, ANT provides the <parallel> tag. Unfortunatley this often kills the builds.

Why this?

Because ANT is not well prepared for parallel builds bacause there are build tasks which cannot run in parallel, like <junit>, <cobertura> and more. E.g. if those tasks try to update the same resource (e.g. junit wants to update the the test result overview).

thread count property

Basically it's a good idea to parallelize your builds. It shortens the build time, but the output may becomes quite unreadable. If you introduce a parameter to specify the thread count, you can easily supply a thread count of 1 to have serial builds to easy the error search.

  <parallel threadcount="${cpucount}">
    ...
  </parallel>

parallelize your modules

To parallelize your modules you simply follow the module dependency. Let's say you have 4 modules with this dependency:

then you can run B and C in parallel. The main build target could look like

  <target name="build-all">
    <ant antfile="../A/build.xml" dir="../A" target="build" inheritAll="false" />
    <ant antfile="../B/build.xml" dir="../B" target="build" inheritAll="false" />
    <ant antfile="../C/build.xml" dir="../C" target="build" inheritAll="false" />
    <ant antfile="../D/build.xml" dir="../D" target="build" inheritAll="false" />
  </target>

Using the <parallel> mechanism and the knowledge of the dependencies it would look like

  <target name="build-all">
    <ant antfile="../A/build.xml" dir="../A" target="build" inheritAll="false" />
    <parallel threadcount="${cpucount}">
      <ant antfile="../B/build.xml" dir="../B" target="build" inheritAll="false" />
      <ant antfile="../C/build.xml" dir="../C" target="build" inheritAll="false" />
    </parallel>
    <ant antfile="../D/build.xml" dir="../D" target="build" inheritAll="false" />
  </target>

Now you have reached the point where obscure problems can occur which are caused by missing synchronization. Let's assume each build invokes a junit test.

  <target name="test">
    <junit haltonfailure="on" printsummary="on" fork="yes" forkmode="once">
      ...
    </junit>
  </target>

If there is task tool which causes problems using parallel you need some synchronization. Maybe you can utilize the ANT <waitfor> task, but I consider <waitfor> as a hack not as a solution.

What I want - and you want - is a <synchronized> task. And here it is:

package de.bb.tools.ant;
import java.util.HashMap;

import org.apache.tools.ant.taskdefs.Sequential;

public class SyncTask extends Sequential {
    // serialize the access to the LOCKS HashMap
    private final static Object SERIALLOCK = new Object();
    // keep objects for different locks
    private final static HashMap<String, Object> LOCKS = new HashMap<String, Object>();
    // the default lock
    private final static Object NULLLOCK = new Object();

    private String id = null;

    public void execute() {
        final Object lock;
        // obtain the lock from the HashMap
        if (id != null) {
            synchronized (SERIALLOCK) {
                Object l = LOCKS.get(id);
                if (l == null) {
                    // create it on first access
                    l = new Object();
                    LOCKS.put(id, l);
                }
                lock = l;
            }
        } else {
        // or use the default lock
            lock = NULLLOCK;
        }
        // here we go synchronized
        synchronized (lock) {
            super.execute();
        }
    }

    /**
     * Supply an ID for a named lock.
     */
    public void setId(String id) {
        this.id = id;
    }
}

Now we can serialize all tasks where needed:

  <taskdef name="synchronized" classname="de.bb.tools.ant.SyncTask"/>
  ...
  <target name="test">
    <synchronized id="junit" />
      <junit haltonfailure="on" printsummary="on" fork="yes" forkmode="once">
        ...
      </junit>
    </synchronized>
  </target>

You can use the same lock for all tasks or assign one lock to each specific task. Don't forget that using different locks plus nested locks opens the door to introduce dead locks!
You can download the JAR de.bb.tools.ant-1.0.0.jar containing the source and compiled class. It's released using the GNU GPLv3.
Best practice is to avoid nested synchronization.

But if you use nested synchronization, then use one template to build all your modules to stay safe, and/or track the locks (synchronized id values) and its order to avoid locking with different order.

Enjoy and use the saved time wisely.

rev: 1.9