Hudson - Continuous Integration for a Google App Engine application

The last blog post described how to configure a Maven project for a Google App Engine application. 

To build and deploy the Maven artifacts you will need some command line hacking.

Hudson CI can be used to escape the command line for all this. Hudson is a continuous integration system for building and testing your projects. It has some cool features as distributed building and much more.

To install Hudson, or Jenkins as the main fork has rebranded it now after Oracle came into clinch with the open source community, simply download the war-archive from either the Jenkins or Hudson web site. My installation is old so I use a Hudson build.You can either deploy it in a web container such as Tomcat or simply use the built in bootstrap. To bootstrap the war archive

java -Dhudson.udp=32850 -jar hudson.war --httpPort=9090 --daemon --logfile=/home/johan/hudson/hudson.log

Access Hudson via http://localhost:8080/.

Goto Manage Hudson -> Configure System and enable Maven under the Maven subsection. Goto Manage Hudson -> Manage Plugins and install the plugins you need. In my case I have installed Cobertura Plugin for code coverage, Findbugs for static code analysis, Maven2 Project Plugin, checkstyle for Java code validation and the CVS plugin. You might need to bring in some sub dependencies like Static Analysis Collector Plug-In, static Analysis Utilities.

Now create a new job, under your job, click Configure and setup appropriate version control config. In my case CVSROOT=:pserver:rankington:xxxx@213.xxx.xxx.xxx:/cvsrepo

and correct CVS module and branch.

Under Build, choose correct Maven installation, probably /usr/bin/mvn. Then setup your Maven build goals. In this case the goals are, enable debug, clean, compile, test, create war archive and generate reports.

-X

clean

package

findbugs:findbugs

checkstyle:checkstyle

This will make Maven checkout your code, compile it, run the Maven plugins for creating xml reports for Findbugs and Checkstyle.

Now add Post-build Actions to integrate these reports into Hudson. Enable Publish Findbugs analysis results from file **/target/findbugsXml.xml

Add similar report integrations by enabling publishing the following reports target/surefire-reports/*.xml for JUnit tests, **/target/site/cobertura/coverage.xml for Cobertura code coverage and so forth. 

 

Now build your Hudson job by clicking Build Now and the Maven goals are executed to checkout, build, test and creates reports of the project. Then the post action goals kicks in and updates the dashboards of Hudson to show the results of the build.

Cobertura allows you to see on a package level the unit test coverage and the possibility to drill down on package and file level. Similar graphs and drilling can be done on Findbugs, Checkstyle and JUnit test reports.

Cobertura reporting on file level

JUnit test reports over time, red indicates test cases have failed during those builds.

Checkstyle reports Java code issues and the interface makes drilling easy.

In this example Hudson will generate a war archive ready for deployment to a web or application server.

The Maven build could be extended to deploy the war at a local server to also run the web tests as a part of the Hudson job.

If you wish to also incorporate production deployment in the Hudson process you could use the Google App Engine scripting possibilities. Add a build step which does something like this to upload your newly built war archive to GAE.

appengine-java-sdk\bin\appcfg.cmd --email user@gmail.com --passin password update myapp/war

The same script can be used to download logs from production via

appengine-java-sdk/bin/appcfg.sh request_logs myapp/war mylogs.txt

Theres a bunch of more handy commands for scripting GAE if you run through the docs at http://code.google.com/intl/sv-SE/appengine/docs/java/tools/uploadinganapp.html.

Next time I'll try to describe what the production environment offers in addition to the local development environment.

Development environment for a Google App Engine application

There is integration with Google App Engine for several IDEs and you can also interact with the deployment tools from plain command line if you have that disposition.

I'm an Eclipse guy, and here's how I've done to get a good setup to be able to develop in an efficient way using Eclipsen, Maven, CVS, Hudson and GAE.

Fisrt of all, install the Google App Engine SDK from http://code.google.com/intl/sv-SE/appengine/docs/java/tools/eclipse.html and follow the install instructions.

You should get a new toolbar in Eclipse with icons for creating GAE web projects, profiling them and for deploying to the cloud. If you create a new GAE application you can have it deployed in Googles data centers in a few minutes. All you need is a standard Google Account.

A few tips:

• You access your application at http://localhost:8888/ by default.
• In your dev environment a light weight data store will be kept in WEB-INF\appengine-generated\local_db.bin. This store is permanent over server restarts and must be deleted if you wish a clean state. Indicies are automatically generated in WEB-INF\appengine-generated\datastore-indexes-auto.xml. You access the development console at http://localhost:8888/_ah/admin where you can inspect and make queries on the data store and work with task queues and more.
• In addition to the standard JEE files such as web.xml there are a few specific config files for GAE projects. Inspect WEB-INF\appengine-web.xml where you setup logging and turn on sessions. By default you will not create web sessions when accessing you application

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application>rankington</application>
    <version>1</version>
    
    <sessions-enabled>true</sessions-enabled>
    
    <!-- Configure java.util.logging -->
    <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
    </system-properties>
    
</appengine-web-app> 

If you wish to complicate your environment with some continous integration for compilation, bug, test and coverage reporting I'll outline how you could integrate your GAE application with Hudson.

I've fiddled around with the default directory structure to make a more Maven compliant tree for dependency management. That is no problem since the App Engine plugin uses the Eclipse .classpath to resolve the structure.  So create src/main/java src/main/resources /src/test/java src/test/resources and src/main/webapp. Create a pom.xml in your project root and setup your Maven dependencies. To make this easy, install the Eclipse Maven plugin which makes pom-editing a no-brainer. You will need a few special dependencies for App Engine. Be careful about which version of the plugin you use to match the Maven dependency.

<dependency>
   <groupId>com.google.appengine</groupId>
   <artifactId>appengine-api-1.0-sdk</artifactId>
   <version>1.4.0</version>
   <type>jar</type>
   <scope>compile</scope>
</dependency>

My Maven dependencies are:

It took me some time to figure out how to add unit testing, findbugs reporting and code coverage to my Maven build. So hopefully you can reuse something like this for your pom file. I've removed most of the code dependencies specific for my project for clarity

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>se.noren.rankington</groupId>
  <artifactId>Rankington</artifactId>
  <packaging>war</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Maven Quick Start Archetype</name>
  <url>http://maven.apache.org</url>
  <ciManagement>
  </ciManagement>
  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>
              <configuration>
                  <source>1.5</source>
                  <target>1.5</target>
            </configuration>
          </plugin>
          <plugin>
              <groupId>org.codehaus.mojo</groupId>
              <artifactId>surefire-report-maven-plugin</artifactId>
              <version>2.0-beta-1</version>
          </plugin>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-checkstyle-plugin</artifactId>
              <version>2.6</version>
          </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <version>2.2</version>
            <configuration>
                <formats>
                    <format>xml</format>
                </formats>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>cobertura</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>          
      </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-beans</artifactId>
        <version>3.0.5.RELEASE</version>
        <type>jar</type>
        <scope>compile</scope>
    </dependency>

  </dependencies>
  <reporting>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-checkstyle-plugin</artifactId>
              <version>2.6</version>
          </plugin>
          <plugin>
              <groupId>org.codehaus.mojo</groupId>
              <artifactId>findbugs-maven-plugin</artifactId>
              <version>2.3.1</version>
              <configuration>
                <findbugsXmlOutput>true</findbugsXmlOutput>
                <findbugsXmlWithMessages>true</findbugsXmlWithMessages>
                <xmlOutput>true</xmlOutput>
                  <!-- Optional directory to put findbugs xml report --> 
                  <findbugsXmlOutputDirectory>target</findbugsXmlOutputDirectory>
                  <effort>Max</effort>
                  <threshold>Low</threshold>
            </configuration>
          </plugin>
      </plugins>
  </reporting>

</project>

So once alright in Eclipse, commit your application to a version control system. I use CVS, but this will work fine with Subversion as well. In the example below I commited App Engine project Rankington on branch Develop.

Now you could build your GAE application from a command line system with Maven installed with something like

CVSROOT=:pserver:rankington@213.xxx.xxx.xxx:/cvsrepo
export CVSROOT
cvs login
cvs co -r Develop Rankington
cd Rankington/
mvn package

This will generate a directory structure with your compiled classes, a packaged war-file ready for deployment to Google App Engine, directories with reports of your unit tests, static code analysis bug possibilities and code coverage.

Go to subdirectory target and you'll find everything



drwxr-xr-x 11 johan johan     4096 2011-04-16 14:28 ./
drwxr-xr-x  8 johan johan     4096 2011-04-16 14:28 ../
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 classes/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 cobertura/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 generated-classes/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 maven-archiver/
drwxr-xr-x  8 johan johan     4096 2011-04-16 14:28 Rankington-1.0-SNAPSHOT/
-rw-r--r--  1 johan johan 23081938 2011-04-16 14:28 Rankington-1.0-SNAPSHOT.war
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 site/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 surefire-reports/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 test-classes/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 war/

Next time we'll look at how to integrate this into Hudson.