Dropbox - is it safe to put you files in the cloud?

I really like the simplicity of sharing files with Dropbox. I haven't gone full circle yet, but I have been moving a substantial part of my personal stuff there.

I hadn't thought much about the security of it, but when listening to Steve Gibson and Leo Laporte on the Security Now podcast (http://www.grc.com/securitynow.htm, or search for it on iTunes) examining Dropbox, I got an eye-opener that you can't assume an awesome service to by definition have awesome security. 

First of all, Dropbox have claimed that not even their employees are able to see your data. Great! But in a recent change in the terms of agreement it says that the authorities due to US regulations can ask Dropbox to decrypt your data in certain crime investigations. Allright, I'm a good guy so that's not a problem for me. But that means that Dropbox must keep my encryption key in their vaults instead of me doing a client side encryption/decryption of my data. Interesting, that means that a bad apple Dropbox employee also have the possibility to look at my data without my knowledge. Not to mention what would happen if Dropbox would lose the table of private keys in some master planned hacking or insider heist.

So, the lesson should be. If you have some valuable or sensitive data, you should probably encrypt it before even dropping it into Dropbox.

Well, that applies to companies or people with more valuables than family photos like me.

Issue two might be more concerning, Derek Newton, http://dereknewton.com/2011/04/dropbox-authentication-static-host-ids/, has looked into how your Dropbox client authenticates against the cloud service. It seems like all you need is a config file which is set up at install time. That file contains your hostid which is your authentication token against Dropbox. The bad thing is that if someone by social engineering, a trojan or other malware gets a copy of this file, they can access your Dropbox account from any machine. Changing your password is not enough since this is an access token. You must remove your own machine as a valid host from Dropbox to stop the bad guy from using your account. Most probably you won't even know someone is eavesdropping on you.

These guys also seems to trust the cloud a bit too naively
https://forums.aws.amazon.com/thread.jspa?threadID=65649&tstart=0

Twitter integration using OAuth

So my Rankington application hosted on Google App Engine wants to read tweets containing mentions of keyword 'rankington' and also update the status of the Twitter system user Rankington on certain occations. There are loads of Twitter APIs out there for Java. Here's a short guide on how get going with Twitter4j and some handy knowledge of OAuth.

Download Twitter4j from http://twitter4j.org/en/index.html. If you wish to do only reads against Twitter you don't need to authenticate in any way, but if you wish to post status updates or similar you must provide credentials for the Twitter user you are using. In the past you could authenticate against the Twitter REST APIs using user/password but this has been shut down by Twitter since August 2010. So don't try that via the Twitter4J API which has not deprecated that code yet.

OAuth

Twitter is now using OAuth as access mechanism. If you are familiar with OpenID you could compare OAuth to OpenID in the sense that OpenID is a decentralised identification infrastructure, whereas OAuth solves authorisation in a decentralised way. OAuth is a RFC under standardization of the IETF. For example the Google Docs API and other Google APIs have recently added OAuth as access restriction mechanims for their REST APIs.

An example of using OAuth would be that you have some resources on a site A, say some private photos. You wish to let another site B access those photos in order to incorporate them into a photo stream or whatever, but you don't want to hand your identification credentials for site A to site B for obvious security reasons. Instead you wish to delegate authorisation to site B to access resources on site A.

What will happen is that site A and site B shares a common OAuth secret. Without going into the handshaking details you will surf against site B and an authorisation request will redirect you to site A where you will be asked to grant site B permission to the appropriate resources. Once redirecting back to site B, an OAuth token will be handed to site B which can be used from now on to access precisely the set of resources granted from site A using the access token. Check out the RFC if interested at http://tools.ietf.org/html/rfc5849.

Twitter4J and OAuth

So, in our case, Twitter is site A and Rankington is site B (service provider and client). To create the shared secret which will be generated by site A (= Twitter), go to http://twitter.com/oauth_clients/new and create a pair of secret keys. A consumer key and a consumer secret. 

A very basic test of this could be:

public class TwitterBridge {
    
    private static String key = "abcaasdkj1231231lkj123";
    private static String secret = "asdkj7987asdjl12312lkj4323423423";
    
    public static void main(String[] argv) throws TwitterException {

        Twitter twitter = new TwitterFactory().getInstance();
        twitter.setOAuthConsumer(key, secret);
        RequestToken requestToken = twitter.getOAuthRequestToken();
        System.out.println(requestToken.getAuthorizationURL());
        
        // Breakpoint here and update value of pin from what
        // what you get in browser when surfing against
        // authorisation URL above.
        String pin = "7117195";
        
        AccessToken accessToken = twitter.getOAuthAccessToken(requestToken, pin);
        
        System.out.println("Token: " + accessToken.getToken());
        System.out.println("Token secret: " + accessToken.getTokenSecret());
        
        Query query = new Query();
        query.setQuery("rankington");
        
        QueryResult queryResult = twitter.search(query);
        
        List<Tweet> tweets = queryResult.getTweets();
        for (Tweet t : tweets) {
            System.out.println("From: " + t.getFromUser());
            System.out.println("Time: " + t.getCreatedAt());
            System.out.println("Text: " + t.getText());
        }
        
        twitter.updateStatus("New tweet!");
    }
}

So put in your consumer key and secret in the code above as key and secret. Then debug these lines of code.

System.out.println(requestToken.getAuthorizationURL());

will print something like
http://api.twitter.com/oauth/authorize?oauth_token=51IMwiqF8MfEdcNDZxzUgV3guqCpQK6VbFZasdl

Open that link in a browser and you will be prompted
to identify yourself against Twitter, unless you are logged in automatically. In the same dialog you allow the client application "rankington" to access Twitter in your stead.

A bit confusingly, we login as user "rankington" on Twitter and allow application "rankington" access. This is just conincidence that the names are the same in this example.

When granted access, a verification code will be shown. Copy this code and either rerun the Java program or insert it into via the debugger as the variable "pin".

Continue to run the program and the Twitter4J API will verify the pin code against the Twitter REST API and receive an access token. We print this and it will look something like

Token: 45335176-3NEtmOsdfsacZROM9ow3sdfsdfHm5dfu0ShNGTdN2CKw
Token secret: nOXQish8asfasiq4tZINEOJuDasdYDQC4dBJiAM3k

All good. These are the important values which we can use to create a new AccessToken in the future.
The end of the program makes a query for tweets concerning status updates containing "rankington". That would print:

From: rankington
Time: Sun Apr 17 01:10:48 CEST 2011
Text: Rankington alpha is out! Follow progress at http://macgyverdev.blogspot.com
#rankington

And in the end, the program makes a status update in the name of the Twitter user "rankington".

So, now we got all ingredients, all keys and secrets for OAuth authority delegation. The piece of code we can use in our real application is the following.

private static String key         = "3649LZ3sasdasdpXWFHkHxaWQQ";
   private static String secret      = "aFOb86GmafgKtTasdasq3CpcwQw7bA";
   private static String token       = "28326asdasdasdD1ZxVDDL5Mqe7H";
   private static String tokensecret = "7Klyasdasdasdwpc8Xbtm0IsiRA";

   public static void main(String[] argv) throws TwitterException {
        
        AccessToken accessToken = new AccessToken(token, tokensecret);
        
        ConfigurationBuilder confBuilder = new ConfigurationBuilder(); 
        confBuilder.setOAuthAccessToken(accessToken.getToken()) 
                   .setOAuthAccessTokenSecret(accessToken.getTokenSecret()) 
                   .setOAuthConsumerKey(key)
                   .setOAuthConsumerSecret(secret); 

        Twitter twitter = new TwitterFactory(confBuilder.build()).getInstance(); 
        
        Query query = new Query();
        query.setQuery("rankington");
        
        QueryResult queryResult = twitter.search(query);
        
        List tweets = queryResult.getTweets();
        for (Tweet t : tweets) {
            System.out.println("From: " + t.getFromUser());
            System.out.println("Time: " + t.getCreatedAt());
            System.out.println("Text: " + t.getText());
        }
        
        twitter.updateStatus("New tweet again!");
    }

You see that we create the AccessTokens needed to authenticate against Twitter using the keys and secrets previously negotiated as a one time routine. The application can now tweet in eternity unless the user revokes the authorisation for the client.

With my basic understanding of OAuth, it feels like a great standard for interconnecting all service providers we got in the cloud. 

Hudson - Continuous Integration for a Google App Engine application

The last blog post described how to configure a Maven project for a Google App Engine application. 

To build and deploy the Maven artifacts you will need some command line hacking.

Hudson CI can be used to escape the command line for all this. Hudson is a continuous integration system for building and testing your projects. It has some cool features as distributed building and much more.

To install Hudson, or Jenkins as the main fork has rebranded it now after Oracle came into clinch with the open source community, simply download the war-archive from either the Jenkins or Hudson web site. My installation is old so I use a Hudson build.You can either deploy it in a web container such as Tomcat or simply use the built in bootstrap. To bootstrap the war archive

java -Dhudson.udp=32850 -jar hudson.war --httpPort=9090 --daemon --logfile=/home/johan/hudson/hudson.log

Access Hudson via http://localhost:8080/.

Goto Manage Hudson -> Configure System and enable Maven under the Maven subsection. Goto Manage Hudson -> Manage Plugins and install the plugins you need. In my case I have installed Cobertura Plugin for code coverage, Findbugs for static code analysis, Maven2 Project Plugin, checkstyle for Java code validation and the CVS plugin. You might need to bring in some sub dependencies like Static Analysis Collector Plug-In, static Analysis Utilities.

Now create a new job, under your job, click Configure and setup appropriate version control config. In my case CVSROOT=:pserver:rankington:xxxx@213.xxx.xxx.xxx:/cvsrepo

and correct CVS module and branch.

Under Build, choose correct Maven installation, probably /usr/bin/mvn. Then setup your Maven build goals. In this case the goals are, enable debug, clean, compile, test, create war archive and generate reports.

-X

clean

package

findbugs:findbugs

checkstyle:checkstyle

This will make Maven checkout your code, compile it, run the Maven plugins for creating xml reports for Findbugs and Checkstyle.

Now add Post-build Actions to integrate these reports into Hudson. Enable Publish Findbugs analysis results from file **/target/findbugsXml.xml

Add similar report integrations by enabling publishing the following reports target/surefire-reports/*.xml for JUnit tests, **/target/site/cobertura/coverage.xml for Cobertura code coverage and so forth. 

 

Now build your Hudson job by clicking Build Now and the Maven goals are executed to checkout, build, test and creates reports of the project. Then the post action goals kicks in and updates the dashboards of Hudson to show the results of the build.

Cobertura allows you to see on a package level the unit test coverage and the possibility to drill down on package and file level. Similar graphs and drilling can be done on Findbugs, Checkstyle and JUnit test reports.

Cobertura reporting on file level

JUnit test reports over time, red indicates test cases have failed during those builds.

Checkstyle reports Java code issues and the interface makes drilling easy.

In this example Hudson will generate a war archive ready for deployment to a web or application server.

The Maven build could be extended to deploy the war at a local server to also run the web tests as a part of the Hudson job.

If you wish to also incorporate production deployment in the Hudson process you could use the Google App Engine scripting possibilities. Add a build step which does something like this to upload your newly built war archive to GAE.

appengine-java-sdk\bin\appcfg.cmd --email user@gmail.com --passin password update myapp/war

The same script can be used to download logs from production via

appengine-java-sdk/bin/appcfg.sh request_logs myapp/war mylogs.txt

Theres a bunch of more handy commands for scripting GAE if you run through the docs at http://code.google.com/intl/sv-SE/appengine/docs/java/tools/uploadinganapp.html.

Next time I'll try to describe what the production environment offers in addition to the local development environment.

Development environment for a Google App Engine application

There is integration with Google App Engine for several IDEs and you can also interact with the deployment tools from plain command line if you have that disposition.

I'm an Eclipse guy, and here's how I've done to get a good setup to be able to develop in an efficient way using Eclipsen, Maven, CVS, Hudson and GAE.

Fisrt of all, install the Google App Engine SDK from http://code.google.com/intl/sv-SE/appengine/docs/java/tools/eclipse.html and follow the install instructions.

You should get a new toolbar in Eclipse with icons for creating GAE web projects, profiling them and for deploying to the cloud. If you create a new GAE application you can have it deployed in Googles data centers in a few minutes. All you need is a standard Google Account.

A few tips:

• You access your application at http://localhost:8888/ by default.
• In your dev environment a light weight data store will be kept in WEB-INF\appengine-generated\local_db.bin. This store is permanent over server restarts and must be deleted if you wish a clean state. Indicies are automatically generated in WEB-INF\appengine-generated\datastore-indexes-auto.xml. You access the development console at http://localhost:8888/_ah/admin where you can inspect and make queries on the data store and work with task queues and more.
• In addition to the standard JEE files such as web.xml there are a few specific config files for GAE projects. Inspect WEB-INF\appengine-web.xml where you setup logging and turn on sessions. By default you will not create web sessions when accessing you application

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application>rankington</application>
    <version>1</version>
    
    <sessions-enabled>true</sessions-enabled>
    
    <!-- Configure java.util.logging -->
    <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
    </system-properties>
    
</appengine-web-app> 

If you wish to complicate your environment with some continous integration for compilation, bug, test and coverage reporting I'll outline how you could integrate your GAE application with Hudson.

I've fiddled around with the default directory structure to make a more Maven compliant tree for dependency management. That is no problem since the App Engine plugin uses the Eclipse .classpath to resolve the structure.  So create src/main/java src/main/resources /src/test/java src/test/resources and src/main/webapp. Create a pom.xml in your project root and setup your Maven dependencies. To make this easy, install the Eclipse Maven plugin which makes pom-editing a no-brainer. You will need a few special dependencies for App Engine. Be careful about which version of the plugin you use to match the Maven dependency.

<dependency>
   <groupId>com.google.appengine</groupId>
   <artifactId>appengine-api-1.0-sdk</artifactId>
   <version>1.4.0</version>
   <type>jar</type>
   <scope>compile</scope>
</dependency>

My Maven dependencies are:

It took me some time to figure out how to add unit testing, findbugs reporting and code coverage to my Maven build. So hopefully you can reuse something like this for your pom file. I've removed most of the code dependencies specific for my project for clarity

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>se.noren.rankington</groupId>
  <artifactId>Rankington</artifactId>
  <packaging>war</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Maven Quick Start Archetype</name>
  <url>http://maven.apache.org</url>
  <ciManagement>
  </ciManagement>
  <build>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>
              <configuration>
                  <source>1.5</source>
                  <target>1.5</target>
            </configuration>
          </plugin>
          <plugin>
              <groupId>org.codehaus.mojo</groupId>
              <artifactId>surefire-report-maven-plugin</artifactId>
              <version>2.0-beta-1</version>
          </plugin>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-checkstyle-plugin</artifactId>
              <version>2.6</version>
          </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>cobertura-maven-plugin</artifactId>
            <version>2.2</version>
            <configuration>
                <formats>
                    <format>xml</format>
                </formats>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>cobertura</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>          
      </plugins>
  </build>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.4</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-beans</artifactId>
        <version>3.0.5.RELEASE</version>
        <type>jar</type>
        <scope>compile</scope>
    </dependency>

  </dependencies>
  <reporting>
      <plugins>
          <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-checkstyle-plugin</artifactId>
              <version>2.6</version>
          </plugin>
          <plugin>
              <groupId>org.codehaus.mojo</groupId>
              <artifactId>findbugs-maven-plugin</artifactId>
              <version>2.3.1</version>
              <configuration>
                <findbugsXmlOutput>true</findbugsXmlOutput>
                <findbugsXmlWithMessages>true</findbugsXmlWithMessages>
                <xmlOutput>true</xmlOutput>
                  <!-- Optional directory to put findbugs xml report --> 
                  <findbugsXmlOutputDirectory>target</findbugsXmlOutputDirectory>
                  <effort>Max</effort>
                  <threshold>Low</threshold>
            </configuration>
          </plugin>
      </plugins>
  </reporting>

</project>

So once alright in Eclipse, commit your application to a version control system. I use CVS, but this will work fine with Subversion as well. In the example below I commited App Engine project Rankington on branch Develop.

Now you could build your GAE application from a command line system with Maven installed with something like

CVSROOT=:pserver:rankington@213.xxx.xxx.xxx:/cvsrepo
export CVSROOT
cvs login
cvs co -r Develop Rankington
cd Rankington/
mvn package

This will generate a directory structure with your compiled classes, a packaged war-file ready for deployment to Google App Engine, directories with reports of your unit tests, static code analysis bug possibilities and code coverage.

Go to subdirectory target and you'll find everything



drwxr-xr-x 11 johan johan     4096 2011-04-16 14:28 ./
drwxr-xr-x  8 johan johan     4096 2011-04-16 14:28 ../
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 classes/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 cobertura/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 generated-classes/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 maven-archiver/
drwxr-xr-x  8 johan johan     4096 2011-04-16 14:28 Rankington-1.0-SNAPSHOT/
-rw-r--r--  1 johan johan 23081938 2011-04-16 14:28 Rankington-1.0-SNAPSHOT.war
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 site/
drwxr-xr-x  2 johan johan     4096 2011-04-16 14:28 surefire-reports/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 test-classes/
drwxr-xr-x  3 johan johan     4096 2011-04-16 14:28 war/

Next time we'll look at how to integrate this into Hudson.