Weather Station using Raspberry Pi, Tomcat and CouchDB

As a Christmas gift I've built a remote temperature sensoring system to make it possible to record and keep track of the temperature in a summer house on an island in the Baltic Sea. The background is that the temperature cannot be too cold in the house during the winter, since the piping may freeze. This gizmo could maybe help in tuning the electric heating of the house to avoid making it too warm but not too cold either. It will also help to identify when the power is out in the house and the heating is malfunctioning making it necessary to travel out to the house to avoid the water to freeze.

Overview

The layout is a Raspberry Pi using a DS18B20 digital temperature sensor that reads the temperature in the house. There is a NMT radio router in the house which serves Internet access. On a Tomcat server in the cloud, a Java web app accepts readings from the Raspberry via a REST API and stores the information in a CouchDB database. The web application also presents a single page web app that reads temperature data via the REST API and presents some nice graphs and tables. The frontend is built using Bootstrap to get a responsive design for both desktop and mobile.

The finished web application would look like this on desktop

and on a phone it would scale appropriately as well.

At the top the application will report for how long the sensor has been active (possibly since the last electricity outtake). If the sensor has not reported any readings during 15 minutes, it is assumed to be offline and a warning will be presented that we have no knowledge of the current temperature. If the ekectricity is gone in the house the heat will probably also be gone.

Setup the hardware

What you need:

• Raspberry Pi
• SD card (at least 4GB)
• Micro USB cable
• Ethernet cable
• DS18B20 temperature sensor
• 4.7 kOhm resistor
• Wires
• Piece of PCB bread board
• Some kind of case
• Soldering iron

So first of all let's setup the Raspberry Pi unit. I had the oldest model with 256MB RAM and Ethernet. I also have one of the newer 512MB  models and I've tried the setup successfully on both. Since the 256MB version worked, that's the one I've shipped. Here's how to make it into a temperature sensor.

Try out the connections on a breadboard according to this sketch (borrowed from AdaFruit)

The ports on the Pi are if not obvious in the sketch the 3.3V power port at slot 1, the data pin is pin #4 and the third wire is connected to ground.

DS18B20 water proof temp sensor

There are several versions of the DS18B20 temperature sensor. I used the water proof version, it's only marginally more expensive than the others but since it has a chord in contrast to the other ones it can be put outside a window if you want.

Then install the software according to below and make sure everything works. However, to make the final product more stable and resistent to shakes you should of course solder the wires. I recommend using an experiment board for this with lanes. I bought one of these a long time ago from Swedish Kjell & Company. It makes the soldering easy and you have lanes of conducting copper which makes it easy to overview your structure and debug errors with a voltmeter.

In this build the amount of board needed is tiny so not many cents of hardware needed. I bought a general purpose plastic box as well to mount the parts within. The box I found had predrilled holes within the box suitable for attaching PCBs and other parts. Very practical.






The final assembly looked like this with the top of the box removed

Setup the software

• Burn Raspbian OS to SD card (I had an old 4GB digital camera SD card over.
• Enable SSHD as default. See http://macgyverdev.blogspot.se/2012/08/raspberry-pi-how-to-get-ssh-and-tomcat.html
• Create a script directory and copy the Python program into it. You find the python source code in the GitHub repository https://github.com/johannoren/WeatherStation in the folder raspberrypi/src/weatherstation.py

mkdir weatherstation
scp /WeatherStationServer/raspberrypi/src/weatherstation.py      pi@piaddress:weatherstation/

• So the juice is in the weatherstation.py python script. But to run it by default on start up we create a wrapper bash script. Create a file /home/pi/weatherstation/weatherstation.sh containing

 #!/bin/sh
sleep 10
sudo python /home/pi/weatherstation/weatherstation.py

• Make sure this kicks in when the Raspberry boots by adding this to /etc/rc.local

/home/pi/weatherstation/weatherstation.sh &

• In the weatherstation.py script, adjust the URI to your potential backend server. The lines containing the following should be adjusted to your server and appname. The appname should be changed to the name of your system preferrably so that you may have multiple systems running on the same backend and CouchDB database.

conn = httplib.HTTPConnection("192.168.1.6:8080")

conn.request("POST", "/WeatherStationServer/api/temperature/appname", params, headers)

• Reboot the Raspberry Pi and the script should try to post readings to the URI in the script.

If you want to make your own script or just verify that the sensor is working correctly the important thing about this sensor is that when you add the kernel modules that handle the protocol the sensor is using, specific file handles will be created by the operating system containing the current temperature.
So you could alternatively try

sudo modprobe w1-gpio
sudo modprobe w1-therm
cd /sys/bus/w1/devices
ls
cd 28-xxxx (change this to match what serial number pops up)
cat w1_slave

So, assuming you are using the provided python script, now we must setup a server that can handle these temperature postings. The source code is in the GitHub repository, you can pull via https://github.com/johannoren/WeatherStation.git

Install this in a Java web server, I used Tomcat 7. To install Tomcat on a Linux Ubuntu flavoured server is trivial 

sudo apt-get install tomcat7 tomcat7-admin tomcat7-common

I personally use Eclipse to build my code, and then export the Web module as war archive (Right-click on the module in Eclipse and choose Export -> WAR archive). Then you can deploy the war file directly in the Tomcat web admin console which is usually located here http://serveraddress/manager/html.

However, the application must store the temperature data in a database and it is configured to use HTTP RESTful calls to store and retrieve JSON data. CouchDB will make a perfect backend for this.

Some notes on installation and how to work with CouchDB using curl in this blog post http://macgyverdev.blogspot.se/2013/12/couchdb-on-linux-mint.html, but it is trivial, apt-get to install.

sudo apt-get install couchdb -y

Create a database for the particular temperature sensor application via

curl -vX PUT http://127.0.0.1:5984/nameofapp

Depending on where you have deployed the CouchDB databse the Java application must be adjusted to communicate to the correct address and port. Before building and exporting the war archive to deploy in Tomcat make sure this constant is correct

WeatherStation /src/se/noren/weatherstation/WeatherStationServiceImpl.java

private static final String COUCHDB_SERVER = "http://johanhtpc:5984";

That's it, now the web app on the server can start posting readings to the database via essentially doing POSTs to the same URL. I used Spring Templates in the Java code which makes REST calls a no brainer. Look at this source code to figure out how to wrap the Spring Template to handle cookies, error handling etcetera https://github.com/johannoren/WeatherStation/blob/master/src/se/noren/weatherstation/adapter/CouchDBAdapter.java

If you wish to see the contents of the database you can do something like this

curl -X GET http://127.0.0.1:5984/nameofapp/_all_docs?include_docs=true

And if you want to delete it to reset testing, you can guess what you need

curl -vX DELETE http://127.0.0.1:5984/nameofapp

More commands in the previous mentioned post.

All good. That's about it I think, if interested in more details on some part, please comment.

If you want to use Fahrenheit or some other system of measurement for temperatures, check out my new hobby project All About Units where there's a lot of goodies on units.

You can buy your Raspberry Pi from many places on the net. Amazon has got both new and used ones for a cheaper prize.




Addition 2015-01-06:
This setup has been extended with another temperature sensor to monitor outside temperature as well. See this blogpost for the additions: http://macgyverdev.blogspot.se/2015/01/raspberry-pi-weather-station-with.html


Just for reference:
I added Google Analytics tracking to be able to see the utilization of the sensor. Here I learned a new thing as well about Analytics. I had the probe run for some time at home before christmas to see that it behaved stable and didn't crash after a long running time, which it didn't, but I unfortunately noticed that the first deployment I made in Google App Engine of the server backend used up to much resources and went over the free quota limit making the server unusable. So I moved the server backend to another server residing on a different domain. Suddenly the Analytics stats disappeared on december 22nd. I had already started wrapping the present. :-/

So the problem is that Google Analytics is specifying the cookie domain in the Javascript. I think this is new, I have no domains in my old Analytics scripts. So what you need to do if you encounter this problem is to manually change this line in the Google Analytics Javascript snippet

	(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
	(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
	m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
	})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
	ga('create', 'UA-46361998-1', 'newdomain.com');
	ga('send', 'pageview');

Create Spring REST service for Google App Engine in 15 minutes

Here's how you setup a REST service deployed in the Google App Engine cloud in 15 minutes. The use case in this example is a highscore backend service for my Android game Othello Legends.

Requirements:
We want to create a REST interface for these resources representing a highscore service.


GET http://myapp.appspot.com/api/highscores/
Fetch all applications backed by the highscore service since we want to reuse this for multiple games.

GET http://myapp.appspot.com/api/highscores/myappname
Fetch a sorted listed of highscores for a particular application myappname.

POST http://myapp.appspot.com/api/highscores/myappname
Post a potential new highscore to service. If it makes it to the highscore list it will be saved in database. The data will be sent as query parameters.

Ingredients of the solution:
Google App Engine runs Java and Python. This example will use the Java infrastructure.
So what we'll do is to create a standard Java J2EE web application built for deployment in App Engine backed by a simple DAO to abstract the Google BigTable databases. By using Spring REST together with Jackson we can communicate with JSON in a RESTful manner with minimum effort.

Sounds complicated? Not at all, here's how you do it!

Prerequisities:

• Java knowledge
• Eclipse and some basic skills of working in it.
• A Google account
• The Eclipse plugin for Google App Engine. If you ain't got it, the install is easy, follow the instruction at http://code.google.com/intl/sv-SE/appengine/docs/java/tools/eclipse.html

REST Implementation:

So to create an App Engine web app, click the New Web Application Project icon. Deselect Google Web Toolkit if you don't intend to use it.

Now, we're going to use Spring REST for the REST heavy weight lifting. Download Spring Framework 3 or later from http://www.springsource.org/download. While at it, download the Jackson JSON library from http://jackson.codehaus.org/. Put the downloaded jars in the /war/WEB-INF/lib/ folder and add them to the classpath of your web application.

Now, to bootstrap Spring to handle your incoming servlet requests you should edit the web.xml file of your web application found in war/WEB-INF/.

   api
   
      org.springframework.web.servlet.DispatcherServlet
   
   1

  

   api
   /api/*



   index.html

That will put Spring in charge of everything coming in under path /api/*. Spring must now which packages to scan for Spring annotated classes. We add a Spring configuration file for this and also add some Spring/Jackson config for specifying how to convert from our Java POJOs to JSON. Put this stuff in a file called api-servlet.xml in war/WEB-INF.

Without going into detail, this config pretty much tells Spring to convert POJOs to JSON as default using Jackson for servlet responses. If you're not interested in the details just grab it, but you must adjust the <context:component-scan base-package="se.noren.othello" /> to match your package names.

Now to the fun part, mapping Java code to the REST resources we want to expose. We need a controller class to annotate how our Java methods should map to the exposed HTTP URIs. Create something similar to

import java.util.Date;
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.validation.BindingResult;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.servlet.ModelAndView;

/**
 * Controller for Legends app family highscore services.
 */
@Controller
@RequestMapping("/highscores")
public class LegendsHighScoreController {
 private static final long serialVersionUID = 1L;

 @Autowired
 HighScoreService highScoresService;

 /**
  * @return Fetch all registered applications in the highscore database.
  */
 @RequestMapping(value = "/", method = RequestMethod.GET)
 public ModelAndView getAllApplications() {
  List<String> allApplications = highScoresService.getAllApplications();
  return new ModelAndView("highScoresView", BindingResult.MODEL_KEY_PREFIX + "applications", allApplications);
 }
 
 /**
  * Fetch all highscores for a particular application.
  * @param application Name of application
  * @return
  */
 @RequestMapping(value = "/{application}", method = RequestMethod.GET)
 public ModelAndView getAllHighScores(@PathVariable String application) {
  List<HighScore> allHighScores = highScoresService.getAllHighScores(application);
  return new ModelAndView("highScoresView", BindingResult.MODEL_KEY_PREFIX + "scores", allHighScores);
 }
 
 /**
  * Add a new highscore to the database if it makes it to the high score list.
  * @param application Name of application
  * @param owner Owner of the highscore
  * @param score Score as whole number
  * @param level Level of player reaching score.
  * @return The created score.
  */
 @RequestMapping(value = "/{application}", method = RequestMethod.POST)
 public ModelAndView addHighScores(@PathVariable String application,
                             @RequestParam String owner,
                             @RequestParam long score,
                             @RequestParam long level
                             ) {
  
  HighScore highScore = new HighScore(owner, score, application, new Date().getTime(), level);
  highScoresService.addHighScores(highScore);
  return new ModelAndView("highScoresView", BindingResult.MODEL_KEY_PREFIX + "scores", highScore);
 }
}

So what's the deal with all the annotations? They're pretty self explanatory once you start matching the Java methods to the three HTTP REST URIs we wanted to create, but in short:

• @Controller - The usual Spring annotation to tell Spring that this is a controller class that should be managed by the Spring container. All RESTful stuff is contained within the this class.
• @RequestMapping("/highscores") - This means that this controller class should accept REST calls under the path /highscores. Since we deployed the servlet under servlet mapping /api in the web.xml this means all REST resources resides under http://host.com/api/highscores
• @Autowired HighScoreService highScoresService - Our backing service class to do real business logic. Agnostic that we're using a RESTful front.
• @RequestMapping(value = "/{application}", method = RequestMethod.GET) public ModelAndView getAllHighScores(@PathVariable String application) -  A method annotated like this creates a REST resource /api/highscores/dynamicAppName where the value given for dynamicAppName is given via the path variable application. The request method specifies that this Java method will be called if this URI is requested via HTTP GET. All ordinary HTTP verbs are supported.
• @RequestParam String owner - If you wish to pass query parameters like myvar1=foo&myvar2=bar you can use the request param annotation.
• The Java class returned in the ModelAndView response will be automatically marshalled to JSON by Jackson on the same structure as the Java POJO.

Database

Google App Engine uses the Google BigTables behind the scenes to store data. You can abstract this by using the standard JPA annotations on your POJOs. The similar JDO standard can be used as well. I've used JDO in previous projects and it works very well. For this simple server application we will however use the query language to directly access the document database. Here's the code for the first method to fetch all highscores for a particular Legends application. The database can filter and sort via API methods in the query.

@Service
public class HighScoreServiceImpl implements HighScoreService {

 @Override
 public List<HighScore> getAllHighScores(String application) {
  ArrayList<HighScore> list = new ArrayList<HighScore>();
  DatastoreService datastore = DatastoreServiceFactory
    .getDatastoreService();

  // The Query interface assembles a query
  Query q = new Query("HighScore");
  q.addFilter("application", Query.FilterOperator.EQUAL, application);
  q.addFilter("score", FilterOperator.GREATER_THAN_OR_EQUAL, 0);
  q.addSort("score", SortDirection.DESCENDING);

  // PreparedQuery contains the methods for fetching query results
  // from the datastore
  PreparedQuery pq = datastore.prepare(q);

  for (Entity result : pq.asIterable()) {
   String owner = (String) result.getProperty("owner");
   Long date = (Long) result.getProperty("date");
   Long score = (Long) result.getProperty("score");
   Long level = (Long) result.getProperty("level");
   list.add(new HighScore(owner, score, application, date, level));
  }

  return list;
 }

That's pretty much it. Run the project locally by right-clicking it and choose Run As -> Web application. Once you are ready to go live create a cloud application by going to https://appengine.google.com/ and Create new application

Now in Eclipse, right click on your project and choose Google -> Deploy to Google App Engine.

You will be asked to supply the name you created in the App Engine administration interface. Wait a few seconds and the application will be deployed in the cloud.