Sunday, 14 April 2013

Simple backups with rsync

Here's my notes on how I setup my home network backup system in case I forget it.
I was first thinking of backing everything to some cloud service like Google Drive or Dropbox, but it will be quite expensive since I have too much home movies, images and sound recordings. So what I've done is that I have one backup of the stuff on a USB harddrive and another duplicate on the HTPC server. The same technique could be used to backup to a server at a friends house to make the backups completely fire proof.

Using the old Unix command rsync it is really easy to automate this. I installed rsync via cygwin, google for cwrsync which is rsync for Windows. My HTPC is a Linux Mint server so I created a RSA keypair for my Linux user and stored the keys on my Windows desktop machine in directory c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc.

So now we can authenticate against the HTPC with ssh which rsync supports and I only need to create a configuration of what I want to synchronize.

I created a config command file like this.


@ECHO OFF

SETLOCAL

SET CWRSYNCHOME=%ProgramFiles(x86)%\CWRSYNC

SET CYGWIN=nontsec

SET HOME=%HOMEDRIVE%%HOMEPATH%

SET CWOLDPATH=%PATH%

SET PATH=%CWRSYNCHOME%\BIN;%PATH%



rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/e/Dokument" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/e/Musik" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/e/Programmering" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/e/Bilder" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/e/Audiobooks" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/c/ws" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/c/recordings" johan@johanhtpc:PCBackups

rsync -av --chmod u+rwx -e "ssh -i c:\docume~1\Johan\.ssh\id_rsa_rsync_johanhtpc" "/cygdrive/c/render" johan@johanhtpc:PCBackups


So when run, rsync will check each folder on the Windows harddrive or external harddrive against the directory on the backup server. If new files have been created or existing files are updated they will be synchronized. No action will be taken for directories or files that are unchanged.

To automate this, I used the Windows scheduler. I added a task that will run each night at 00:59. It will simply run the above command.






Saturday, 2 February 2013

Create server backdoors using SQL Injection

If you're a web programmer you are probably aware of the most common security mistakes we make. OWASP keeps statistics on what exploits are the most common. If you're not familiar with these, this should be mandatory reading https://www.owasp.org/index.php/Top_10_2010-Main.

As shown, injections are still one of the worse problems. If you're not familiar with SQL injection check out some basic ways to exploit it on vulnerable site. The examples are often about trying to select some sensitive data and getting the data to be rendered on the vulnerable site. 

I'm not a black hat hacker so I've always thought about SQL injection as something primarily putting the site and it's data into danger. But tag along to see that SQL injection can be the entry point of pwning the complete server and getting inside the firewall and the internal network.

Now, I was reading up on SQL the other day for a project at work and stumbled upon some SQL syntax I didn't know about. Combining this with an SQL injection vulnerability could be dynamite.

So first, assume you found a weakness on a site. There are tools for that, but basically try to append code to request parameters like ' or 'foo'='1 or similar to look for server crashes giving you a hint of SQL injection problems like Unknown column 'foo' in 'where clause'. Now you would "normally" start the tiresome work of finding something valuable in the database.

But, with the SQL syntax INTO FILE you can write files. Nice. So depending on what technology the server is based on you could write files that can be accessed via the web interface. If the file names in the URI don't give away what technology used look at the HTTP header value of Server. If for example the header talks about JBoss you can guess that the site is Java based and we could try to create JSP files. Similarly you could aim for creating PHP script files etc if that's whats dished out by the server.

So, by using something like this in the injection exploit (the red is what is supplied by the hacker via the request parameter injection)

SELECT a, b FROM someunknowntable WHERE someunknowncolumn = '' UNION SELECT '<?php system($_GET["cmd"]); ?>' INTO OUTFILE '/var/www/htdocs/pwn.php'; --

or similar for other scripting technologies like JSP, ASP etc you have created a public backdoor.
Catastrophe! 
Point you're browser to http://thesite.com/pwn.php?cmd=pwd
to print out the currect working directory of the web server process.

Now only imagination stops you. cmd=cat /etc/passwd /etc/shadow to dump all user credentials. If the web server is running as root it's too easy to start creating misery like keylogging the other users or dumping all databases.

The lesson from this is that if you have five web servers and databases hosting different sites on your server it is the weakest link of them that defines the total security. So by hacking the not so important server with an SQL injection weakness you can get to the data of the highly secured applications with no SQL injection weaknesses via a backdoor.

Some thought on avoiding this kind of problems
  • Always use frameworks and libraries that removes the possibility of SQL injection. Java has prepared statements or some of the ORM technologies. The other languages has their own ways.
  • Run the database process as a user with low file access priviligies so that it can't write files anywhere it shouldn't be able to. Or even better, run it on a separate machine.
  • Same for the web server, don't run it as root. There are other ways of hacking web or application servers to gain shell access.
  • A general good thought is not to reveal to much information about what technology serves the site if possible to make it harder to exploit knowledge about how it behaves. For example, don't show server versions in HTTP headers, don't show crash stacktraces in server responses in production mode and so on.







Friday, 4 January 2013

Sink Hole Animation

I made a little animation which fakes a sink hole in the platform outside the Umeå Östra train station. This relies on camera tracking which takes a real scene and incorporates it in a simulated 3D environment.


Prerequisities 

A camera, (in my case a Canon 550D), some bright post-its, Blender (free and open source), Gimp or other image processing software and a few hours of modelling time and a nights sleep for rendering the scene.

Pick a scene

Figure out a scene. To make the camera tracking easiser to automate, pick a bright scene so that you will have short shutter times. Also, (which I failed to follow) try to keep your camera steady and make pans slow and steady to avoid blurry and unsharp images in your film. Record in highest possible resolution of your camera.

Object tracking

You must find some high contrast objects in the scene which you will use to track during the movement of the real camera in order for Blender to figure out the relative position of these objects. Preferrably use objects which will have parallax shifts when you move around in the scene. Also use a few objects that lie in your ground plane (the train platform) so that you can create a correct coordinate system.

In this scene, I want good resolution around the sink hole, but there are few high contrast objects, so I scattered some yellow pieces of post it notes around the area. 

Record your movie, import it into Blender and use the new Movie Clip Editor, which has come in one of the latest releases, to tag tracking objects. In the best of worlds, the software can follow the movement of the high contrast objects through the whole movie clip. Didn't work like a charm for me, had to manually help the tracker when the camera panned fast, so that's a note to self for future trackings. When you solve the system, Blender will figure out how the camera has moved during the photo shoot and create a camera animation for that for the virtual camera used later when rendering.

Modelling and rendering

I made a real simple hole model in Blender with some gravel textures overlaying eachother. Then by using the composition features of Blender you can merge each frame in the original movie clip with the rendered model with the correct camera position. In this example, the large disk in the image representing the ground is not rendered in the final composition, but it can receive renderable shadows, so that you can have your own 3D objects cast shadows onto the ground in the movie.

For reference, the original movie looks like


This is based in an excellent explanation of the Blender camera tracking functionality by Andrew Price on BlenderGuru.




Saturday, 29 December 2012

Hermanizer Power Pedal - A DIY Fuzz Box

For christmas gift to my brother I thought I'd try to make a guitar effect pedal. The main feature would be to have a nice sounding distortion effect, classic rock fuzz. I also found an article on Instructables by Harrymatic discussing adding a timer IC to the circuit to create a sound slicing/chopping kind of effect. This construction is based on his ideas.

So what you need is an amplifier IC, I used an LM386 circuit which can amplify a signal up to 200 times. To be able to control the amplification and the output signal, two potentiometers (resistors with variable resistance) acts as gain and volume controls. The sound slicing effect is created by a NE555 timer circuit which flips the voltage on its output pin high and low in a repeating fashion. The frequency of this is determined by a third potentiometer which is glued to a "effect rate" knob.

Circuit diagram of the effect pedal - click for larger picture
Depending on the state of the FX On/Off switches, pin 6 of the amplifier IC is fed by either a constant feed from the power source or a chopped up square wave from the timer IC.

So first step to try this out was connect everything on a breadboard.

If you buy your ICs with DIL8 sockets they will fit nicely on a standard breadboard. An important lesson from this setup is that you must ground both your input jack coming from your instrument and the output jack going to headphones/speakers/amplifiers or else you will get very disturbing hizzes and noises.

Once everything worked out on the breadboard I printed and soldered the circuits on a stripboard. The soldering is not shown in this picture since the circuits are on the backside of the stripboard.

For a case I bought a metal case and drilled the necessary holes for the potentiometers, switches and the power LED.

I bought the stuff I didn't already have at home from ELFA. It amounts to somewhere around 200 SEK. The integrated circuits are about 8 SEK each but the casing and the knobs turned out to be the most expensive stuff.

After stuffing everything inside the pedal works fine. When only using the distortion effect no hearable background fuzz is noticed. However, when enabling the chopping effect there is a constant background noise which is evident if no signal is coming from the instrument.
The final effect box
So how does the pedal sound? Well, this is the sound when applying quite a lot of distortion effect on an otherwise clean organ sound. I'm not a guitar player like my brother, but here's a few chords from my old Squire guitar lined directly into the effect pedal with pretty much distortion.

Here's a synth line which after a few seconds get the the chopper effect applied. In the end of the clip the rate of the effect is increased to maximum.



Saturday, 6 October 2012

Functional programming in Java?

After more toying with Scala on my spare time while hacking Java in office hours I miss out on using all these functional idioms in my Java programs. So with Java 8 coming closer I got interested in how the JDK team is planning on utilizing the new Java closure syntax in for example the collections libraries to support a more functional style of programming. While I have it fresh in memory, let me tell you some cool aspects of this!

Closures in Java 

Closures are anonymous functions also known as lambda expression. In languages where functions are first class members you could assign a lamba expression to a function variable. However, this is not entirely true for the Java implementation. In order not to mess up the type system to much, the engineers behind the implementation of the Lambda JSR 335 in OpenJDK has taken the approach to use interfaces containing one method to become special in the language. Such an interface is called a functional interface. An example of such an interface is ActionListener
public interface ActionListener { 
    void actionPerformed(ActionEvent e);
}
or Runnable or Comparator.
So in JDK 7 or earlier syntax, this is how you would declare for example an ActionListener
ActionListener l = new ActionListener() { 
  public void actionPerformed(ActionEvent e) { 
    doWork(e.getModifiers());
  }
};

However, with the new Java 8 syntax, you'd write it as
ActionListener l = (ActionEvent e) -> doWork(e.getModifiers());
So you get rid of all the unnecessary boilerplate code inherent in anonymous class creations so common in Java. More pleasing examples of usages of this syntax
Comparator<String> c = (s1, s2) -> s1.compareToIgnoreCase(s2);
FileFilter java = f -> f.getName().endsWith(".java");

There's a lot of interesting things going on here like how lexical scopes work, variable binding, type inference etcetera. Check out project lead Brian Goetz report on how these things works.

Java Collection libraries 

So how will this be used in the collection libraries which are probably the most used toolkit of the Java platform? First of all, the Collections team has decided not to restart by rewriting the collections libraries from scratch. While mentioning that this might be a candidate for future JDK versions, version 8 will instead provide an evolutionary step forward in the Collections by adding extension methods to existing interfaces (List, Set, Iterable) and retrofit existing classes with new interfaces such as Stream.

A major shift will be from the imperative style of external iteration to the more functional style of internal iteration. For example, the recommended idiom in Java 5+ to change a property on all objects in a collection uses 
for (Car c : cars) {
    c.setState(MOVING);
}
With closures you'll write this as
cars.forEach(c -> { c.setState(MOVING); });
What are the benefits of this idiom? You move the control flow to the library instead of the client code. In this way the library can decide on potentially use laziness, parallelism and out-of-order execution to improve performance which will be showed in later examples.

You can pipeline operations. In this example the filter operation uses a predicate to decide which objects in the collection to pipe to the final forEach clause.
cars.filter(c -> c.getWeight() > 2000)
      .forEach(c -> { c.setState(STOPPED); });
And to store results from computations using for example the map operation which operates on each value in the piped collection use something like
Set<Engine> smallEngines = cars.filter(c -> c.getMaxSpeed() < 100)
                .map(c -> c.getEngine())
                .into(new HashSet<>());
or to sum them. This is the well known functional idiom of map reduce.
int sum = cars.filter(c -> c.getState() == MOVING)
                .map(c -> c.getWeight())
                .sum();

So, all these operations will not create temporary new collections and pass on to the next operation. Instead they operate lazily and stream values between the control blocks. This implies good performance when for example searching for the first object that satisfies some condition. The upstream iterator in this example will not continue the iteration when getFirst() has found a match.
Car fastCar = cars.filter(c -> c.getSpeed() > 120).getFirst();
When used to these constructs, a lot of boilerplate code should be possible to remove. Here Brian Goetz shows an example of a method in java.lang.Class as today
 for (Method m : enclosingInfo.getEnclosingClass().getDeclaredMethods()) {
     if (m.getName().equals(enclosingInfo.getName()) ) {
         Class<?>[] candidateParamClasses = m.getParameterTypes();
         if (candidateParamClasses.length == parameterClasses.length) {
             boolean matches = true;
             for(int i = 0; i < candidateParamClasses.length; i++) {
                 if (!candidateParamClasses[i].equals(parameterClasses[i])) {
                     matches = false;
                     break;
                 }
             }

             if (matches) { // finally, check return type
                 if (m.getReturnType().equals(returnType) )
                     return m;
             }
         }
     }
 }

 throw new InternalError("Enclosing method not found");

and how it could be rewritten without all the temporary variables making it both more readable and less error prone.
Method matching =
  Arrays.asList(enclosingInfo.getEnclosingClass().getDeclaredMethods())
    .filter(m -> Objects.equals(m.getName(), enclosingInfo.getName())
    .filter(m ->  Arrays.equals(m.getParameterTypes(), parameterClasses))
    .filter(m -> Objects.equals(m.getReturnType(), returnType))
    .getFirst();
if (matching == null)
    throw new InternalError("Enclosing method not found");
return matching;

There's a lot of more cool features on the project site, but before ending you should see how easy parallel computation can become. By streaming the pipeline via parallel() the library will try to divide the pipeline stream of operations to all your cores. 
int sum = cars.parallel()
                .filter(c -> c.getState() == MOVING)
                .map(c -> c.getWeight())
                .sum();
Via the new interface Splittable you can also very easily use the Fork/Join framwork for divide and conquer tasks.

Timeplans

This looks awesome. But when can we use it? According to the milestone plan the public review version should be available in January 2013. The JDK 8 timeplan currently looks like

2012/7 Expert Group formation
2012/9 Early Draft Review
2013/1 Public Review
2013/6 Proposed Final Draft
2013/8 Final Release
http://openjdk.java.net/projects/jdk8/spec/

But you can download bleeding edge versions of the OpenJDK 8 binaries today and play around with the lambda language construct and the current implementations in the collections libraries as well as a lot of the other libraries. http://jdk8.java.net/download.html

Alternative functional libraries

If you can't wait there are libraries for functional programming in Java that will work with JDK5 or newer.

Guava

Googles Guava libraries have support for functional idioms. However without language support the code easily becomes messed up with boilerplate. On the other hand, there are a lot of good stuff in Guava like a richer set of collection constructs and easier use of immutable collections types. Here
Multiset<Integer> lengths = HashMultiset.create(
  FluentIterable.from(strings)
    .filter(new Predicate<String>() {
       public boolean apply(String string) {
         return CharMatcher.JAVA_UPPER_CASE.matchesAllOf(string);
       }
     })
    .transform(new Function<String, Integer>() {
       public Integer apply(String string) {
         return string.length();
       }
     }));

Lambdaj

Another interesting library is Lambdaj which uses static imported methods to hide give a nicer looking syntax. This is some typical Java code to sort a list of persons according to age
List<Person> sortedByAgePersons = new ArrayList<Person>(persons);
Collections.sort(sortedByAgePersons, new Comparator<Person>() {
        public int compare(Person p1, Person p2) {
           return Integer.valueOf(p1.getAge()).compareTo(p2.getAge());
        }
});
With Lambdaj, you could express this as
List<Person> sortedByAgePersons = sort(persons, on(Person.class).getAge());
Check out more features at http://code.google.com/p/lambdaj/wiki/LambdajFeatures.

FunctionalJava

FunctionalJava solves the syntax verbosity problem by using the Java 7 BGGA proposal syntax. This adds closures as a part of the language dialect. However, this requires a pass with a pre compiler to render compilable Java code.
This is an example of code that adds 42 to each element in the array.


  1. final Array<Integer> a = array(123);  
  2. final Array<Integer> b = a.map({int i => i + 42});  
  3. arrayShow(intShow).println(b); // {43,44,45}  

Noteworthy is that solution also heavily relies on static imports. Check out more example at http://functionaljava.org/examples/1.5/

My thoughts

In the end though, due to the lack of anonymous functions in Java today, the best choice to program in a functional way is probably to stick to Scala, Clojure, Groovy or another of the JVM languages that has inherent support for this style until Java 8. With what we got today in Java you can still use many of the functional concepts like preferring immutable data, avoiding side effects and more.

The above mentioned alternatives are just a few of what's out there. But common among them are either that you must rely on preprocessing some functional dialect of Java to produce compilable Java code or to use a verbose syntax like Guava. In my opinion these tradeoffs effect on readability, maintainability and possibly portability just isn't worth it.

By the way, Brian Goetz appears 20 mins into the Java One 2012 Technical keynote to show off some collection examples of Java 8. http://medianetwork.oracle.com/video/player/1871712019001


Monday, 3 September 2012

Debugging Knockout.js

Knockout.js is really powerful. A consequence of being awesome is unfortunately that its magic makes it hard to debug when it doesn't behave as you expect.

Here's a trick that might come in handy if a data binding does not behave properly. You can print the the variables of the view model that is in the current scope by inserting this debugging tag in your HTML source.
<pre data-bind="text: JSON.stringify(ko.toJS($data), null, 3)"></pre>
Yielding a print like this in the rendered page
{
   "myViewModelVar1": "someValue",
   "myViewModelVar2": "someOtherValue"
}
You know any other handy tricks?

Sunday, 19 August 2012

Learning Scala - Photo collage creator

I finally took the time this summer to read a book on Scala. I bought Programming in Scala by Martin Odersky, the father of the language, which I think was a good choice. Regardless whether I'll write a lot of Scala programs in the future I learnt some new general programming techniques and a well needed recap on  programming language fundamentals from school. After reading it and applied it on a couple of hobby projects I must say that I feel excited.

My first project to play around with the language is a photo collage creator where you supply the program a motive and a set of images to create a collage from. The algorithm tries to puzzle the images together to create a collage that best fits the motive.

The motive to create collage from.
The motive is divided in segments and the image catalogue is searched for best fitting images to puzzle together.

The final collage in low-res.
When printing the collage in high resolution, for example 16384 x 10922 pixels the effect becomes quite cool when you approach the collage from a distance to a near closeup.

Let me just show you an arbitrary Scala function in this program that demonstrates a few of the things that I like compared to my daily work horse Java.

  /**
   * Calculate the average brightness of a portion of an image.
   * 
   * @param img Image to analyse for average brightness.
   * @param startx Start x coordinate of image subset to analyze 
   * @param starty Start y coordinate of image subset to analyze
   * @param stopx Stop x coordinate of image subset to analyze
   * @param stopy Stop y coordinate of image subset to analyze
   * @return Average brightness of the subset of the image
   */
  def brightness(img: BufferedImage, startx: Int, starty: Int, stopx: Int, stopy: Int): Int = {

    @tailrec
    def estimateBrightness(x: Int, y: Int, maxx: Int, maxy: Int, aggr: Int): Int = {
      if (y == maxy)
        aggr
      else if (x == maxx)
        estimateBrightness(startx, y + 1, maxx, maxy, aggr)
      else
        estimateBrightness(x + 1, y, maxx, maxy, aggr + rgb2gray(img.getRGB(x, y)))
    }
   
    /*
     * Average the brightness of the number of evaluated pixels with 
     * the aggregate of their calculated values.
     */
    val aggregatedBrightness = estimateBrightness(startx, starty, stopx, stopy, 0)
    aggregatedBrightness / ((stopx - startx) * (stopy - starty))
  }



As you can see, Scala is statically typed, but the compiler tries to infer as much as possible. You can create a constant variable with the val keyword. But it will in this example figure out that the variable aggregatedBrightness must be of the type (or subclass) Int since it is evaluated via the function estimateBrightness(). You will save yourself a lot of boilerplate declarations.

But what about the function estimateBrightness? It is declared inside the scope of the function brightness(). In Scala a function is on par with any plain old objects and can be referenced via variables or passed as arguments to functions and also be declared inside functions as a consequence. Why wouldn't it always be so?

Everything has a value, even a for loop or an if clause will result in something that can be passed to a variable or statement. This makes for concise and beautiful code.

Scala is basically a functional language but with all the imperative concepts around to make it easy for imperative people like me to make the transition to a more functional style in the tempo that suits me. In this example I made my calculations in a functional style using recursion instead of using loop constructs. I tried to write the program without any for or while loops at all but my conclusion is that just because it is nice to make everything recursive and functional it must not inherently be readable and understandable. I'll stick to my imperative guns when I need them for some time more.

An interesting annotation is @tailrec on the local function declaration. It forces the compiler to verify that this recursive function will be tail-call optimized, meaning that you can be sure that this function will not create stack frames for each invocation in the recusive loop. If so you would be running out of stack after some 10 000 invocations depending on your JVM startup flags.

To be able to write efficient and understandable functional programs my impression is that the requirements of the programmer are heightened compared to programming in plain old Java/C/C++. A challenge I'm gladly willing to continue with.

Instead of me trying to convince you that Scala seems like a great contribution to the Java VM family I strongly recommend you to read the book. You'll definitively become a better C# or Java programmer as well afterwards.



If you want to play around with the Photo Collage creator program and generate som collages of your own clone the code from GitHub.

https://github.com/johannoren/PhotoCollage

I have used Eclipse with the Scala plugin which make the program run without any hazzle. To configure it without any code changes, create a directory photos in the module, and put one image in that directory as your motive and name if motive.jpg. Put all your images that will be part of the puzzle in a subdirectory called inputphotos. Run PhotoCollage and monitor the standard out until the program is finished. Run time depends mainly on the number of images in the directory inputphotos.