Australian Weather Hacking Project

Published: May 25, 2008
Tags: google maps javascript programming weather data hacking getauweather

Ever since I wrote my getauweather program in early 2006 I have been meaning to put it to some sort of good use and actually do something with it. Just a little overdue, I've spent the last week working to this end and today am ready to announce my Australian weather data hacking project. At this page I hope to make available progressively more complicated and interesting applications of the huge amount of data that the Bureau of Meteorology make available. Perhaps more importantly, I am going to make every effort to make that data available to the community at large so that people who aren't me can do cool stuff with it as well.

At the moment, there is not an awful lot there. Every hour, a cron job runs a little Python script which uses getauweather (not the version you can currently download from my software page, a newer, better version that I'll release in a few days when I am confident that it is working) to grab the latest data from all the weather stations and then:

1. Reformats the data into CSV, XML and YAML, which anybody can then use in their own applications, for whatever purpose. The .csv file comes in at just under 100 kb and the two markup language files are in the area of 250 kb.
2. Updates this Google map of weather stations. This little visualisation really is my first use of either Javascript or the Google Maps API, so please forgive the fact that it really does suck. The page is a whopping 160 kb in size and, on most computers I've tried it on, Firefox will complain about how long it runs - just tell it to keep going and you'll see results soon enough. I plan to use this map as a means to develop my Javascript skills, so hopefully it won't suck for too much longer and will instead be powered by lightweight AJAX goodness. Of course, if you are already a Javascript wizard feel free to upstage me by making the coolest thing you can using those three freely available files above.

The next logical step for this project is to start logging these hourly weather results into a database which can be queried via a HTTP API. This will give me an excuse to finally try out CherryPy. Once such a database is up, AJAX magic will allow all sorts of awesome applications, and if people can download monthly dumps of the db then they can perform all sorts of fancy statistical analysis and the like. It should be good. I will note in passing that this undertaking is probably legally shaky for now. Unlike the situation in the US, which has the eminently sensible rule that the federal government is not permitted to claim copyright on anything they produce, the Commonwealth of Australia feels quite entitled to copyright the BoM observation data. Apparently they are very easy going about granting licenses to repackage and redistribute the freely available stuff, so I'll try to go down that route soon and, I suppose, pull the project if the BoM ends up objecting. Personally, I am not entirely convinced that their claim of copyright can be legitimate. My understanding is that copyright law allows for protecting a particular expression of an idea, not the idea itself, which can only be protected by a patent. It's not clear to me how this distinction applies to raw numbers, like weather station data, which can only possibly be expressed in one sensible possible way. At any rate, I don't expect any trouble to show up on this front.

On a closing note, returning to the subject of improving my Javascript skills: In a previous entry I posted a link to a level of Super Mario World implemented in 14 kb of Javascript. I learned the other day that the same guy has gone ahead and started working on implementing Super Mario Kart the same way, and has previously done Wolfenstein 3D! The blog author, one Jacob Seidelin, is in fact quite the Javascript hacker, in the most traditional sense of the word - coding for fun with no specific goal or direction, simply a desire to push boundaries and overcome limitations, which he is certainly doing. I'll have to keep an eye on his work.