Steaming Ahead

Firstly, I'm really pleased to see that I'm getting some interesting comments - shame I haven't moderated them for ages. I'll get back to the issues raised in a later post, but here I want to tell you how I'm now charging ahead.

I took a huge leap and started on another rebuild. I've canibalised code from every version, live and experimental, but the key changes are that I'm building it on an SQLite database and using ClearSilver templates to generate the HTML.

There are a lot of advantages to this approach. I'll talk more about them later as well, but just for starters:

• It runs on a completely live database. Add a place and it will, instantly, be available for route planning and will appear in the drop-down hints for place entry
• Anyone who knows SQL can understand the database and add to it - and I haven't got to document these two large chunks of it, someone else has already done it.
  
• There are utilities to help me - the command line SQLite interface and a great Firefox tool to browse and edit
• Using templates means that pages either generate or fail, they don't break down half way though.
• The discipline of the templates really helps in design.
• It's not all written in my own script, and so it's much more capable of being taken forward by someone else. It would be a real shame for the project to rot if I was unable to keep it going for some reason.
• The projects are maintained - SQLite is in very active development indeed and while ClearSilver is quieter, it's not dead.
• They both are major projects - ClearSilver is behind Google Groups and Orkut, SQLite3 is everywhere, including in Google Gears, the Mac desktop, and the iPhone.
  

So why didn't I do it like this in the first place. Well, ClearSilver was first released on April 2006, SQLite3 - the first version with the features I need - came out in June 2004. CanalPlanAC has been live since Autumn 2001. I had to do it myself. But it was clearly meant to happen. I decided to use SQLite one day, and the very next they released a version with rtree support which is just what I need for POI data.

So why haven't I gone the whole hog and used a standard programming language as well? Mainly speed - I'm able to cut-and-paste large chunks of code from working versions because I keep the same scripting language, just chainging the in- and out- interfaces. Rewriting all this, and changing all the C to work with the other languages (the core route planning has to take place in something like C - it really is the grunt work).

As I am rebuilding from scratch, I'm making a number of other big changes which will support future enhancements:

• Internationalisation. The way I'm using and structuring the templates will make it easy to produce versions in other languages. That's very important in conjunction with...
• Expanded Geography. The core coordinate system is now latitude and longitude rather than OS grid. So it can support waterways anywhere in the world. In conjunction with ...
• The waterway sizes are stored in the database, not the code, so new waterway guages can be added and used. In conjunction with ...
• Different default options for different people (larger boats for example) because ...
  
• Boat sizes can be metric and imperial

So - there we go. An international waterway planner is only a few dozen steps away.

I'll be releasing a test version in a week or two, and will point you at it from here. When I do that I'll be starting another blog elsewhere which will function as release notes (and be a very good place for suggesting enhancements) while continuing with the bigger, stuff here.

The Sound of Silence

Things have been deathly quiet in the world of CanalPlanAC for a while now. No posts here for months, and no updates to the site since September.

Partly I have been doing a bit less - the rest of life has been getting in the way - but also I've finally found the right way forward for the site. As I said last time, I was working on a complete rewrite, and battling IE along the way. The problem with that was that I was really going to have to rewrite a huge amount of very successful code, something I didn't really want to do. At the same time, I was thinking more and more about the huge success of interactive sites - wikis in particular. Then at Christmas my motherboard died, so I took the opportunity to rebuild the computer from the ground up - I'm now running 64 bit Ubuntu; the best OS I've ever used (and I've used a few!).

So that provided the perfect excuse to redesign CanalPlanAC - copying files across one at a time, and removing all the cruft that has accumulated nearly 8 years of operational use. CanalPlan AC is a pretty good example of a three tier architecture (although the application and data tier are glued a little closely together) and I'm reworking it deeply, but only to the extent necessary at each layer:

1. The old version used static text files for the data, and built databases and quick-access files from them at build time. Any adjustments that were made were either queued for later (new places for example) or were cobbled on (like comments or changes to coordinates) and didn't get into the real data without a lot of mandraulics. The new version lives on a layer of databases: if a new place is added it is immediately available for route planning.
2. The old version used a pile of HTML versions, improper doctypes etc. I'm not aiming a full compliance with the new version (I want to use non-standard extensions for things like turning auto-complete off), but I am aiming for three green ticks on the Firefox web developer toolbar. As you can now run Internet Explorer on Linux(!) I can check that the pages work, and stick all the nasty sticks and props in that are necessary to work around IE's rather strange habits (like hiding all floats inside divs without a width, for example). I'm improving the appearance of the pages without going over the top.
   
3. I'm cleaning out the middle layer - removing old code (support for very old browsers, experimental features that never got turned on, that sort of thing), pulling things together (the live version has two different user account systems - one for photos and one for saving routes - neither of which is bug free).
   

I'm putting a few constraints on the user here. They must have reasonably modern browser with Javascript and session cookies.

Here's a screenshot of the index page on the development version:



This is pretty representative of the new version. Highlights include:

• It still looks like CanalPlanAC - the yellow background (for route planning; blue for gazetteers and a new pale red for editing) is still there - I'm not gratuitously changing the user experience.
  
• A new logo donated by a user
• Rounded coloured bars for menus
• "Edit" or "Add" boxes for the quotes, events and photo data (the "Edit" links with the arrows produce a drop-down menu for "Edit/Add/Delete".
• The quote, photo and events are pulled into a static HTML page using AJAX techniques.
• All pages tested to work down to 800x600 - below that you have to take your chances.

So that's where I'm at. Back to coding!

That is...

Well IE must stand for something sensible.

Having let a few select people (everybody who reads uk.rec.waterways) know about the development version of CanalPlanAC (Version 9) I've been fighting an uphill battle against the foibles of IE and it not displaying properly.

So I dragged an old Windows 98 computer out, hooked it up to a KVM switch and the router and started to get it ready to work.

I couldn't remember the password, but of course you just log on to this machine by pressing "cancel" anyway.

Well the first problem is that it didn't connect. So I found the button to change it from fixed IP to DHCP. Of course, it had to reboot then.

Then I needed to upgrade from IE4.0 (I said it was an old machine). Guess what - Microsoft's own update page doesn't render properly in IE4. You get Javascript errors, and pop-ups advertising IE7 obscuring the system description for IE6 (although it does note elsewhere on the page that you can't run IE7 on this O/S anyway).

So I upgraded to IE6. It confidently predicted a multi-hour download as it hadn't noticed the network connection. Then it rebooted again (without bothering to ask this time).

Then it told me, at reboot, that some component was too old and it some things might not work until I rebooted (remember, it just had rebooted).

Then it spend about 10 minutes showing me an animation of some drumsticks.

And people ask me why I don't use Windows! And I know that this is 98 and so must be around 7 or 8 years old now, but that's not the point. The point is, it's rubbish and always has been.

Interestingly, the tabs that have been causing all the problems appeared nicely under IE4, even if the Javascript failed! Under IE5 they have vanished. Arrrghgggggghhhhh.

Firebug

I'd like to rave briefly about a tool called Firebug. It is absolutely astonishing, and if you are doing complicated web development, almost invaluable.

Lots of the new stuff I'm doing on CanalplanAC involves Javascript interacting with the document. Firebug lets you see the structure of the document as it is at this moment, and when it changes, what you see changes.

That, alone, would be wonderful. But what Firebug gives you is a live editor for the page - not just the HTML but the CSS as well. So you can see what happens when you change the background colour, or padding, or floatiness of a particular element.

The time saved by tweaking the page, rather than editing and reloading is astonishing.

If you are doing almost anything on web pages, you'll find it worth giving a try.

This is an entirely unsolicited testimonial.

Meanwhile, on the western front

Not much has been happening with CanalplanAC lately - and so there isn't much to say here. The broadest explanation is simply life getting in the way. For a pile of reasons to do with home life, holidays, boat and the like I don't seem to have much spare time these days. And I've taken up a new job on promotion; which is great, but leaves me with no mental energy (no energy at all sometimes) at the end of the day for programming.

There are, as ever, a pile of ideas floating around my head. Improving the interface in various ways - while adding new features - is the obvious one. I like the new gazetteer, and making more of the site like that will be nice. Also, I've got a rather neat "route planning wizard" for novice users that I'll try to integrate some time over the next few weeks

In a recent article, on his always readable blog, Andrew Denny is discussing the virtues of Linux for running web sites and says "If you really want to see a state of the art Linux web site, try Nick Atty's Canalplan with its wonderful drop-down placenames in the Gazetteer (how cool is that? find a canal location as you type!),"

I'm glad Andrew likes this - it's a neat feature I'm quite proud of (although it's still not as good as it should be).

What's interesting is that he's picked up on something that is a wonderful example of the strengths of Open Source Software. Here's the story behind how that feature came to be written...

I subscribe to the mailing list for the Google Maps API, and a while ago someone mentioned a neat tool in Google Labs called suggest. I really liked the look of it, and decided I'd steal adapt it for CanalplanAC.

But before sitting down to write something as complicated as this I thought I'd do a search and see if someone had already written it. And, lo and behold, I quickly located actb which does almost all that I wanted. So that was all the messy interaction with the DOM dealt with. Here was a bit of Javascript that I could stick into my page and I'd get drop-down selectors.

However, the project was clearly dead - no updates since early 2005 - and didn't do everything I wanted. In particular, it required you to give it the list of things it was going to search in before it started. But I've got 7000 places in my database - I hardly want each user to have to download that each time.

If this had been a commercial package I'd be stuck. But it wasn't - it was open source (it's under a very liberal Creative Commons license). So I took the source code and hacked around with it - I grabbed a bit of example code for how to use XMLHttpRequest from the web, wrote a bit of glue to code (so that it only goes off and gets suggestions when it has three letters, and only goes back to the server when they change). And that was it - one Javascript file to stick into each page when I wanted to add the new auto-suggest. So that's the "client" end - what about the server?

Google no doubt have no problems powering Suggest, but then they do have a lot of computing power to throw at the problem. I've got one PC that runs the whole site. So I knocked up a bit of C to produce suggestions as quickly and efficiently as possible (a list is produced at build time that contains lower case and correctly formatted versions of each name, so that it can quickly match candidate suggestions but return an attactive version). And there we are. It took about a days work to implement.

It's not Linux that makes CanalplanAC great - it's the philosophy behind it. If you'd like to know more about this, and about how us web geeks work, have a look at Eric Raymond's The Art of Unix Programming. It's only his computing philosophy I'm espousing here, btw.

And, of course, it's not finished. But then software is alive, and the only time it stops growing is when it's dead. In particular I'm unhappy with the order that suggestions appear. A recent tweak helped - it now does two searches, the first for matches at the start of the string and the second for anywhere (so if you type "wig" it finds "Wigan Pier" ahead of "Parbold to Wigan Railway Bridge"). But it still could be improved - if you give it "wolverh" it comes back with "Wolverhampton Boat Club", then "Wolverhampton Bottom Lock", then the rest of the locks in sorted order (10-19, 2, 20, 21, 3, 4, 5, 6, 7, 8, 9, Top), then a random selection of railway bridges and then - finally - Wolverhampton itself.

Ah well - it doesn't look like I'll have to find another hobby for a while.

More archaobugology

They do keep coming out of the woodwork. I've just found, and fixed, another 5 or 6 years old bug lurking in a key bit of software ('=' signs appearing in the value part of post or get data were being dropped). I'm starting to realise just why so much of our software is a fragile as it is, and getting increasingly frightened by our reliance on computers for critical things. Mind you, I've been reading comp.risks for years, so shouldn't be that surprised.

On the details of the work, the new gazetteer is coming along apace. I plan to go live in the second half of February. It won't be finished, and will still have some bugs in but (putting aside the fact that - as shown above - everything still has bugs in it) it will be less of an abomination, and cause less harm than Google's recent changes to their interface to Usenet. And, again unlike Google, I won't be breaking anything that was long established before they came along.

More on the new gazetteer

Work on the new gazetteer procedes apace, although I doubt it will go properly live until a month or two into the new year. But I hope to have a prototype up one side of Christmas or the other for people to play with.

I've made a few interesting design choices along the way. It's quite clear, for example, that the gazetteer could be a single HTML page that pulled all the appropriate information from the server using AJAX techniques. But that would really mess up search engines; there are a lot of interesting articles on the tension between AJAX and SEO around the web.

So I'm content that there will still be a separate page for each location (and I'll be making sure it remains compatible with mapping from cgi parameters to fake directories for the future). But each page will do a lot more with AJAX-like tecniques.

I'm using the rather nice tags from acme.com for selecting the map or the photos (or I intend to - this bit hasn't really been started yet), for a very neat way using nested tab blocks to show POI data without filling the page with endless items (which is part working - enough for me to be convinced that this is the way to go) and for links, including tagged links (which is working). What are tagged links you ask? Read on.

There are two entirely new features in this version. The first is a new way to select place - this is the default if you start the gazetteer without giving it a place. This uses the same selection tree as you get if you use one of the region buttons on the current gazetteer start screen, but instead of it being page by page, with lots of buttons, it is all on one screen with drop down lists for region, waterway, [optional] part of waterway, [optional] sub part of waterway and place. This is all done by AJAX-like techniques, so the transitions and changes of the drop-downs as smooth, but there isn't a huge delay because it doesn't need to download thousands of selection lists before starting.

The second new feature is tagged links. This allows users to associate a link to an external site (web site, page, blog entry, picture, sound clip, video, whatever) with a "tag" and the link will be shown on all gazetteer entries that match the tag. Tags exist for each place by itself, for waterways, for identified parts of waterways and for all the rings that CanalplanAC knows about. So, for example, the "Stourport Ring in 10 minutes" video can be linked to every location on the Stourport Ring in a few clicks of the mouse. At the moment tags are not user definable, but if someone comes up with a good use for user creatable ones it would be straightforward to implement them. Again, this is all interacive using AJAX-like processes to get and send data between the browser and the server.

Of course there is lots going on to support this. A full re-engineer of the POI data structures is only part of it. But I'm confident that I'm going to end up with a smoother, quicker, more functional and more attractive gazetteer that's a really good example of webv2.0 technologies.

One footnote. Why to I keep saying "AJAX-like" up there. We'll because while there is a lot of Asynchronous (which is a fun learning experience for me), Javascript (which I'm actually starting to like as a programming language) and an awful lot of And there isn't much XML. You might think this odd; CanalplanAC was an fairly early adopter of XML for internal data formats and has used them since The Great XML Port of Spring 2002. The problem is that when you start sticking XML into the Javascript DOM, you get into some fairly hairy browser differences stuff. So although I use the XML request methods to get stuff from the server, I've been using fairly straight forward text formats for - say - the drop down menus.

For the POI stuff I was expecting to have to knuckle down and get dirty with XML and DOMS when I came across JSON. According to www.json.org, JSON is based on data structures that "Virtually all modern programming languages support ... in one form or another." Absolutely right. Canalplan Basic certainly does. It took me only an hour to produce a "write this data structure in JSON" feature into the interpreter - it's a lot cleaner than the corresponding XML writer. And it works very nicely indeed. So we are certainly AJA here, but the X seems to be taking a bit of time off.

More on layout

I think this deserves a post of its own, rather than another comment. But I'll address the side issue of links to the pages etc in a comment on that "thread".

Tasha's photo collage idea is really stimulating - but I think this is yet another thing for the WIBNI list - perhaps as a placefinder navigation device or something.

That's a strong vote of confidence from Paul for the menu and reduced text idea, so I'll defintely continue in that direction. I think for things that are cheap to generate I'll send them down the wire with some stylesheet/javascript to turn them on and off (the way the "show/hide list of excluded waterways works in the options page). For more expensive things (which find nearest certainly is, which rules out having "the nearest waterpoint is at..." on each page (also part of this upgrade involves extending this to "fine nearest pub" etc using the POI data)) I think an AJAX approach to seamlessly fetch and display them would work well.

I think, also, that user configuration of what is shown would be good, with two or three predefined layouts including "everything" and "low bandwidth".

What I might well do is to summarise local facilities ("Show map with pubs, shops and railway station"; "See list of local shops, museums and wireless hotspots") when they aren't displayed. I've also always planned to allow people to chose what things are of interest to them (I will always want to see lists of pubs and second-hand bookshops; others may be more interested in burger bars and wireless hotspots).

I'm having interesting thoughts about a randomly, but weighted, selected photo with buttons to vote on how much you like it (which affects the weighting) as well.

Giving an approx walking time for the distances is a nice idea. They are "as the crow flies" ones, so I'll have to see if that comes up with anything particularly silly.

I'll just keep plodding away. I'm creating this as a separate script, so as soon as I have something half decent I'll publish it and point blog readers to it.

Thoughts on Layout

I'm working on a new page layout for the gazetteer - this is an essential preliminary to the new stuff I'm doing with POI data (since this is where most of it appears). This is the most messy page, and has grown the most during development.

Graphic design and layout aren't my strongest points, and I'm getting a bit stuck. So this is an appeal to all my readers to suggest something a bit better.

My current thoughts have a menu bar down the left, with things like "search", "home page", "set options", "sunrise/sunset","find nearest" and so on. It will also include a data panel with the postcode, OS grid ref, current sunrise/sunset, lat/long etc.

Then, on the right, the main page.

This needs to include at least the following:
Text about the place (such as historical stuff)
Generated text about the place ("... is on the ... canal, 3 miles from ...")
Photographs (sometimes none, one, dozens)
Google maps (if wanted)
Preprogrammed searches (restaurants etc)
Local stuff like shops - optionally tied to the google maps.

Does anyone have any elegant thoughts for what this should look like. Ideally, if you want my neverending gratitude, you could mock-up a page for a few places. You can just grab the HTML for any page and hack it around as much as you like, or make something up with some lorem ipsum and stock photos. I don't really care.

What matters is that the new layout should be able to show all the data included in all the following gazetteer entries, and ideally be easy to add much more to if I want to.

• York
• Windsor
• Williamson Bridge
  

So, not asking much really, am I!

This is also a good time to chuck in any other thoughts on how to make the gazeetteer work better or add thoughts for new gazetteer features.

You can reply by commenting here, or by sending me an email. Googling for "Nick Atty" on uk.rec.waterways will find a currently valid email address for me.