CanalplanAC

The Sound of Silence

2008-03-01T07:07:00.006Z

Things have been deathly quiet in the world of CanalPlanAC for a while now. No posts here for months, and no updates to the site since September.

Partly I have been doing a bit less - the rest of life has been getting in the way - but also I've finally found the right way forward for the site. As I said last time, I was working on a complete rewrite, and battling IE along the way. The problem with that was that I was really going to have to rewrite a huge amount of very successful code, something I didn't really want to do. At the same time, I was thinking more and more about the huge success of interactive sites - wikis in particular. Then at Christmas my motherboard died, so I took the opportunity to rebuild the computer from the ground up - I'm now running 64 bit Ubuntu; the best OS I've ever used (and I've used a few!).

So that provided the perfect excuse to redesign CanalPlanAC - copying files across one at a time, and removing all the cruft that has accumulated nearly 8 years of operational use. CanalPlan AC is a pretty good example of a three tier architecture (although the application and data tier are glued a little closely together) and I'm reworking it deeply, but only to the extent necessary at each layer:

The old version used static text files for the data, and built databases and quick-access files from them at build time. Any adjustments that were made were either queued for later (new places for example) or were cobbled on (like comments or changes to coordinates) and didn't get into the real data without a lot of mandraulics. The new version lives on a layer of databases: if a new place is added it is immediately available for route planning.
The old version used a pile of HTML versions, improper doctypes etc. I'm not aiming a full compliance with the new version (I want to use non-standard extensions for things like turning auto-complete off), but I am aiming for three green ticks on the Firefox web developer toolbar. As you can now run Internet Explorer on Linux(!) I can check that the pages work, and stick all the nasty sticks and props in that are necessary to work around IE's rather strange habits (like hiding all floats inside divs without a width, for example). I'm improving the appearance of the pages without going over the top.
I'm cleaning out the middle layer - removing old code (support for very old browsers, experimental features that never got turned on, that sort of thing), pulling things together (the live version has two different user account systems - one for photos and one for saving routes - neither of which is bug free).

I'm putting a few constraints on the user here. They must have reasonably modern browser with Javascript and session cookies.

Here's a screenshot of the index page on the development version:

This is pretty representative of the new version. Highlights include:

It still looks like CanalPlanAC - the yellow background (for route planning; blue for gazetteers and a new pale red for editing) is still there - I'm not gratuitously changing the user experience.
A new logo donated by a user
Rounded coloured bars for menus
"Edit" or "Add" boxes for the quotes, events and photo data (the "Edit" links with the arrows produce a drop-down menu for "Edit/Add/Delete".
The quote, photo and events are pulled into a static HTML page using AJAX techniques.
All pages tested to work down to 800x600 - below that you have to take your chances.

So that's where I'm at. Back to coding!

That is...

2007-07-30T16:12:00.000+01:00

Well IE must stand for something sensible.

Having let a few select people (everybody who reads uk.rec.waterways) know about the development version of CanalPlanAC (Version 9) I've been fighting an uphill battle against the foibles of IE and it not displaying properly.

So I dragged an old Windows 98 computer out, hooked it up to a KVM switch and the router and started to get it ready to work.

I couldn't remember the password, but of course you just log on to this machine by pressing "cancel" anyway.

Well the first problem is that it didn't connect. So I found the button to change it from fixed IP to DHCP. Of course, it had to reboot then.

Then I needed to upgrade from IE4.0 (I said it was an old machine). Guess what - Microsoft's own update page doesn't render properly in IE4. You get Javascript errors, and pop-ups advertising IE7 obscuring the system description for IE6 (although it does note elsewhere on the page that you can't run IE7 on this O/S anyway).

So I upgraded to IE6. It confidently predicted a multi-hour download as it hadn't noticed the network connection. Then it rebooted again (without bothering to ask this time).

Then it told me, at reboot, that some component was too old and it some things might not work until I rebooted (remember, it just had rebooted).

Then it spend about 10 minutes showing me an animation of some drumsticks.

And people ask me why I don't use Windows! And I know that this is 98 and so must be around 7 or 8 years old now, but that's not the point. The point is, it's rubbish and always has been.

Interestingly, the tabs that have been causing all the problems appeared nicely under IE4, even if the Javascript failed! Under IE5 they have vanished. Arrrghgggggghhhhh.

Firebug

2007-06-23T09:56:00.000+01:00

I'd like to rave briefly about a tool called Firebug. It is absolutely astonishing, and if you are doing complicated web development, almost invaluable.

Lots of the new stuff I'm doing on CanalplanAC involves Javascript interacting with the document. Firebug lets you see the structure of the document as it is at this moment, and when it changes, what you see changes.

That, alone, would be wonderful. But what Firebug gives you is a live editor for the page - not just the HTML but the CSS as well. So you can see what happens when you change the background colour, or padding, or floatiness of a particular element.

The time saved by tweaking the page, rather than editing and reloading is astonishing.

If you are doing almost anything on web pages, you'll find it worth giving a try.

This is an entirely unsolicited testimonial.

Meanwhile, on the western front

2007-06-09T07:44:00.001+01:00

Not much has been happening with CanalplanAC lately - and so there isn't much to say here. The broadest explanation is simply life getting in the way. For a pile of reasons to do with home life, holidays, boat and the like I don't seem to have much spare time these days. And I've taken up a new job on promotion; which is great, but leaves me with no mental energy (no energy at all sometimes) at the end of the day for programming.

There are, as ever, a pile of ideas floating around my head. Improving the interface in various ways - while adding new features - is the obvious one. I like the new gazetteer, and making more of the site like that will be nice. Also, I've got a rather neat "route planning wizard" for novice users that I'll try to integrate some time over the next few weeks

2007-02-26T20:44:00.000Z

In a recent article, on his always readable blog, Andrew Denny is discussing the virtues of Linux for running web sites and says "If you really want to see a state of the art Linux web site, try Nick Atty's Canalplan with its wonderful drop-down placenames in the Gazetteer (how cool is that? find a canal location as you type!),"

I'm glad Andrew likes this - it's a neat feature I'm quite proud of (although it's still not as good as it should be).

What's interesting is that he's picked up on something that is a wonderful example of the strengths of Open Source Software. Here's the story behind how that feature came to be written...

I subscribe to the mailing list for the Google Maps API, and a while ago someone mentioned a neat tool in Google Labs called suggest. I really liked the look of it, and decided I'd ~~steal~~ adapt it for CanalplanAC.

But before sitting down to write something as complicated as this I thought I'd do a search and see if someone had already written it. And, lo and behold, I quickly located actb which does almost all that I wanted. So that was all the messy interaction with the DOM dealt with. Here was a bit of Javascript that I could stick into my page and I'd get drop-down selectors.

However, the project was clearly dead - no updates since early 2005 - and didn't do everything I wanted. In particular, it required you to give it the list of things it was going to search in before it started. But I've got 7000 places in my database - I hardly want each user to have to download that each time.

If this had been a commercial package I'd be stuck. But it wasn't - it was open source (it's under a very liberal Creative Commons license). So I took the source code and hacked around with it - I grabbed a bit of example code for how to use XMLHttpRequest from the web, wrote a bit of glue to code (so that it only goes off and gets suggestions when it has three letters, and only goes back to the server when they change). And that was it - one Javascript file to stick into each page when I wanted to add the new auto-suggest. So that's the "client" end - what about the server?

Google no doubt have no problems powering Suggest, but then they do have a lot of computing power to throw at the problem. I've got one PC that runs the whole site. So I knocked up a bit of C to produce suggestions as quickly and efficiently as possible (a list is produced at build time that contains lower case and correctly formatted versions of each name, so that it can quickly match candidate suggestions but return an attactive version). And there we are. It took about a days work to implement.

It's not Linux that makes CanalplanAC great - it's the philosophy behind it. If you'd like to know more about this, and about how us web geeks work, have a look at Eric Raymond's The Art of Unix Programming. It's only his computing philosophy I'm espousing here, btw.

And, of course, it's not finished. But then software is alive, and the only time it stops growing is when it's dead. In particular I'm unhappy with the order that suggestions appear. A recent tweak helped - it now does two searches, the first for matches at the start of the string and the second for anywhere (so if you type "wig" it finds "Wigan Pier" ahead of "Parbold to Wigan Railway Bridge"). But it still could be improved - if you give it "wolverh" it comes back with "Wolverhampton Boat Club", then "Wolverhampton Bottom Lock", then the rest of the locks in sorted order (10-19, 2, 20, 21, 3, 4, 5, 6, 7, 8, 9, Top), then a random selection of railway bridges and then - finally - Wolverhampton itself.

Ah well - it doesn't look like I'll have to find another hobby for a while.

More archaobugology

2007-02-01T07:56:00.000Z

They do keep coming out of the woodwork. I've just found, and fixed, another 5 or 6 years old bug lurking in a key bit of software ('=' signs appearing in the value part of post or get data were being dropped). I'm starting to realise just why so much of our software is a fragile as it is, and getting increasingly frightened by our reliance on computers for critical things. Mind you, I've been reading comp.risks for years, so shouldn't be that surprised.

On the details of the work, the new gazetteer is coming along apace. I plan to go live in the second half of February. It won't be finished, and will still have some bugs in but (putting aside the fact that - as shown above - everything still has bugs in it) it will be less of an abomination, and cause less harm than Google's recent changes to their interface to Usenet. And, again unlike Google, I won't be breaking anything that was long established before they came along.

More on the new gazetteer

2006-12-03T19:13:00.000Z

Work on the new gazetteer procedes apace, although I doubt it will go properly live until a month or two into the new year. But I hope to have a prototype up one side of Christmas or the other for people to play with.

I've made a few interesting design choices along the way. It's quite clear, for example, that the gazetteer could be a single HTML page that pulled all the appropriate information from the server using AJAX techniques. But that would really mess up search engines; there are a lot of interesting articles on the tension between AJAX and SEO around the web.

So I'm content that there will still be a separate page for each location (and I'll be making sure it remains compatible with mapping from cgi parameters to fake directories for the future). But each page will do a lot more with AJAX-like tecniques.

I'm using the rather nice tags from acme.com for selecting the map or the photos (or I intend to - this bit hasn't really been started yet), for a very neat way using nested tab blocks to show POI data without filling the page with endless items (which is part working - enough for me to be convinced that this is the way to go) and for links, including tagged links (which is working). What are tagged links you ask? Read on.

There are two entirely new features in this version. The first is a new way to select place - this is the default if you start the gazetteer without giving it a place. This uses the same selection tree as you get if you use one of the region buttons on the current gazetteer start screen, but instead of it being page by page, with lots of buttons, it is all on one screen with drop down lists for region, waterway, [optional] part of waterway, [optional] sub part of waterway and place. This is all done by AJAX-like techniques, so the transitions and changes of the drop-downs as smooth, but there isn't a huge delay because it doesn't need to download thousands of selection lists before starting.

The second new feature is tagged links. This allows users to associate a link to an external site (web site, page, blog entry, picture, sound clip, video, whatever) with a "tag" and the link will be shown on all gazetteer entries that match the tag. Tags exist for each place by itself, for waterways, for identified parts of waterways and for all the rings that CanalplanAC knows about. So, for example, the "Stourport Ring in 10 minutes" video can be linked to every location on the Stourport Ring in a few clicks of the mouse. At the moment tags are not user definable, but if someone comes up with a good use for user creatable ones it would be straightforward to implement them. Again, this is all interacive using AJAX-like processes to get and send data between the browser and the server.

Of course there is lots going on to support this. A full re-engineer of the POI data structures is only part of it. But I'm confident that I'm going to end up with a smoother, quicker, more functional and more attractive gazetteer that's a really good example of webv2.0 technologies.

One footnote. Why to I keep saying "AJAX-like" up there. We'll because while there is a lot of Asynchronous (which is a fun learning experience for me), Javascript (which I'm actually starting to like as a programming language) and an awful lot of And there isn't much XML. You might think this odd; CanalplanAC was an fairly early adopter of XML for internal data formats and has used them since The Great XML Port of Spring 2002. The problem is that when you start sticking XML into the Javascript DOM, you get into some fairly hairy browser differences stuff. So although I use the XML request methods to get stuff from the server, I've been using fairly straight forward text formats for - say - the drop down menus.

For the POI stuff I was expecting to have to knuckle down and get dirty with XML and DOMS when I came across JSON. According to www.json.org, JSON is based on data structures that "Virtually all modern programming languages support ... in one form or another." Absolutely right. Canalplan Basic certainly does. It took me only an hour to produce a "write this data structure in JSON" feature into the interpreter - it's a lot cleaner than the corresponding XML writer. And it works very nicely indeed. So we are certainly AJA here, but the X seems to be taking a bit of time off.

More on layout

2006-11-14T07:27:00.000Z

I think this deserves a post of its own, rather than another comment. But I'll address the side issue of links to the pages etc in a comment on that "thread".

Tasha's photo collage idea is really stimulating - but I think this is yet another thing for the WIBNI list - perhaps as a placefinder navigation device or something.

That's a strong vote of confidence from Paul for the menu and reduced text idea, so I'll defintely continue in that direction. I think for things that are cheap to generate I'll send them down the wire with some stylesheet/javascript to turn them on and off (the way the "show/hide list of excluded waterways works in the options page). For more expensive things (which find nearest certainly is, which rules out having "the nearest waterpoint is at..." on each page (also part of this upgrade involves extending this to "fine nearest pub" etc using the POI data)) I think an AJAX approach to seamlessly fetch and display them would work well.

I think, also, that user configuration of what is shown would be good, with two or three predefined layouts including "everything" and "low bandwidth".

What I might well do is to summarise local facilities ("Show map with pubs, shops and railway station"; "See list of local shops, museums and wireless hotspots") when they aren't displayed. I've also always planned to allow people to chose what things are of interest to them (I will always want to see lists of pubs and second-hand bookshops; others may be more interested in burger bars and wireless hotspots).

I'm having interesting thoughts about a randomly, but weighted, selected photo with buttons to vote on how much you like it (which affects the weighting) as well.

Giving an approx walking time for the distances is a nice idea. They are "as the crow flies" ones, so I'll have to see if that comes up with anything particularly silly.

I'll just keep plodding away. I'm creating this as a separate script, so as soon as I have something half decent I'll publish it and point blog readers to it.

Thoughts on Layout

2006-11-12T15:56:00.000Z

I'm working on a new page layout for the gazetteer - this is an essential preliminary to the new stuff I'm doing with POI data (since this is where most of it appears). This is the most messy page, and has grown the most during development.

Graphic design and layout aren't my strongest points, and I'm getting a bit stuck. So this is an appeal to all my readers to suggest something a bit better.

My current thoughts have a menu bar down the left, with things like "search", "home page", "set options", "sunrise/sunset","find nearest" and so on. It will also include a data panel with the postcode, OS grid ref, current sunrise/sunset, lat/long etc.

Then, on the right, the main page.

This needs to include at least the following:
Text about the place (such as historical stuff)
Generated text about the place ("... is on the ... canal, 3 miles from ...")
Photographs (sometimes none, one, dozens)
Google maps (if wanted)
Preprogrammed searches (restaurants etc)
Local stuff like shops - optionally tied to the google maps.

Does anyone have any elegant thoughts for what this should look like. Ideally, if you want my neverending gratitude, you could mock-up a page for a few places. You can just grab the HTML for any page and hack it around as much as you like, or make something up with some lorem ipsum and stock photos. I don't really care.

What matters is that the new layout should be able to show all the data included in all the following gazetteer entries, and ideally be easy to add much more to if I want to.

So, not asking much really, am I!

This is also a good time to chuck in any other thoughts on how to make the gazeetteer work better or add thoughts for new gazetteer features.

You can reply by commenting here, or by sending me an email. Googling for "Nick Atty" on uk.rec.waterways will find a currently valid email address for me.

Why can't we just kill them?

2006-11-04T08:58:00.000Z

What will I be doing on my computer today? Will I be improving CanalplanAC? Will I be writing stuff to help the campaign to save the waterways? No, I'll be doing yet more anti-spammer stuff.

There are clearly programs out there that run down through sites, find forms like the canalplan main input page, scan the page code, extract the fields, construct fake inputs for all the fields, and then push them back to the server.

So everyone who uses CanalplanAC gets slower response just because of the server load. If it was hosted on a pay-per-byte channel I'd be paying for the bandwidth. And I'm now having to add code to stop these malformed inputs breaking the code (safely - it thinks there is a bug elsewhere (as nothing could ever really) look like this and reports it to me). This adds more code, which slows down the whole thing slightly, adds a small processing load everytime it's run, adds more new code which is always prone to have bugs in it, and most importantly it's making me spend time effort and ingenuity on the program that no-one gets anything in return for.

It's Bonfire night tomorrow - perhaps we need to make it annual "burn a spammer" day,

Places of Interest - or interesting places?

2006-10-07T11:09:00.000+01:00

There are two fundamental approaches to mapping something like the waterways system: a topological one where you concentrate on the network structure - what is linked to where and how; and a geographical one where you concentrate on where things physically are.

The first of these means that your software is, at heart, an exercise in applied graph theory. The second means your program is a - specialised - geographic information system (a GIS).

CanalplanAC falls firmly into the first of these camps. The (apparently dead) Waterscape route planner gave every impression of being the second.

The problem is, neither of them can quite cut the mustard as you get into the details.

Waterscape's planner had the fundamental limitation that only one thing could be in one place at the same time. Which makes sense at first blush. But unfortunately it meant that every aqueduct was in fact a link between navigable waterways and it would cheerfully route you from Leigh to Manchester via the MSC - moving from one to the other at Barton.

Canalplan's approach - where places are nodes in a graph, and just happen to have some basic geographical information (in particular, their coordinates) bolted onto them is - I am convinced - the right approach. But it's not quite enough.

It starts to break down as you start to give more information about things around the waterways. In particular, as you get more and more places in the data, the idea that a pub - particularly one that isn't bang on the towpath - "belongs" to a particular place stops being sensible. You may have noticed that I've started to add more POI (Place Of Interest) data. In doing this, I've had to fiddle things by adding the POI items to all places within a certain range. This is inefficient - it's slow at build time, difficult to maintain if things change, very unamenable to user adjustment (something I want to continue to expand the use of) and just general not the Right Thing to do.

So I've started to migrate to a two layer approach. Waterways places will remain as a graph with coordinate information, but Places of Interest will live in a simple geographical database and be added as appropriate.

To help with this I've written a very nice (though I say it myself) tree based data structure and built a new sort of loop into the programming language that can search through several thousand places and find only those within a certain range of a place in almost no time at all. In the next release or so I'll have added this, and migrated all the existing pub, museum, shop etc data, so you won't see much difference, but it will all be working better.

It will then be that - as with photos - the data structure on the server is the true data, allowing people to modify and add places of interest with the changes taking place as soon as approved.

But, of course, it's never as simple as this. What, for example, is a boatyard? Is it a place on the canals, or is it a POI? If the former, then, unless each is added as its own place, we still have places with POI-type information associated with them. If the latter, then how will I ever implement "find nearest boatyard with gas for sale" for example?

I have a way to make the second of these work, but for the moment I'm suspending judgement on which way I will actually jump.

All of this is part - as I mentioned in the article about distances - of a gradual redesign of some of the underlying data structures. This is always a risky thing to do with something as large and evolved as CanalplanAC, but sometimes it just has to be done.

Canalplan is getting a bit long in the teeth and software rot is starting to set in in a few places. After all, I see from my change log that the gazetter, and the move to OS coordinates from purely arbitrary ones (what was I ever thinking of) came in in November 2000. In Internet terms, that's ancient. In that time the number of places in the database has gone up by an order of magnitude - and what worked fine then is really starting to creak. And there are new things out there. Satellite navigation systems having taken off means that geographical data is suddenly of use to lots of people, not just to a few wierdo programmers - and so it becomes available and affordable. I have to be able to move with this.

Distances - not always as straightforward as you'd think

2006-10-01T09:48:00.000+01:00

So I started putting distance table information in, and hit a snag along the Ashton when it started objecting to discrepencies between the distance table and the calculated distances.

Using a convenient Google Maps pedometer I crossed checked the distances.

For the fist 5 or 6 places it was really good:

Place	Distance table	Pedometer
Ducie Street, Manchester	0.0	0.0
Lock No 1 Ancoats Junction	.4	0.391
Lock No 3 Ancoats	.5	0.495
Lock No 4 Bewicks Locks	1.3	1.294
Lock No 8 Clayton Bottom Lock	2.0	2.036
Clayton Junction	2.4	2.485
Clayton Lock No 13	2.8	2.727
The last couple of these are showing slightly bigger errors, but it's still all well within what is needed for canal journeys. In fact, for Clayton Junction the data table merges it with Clayton Top Lock, which is a bit earlier, so the extra on the pedometer is explicable.
But now it starts to go wrong:
Clayton Top Lock No 16	3.5	3.033
We're out by nearly half a mile. Looking at any map it's clear that Clayton lock 13 is nowhere near that far from Clayton Junction. By the time we get to the end of this stretch, we're back on track
Fairfield Junction	3.8	3.735

So the moral seems to be not to trust the distance tables explicitly. It also shows that the pedometer approach may well work for getting pretty exact distances for all places. Now I'm not doing that for all 7000 places, but it could get build into the lengthsman feature at some stage.

In passing, the overall discrepency between the pedometer and the distance table is 128 yards. At 3mph this is less than 1½ minutes! I can live with that.

Measuring Up

2006-09-30T16:09:00.000+01:00

Some of you may have noticed my appeal in uk.rec.waterways for information on the lengths of a few tunnels, and wondered what I was up to.

Well here's what I'm doing. I created my data originally using, mainly, the route tables on Jim Shead's web site - which seem accurate and match whenever I cross-check anything.

When I interpolated a place, I guessed how to split the distance.

Over time this had lead to the distances being a bit arbitrary - the overall distances tend to be right, but some of the intermediate ones are really rather silly. And sometimes people complain about this.

Also, of course, people have been moving places around to get the coordinates spot on - yet this has no affect on the distances, when it ought to do.

So what I'm doing is to only keep the overall distances and to calculate the intermediate places during the data build. I'll compute the pythagorean distance between the places, total those, and then distribute the actual distance in proportion to the coordinate spacing (producing an error whenever there is a significant discrepency between the two totals).

The nice thing about this is that it keeps the totals right, and keeps all the intermediate places at the correct relative spacing, and always adjusts itself whenever new places are added or places haev their locations tweaked.

But I want known exact distances to remain exact: so tunnels in particular are flagged as having a fixed distance, which isn't affected by the adjustment.

I've built this, and tested it for one stretch of waterway - I'll add a couple more and then it can become a bit of background work.

Why am I doing this particular thing now? - well because it's part of a general attempt to strengthen the data and make it into a firmer foundation to build more sophisticated calculations etc on top of.

An embarrassment of riches

2006-09-21T07:55:00.000+01:00

Sometimes it's a case of "be careful what you wish for". Having put up my experimental lengthsman feature, several people have tried it, despite it being hidden well away. At the same time, photographers have been very active, not just submitting some truely stunning images, but also adding new places, and suggesting that photographs should be moved around as well.

And they've been doing it faster than I can keep up. So I've bitten the bullet and put updating things on hold somewhat while I automate the process. I suspect there's a technical term for this in project management terminology, but I view it as leaving the alligators alone and getting on with draining the swamp!

Not that this is, in any way, meant to be a complaint about the volumes of data. It's incredibly rewarding to see the amount of effort people are prepared to put into providing stuff for the site, and every bit is appreciated. It's just sometimes it gets hard to find the time out from real life to do anything with it! But please, keep it coming.

It all comes together

2006-08-24T19:37:00.000+01:00

I've just had one of those brilliant ideas when you suddenly see how half a dozen disparate things can all fit together.

I was having an email chat with Geoff Farmer about some ideas for CanalplanAC, mainly on the lines of what to do next (other than all the stuff I've written about here before, of course).

I was mentioning all the really neat interactive stuff Google now does that could be integrated in some way and mentioned the idea of place searching; imagine people being able to search for pubs and
restaurants, and then add them to the map for the benefit of people
coming along later.

Or how about a canalplan Google Gadget that provides Canalplan gazetteer
stuff.

In his reply he reminded me that the thing everybody really wants is more sophisticated overnight scheduling, and then said (I've taken the liberty of quoting him here):

For me though, the thing that will really make Canalplan a killer app will be itinerary planning.

Whilst Canalplan can plan a journey, the places it suggests I end my day at are not ideal - but then, I guess, Canalplan has no idea what a good stopping place is.

If you are aiming at getting Canalplan to produce itineraries, I would suggest that some way is introduced to record the suitability of stopping places. This would probably require the addition of visitor mooring places etc. and some sort of 'suitability as a stopping point' from zero for locks/bridges/winding holes etc to 10 for somewhere really nice. Users could select a minimum stopping suitability rank when specifying there journey details.

Then again, other criteria could be used like 'maximum distance to pub' or 'minimum number of real ales at local pub'

At that I wandered off to bed, mentally planning my reply which would go something like the one I've sent to others in the past:

That's a wonderful idea, but I'm far from convinced that anyone would go to the effort to put places in (Geoff has contributed heaps to CanalplanAC, and may think everybody is like him) and without all that data, it wouldn't work. And data bashing like that is not something I'd want to do.

And then, in the middle of the night, it all came together. If you merge these two thoughts with what I was saying a few weeks ago about the benefit of being on line, you get this - my super new idea for the future:

People can produce their own adjusted routes. They chose where to stop, but - if it has the data - Canalplan AC will suggest a few local places itself. And then -- the clever bit -- when they pick places they have to say why they are stopping there, and CanalplanAC keeps the data. So everytime you plan a route and select your stopping places, you are also helping the program select stopping places for everybody else, without doing anything different.

It's real next generation web stuff. Its one of the things that only an on-line program can do. And everytime the program is used, it gets better, all by itself!

None of this will happen tomorrow. But it is clearly the way to go. I'll keep you posted.

Medium term plans

2006-08-16T07:41:00.000+01:00

Working on the options screen for the POI data, I'm becoming aware that the gradual evolution of CanalplanAC has lead to some rather messy and inconsitent internals.

There are two different sets of options that can be saved in cookies: the route options and the gazetteer options. There are two different bits of the site you can log into (with different userids and passwords!): save and load routes and add photographs. There are bits that ought to be settable but aren't (your default starting position for example).

There is a also a weakness in the separation of the gazetteer and the route. This has lead to things like the Virtual Cruise, which is really a cut-down gazetteer being invented. Virtual Cruise is great, but the only reason it exists is that the gazetteer doesn't have access to the generated route.

So here's what is going to happen over the next few months, in order. It's the nearest thing CanalplanAC gets to a project plan. Italics is a policy change that will come out of things at around that stage, rather than a specific piece of work.

Gazetteer POI options will be implemented
The gazetteer will be redisgned to look more like the home page and photo pages with a left-hand menu bar. Some common code will be created and used for this
The log in and account process will be enhanced with things like "remind me of my password"
Users will be able to log on (with the existing "remember me") option from the home page
Only registered users will be able to load./save routes and options
Once logged in you will be able to load/save routes, add photos etc without any further verification
User options from the route planner and from the gazetteer will be saved in a central database by userid. If you are logged on, you will be offered this.
A new "options" screen, probably with tabs, will be created to set all options; exising route, existing gazetter, new ones

This will make the code more robust (there is clearly something dodgy about the user accounts for loading and saving at the moment: moving them in with the reliable photo ones will help), make it easy to rationalise things (the new style gazetteer will be flexible enough to work as the virtual cruise), and will be a good foundation for planned future work like the adjustable scheduler I have planned (since it will give me somewhere to keep the huge quantities of intermediate data).

So watch for this slowly appearing. I don't promise not to do other fun stuff along the way though!

Down at the coal-face

2006-08-07T20:59:00.000+01:00

Over the weekend I sucessfully rewrote a key part of the unknown place matching code in C. There is a lot of C in CanalplanAC (about 34000 lines) but these days very little programming takes place in it. Instead, I do almost all my development in the custom basic-like scripting language that this C code implements.

But as I mentioned before, place matching was getting slow, and to add matching of newly added places would hvae made it even lot slower. So a added a new "placematch" function that can be called from the scripts and which returns a list of places that match according to the specific pattern passed to it.

This is a great example of the tension between compiled and scripting languages. I can develop at about 5 to 10 times the speed in my scripting language than I can in C, but C runs at between 10 and 100 times as fast. Nearly always the first of these is more important than the second.

There are a couple of bugs in there - including one that has manifested itself twice today and which I cannot duplicate (CanalplanAC mails me when it fails, including all the input paramters so that I can reproduce the error). This is very odd and may keep me busy in the future - watch this space.

I also hope to start writing a bit about things other than the deep workings of the code - although as that's most of what I'm doing at the moment, it's not that likely to change - unless you tell me otherwise!

Crawling out of the woodwork

2006-08-01T07:26:00.000+01:00

Now this is weird. To say the least.

Over the weekend I tickled the sourcecode very slightly - I added a single new function to do fuzzy matching inside the code rather than externally (as mentioned in my last post). It's not working yet, this is just the hook that lets me develop it. Four lines changed in the code, none of them difficult, or with anything unusual about them.

When I released last night I started getting bug reports from the placefinder. It was blowing up in a line that multiplied a score by 100. When I adjusted that to stop it happening, the same happened in another line that multipled a different value by 100.

I got up early this morning as this was driving me bonkers and got down to work on it. The errors were coming out of the parsing part of the code - which hasn't been touched since September 2004. The adjustments I'd made to the code weren't the sort to have introduced any memory corruption bugs (plus the fact it was doing exactly the same on the development and server machine, and was utterly consistent), so what the heck was happening?

A bit of experiment showed that:
print 2 * 95
worked just as it should, as did
print 2 * 101
but try to do anything between 95 and 100 and it died. It wasn't the multiplication that did it, just the same happened with addition etc.

I know that it won't help, but I fruitlessly search for 94s and 95s, 100s and 99s in the code.

When I was first developing the CanalplanAC interpreter, which is far and away the most complicated thing I've ever written, I took the trouble to build a dedicated debugging and tracing feature into it, and left all the calls in the code - but they are excluded normally by some macro hackery.

So I changed a constant, rebuilt from clean, and this time, before typing

x = 2 * 95
I typed

trace "EF"

'E' means trace expressions, and 'F' means trace syntactical flow.

In another console I did the same with x = 2 * 10 and I compared the outputs. Each generated a page that looked like this (the full trace for a successful evaluation and assignment of 20 to x)


syntax.c (147) Read_Everything  Read: x = 2 * 10 as 1
syntax.c (162) Read_Everything  Identifier token is: x
syntax.c (187) Read_Everything  Parsing [SET/GOSUB]
syntax.c (188) Read_Everything  Syntax table reads: %IK#
syntax.c (190) Read_Everything  Using syntax table entry %
syntax.c (190) Read_Everything  Using syntax table entry I
syntax.c (853) Match_Simple  Read identifier - Identifier token is: x
syntax.c (190) Read_Everything  Using syntax table entry K
syntax.c (301) Read_Everything  Next token read: Token is operator: =
syntax.c (530) Read_Everything  Keyword changed to [IMPLICIT SET]
syntax.c (190) Read_Everything  Using syntax table entry #
syntax.c (285) Read_Everything  Spliced to syntax table: #%A~X
syntax.c (190) Read_Everything  Using syntax table entry %
syntax.c (190) Read_Everything  Using syntax table entry A
token.c (389) Syntactic_Unit  First token: Identifier token is: x
token.c (421) Pull_Expression  Expression token: Token is operator: =
syntax.c (747) Match_Simple  Item is an expression: x
syntax.c (190) Read_Everything  Using syntax table entry ~
syntax.c (190) Read_Everything  Using syntax table entry X
NUMBER: 2
token.c (389) Syntactic_Unit  First token: Integer constant is: 2
token.c (421) Pull_Expression  Expression token: Token is operator: *
NUMBER: 10
token.c (421) Pull_Expression  Expression token: Integer constant is: 10
token.c (421) Pull_Expression  Expression token: empty token
syntax.c (731) Match_Simple  Item is an expression:  2 * 10
syntax.c (595) Read_Everything  ?Separator empty token
syntax.c (606) Read_Everything  End of line: x = 2 * 10
syntax.c (607) Read_Everything  Lastpushed: 0, depth: 0
syntax.c (634) Read_Everything  About to resolve procedure calls
expression.c (313) Real_Evaluate  Caching this expression
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - Identifier token is: x
variables.c (148) Create_Global_Variable  Creating global variable x
variables.c (97) hashdup  Making item of size 10 to hold string 'x' - allocated at 0x86bc5b8
variables.c (103) hashdup  The name item points at 0x86bc5c0
variables.c (105) hashdup  Copied in
variables.c (152) Create_Global_Variable  Done!
expression.c (958) Push_Variable_Onto_Valstack  Pushing variable called x onto the stack
expression.c (959) Push_Variable_Onto_Valstack  It holds [NULL]
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - empty token
expression.c (560) Real_Evaluate  Returning from Evaluate
expression.c (1069) Drop_Valstk  Dropping value stack
expression.c (188) Evaluate_String  - EXPRESSION TO EVALUATE IS  2 * 10
expression.c (313) Real_Evaluate  Caching this expression
expression.c (336) Real_Evaluate  Getting token from expression into new storage
NUMBER: 2
expression.c (354) Real_Evaluate  Evaluating - Integer constant is: 2
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 2 onto the stack
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - Token is operator: *
expression.c (336) Real_Evaluate  Getting token from expression into new storage
NUMBER: 10
expression.c (354) Real_Evaluate  Evaluating - Integer constant is: 10
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 10 onto the stack
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - empty token
expression.c (557) Real_Evaluate  End-loop.  Have popped *
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 10
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 2
expression.c (663) Operate_On_Stack  Calculating 2 * 10
expression.c (665) Operate_On_Stack  Evaluates to 20
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 20 onto the stack
expression.c (560) Real_Evaluate  Returning from Evaluate
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 20
expression.c (210) Evaluate_String  -END OF EXPRESSION- returning 20
expression.c (1069) Drop_Valstk  Dropping value stack

The faulty calculation stopped dead with an error after the line I've marked in red.

Armed with this it was easy to see what was happening. The code was wrong, it was as simple as that - a test that should only have been performed when checking an "operator" was being performed for numbers as well. CanalplanAC stores all symbols in what is called a "union" - where different sorts of things are stored in the same space.

Now the first lesson. The time and effort I put into writing the trace module and embedding trace statements in the code this has paid off just today. This part of the code runs like clockwork, and I've never looked at it for ages. It's like debugging someone else's code, not mine now. Without the debugging this problem would have taken days to find.

Lesson two - you are not clever enough to use "fall-through" in C, even if you think you are. Even if it's one of the accepted reasons for doing it, and you comment it. It will still bite you on the arse.

When I added the new function I increased the size of the table that holds function names, and this in turn changed the values of the operators by one. This meant that the largest operator now had a value of 100.

Would you believe it, but for the last two years Canalplan has been incapable of evaluating any expression that ended with a literal number between 94 and 99? Well it has. And at no time at all has anyone tried to do this. There aren't any of them, anywhere in the code. But there are several 100s.

Lesson three - nobody ever tests their code enough.

And on that cheery note, I'm off to work.

Bitrot, or, Be Sure Your Bugs Will FInd You Out

2006-07-29T08:48:00.000+01:00

The Jargon File defines bitrot as the observation that unused programs or features will often stop working after sufficient time has passed, even if ‘nothing has changed’. If you ask me that "unused" is superfluous. In the last week two frequent users of CanalplanAC have reported the same bug, in an existing piece of code, that must have been there for the best part of a year. This is not the first time this has happened. And I'm not talking about an existing bug that has been tickled into life by some other change here, this is something that is doing just what it has always done ever since the feature was implemented (the actual bug is discrepencies between the counts of photographs submitted and the "my photographs" page).

In the same week I've found a quasi-bug in the gazetter options (there is developmental stuff in there that doesn't do anything apart from confuse people). I've also had so many new places submitted that I am finally having to get round to writing a proper, automated, way of adding them rather than cutting-and-pasting into the master data file.

It's also been pointed out to me while new places are in limbo (they've been submitted but not added) they cannot be found in a gazetter search (that's not entirely true, they can be found if you happen to remember their ID, but there is no way to find it out if you don't know it). To fix this will require a lot of work in the never-stable innards of the placefinder. It will complicate them so much that this is the prod I need to re-write this critical bit of code in C (it's now taking noticeable numbers of seconds to do a search).

And that's a week's worth. I'm not complaining - it's great that I have so many enhusiastic users who are busy adding photos, correcting coordinates, adding places and just generally picking nits off the program. But sometimes the net effect is that the your current taget (adding a new otions feature to limit POIs) is getting futher away, not closer!

Standalone CanalplanAC - what the heck is happening?

2006-07-24T19:21:00.000+01:00

Chris Hunter asked the above question - in a more polite way - in the first ever comment to this blog. It was one I was reserving for when I wasn't feeling inspired to write about what I was doing, and hadn't posted for a while. But it wouldn't be fair to leave it unanswered until then.

In short, it's stalled at the alpha-test version: three selected people - experts at breaking Web CanalplanAC (WCAC from now on) were sent disks and asked to do their worst. They sent me lots of comments, and gave me lots to think about.

Then my only Windows machine went belly-up the way only a Windows machine can, and I've not got round (or, frankly, felt justified in chucking several hundred pounds of the family silver at Standalone CanalplanAC (SCAC) development - as that's the only thing I'd need it for) to replacing it. My spare time over the last year has been like Chapter 13 of a Jasper Fford novel.

That put an immediate hold on things. In the meantime, three other developments took place, all of which deserve - and will eventually get - articles of their own. The first of these was the addition of Google Ads to WCAC; this funded a broadband connection which in turn lead me into the wonderful world of Google Maps - the second new development. The third was my licensing of POI data to improve the database.

Now all of those provide arguments against me putting effort into SCAC. The first is a far nicer way to earn money from the site than selling the software, the second won't work without a broadband Internet connexion, and distributing the data behind the third would break my licensing conditions.

There are so many nice things about developing on-line software like WCAC, and so many horrible things about developing stand-alone software like SCAC. Here are the first that spring to mind:

Release cycles. Stand-alone software needs to be nicely packaged, stable, without experimental features or egregious bugs (obligatory anti-Microsoft joke omitted). Web software can be the opposite: if you break it, never mind - you can fix it tomorrow. You can put a half-baked idea up and see what happens.
Support. The web software is free. If it doesn't do what you want, then that's fine - you've lost nothing but a bit of time. If I sell software I'll feel a moral obligation to support people, send out patches for bugs etc (oa-Mjo).
Peeking behind the scenes. You can do really cool things when the site is live and you own it. For example, every time the placefinder kicks in and suggests a match this is logged. At intervals I pour over the logs, then dry them out and tweak the auto-match code to make it actually reflect what people are doing.
Consistency. You are developing for a single system. You have to deal with a collection of web browsers, but that's all you have to deal with. You're not dealing with the niceties of SP2, home v professional, Windows 95, 98, 2000, XP, Vista, whatever. None of which you have, and all of which you have near religious objections to paying money for.
Independence. SCAC relies on the Abyss webserver. I've just heard that Aprelium have released a new version, and (surprise!) the patching of the configuration file that CanalplanAC's installation does fails miserably. I'ts not their fault - they don't know I'm relying on them, and even if I were, so what? But it just sums up the complexities of putting things on other machines with other software, rather than on your own server.

What it ulitmately boils down to is fun. This isn't what I do for a living and probably isn't what I want to. I get very little reward for it (and don't want too much - as long as I can justify all my income from the software on expenditure on hardware I can keep the taxman from taking his hunk out of it). Hacking together a new feature, throwing it out, watching it break, fixing it, getting nice comments from people (keep them coming - they really do inspire me to keep working) and getting "wouldn't it be nice if it did this) is a great way to spend your time. Working your way systematically down a list of known bugs and misfeatures, fixing them and not touching anything else interesting at the same time, regression testing the whole lot, etc, etc is the right way to do it if you're getting paid for it. But it's not much of a hobby.

I'm not saying I'm giving up the idea of ever releasing a stand-alone version. But I think it would need a major shift in something before I did. And a stand-alone would never be as featureful, data-rich, up-to-date or responsive as the web site.

More on POI options

2006-07-24T08:12:00.000+01:00

I've now manually assigned all the existing POI sets to one of a small number of "groups", and have automatically given each set an ID (I do like having an editor in which I can write keyboard macros with incrementing numbers in them!). These are output during POI data creation and then run through a bit of code to collate and sort them by group. The IDs will be used to save your preferences, which means that if I move a set from one group to another, or rename it, your options settings will not break.

The next thing is to go from the collated sets to an HTML page.

What I'm doing - places of interest

2006-07-23T13:41:00.000+01:00

One of the big efforts I'm making at the moment is to integrate all the POI data I've got into the database. What this involves is creating, and testing, rules that say what format the raw data files are in (in particular, some contain telephone information that can be parsed out using a regular expression) and how to convert them into entries in the place database. Having licensed the whole lot of data it seems silly not to use as much of it as possible.

But it's becoming obvious that for some areas it is starting to get a bit silly, and the map starts to vanish under the symbols. So I'm going to have to write an "option" selector that lets you chose which sorts of places you want showing (probably both at a "group" level - so you can exclude all historic sites for example, and at an individual level - so you can exclude a particular pub chain for example).

This won't be a particularly difficult piece of work, but it's fairly complicated, involving making changes to the data generation to automatically generate the configuration file - so that there is no reprogramming involved when new POI sets are added; writing code to generate the option file from the configuration file, and display it, and to load and save from a database somewhere; and to the gazetteer code to only show (both as maps and as text) places that you are interested in.

Should be a few week's work when fitted in along with everything else in life (a lot of next weekend will be lost to a beer festival!).

First thoughts

2006-07-23T12:51:00.000+01:00

I've been in the habit of embedding chatty bits in various bits of the CanalplanAC website for some time now, but what with the news, the todo list, the FAQs etc, it all gets a bit messed up. Then there is the guestbook, discussions on uk.rec.waterways and so on.

Suddenly I realised that a blog would be a much better and easier way of doing things, so here it is. I'm going to use it to put up information on what I'm thinking of doing and working on; what's bugging me at the moment, notes of really nice things that people have done and so on. I've sent an informal target of a couple of posts a week minimum - let's see how I do.

It's also a great opportunity for you, by commenting on my posts, to influence the future development of CanalplanAC.