It all comes together

I've just had one of those brilliant ideas when you suddenly see how half a dozen disparate things can all fit together.

I was having an email chat with Geoff Farmer about some ideas for CanalplanAC, mainly on the lines of what to do next (other than all the stuff I've written about here before, of course).

I was mentioning all the really neat interactive stuff Google now does that could be integrated in some way and mentioned the idea of place searching; imagine people being able to search for pubs and
restaurants, and then add them to the map for the benefit of people
coming along later.

Or how about a canalplan Google Gadget that provides Canalplan gazetteer
stuff.

In his reply he reminded me that the thing everybody really wants is more sophisticated overnight scheduling, and then said (I've taken the liberty of quoting him here):

For me though, the thing that will really make Canalplan a killer app will be itinerary planning.

Whilst Canalplan can plan a journey, the places it suggests I end my day at are not ideal - but then, I guess, Canalplan has no idea what a good stopping place is.

If you are aiming at getting Canalplan to produce itineraries, I would suggest that some way is introduced to record the suitability of stopping places. This would probably require the addition of visitor mooring places etc. and some sort of 'suitability as a stopping point' from zero for locks/bridges/winding holes etc to 10 for somewhere really nice. Users could select a minimum stopping suitability rank when specifying there journey details.

Then again, other criteria could be used like 'maximum distance to pub' or 'minimum number of real ales at local pub'

At that I wandered off to bed, mentally planning my reply which would go something like the one I've sent to others in the past:

That's a wonderful idea, but I'm far from convinced that anyone would go to the effort to put places in (Geoff has contributed heaps to CanalplanAC, and may think everybody is like him) and without all that data, it wouldn't work. And data bashing like that is not something I'd want to do.

And then, in the middle of the night, it all came together. If you merge these two thoughts with what I was saying a few weeks ago about the benefit of being on line, you get this - my super new idea for the future:

People can produce their own adjusted routes. They chose where to stop, but - if it has the data - Canalplan AC will suggest a few local places itself. And then -- the clever bit -- when they pick places they have to say why they are stopping there, and CanalplanAC keeps the data. So everytime you plan a route and select your stopping places, you are also helping the program select stopping places for everybody else, without doing anything different.

It's real next generation web stuff. Its one of the things that only an on-line program can do. And everytime the program is used, it gets better, all by itself!

None of this will happen tomorrow. But it is clearly the way to go. I'll keep you posted.

Medium term plans

Working on the options screen for the POI data, I'm becoming aware that the gradual evolution of CanalplanAC has lead to some rather messy and inconsitent internals.

There are two different sets of options that can be saved in cookies: the route options and the gazetteer options. There are two different bits of the site you can log into (with different userids and passwords!): save and load routes and add photographs. There are bits that ought to be settable but aren't (your default starting position for example).

There is a also a weakness in the separation of the gazetteer and the route. This has lead to things like the Virtual Cruise, which is really a cut-down gazetteer being invented. Virtual Cruise is great, but the only reason it exists is that the gazetteer doesn't have access to the generated route.

So here's what is going to happen over the next few months, in order. It's the nearest thing CanalplanAC gets to a project plan. Italics is a policy change that will come out of things at around that stage, rather than a specific piece of work.

• Gazetteer POI options will be implemented
• The gazetteer will be redisgned to look more like the home page and photo pages with a left-hand menu bar. Some common code will be created and used for this
• The log in and account process will be enhanced with things like "remind me of my password"
  
• Users will be able to log on (with the existing "remember me") option from the home page
• Only registered users will be able to load./save routes and options
• Once logged in you will be able to load/save routes, add photos etc without any further verification
• User options from the route planner and from the gazetteer will be saved in a central database by userid. If you are logged on, you will be offered this.
• A new "options" screen, probably with tabs, will be created to set all options; exising route, existing gazetter, new ones

This will make the code more robust (there is clearly something dodgy about the user accounts for loading and saving at the moment: moving them in with the reliable photo ones will help), make it easy to rationalise things (the new style gazetteer will be flexible enough to work as the virtual cruise), and will be a good foundation for planned future work like the adjustable scheduler I have planned (since it will give me somewhere to keep the huge quantities of intermediate data).

So watch for this slowly appearing. I don't promise not to do other fun stuff along the way though!

Down at the coal-face

Over the weekend I sucessfully rewrote a key part of the unknown place matching code in C. There is a lot of C in CanalplanAC (about 34000 lines) but these days very little programming takes place in it. Instead, I do almost all my development in the custom basic-like scripting language that this C code implements.

But as I mentioned before, place matching was getting slow, and to add matching of newly added places would hvae made it even lot slower. So a added a new "placematch" function that can be called from the scripts and which returns a list of places that match according to the specific pattern passed to it.

This is a great example of the tension between compiled and scripting languages. I can develop at about 5 to 10 times the speed in my scripting language than I can in C, but C runs at between 10 and 100 times as fast. Nearly always the first of these is more important than the second.

There are a couple of bugs in there - including one that has manifested itself twice today and which I cannot duplicate (CanalplanAC mails me when it fails, including all the input paramters so that I can reproduce the error). This is very odd and may keep me busy in the future - watch this space.

I also hope to start writing a bit about things other than the deep workings of the code - although as that's most of what I'm doing at the moment, it's not that likely to change - unless you tell me otherwise!

Crawling out of the woodwork

Now this is weird. To say the least.

Over the weekend I tickled the sourcecode very slightly - I added a single new function to do fuzzy matching inside the code rather than externally (as mentioned in my last post). It's not working yet, this is just the hook that lets me develop it. Four lines changed in the code, none of them difficult, or with anything unusual about them.

When I released last night I started getting bug reports from the placefinder. It was blowing up in a line that multiplied a score by 100. When I adjusted that to stop it happening, the same happened in another line that multipled a different value by 100.

I got up early this morning as this was driving me bonkers and got down to work on it. The errors were coming out of the parsing part of the code - which hasn't been touched since September 2004. The adjustments I'd made to the code weren't the sort to have introduced any memory corruption bugs (plus the fact it was doing exactly the same on the development and server machine, and was utterly consistent), so what the heck was happening?

A bit of experiment showed that:
print 2 * 95
worked just as it should, as did
print 2 * 101
but try to do anything between 95 and 100 and it died. It wasn't the multiplication that did it, just the same happened with addition etc.

I know that it won't help, but I fruitlessly search for 94s and 95s, 100s and 99s in the code.

When I was first developing the CanalplanAC interpreter, which is far and away the most complicated thing I've ever written, I took the trouble to build a dedicated debugging and tracing feature into it, and left all the calls in the code - but they are excluded normally by some macro hackery.

So I changed a constant, rebuilt from clean, and this time, before typing
x = 2 * 95
I typed
trace "EF"
'E' means trace expressions, and 'F' means trace syntactical flow.

In another console I did the same with x = 2 * 10 and I compared the outputs. Each generated a page that looked like this (the full trace for a successful evaluation and assignment of 20 to x)

syntax.c (147) Read_Everything  Read: x = 2 * 10 as 1
syntax.c (162) Read_Everything  Identifier token is: x
syntax.c (187) Read_Everything  Parsing [SET/GOSUB]
syntax.c (188) Read_Everything  Syntax table reads: %IK#
syntax.c (190) Read_Everything  Using syntax table entry %
syntax.c (190) Read_Everything  Using syntax table entry I
syntax.c (853) Match_Simple  Read identifier - Identifier token is: x
syntax.c (190) Read_Everything  Using syntax table entry K
syntax.c (301) Read_Everything  Next token read: Token is operator: =
syntax.c (530) Read_Everything  Keyword changed to [IMPLICIT SET]
syntax.c (190) Read_Everything  Using syntax table entry #
syntax.c (285) Read_Everything  Spliced to syntax table: #%A~X
syntax.c (190) Read_Everything  Using syntax table entry %
syntax.c (190) Read_Everything  Using syntax table entry A
token.c (389) Syntactic_Unit  First token: Identifier token is: x
token.c (421) Pull_Expression  Expression token: Token is operator: =
syntax.c (747) Match_Simple  Item is an expression: x
syntax.c (190) Read_Everything  Using syntax table entry ~
syntax.c (190) Read_Everything  Using syntax table entry X
NUMBER: 2
token.c (389) Syntactic_Unit  First token: Integer constant is: 2
token.c (421) Pull_Expression  Expression token: Token is operator: *
NUMBER: 10
token.c (421) Pull_Expression  Expression token: Integer constant is: 10
token.c (421) Pull_Expression  Expression token: empty token
syntax.c (731) Match_Simple  Item is an expression:  2 * 10
syntax.c (595) Read_Everything  ?Separator empty token
syntax.c (606) Read_Everything  End of line: x = 2 * 10
syntax.c (607) Read_Everything  Lastpushed: 0, depth: 0
syntax.c (634) Read_Everything  About to resolve procedure calls
expression.c (313) Real_Evaluate  Caching this expression
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - Identifier token is: x
variables.c (148) Create_Global_Variable  Creating global variable x
variables.c (97) hashdup  Making item of size 10 to hold string 'x' - allocated at 0x86bc5b8
variables.c (103) hashdup  The name item points at 0x86bc5c0
variables.c (105) hashdup  Copied in
variables.c (152) Create_Global_Variable  Done!
expression.c (958) Push_Variable_Onto_Valstack  Pushing variable called x onto the stack
expression.c (959) Push_Variable_Onto_Valstack  It holds [NULL]
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - empty token
expression.c (560) Real_Evaluate  Returning from Evaluate
expression.c (1069) Drop_Valstk  Dropping value stack
expression.c (188) Evaluate_String  - EXPRESSION TO EVALUATE IS  2 * 10
expression.c (313) Real_Evaluate  Caching this expression
expression.c (336) Real_Evaluate  Getting token from expression into new storage
NUMBER: 2
expression.c (354) Real_Evaluate  Evaluating - Integer constant is: 2
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 2 onto the stack
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - Token is operator: *
expression.c (336) Real_Evaluate  Getting token from expression into new storage
NUMBER: 10
expression.c (354) Real_Evaluate  Evaluating - Integer constant is: 10
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 10 onto the stack
expression.c (336) Real_Evaluate  Getting token from expression into new storage
expression.c (354) Real_Evaluate  Evaluating - empty token
expression.c (557) Real_Evaluate  End-loop.  Have popped *
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 10
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 2
expression.c (663) Operate_On_Stack  Calculating 2 * 10
expression.c (665) Operate_On_Stack  Evaluates to 20
expression.c (921) Push_Value_Onto_Valstack  Pushing value of 20 onto the stack
expression.c (560) Real_Evaluate  Returning from Evaluate
expression.c (1018) Pull_Top_Value_Off_Stack  Result was a new value 20
expression.c (210) Evaluate_String  -END OF EXPRESSION- returning 20
expression.c (1069) Drop_Valstk  Dropping value stack

The faulty calculation stopped dead with an error after the line I've marked in red.

Armed with this it was easy to see what was happening. The code was wrong, it was as simple as that - a test that should only have been performed when checking an "operator" was being performed for numbers as well. CanalplanAC stores all symbols in what is called a "union" - where different sorts of things are stored in the same space.

Now the first lesson. The time and effort I put into writing the trace module and embedding trace statements in the code this has paid off just today. This part of the code runs like clockwork, and I've never looked at it for ages. It's like debugging someone else's code, not mine now. Without the debugging this problem would have taken days to find.

Lesson two - you are not clever enough to use "fall-through" in C, even if you think you are. Even if it's one of the accepted reasons for doing it, and you comment it. It will still bite you on the arse.

When I added the new function I increased the size of the table that holds function names, and this in turn changed the values of the operators by one. This meant that the largest operator now had a value of 100.

Would you believe it, but for the last two years Canalplan has been incapable of evaluating any expression that ended with a literal number between 94 and 99? Well it has. And at no time at all has anyone tried to do this. There aren't any of them, anywhere in the code. But there are several 100s.

Lesson three - nobody ever tests their code enough.

And on that cheery note, I'm off to work.