Recently I gave my first of two talks at the London Node User Group about the concept of protocols and lazy sequences as they apply to JavaScript (and Node). Despite a some issues with my slides the talk went pretty well and I should be on course to do part two in the not too distant future.
Here’s the video:
Here are the slides:
Before I started this blog I had a strong desire to share the knowledge, experience and techniques I’ve learned while developing VoiceXML applications for leading UK speech solutions provider Eckoh where I am a developer. From talking to a few people I soon realised that the issues I was planning to tackle and the problems that I had already solved are general issues for VoiceXML developers. I then came to the conclusion that what was missing was a toolkit, that anyone could use and contribute to, with the specific goal of increasing the development speed and flexibility of VoiceXML applications using modern programming/software techniques. And so the VoiceXML Toolkit (VXTK) project was born.
As an introduction to the project what follows is the first draft of the VXTK Manifesto:
VXTK is the VoiceXML Toolkit, it’s primary goals are to:
VoiceXML 2.x has been around for a few years now (from 2004) and a number of solutions/frameworks have been released for building applications. It seems, however, that many of these utilise a very specific solution to what is a very general problem. The general trend seems to be toward “home-grown” frameworks or propitiatory libraries. Parallel to this, there has been much movement in the Javascript/ECMAScript world both on and off the browser. VXTK tries to join these worlds together to add previously missing functionality to the VoiceXML browser and provide a common interface to the backend.
While VXTK will provide the basis for a full application development environment it would not replace any existing frameworks, rather it should enhance those that already exist and unify how they might interoperate. Additionally it will, hopefully, provide a home for the developers of those libraries and frameworks to share their expertise in a way that will benefit the community that, currently, seems to be absent.
VXTK is currently under active development and is available via Google Code at http://code.google.com/p/vxtk/.
At the moment development is closed, this will only be until the 0.0.1 release at which the project will be opened. However, I would encourage those interested to submit bugs and feature requests to the Google Code site.
Subroutines are an important part of many useful languages and so it seems, to me, a little bit of a blow that VoiceXML doesn’t give us a useful method of calling subroutines. It could be argued that this is, in fact, exactly what subdialogs are for and while I’d concede that this is generally true there are some limitations of subdialogs that make them a lot less useful than they appear on the face of it.
Last year I built a library that would do some queuing on behalf of an application and then call a custom page provided by the calling application. The application would call my library with a subdialog and then my library would call the application’s custom page using a subdialog. So I wrote the library and tested it using a simple test application but, when integrated into a production application, it would always break when calling the user supplied page.
It turned out that the calling application had used subdialogs to achieve its code reuse so, by the time they had called my code we were already 4 or 5 subdialogs deep and our platforms limit was set to 6. We were forced at this point to increase the browser limit to 10 but, if we were not able to do so, we would have been in serious trouble having to rewrite large chunks of both the application and the library code.
So when I came to write my next application and was faced with the same problem of code reuse I tried to find a way to do so without invoking a, potentially long, chain of subdialogs. In the back of my mind I had remembered a concept that could, in theory, solve my problem. The concept was that of continuations and, in specific, the first-class continuation.
A continuation is the ability to freeze a program’s entire state state, call some other code and then resume the original program from where it left off. A first-order continuation is slightly simpler in that it only saves the program’s execution state and not all of its state information (kind of like GOSUB in BASIC). A good example was posted on the perl6 message boards by Luke Palmer:
Say you’re in the kitchen in front of the refrigerator, thinking about a sandwitch. You take a continuation right there and stick it in your pocket. Then you get some turkey and bread out of the refrigerator and make yourself a sandwitch, which is now sitting on the counter. You invoke the continuation in your pocket, and you find yourself standing in front of the refrigerator again, thinking about a sandwitch. But fortunately, there’s a sandwitch on the counter, and all the materials used to make it are gone. So you eat it. 🙂
Another example of a first-class continuation is throw/catch in VXML. When an event is thrown execution passes to the catch handler which is then free to modify any of the current state of the application and then, when the handler completes, control is returned to the point at which the event was thrown. This is an effect I exploited to achieve dynamic loading in a earlier post.
After some digging, it looked as if all this was pretty much the sort of thing I wanted to do and so I went about implementing something similar in VXML and Javascript.
The basic idea of a first-class continuation is the ability to save the current position of a program and then goto some other position in the same program. The saved position is then used to work out where to continue from when the “other” bit of code has completed. This is pretty tricky in VXML as we don’t have access to the current execution stack. What’s less tricky is saving where the program will go next, instead of where it is now. We know this at design time as, quite simply, that’s the way we can write it:
<vxml application="root.vxml"> <form id="form1"> <field name="field1_1"> </field> <field name="field1_2"> </field> <block> <!-- invoke continuation --> </block> </form> <form id="form2"> <block> <!-- return from continuation --> </block> <field name="field2_1"> </field> <field name="field2_2"> </field> </form> </vxml>If we split our code into two forms then we know exactly where we want execution to continue from because it’s the top of the next form. So let’s create a function called gosub which takes 2 arguments, the first will be where we’re going to and the second is where we want to return to (i.e. the next form). We can then use the goto tag to tell the browser to goto our destination code:
<block> <!-- invoke continuation --> <goto expr="gosub('otherPage.vxml', 'thisPage.vxml#form2')" /> </block>The gosub function just needs to push the next form’s URI onto a stack and then return the value of the destination argument:
var stack = []; function gosub(dest, next) { stack.push(next); return dest; }Now we can goto some other VXML page (and form) and save the state so we can return later, now all we need now is some way to get back. This turns out to be pretty much the same solution just in reverse. At the end of our destination page we call the goto tag, popping of the next URI from the stack:
<block> <!-- return from the continuation --> <goto expr="gosub.back()" /> </block>Calling gosub.back (we use back because return is a reserved word) does the job of popping a URI off the stack and returning it to the goto expr:
gosub.back = function() { return stack.pop(); }The only thing missing now is to make it as useful as a subdialog. In order to do that we need to be able to pass arguments in and return some result. Here’s how our original example would look if we could pass arguments and get a result:
<vxml application="root.vxml"> <form id="form1"> <field name="field1_1"> </field> <field name="field1_2"> </field> <block> <!-- invoke continuation --> <goto expr="gosub('otherPage.vxml', 'thisPage.vxml#form2', { param1: value1, param2, value2 })" /> </block> </form> <form id="form2"> <block> <!-- return from continuation --> <var name="result" expr="gosub.result" /> </block> <field name="field2_1"> </field> <field name="field2_2"> </field> </form> </vxml>So we added an extra argument to gosub which receives the argument(s) to pass to the continuation and a parameter which contains the result of the continuation. From an implementation point of view this is pretty simple if we make the assumption that gosub.result only needs to contain the last result returned from a gosub. We can also make the same true from the point of view of the arguments by providing a gosub.args property which will contain the arguments from the last invocation of gosub:
var stack = []; function gosub(dest, next, args) { stack.push(next); gosub.args = args; return dest; } gosub.back = function(result) { gosub.result = result; return stack.pop(); }So our called page/form would look something like:
<vxml> <form id="subForm"> <var name="args" expr="gosub.args" /> <var name="result" expr="{}" /> <field> <!-- Do something here --> </field> <block> <!-- return from the continuation --> <goto expr="gosub.back(result)" /> </block> </form> </vxml>Given this we can now get the effect of calling a subdialog without the need to start a new browser instance. This method does, however, have a similar limitation to a subdialog which is that, in a subdialog, you cannot store anything it the calling page’s global state and, using this method, you cannot store anything in the page’s (or form’s) local state. I’ll post later on how we might work around this restriction but, for most uses this should not be an issue.
Update: The code represented here is now available in the current trunk of VXTK.
In the last post we discussed the CommonJS module loading spec and how it can be modified for asynchronous loading. It turns out that this is really useful for adding dynamic loading to VXML pages and in this post I’ll discuss exactly how that can be achieved.
At the end of the last post I left you with the following example implementation:
function loadScript(uri, callbacks) {
// Put your loader here
}
function resolve(loader) {
var next = require.resolve.next());
var uri = require.resolve.find(next);
loader(uri, {
onSuccess: function() {
if (next === require.resolve.next()) {
// The module did not load properly
// handle the error
} else {
// The module loaded successfully
resolve(loader);
}
},
onError: function() {
// handle error
}
});
}
// Setup the paths
require.resolve.path('', '/js');
require.resolve.path('thirdPartyModule', 'http://path.to.third.party/module');
// Initialise the dependency stack
require.resolve.load('myModule');
// load the module(s)
resolve(loadScript);
VXML has some important restrictions that will prevent us from directly implementing the code above:
Fortunately the restrictions are not insurmountable and we can exploit a few features of VXML that will enable us to work around the apparent issues:
The target code above has a number of discrete steps that we’ll deal with in turn to develop our loader, they are:
The first 2 steps can be easily achieved using a throw:
<vxml>
<catch event="require.resolve">
<!-- Resolve dependencies -->
</catch>
<form id="loader">
<block>
<throw event="require.resolve" />
</block>
<block>
<!-- Your code here -->
</block>
</form>
</vxml>
So long as the loader form is the first in our document, the require.resolve event gets thrown immediately and when it completes execution is passed to the next block.
We can achieve the 3rd step by calling require.resolve’s next and find methods along with a script src expression in our catch block:
<catch event="require.resolve">
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<script expr="require.resolve.find(next)" />
</if>
</catch>
If the load fails we’ll get a error.badfetch event required by step 4, however, if it succeeds but the loaded file doesn’t define our module we still need to catch that. In order to do that we’ll exploit the behaviour of require.resolve.next which, if our module was not defined, will return the same module name as it did the first time:
<catch event="require.resolve">
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<script expr="require.resolve.find(next)" />
<if cond="next == require.resolve.next()">
<throw event="error.badfetch" messageexpr="'Could not load module ' + next"/>
</if>
</if>
</catch>
Now, if we get back the same module name, we throw our own error.badfetch.
Finally we need recursively call the resolver to resolve the next dependency, to do this we just throw the event again:
<catch event="require.resolve">
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<script expr="require.resolve.find(next)" />
<if cond="next == require.resolve.next()">
<throw event="error.badfetch" messageexpr="'Could not load module ' + next"/>
</if>
<throw event="require.resolve" />
</if>
</catch>
The catch block can now be moved into a root document where we can also load our implementation of require and initialise the loader:
<vxml>
<script src="/js/require.js" />
<script>
require.resolve.path('', '/js/');
require.resolve.load('myApp');
</script>
<catch event="require.resolve">
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<script expr="require.resolve.find(next)" />
<if cond="next == require.resolve.next()">
<throw event="error.badfetch" messageexpr="'Could not load module ' + next"/>
</if>
<throw eventexpr="_event" />
</if>
</catch>
<catch event="require.resolve" count="5">
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<log>More modules to load make sure you throw <value expr="_event" /></log>
</if>
</catch>
</vxml>
You’ll notice that in the above code I’ve added an explicit catch for the 5th time the event is thrown. The reason for this is that some VXML browsers restrict the number of times you can throw an event from itself. To get around this we terminate the load process on the 5th throw. This does mean that you can return from the loader without loading all your modules, we can solve this by adding a little bit of code to restart the loop in our VXML document:
<vxml application="root.vxml">
<form id="loader">
<block>
<throw event="require.resolve" />
</block>
<block>
<var name="next" expr="require.resolve.next()" />
<if cond="next">
<goto next="#loader" />
</if>
</block>
</form>
<form id="myForm">
<block>
<prompt><value expr="require('myApp').message" /></prompt>
</block>
</form>
</vxml>
So, now, if the loader returns before all the modules have been loaded then the loop is started up again. When all the modules are loaded the browser falls to the next form which starts running your code.
In January 2009 Kevin Dangoor started the ServerJS project, later that year it was renamed to CommonJS. The goal of the project was to create an ecosystem to enable developers to write javascript code to run outside of the browser and, at the time of writing, Wikipedia lists 15 implementations including node.js, CouchDB and XULJet and 4 proposals have become specifications: modules, packages, system and promises (although the latter does not seem to be reflected on the CommonJS site). Today we’ll look at the module specification and how we can get it to load modules asynchronously.
The specification is at version 1.1.1 right now and the general theory of operation is pretty straight forward. It defines a single free function called require which is responsible for loading and managing any dependencies outside of your current .js file.
var add = require('math').add
var increment = function(val) {
return add(val, 1);
};
In the above example we require the math library and then use the add method in our increment function. The math library itself defined the add function by “exporting” it. This is done by adding it as a property of an exports object that is created for it by the module loader:
exports.add = function(a, b) {
return a+b;
};
It’s the job of require to load the math.js file and provide it with the exports object which it [math.js] can then populate. This all happens synchronously when require is called which means that implementations without the ability to directly load files are unable to use this version of the specification.
There’s been a lot of talk in the CommonJS community about how to tackle just this issue and they’ve come up with the transport addition to the spec. It’s still in the proposal phase right now and has 4 sub proposals each of which has various levels of support. Transport proposal D seems to include many of the features of the other proposals and also seems [in my view] to be garnering the most acceptance, or at least the most discussion.
The basic theory of the transport proposal is that, instead of a file that just starts populating the exports object, we defer calling the body of the module until later. We do this by calling a require method which takes care of resolving dependencies and initialising the module.
Lets look at an example, the code below defines the math module and the add method:
require.define({
"math": function (require, exports, module) {
exports.add = function(a, b) {
return a+b;
}
}
}, []);
So now in our module instead of starting straight away with our exports we make a call to require.define which sets up the definition of our module. The definition contains a set of the modules defined in the file and the factory methods needed to instantiate them. When we then call require(‘math’) later in our code the factory function is call and the exports object is returned.
The last argument of require.define is a list of dependencies for the modules defined in the preceding definition, before any call to require can be made the dependencies listed in the final argument must have been loaded. If not then require will throw an exception.
The main advantage of the above method for defining our modules is the ability to know all the dependencies of a module upfront. Additionally we have the ability to delay initialising any module until it’s at which point we should have loaded it’s dependencies. If you put these two together then you have the ability to asynchronously load your modules.
The proposal leaves the nuts and bolts of module loading up to the implementation so from here on we’re going to go “off-road” and define an additional set of methods that we’ll use to load our modules. To make things neat we’ll put all these new methods into an object called require.resolve so the chance of colliding with changes to the spec is minimal.
First up is require.resolve.next, this will check the dependency list and return the first unloaded item in the dependency list. Until the module becomes defined, calling the method again will return the same value. This will be useful for checking to see if a load was successful as we can check to see if the next module has changed after the load.
Next we’ll add a method called require.resolve.find(id) which, given a module id, will return the URI we need to use in our script tag. Additionally we’ll need a require.resolve.path(prefix, path) which will allow us to match top level module names to specific paths, this will become useful if a module is located somewhere other that the current host or we want to change the base path to load from.
Finally we need some way to tell require what modules to start with (bootstrapping). For this we’ll use require.resolve.load(id) which will push a module onto the dependency stack.
Using the new methods we can poll the dependency stack and load modules. Loading new modules should update the dependency stack and we can keep going till all modules have been loaded. Below is some sample code for how you might use these new methods to implement a loader/resolver:
function loadScript(uri, callbacks) {
// Put your loader here
}
function resolve(loader) {
var next = require.resolve.next());
var uri = require.resolve.find(next);
loader(uri, {
onSuccess: function() {
if (next === require.resolve.next()) {
// The module did not load properly
// handle the error
} else {
// The module loaded successfully
resolve(loader);
}
},
onError: function() {
// handle error
}
});
}
// Setup the paths
require.resolve.path('', '/js');
require.resolve.path('thirdPartyModule', 'http://path.to.third.party/module');
// Initialise the dependency stack
require.resolve.load('myModule');
// load the module(s)
resolve(loadScript);
Look out for my next post which will show you how to implement the above code in VXML.
Before starting any real work on VXTK I wanted to try to adopt one of the mantras of test driven development which is “test early and test often.”
There do seem to be quite a few test frameworks available for javascript most of which run in the browser. This is less than useful if you want to integrate your tests into a continuous integration environment. After a long search on Google the Jasmine BDD framework kept coming up and, more specifically, the jasmine-reporters project. I quite like the idea of behaviour driven development and feel that it would lend itself well to the project. However, I may be completely wrong and may well find myself rewriting all my tests for JSUnit or similar.
Expect a later post on my CI setup, but for the moment I’ll leave you with my first milestone. I’ve tagged my initial project skeleton with a fully working setup of jasmine-reporters at http://code.google.com/p/vxtk/source/browse/?r=skeleton. I’ve opted for mercurial rather than SVN I’ll be writing most of the code on my netbook which won’t be on line all the time and I still want the ability to commit and branch code as I write it.
To run the test’s you’ll need java installed but that should be all, jasmine-reporters does a pretty good job of bundling all it’s dependencies so you should be able to just execute the run.sh file in the test directory. I’ve included a simple suite in tests.html just so you can verify if everything’s working
As it stands it’s a pretty good starting point for any javascript project which is why I’ve created the tag.
Happy testing.
In this series I’ll explain how to create dynamic and reusable VoiceXML without the need to generate any pages on the sever side. After each post the code will be uploaded to http://code.google.com/p/vxtk/ where it will form the basis for VXTK (VoiceXML Tool Kit). The code will be licensed under GPLv2 and the blog posts under creative commons Attribution-ShareAlike.
From the outset VoiceXML look to be a promising idea, effectively taking the lessons learned from the Web and applying them to voice. When the 2.0 spec came out I must admit to being less than impressed as some fundamental features appeared to be missing forcing vendors to produce non-standard extentions. While the 2.1 spec addressed some of these issues there are still a few notable exceptions:
1. No ability to load an arbitrary number of scripts.
Without the, now essential, xhr requests we’ve come to know and love in the web 2.0 world there’s no obvious way to load an arbitrary .js file when you need to. This means you need to either load all the scripts you’ll ever need in the root document or your vxml page. In this series I’ll demonstrate a clever solution that gets around the problem and allows for a dynamic loading mechanism that doesn’t break the spec and complies with the CommonJS module specification (transport proposal D).
2. Static properties
A glaring omission is the lack of an expression attribute for the property tag meaning that having dynamic values for important properties like timeouts and fetchaudio is impossible without generating the page server side. Some vendors have added this feature (with some restrictions) but, if your not one of the lucky few then you may still be forced to generate some pages. In this series I’ll offer a simple solution using Apache server side includes and a class to help manage properties.
3. No subroutines
While it’s possible to use subdialogs to create reusable code their behaviour can make using them at best cumbersome and at worst very inefficient. Using the continuation pattern I’ll demonstrate how we can simulate a gosub and then asynchronous method calls.
In parts 1 to 3 I’ll cover each of the above issues and this will become the base of the VXTK library. In the subsequent posts I’ll tackle a specific feature of the library and finally I’ll demonstrate how we can use the library to build a fully dynamic dialogue manager.
The additional topics I’ll cover will include:
The baha beach club 