Johno Scott's Blog: Building Wave Robots in ColdFusion with OpenBD on Google App Engine

Robot development for Google Wave has sparked my imagination lately. You can* write Robots in Java and Python and run them on Google App Engine, but what about ColdFusion ? Well since Open Blue Dragon (OpenBD) can now run on GAE, I thought it would be fun to have a go at a Wave Robot written in ColdFusion.

Update: 1 Jun 2010 - Its working now (Yea !).

I have working Operations being sent back to the server causing changes in the Wave. The profile is also working. The problem was :

The JSON that the wave server is expecting is case sensitive. Key names like childBlipIds which need to be in the precise camel-case.

I had to use a 3rd party CFJSON serializer because the OpenBD built-in one forces upper or lower case key names.
Adding keys to objects you are creating to send back to the wave server need to be the correct case, using the correct method.

This is WRONG :

o.params = StructNew();
o.params.blipData = NewBlip(); // Case of the keyname will change to PARAMS and BLIPDATA

This is CORRECT:

o['params'] = StructNew();
o['params']['blipData '] = NewBlip();

So make sure you create your keys using the [‘keyName’] notation so that the case is preserved

Try it yourself. Invite cfwavebot@appspot.com to a wave.

After reading API docs for the Google Wave, the Robot Wire Protocol is what I decided to implement (partially – as a proof or concept). It is a light-weight message and event protocol that uses JSON to exchange data.

*Update : At I/O 2010 Google announced that Robots can run on servers other that Google App Engine. I haven't tried it myself yet.

This blog post is about my efforts so far, and explains how to get setup the way I did. CFML source code is included

In this article :

What is / isnt working ?

Problems Overcome
What I have / what remains to be implemented
How To Build Your Own ColdFusion Wave Robots

Essential Reading

Where to get help

What is / isnt working ?

jsonrpc message bundles are being received by the robot OK
The robot is replying with operations json :

wavelet.appendBlip

Fixed ! The Wave Server wont:

process the operations json the robot sends back
Display any of the profile details (avatar, profile web link)

You can see how it is currently running from this screencast (doesn’t have sound).

http://www.youtube.com/watch?v=q3h4oU7hGKI

Notice from the video that :

I start a new wave with the robot but nothing from it appears in the wave
From the GAE logs, you can see the robot receives jsonrpc messages and responds with operations like "wavelet_appendBlip"

I conclude (with a guess) that the UPPERCASE field names that ColdFusion creates when using its serializejson() function cant be understood by the Google Wave Server

There are more detailed screenshots of the issues so far below.

Problems Overcome

Logging to the console is the only way to debug. Forget about <cfdumps> to the page, because you are always returning json strings, not HTML. Fortunately good-ole <cflog > works both locally and on Google App Engine.
Debugging with a locally running version of your robot is much quicker that deploying to GAE. You need to fake jsonrpc calls from the Wave Server which you can do with a simple web page that you can cut and paste json messages and then fire them at your robot.
Different JSON http content encoding (binary and text)
What isn't documented

OpenBD can't process requests like /robots/jsonrpc without a URL re-writer
- You need to use the tuckey url rewriter

How the ColdFusion version works

I started with a single cfml template for each of the 3 URLs that the Robot must answer on. ColdFusion has built-in support for JSON serialization (although its insistance on CAPITALIZING field names has be stuck at the moment).

I made use of the URL rewriting technique for OpenBD described on the OpenBD wiki to handle all the required Robot URLS :

URL re-writing maps the required calls as follows :

Robot URL	Mapped to CFML Script
_wave/capabilities.xml	_wave/robot/capabilies.cfm
_wave/robot/profile	_wave/robot/profile.cfm
_wave/robot/jsonrpc	_wave/robot/jsonrpc.cfm

Capabilites Hash - The Robot needs to provide a hash of its current capabilites each time it replies to the Wave Server. I think the idea here is that this should be changed when you decide to add or remove one of the events in capabilities.xml. It is 16 hex characters long so I used a built-in hash generator and trimmed it.

Event Handling - there is a <cfswitch> statement that sends events to individual event handler functions. Simple for now.

Operations Queue - A simple queue (an Array) of Operation objects is built for the event handlers to generate operations. These are bundled up and serialized as JSON and sent back to the wave server.

What remains to Implement

The ColdFusion code I have so far is really just proof-of-concept. Once I have a working robot I plan to do the following :
- Implement classes (using CFCs) to mimick the Wave Robot Java Client Library

Essential Reading

Official Wave API docs from Google
- Google Wave API OverView Has a description of Wave Entities (and a diagram to help explain them). You need to know and undestand these key concepts :
  - Wave
  - Wavelet
  - Blips and Data Documents

The Events and Operations detail in the API explain the interactions between Robots and the Wave Server

The Wave Robot Wire Protocol is what I set out to implement. It explains the JSON-RPC message passing.

Google App Engine on the OpenBD Wiki has some important information about what to expect when running ColdFusion on GAE. The essential things are :
- No <CFFile> access or writing to the file system

Debugging Wave Robots : This essential article explains the technique of debugging your robot on a local development server and "faking" Wave JSON message bundles from your browser. It also explains the logs viewer on Google App Engine.

How To Build Your Own ColdFusion Wave Robots

Setup Eclipse with the Google plug-in for Eclipse
- There are good instructions for doing this :
  - Setting Up App Engine, Google Wave Robots API: Java Tutorial

Download OpenBD for Google App Engine (make sure you get the Google Ap Engine bundle)

TIP: The configuration instructions are in the zip file that you dowload ( NOT in the Wiki)

Create a GAE Project and add the JARs and configure OpenBD using the instruction above.

Configure the project for a ColdFusion Wave Robot

Copy the CFML source files to /war so that you have :

/war/_wave/robot/

capabilities.cfm
jsonrpc.cfm

profile.cfm

/war/_wave

index.cfm

/war/images

avatar.png

/war/static

profile.html

json2.js

Edit /war/WEB-INF/web.xml and add the following :

<filter-name>UrlRewriteFilter</filter-name>

<filter-class>org.tuckey.web.filters.urlrewrite.UrlRewriteFilter</filter-class>

</filter>

<filter-mapping>

<filter-name>UrlRewriteFilter</filter-name>

<url-pattern>/*</url-pattern>

</filter-mapping>

<!-- set the amount of seconds the conf file will be checked for reload

can be a valid integer (0 denotes check every time,

empty/not set denotes no reload check) -->

<init-param>

<param-name>confReloadCheckInterval</param-name>

<param-value>0</param-value>

</init-param>

And this is worth doing to :

<!-- Comment out the welcome file list. Not essential, but less confusing with 404 errors

<welcome-file-list>

<welcome-file>index.cfm</welcome-file>

</welcome-file-list>

-->

Turn verbose logging on (.level = ALL) in /war/WEB-INF/logging.properties

HINT : If you don’t do this then you wont see any <cflog > output in your Google App Engine logs

# Set the default logging level for all loggers to WARNING

.level = ALL

Here is what your project layout should look like in Eclipse

Run and Debug your Robot Locally

I have a slightly modified version of the browser-based JSON message POST tool in this article by Pamela Fox. The key to making this work for you is getting your hands on some real JSON message bundles from a wave session.

I created a second Robot sample in Java and implemented just the methods I wanted to get working in my ColdFusion version. I was able to copy the JSON message bundles from the Google App Engine Logs, and then use them locally.

<cflog text="#message#"> is your friend here. Log output displays in real-time locally in the Eclipse console, and quite quickly after a page refresh in the App Engine Logs (but make sure you have enabled verbose logging in the WEB-INF/logging.properties file)

Its ironic that JSON serialization turns out to be a great way to dump messages sent and received to the console. You can dump the contents of a struct (say called properties) like this :

Deploy to Google App Engine and Debug

Once you have a local version running you will want to deploy to App Engine. This can take anywhere from 15 secs to 2 minutes so I don’t do it very often.

One difference between the App Engine and local environments is that the Wave Server encodes a binary version of the HTTP POST JSON content which I had to deal with like this :

<!--- The JSON request is packed differenty depending how its called :

* Google Wave Servers send the request as binary

* Browser testing with XMLHTTPRequest sends request as text

--->

</cfif>

<cflog text="UnpackJSONRequest: #pageContentStr#">

</cffunction>

TIP : Click the radio button Show: All requests to refresh the logs view.

Add your Robot to a New Wave and watch the Google App Engine logs

Update 1 June : My robot is working and does the following.

Responds to a SELF_ADDED message with an introduction (which is a wavelet.appendBlip operation)
Responds to a BLIP_SUBMITTED event (that when you click Done or type Shift-Enter when typing ) event with a greeting (another wavelet.appendBlip operation)

CFML Source Code

Here is the ColdFusion source for the main files :

/war/_wave/robot/

capabilities.cfm

jsonrpc.cfm
profile.cfm

Capabilities.cfm

</cfsilent><cfoutput>

<w:robot xmlns:w="http://wave.google.com/extensions/robots/1.0">

<w:version>#versionHash#</w:version>

<w:protocolversion>0.21</w:protocolversion>

<w:capabilities>

<w:capability name="WAVELET_SELF_ADDED"/>

<w:capability name="BLIP_SUBMITTED"/>

<w:capability name="WAVELET_PARTICIPANTS_CHANGED"/>

</w:capabilities>

</w:robot>

</cfoutput>

Profile.cfm - updated 1 June

<cfsilent><cfcontent type="application/json">

<cfset operationsObj = StructNew()>

<cfset operationsObj['name'] = "cfwaverobot">

<cfset operationsObj['imageUrl'] = "http://cfwavebot.appspot.com/images/avatar.png">

<cfset operationsObj['profileUrl'] = "http://cfwavebot.appspot.com/static/profile.html">

<cfinvoke component="json"

method="encode"

data= "#operationsObj#"

returnvariable="returnJSON" />

</cfsilent><cfoutput>#returnJSON#</cfoutput>

Jsonrpc.cfm - updated 1 June

<!--- JSONRPC.cfm

A Google Wave Robot written in ColdFusion

INPUT :

Message Bundle JSON (Expected as POST)

OUTPUT :

Operations JSON (MIME TYPE: application/json)

Requires :

cfjson CFC

http://www.epiphantastic.com/cfjson/

Find Out More :

http://blog.johnoscott.com/2010/05/building-wave-robots-in-coldfusion-with.html

--->

<!--- <cflog text="#logpage# cgi.script_name= #cgi.script_name#">

<!--- <cflog text="StructCount(Form)=#StructCount(Form)#">

</cfloop> --->

<!--- <cflog text="StructCount(HttpRequestData)=#StructCount(HttpRequestData)#">

</cfloop> --->

<!--- <cfif IsBinary(HttpRequestData.CONTENT)>

</cfif>

<!---

</cfloop>

--->

</cffunction>

<!--- The JSON request is packed differenty depending how its called :

* Google Wave Servers send the request as binary

* Browser testing with XMLHTTPRequest sends request as text

--->

</cfif>

<cflog text="UnpackJSONRequest: #pageContentStr#">

</cffunction>

<cflog text="Event[#i#] = #CustomSerializeJSON(events[i])#">

<cftry>

<cflog text="** ProcessEvents: Event:#i# cfcatch caught error: #cfcatch.Message# *********">

</cfcatch>

</cftry>

</cfloop>

<cflog text="#logpage# NO EVENTS" >

</cfif>

</cffunction>

<!--- An Event looks something like this :

{ "MODIFIEDBY":"johnoscott@googlewave.com",

"PROPERTIES": {"BLIPID":"b+N-PjdA1UO","PARTICIPANTSREMOVED":[],"PARTICIPANTSADDED":["cfwavebot@appspot.com"]},

"TIMESTAMP":1.27519999508E+12,

"TYPE":"WAVELET_PARTICIPANTS_CHANGED"

}

--->

</cfcase>

</cfcase>

</cfcase>

</cfswitch>

</cffunction>

<!--- Sample Event

{

"type": "BLIP_SUBMITTED",

"modifiedBy": "user@example.com",

"timestamp": 1255935016481,

"properties": {

"blipId": "b+ja8F_Hw4J"

}

--->

<cflog text="BlipSubmittedEvent:" >

<cflog text="BlipSubmittedEvent: blipID=#blipID#" >

<cflog text="BlipSubmittedEvent: content = #content#">

</cfif>

</cffunction>

<!--- Event looks something like this :

{ "MODIFIEDBY":"johnoscott@googlewave.com",

"PROPERTIES": {"BLIPID":"b+N-PjdA1UO","PARTICIPANTSREMOVED":[],"PARTICIPANTSADDED":["cfwavebot@appspot.com"]},

"TIMESTAMP":1.27519999508E+12,

"TYPE":"WAVELET_PARTICIPANTS_CHANGED"

}

--->

<cflog text="WaveletSelfAddedEvent:" >

<cfset replyStr = "Howdy, I'm a ColdFusion robot.\n" &

"I only do one thing right now, and that is to reply to " &

"a BLIP_SUBMITTED event if it contains a greeting (hint : starts with hel..). " &

"Find out how i came into being here : http://bit.ly/dhQRQn"

</cffunction>

<!--- Event looks something like this :

{ "MODIFIEDBY":"johnoscott@googlewave.com",

"PROPERTIES": {"BLIPID":"b+N-PjdA1UO","PARTICIPANTSREMOVED":[],"PARTICIPANTSADDED":["cfwavebot@appspot.com"]},

"TIMESTAMP":1.27519999508E+12,

"TYPE":"WAVELET_PARTICIPANTS_CHANGED"

}

--->

<cflog text="WaveletParticipantsChangedEvent:" >

</cffunction>

<!--- RequestMessageBundle to return look like this, an array of operations

[ { "method":"robot.notifyCapabilitiesHash",

"id":"op1",

"params":{"protocolVersion":"0.21","capabilitiesHash":"ffffff79d8b7888c"}

{ "method":"wavelet.appendBlip",

"id":"op2",

"params":{ "waveId":"googlewave.com!w+jH65e-xWW",

"waveletId":"googlewave.com!conv+root",

"blipData":{"annotations":[],

"elements":{},

"blipId":"TBD_googlewave.com!conv+root_1771192937",

"childBlipIds":[],

"contributors":[],

"creator":null,

"content":"\nHi everybody! MY JOB IS TO STOP EVERYONE FROM SHOUTING",

"lastModifiedTime":-1,

"parentBlipId":null,

"version":-1,

"waveId":"googlewave.com!w+jH65e-xWW",

"waveletId":"googlewave.com!conv+root"}

}

]

--->

</cffunction>

<cflog text="QueueOperation: id=#operation.ID# json=#CustomSerializeJSON(operation)#">

</cffunction>

<!--- Example :

{

"method":"undefined",

"id":"op1",

"params":{}

}

--->

</cffunction>

<!--- Example :

"method":"undefined",

"id":"op1",

"params":{"protocolVersion":"0.21","capabilitiesHash":"ffffff79d8b7888c"}

--->

</cffunction>

<cflog text="GetQueuedOperationsJSON: #ArrayLen(OperationsArray)# Operation(s) to return :">

<cflog text="-- Operation: #i# #o.Method# as json:#CustomSerializeJSON(o)#">

</cfloop>

</cffunction>

try {

return RequestMessageBundle.blips[blipId].content;

} catch (Any e) {

return "";

}

</cfscript>

</cffunction>

<!--- Example :

{

"waveId":"googlewave.com!w+jH65e-xWW",

"waveletId":"googlewave.com!conv+root",

}

--->

<!---

Get the Wavelet object from the Message Bundle

It looks something like this

"wavelet": {

"creationTime": 1255934856713,

"creator": "user@example.com",

"lastModifiedTime": 1255935016481,

"participants": [ "user@example.com","user2@example.com" ],

"rootBlipId": "b+ja8F_Hw4J",

"title": "",

"version": 11,

"waveId": "example.com!w+ja8F_Hw4I",

"waveletId": "example.com!conv+root",

"dataDocuments": { }

}

--->

// Create the return structure

<cflog text="GetWavelet: *** ERROR *** Cant get Wavelet">

</cfif>

</cffunction>

<!--- Example :

{

"blipId":"TBD_googlewave.com!conv+root_-1113654217",

"annotations":[],

"elements":{},

"childBlipIds":[],

"contributors":[],

"content":"\nHi everybody, I just got added to the wave.",

"lastModifiedTime":-1,

"version":-1,

"waveId":"googlewave.com!w+N-PjdA1UX",

"waveletId":"googlewave.com!conv+root"

}

--->

blip = StructNew();

blip['blipId'] = "#blipID#"; // Not sure if this should be the same

blip['annotations'] = ArrayNew(1);

blip['elements'] = StructNew();

blip['childBlipIds']= ArrayNew(1);

blip['contributors']= ArrayNew(1);

blip['content'] = Chr(10) & "#reply#"; // Must start with a newline

blip['lastModifiedTime'] = -1;

blip['version'] = -1;

blip['waveId'] = wavelet.waveId;

blip['waveletId']= wavelet.waveletId;

return blip;

</cfscript>

</cffunction>

<!--- Example Operation looks like this

{ "method":"wavelet.appendBlip",

"id":"op2",

"params":{ "waveId":"googlewave.com!w+jH65e-xWW",

"waveletId":"googlewave.com!conv+root",

"blipData":{"annotations":[],

"elements":{},

"blipId":"TBD_googlewave.com!conv+root_1771192937",

"childBlipIds":[],

"contributors":[],

"creator":null,

"content":"\nHi everybody! MY JOB IS TO STOP EVERYONE FROM SHOUTING",

"lastModifiedTime":-1,

"parentBlipId":null,

"version":-1,

"waveId":"googlewave.com!w+jH65e-xWW",

"waveletId":"googlewave.com!conv+root"}

}

} --->

// Start with an empty operation

o = NewOperation();

// Set the method

o['method'] = "wavelet.appendBlip";

// Set the Wave and Wavelet

w = GetWavelet();

o['params']['waveId'] = w.waveID;

o['params']['waveletId'] = w.waveletId;

// Add the Blip with our reply in it

// blipID = event.properties.blipID; // get the blipID from the event

blipID = "TBD_#w.waveletId#_-239145996"; // TBD_googlewave.com!conv+root_-239145996

o['params']['blipData'] = NewBlip(blipID, w, reply);

// Add it to the queue

QueueOperation(o);

</cfscript>

</cffunction>

<cfinvoke component="json"

method="encode"

data= "#o#"

returnvariable="returnJSON" />

</cffunction>

* MAIN PAGE EXECUTION

// --- UNPACK THE MESSAGE BUNDLE ---

RequestMessageBundle = UnpackJSONRequest(); // sets the global RequestMessageBundle

// --- Initialise the Operations Queue ---

InitOperationsQueue();

// --- Handle the Events ---

ProcessEvents();

</cfscript>

<cflog text="GetQueuedOperationsJSON: ">

<cflog text=">>>> We are done ! Returning Operations as JSON: #ResponseOperationsJSON# ">

</cfsilent><cfoutput>#ResponseOperationsJSON#</cfoutput>

<cfcatch type="any"><cfoutput>[{"error":"Please POST me a JSON Message Bundle"}]</cfoutput>

<cflog text="******************** cfcatch caught error: #cfcatch.Message# ********************************">

</cfcatch>

</cftry>

Johno Scott's Blog

Sunday, May 30, 2010

Building Wave Robots in ColdFusion with OpenBD on Google App Engine

2 comments:

Followers

About Me