The Tortoise

 

I'm keeping the animal theme going after @tingenek's last post about riding the camel, with a reference to Aesop's fable about slow and steady winning the race.

It's a good year since we got funding thanks to Geovation and we are finally on the verge of launching the first app for android and accompanying website.

Barring any last-minute hiccups, the "Get Community Payback" app will be available to download from Google Play for android devices from October 1st 2013.

Users will be able to take photos of fly-tipping or other ugly sights, tag the location on a map and submit the information to their local Probation Trust. We'll forward nominations from other areas to the Probation Trust concerned.

In Staffordshire & West Midlands, staff can assess and review projects on a website we've developed. As projects are updated, automatic notifications go back to the person who suggested the project.

Other Probation Trusts are free to use our website to track projects in their area. The code is open source anyway so they can adopt it, adapt it or completely ignore it! Luckily, most seem interested.

People can keep track of projects on the public-facing website. There they can search a map and click on pins to reveal photos and project descriptions.

But while we've been treading carefully, learning and developing our skills, we have been overtaken by a company offering a rival service!

Paradex released a Community Payback project reporting app a few months ago, offering to e-mail project nominations to Probation Trusts for a monthly charge. They didn't ask if we were interested at Staffordshire & West Midlands, nor did they ask permission to add our video to their website!

That website ends with "org", despite Paradex looking to make a profit, and they use "official" Ministry of Justice and Community Payback logos and colour scheme. If you look very closely, you will see the small print admitting Paradex is "independent of Probation and the Ministry of Justice".

I don't have any beef with small software companies trying to identify opportunities to make a few quid, but asking cash-strapped public services for a fee so you can send them the odd e-mail?

In the end, if there is demand for the service and two apps are available, then people will use the best one and only they will be the judge of that.

We are hopeful that our methodical approach to service design and user experience will ultimately pay off as the finish line comes into view.

Keep an eye on @swmcpvisibility on Twitter for more updates in the next few weeks.

Riding the Camel

When I originally thought about the technical design for the CP Visibility application, I had in mind a series of blocks that were loosely coupled together for maximum flexibility;

• An xml database (eXist)
• A public search engine (SOLR)
• A publication mechanism to push updates to the phones (MQTT)

However, at that time I hadn't considered exactly how that was going to happen - I'd assumed an api or library or REST game of some sort would present itself (optimism is an integral part of professional IT).

Move on a few months and it's clear that although there are well documented ways to do this in Java: the Paho client library for MQTT and the SOLRJ library for SOLR, each library introduces it's own complexity layer:

• The library needs to be understood. 
• Extra java code needs to be written and tested,
• It has to work in eXist as a module.

Hold on though, each new use of the data, like a twitter or  a websocket feed would, again, require it's own mini 'solution' - created, from scratch, each time. That doesn't sound like a good idea.

Enter Apache Camel, which I'd used in another project at work in the meantime and seemed ideal for this task. Basically Camel knows both how to connect to a load of technologies and has a ruleset that tells it what to route to where and when. It's a 'mediation engine'.
[Have a look at their web site, it does a much better job of describing what Camel does.]


Now for this project the important aspects of Camel are:

• It can be deployed as a servlet, so slots right in to existing stack.
• It understands both MQTT and SOLR.
• It has XQuery/XPath etc built in so it can understand our project files.

What it doesn't have is any understanding of eXist. However, that turns out not to be insurmountable. eXist has a 'file' module that lets us write out an xml project file to a folder on the server. It's not pretty, but it also solves the problem of queuing up updates. Sometimes old-school is the way to go.

We start then by writing an xml file to a folder on the server called camel-in each time there is a new or updated project. Next comes the Camel magic.

Camel consists of <routes> which we can configure in a simple xml file on the server.
First part of the route, the <from> tag picks up any files that appear in camel-in and deletes them after successful processing; the xml content becoming the 'body' or payload for the rest of the route:

<route>

    <from uri="file:/opt/tomcat/temp/camel-in?delete=true"/>

    <setHeader headerName="myid">
        <xpath>/document/id/text()</xpath>
    </setHeader>

    <multicast stopOnException="true">

<to uri="direct:solr"/>
        <to uri="direct:mqtt"/>
    </multicast>

</route>  

Next, we set up a header variable with the id of the project as we need it for mqtt later. Notice <xpath> is built in, so we can read it out directly.  Lastly, we 'multicast' to other routes for mqtt and solr, a bit like calling a sub-routine.
Multicasting is worth explaining; in Camel, a normal route with a couple of steps acts like a pipeline, pouring the body from the output of one into the input of the next. Usually, this is fine, unless you want to alter the body within the route. If you do that, the altered body gets poured into the next step, not the original. We need very different body data for mqtt (text) and solr (xml), so we have to <multicast>. This makes sure a separate copy of the body is sent to each route. First up is SOLR:

<route>
     <from uri="direct:solr"/>

     <to uri="xslt:file:/opt/tomcat/temp/camel-xsl
     /proj2solr.xsl"/>

     <log message="SOLR Update"/>

     <convertBodyTo type="java.lang.String"/>

     <setHeader headerName="SolrOperation">
        <constant>INSERT</constant>
     </setHeader>

     <to uri="solr://localhost/solr/cpsv0"/>
</route>

This route takes a project and uses xslt to put together the right xml format for a solr insert:

<add>
   <doc>
      <field name="id">538</field>
       more fields......
      <field name="location">51.6812,-2.23541</field>
   </doc>
</add>

Next, the body gets converted to a string (rather than xml) and sent to the solr end-point with the appropriate instructions i.e insert the data.

Now, this might seem a little complex, but look at the advantages. No-one needs to know SOLRJ, no code is written that we have to maintain and it's dead easy to alter. Next up is MQTT:

<route>
    <from uri="direct:mqtt"/>

    <to uri="xslt:file:/opt/tomcat/temp/camel-xsl
    /proj2mqtt.xsl"/>

    <log message="MQTT Update"/>

        <recipientList ignoreInvalidEndpoints="false" >

        <simple>
            mqtt:camel?host=tcp://localhost:1883
            &amp;publishTopicName=projects/${header.myid}
        </simple>

        </recipientList>
</route>

This route is a little more complex, but not by much. For the mqtt side we're publishing a message to the /projects/N topic, where N is the project number. The message content is created using the same approach that we used before, since xslt will also output plain text.
Curiously, one thing Camel doesn't do easily is allow you to just drop a variable into a url. Instead, you make up a <recepientList> which allows for route end-point strings to be calculated at run-time . The <simple> tag is the language we're using to make up that string. It could just as easily be <javascript> or <xpath> etc, there's a few you can use.

That's it, in a few lines of xml the updates we need are done. Camel is a very handy bit of kit, we can adapt as the project progresses without having to invest in apis or worry about some bit of code working with another. Best of all, it maintains the loose coupling and flexibility we need.

Technology choices

It's a great moment when after all the hard work you win some funding for your Idea. So, having won the Geovation Challenge we were faced with the, very welcome, problem of "Where do we go from here?" Well, to try and answer that from a technology viewpoint, here's some thoughts:

• You can create a good solution if you can leverage existing technology and experience; as long as you don't fall into the "everything looks like a nail" trap. 
• It's better to be able to produce something quickly - especially if you're doing service design than to go through a big software cycle only to find you've built a "dinosaur".
• It's good if you can "loosely-couple" the solution. If you make it out of blocks, you can replace them if you have scale issues, find a better way, need additional abilities etc without destroying everything.

Now as a Trust, although things like developing on mobile devices is a new area, we do know about data and that's pretty much the place to start: at the back end.

Databases - where does stuff go? 

In the sphere of databases there are a few clear trends: a) SQL system like Oracle or MySQL, b) Non-SQL systems like Hadoop and CouchDb. In fact there is a third: XML native databases. These are a form of no-sql, but are also structured.
Now we use this technology a fair bit at work as Offender information is too loosely structured to hold easily in SQL, but it is nicely modelled in XML which is also pretty flexible about changes. We use the eXist Open Source XML database at work which has a built-in XQuery language at there is no reason not to keep using that: it fits the problem and we have experience. So, the first decision is easy, i.e. we hold each project as a separate piece of XML, like so:

<document use="geovation" version="1.0">
<author>geovation_test</author>
<id>216</id>
<title>Queen St, Sandwell, West Bromwich.
</title>
<created iso="2012-11-21Z">Wed, 21 Nov 2012</created>
<status>in-progress</status>
<position>
<lat>52.519651</lat>
<lng>-1.989832</lng>
<address>Queen St, Sandwell, West Bromwich, West Midlands B70, UK</address></position>
<description>Fly tipping near the canal.</description>
<image>image/15e47c2a-7594-47f7-b96e-b88f1a3f3fd3_big.jpg</image>
<comment>This looks like a great project - let's do it!</comment></document> 

This is the first cut, with fields for position, description, status and somewhere to put images. For efficiency, we've put the image uploaded by the author on the file system, rather than in the database.

We manipulate this data with a language called XQuery. This can not only query the data, but is also a complete language in it's own right. This means we can write an entire application in just XQuery - it saves a lot of time over traditional languages which rely on embedded SQL to talk to the database! As if that isn't enough, we also get thrown in for free XForms via XSLTForms and a nice declarative way to create the web interface for the administrators to work with the projects.

So, we have a database, we can create, save, query projects. Aren't we done?

Search - set the data free!

We have a database and code, but at the end of the day this isn't a closed project; not only our code, but as far as possible our data is Open. What we want from the solution is a system that allows anyone to include the project data in a web-site, mashup, application or whatever. That gives us a few issues:

1. Scale. We don't know how successful this will be.
2. Security. We need to separate the internal project with maybe private text, from whatever data we expose to the outside world. We want to avoid log-ins and api-keys if possible.
3. Geo-search, e.g. "I'm here, what projects are nearby" is a key requirement. This is possible, but isn't really well supported in eXist at the moment - so lets not "treat it like a nail"

We don't really want to construct a whole Search API ourselves, if we can use someone else's to do the job so that leads us to Apache SOLR. SOLR is a very fast search engine built on top of Lucene. It has good scalability, a well written REST api, good geo-search,  and can spit out results in XML, JSON and a few other formats at will. So, what we'll do is this:

1. Projects get created in eXist as XML
2. When they go live, all the fields we want to expose get copied to SOLR - easy.
3. If the project alters, we keep SOLR in step

Now we have a fast search engine with a well-documented API and all the data we wont to expose. No need for extra security, logins etc. 

Pub/Sub Getting the message out.

The last piece of the puzzle we need is how to keep app users up to date with project changes. Admittedly, we could just keep a list on each phone and scan the database occasionally. That's a bit problematic as a solution though and doesn't scale well at all: imagine going to the post office to get a letter and waiting whilst they scoured the place for it. Now imagine a thousand people doing it at the same time. What you need is a pigeon-hole with your letter in it.
That's what pub/sub does. It consists of a Broker, that acts as the post office. The apps on the phone all subscribe to a topic for each project the user has created and any they are following, e.g. /projects/P0121 . When something changes in the project, the server publishes the change to the broker.
The next time the app connects, it gets any messages published for those topics - simple.

We're using a nice light protocol for this called MQTT  though we haven't decided on a broker yet. Having said that, mosquito seems very good!

Where we are now.

So, this is where we are at the moment, an application in three bits:

• a database
• a search engine
• a status messenger

all working together.

This is probably not the last iteration of this system, and once we've built it, we'll probably tweak it quite a bit. But, hopefully, the ideas of loose coupling, quick development and leveraging our strengths will prove themselves in the real world.

Service Design

Community Payback Visibility App – Designing the Service

If you had asked me a year ago what service design was, I would have struggled to guess the answer. In my ignorance, I think I would have been quite sceptical. Probably, arrogantly, I would have said it must be one of those ooshy-mooshy things people talk about in continental cafes.

But I would have been wrong... Well, possibly not about the the cafes, but certainly about the ooshy-mooshiness.

As we pitched our app idea at the GeoVation Challenge, we were introduced to the concept of service design and began to understand the benefits it would bring.

Wikipedia describes service design as “the activity of planning and organizing (sic) people, infrastructure, communication and material components of a service in order to improve its quality and the interaction between service provider and customers.”

In other words, if you are planning to provide a service for people, make it relevant, make it useful, make it responsive,  make it friendly. And don’t think you can do all that without speaking to anyone. You don’t know it all and you will not have thought of everything. Test it, prototype it, talk to users, listen carefully to the feedback. All the time, you will be informing the design.

The principles sound obvious when you think about it, but it’s incredibly easy to find examples of appallingly badly designed services – where the very people expected to use the service are tearing their hair out at how difficult it is to use, how user-unfriendly it is.

I’m sure you can think of at least one occasion when you have had bad service from a company or a public authority. Perhaps the interminable loop of automated phone options that never seems to put you in touch with the people you need to talk to. Maybe the ludicrously inept and inattentive bar person who keeps you waiting for hours to buy a round of drinks.

We don't want our app to redirect you to an auto-moron. Nor do we want it to keep you waiting for hours while it sees to something else. We want our app to be used. We don’t expect people to get beside themselves with excitement, but we don’t want them to be tearing their hair out when they try and use it.

We want the experience to be positive, so people will use it again and will recommend it to friends.

We want our staff to appreciate the design of the internal response bit. We think the actual staff who will be expected to respond to nominations are the best people to design how it works. We don’t pretend to have all the answers. We don’t want to go away, hide and design the backend then dump it on the staff and say, “There you go!” Inevitably, we will have ignored some crucial element that they will point out, by which time it will be too late.

But at the same time, we are not service design experts. We are a data analyst, an IT architect and an operational manager in Community Payback.

David and Sean

So we asked Nonon to help. Sean and David, two service design experts, came on board and agreed to work with us and help us… well, design the service!

After an initial meeting with the project team, Sean and David put together the prototyping bootcamp – a day spent with the actual staff who will be expected to respond to the nominations – working through the principles of service design and how to prototype a service.

We focussed on those elements of the project that we (as a group) thought were the most important to test. We began to put together a blueprint, mapping the user journey, identifying the touchpoints, highlighting the priority areas for prototyping.

When you get service design wrong, or don’t pay attention to it, you take a massive chance. 

You may think you know how everyone would want to engage.

How can you be certain that the customer experience will be positive? They may well get annoyed.

And if they do, you will probably pay the price in terms of continued use or recommendations to friends.

So don't be a sceptic or a know-all. Be smart and involve the users - front and back - in the design, right from the very beginning. Make use of the experts. Or just ignore service design... at your peril!

The GeoVation Challenge

Today we have a really great post from Jason Davies of Staffordshire and West Midlands Probation Trust who won the ‘How can we transform Neighbourhoods in Britain together?’ GeoVation Challenge with the idea for Community Payback Visibility.  Jason’s post is a great way to get a feel for what happens at a GeoVation Camp and the nail-biting climax of the GeoVation Showcase back in June:
Community Payback is unpaid work carried out by offenders on community service and the idea is for the public to nominate sites for Community Payback via an app and track the progress.  The government is currently undertaking a review of the probation service and is encouraging probation trusts to be innovative in responding to fundamental change.

It’s Wednesday afternoon, mid-June and we’re back in Southampton. It’s the final of the GeoVation Challenge. The judges have retired to their chambers. We’ve made our case and it’s out of our hands, but the nerves are jangling now.
This is the culmination of a Staffordshire & West Midlands Probation Trust bid for some GeoVation funding. Ordnance Survey run the GeoVation Challenge with development funding awarded to the best and most innovative ways of combining maps and data to benefit local communities.
We want to address the lack of public awareness in community sentences and feelings of disconnection between the public and authority – a sense of distance from decision-making by developing a mobile phone app to make it easier and more likely for people to nominate sites for Community Payback. We would exploit the rise in smartphones and harness the camera and GPS applications to make it happen. The reward would be far greater visibility of the unpaid work that offenders do to improve their local communities.
Part of our inspiration for the idea came from the Idaho Department of Fish and Game. The IDFG have developed an app that lets drivers report roadkill on Idaho’s roads by taking a picture with their mobile phone and sending the geo-tagged photo in for analysis. Elk and moose lovers, rejoice -this app helps boost survival rates near busy Idaho roads.
We posted the idea on the GeoVation website and waited. 74 other ideas had been submitted, so we were delighted to be shortlisted as one of 20 invited to the GeoVation Camp in May.
We got a small team together to represent the Trust: Mark from IT, Craig from Community Payback and me. We knew we’d have to pitch to the assembled audience and judges, so we got some slides made and put down a few words.
The weekend was challenging, but rewarding and fun. Until Sunday afternoon. On Sunday afternoon it got a bit tense, a bit intense! Sunday afternoon was pecha kucha.
Roughly translated from Japanese as “chit chat”, it’s really anything but. Two minutes. Six slides. 20 seconds per slide. Auto-timed PowerPoint. No room for waffle.

• Slide 1: Intro. 20 seconds to explain to an audience of non-criminal justice people who we are, what we do and what on earth is Community Payback anyway?
• Slide 2: The Problem. We quote from the final report into last summer’s riots. “People want to be involved in improving their areas and more communities should nominate projects for Community Payback… Probation Trusts should publish clearly accessible data on the outcomes of community sentences.”
• Slide 3: The Solution (Part 1). We want to develop a mobile app that members of the public could download for free. The app will let them take a geo-tagged photo of a site they want offenders to work on. OrdnanceSurvey maps on the phone will display the location to the user, who will be able to manually adjust it to give pinpoint accuracy. The app will send the photo automatically to us and we will assess the site for suitability.
• Slide 4: The Solution (Part 2). We will respond to all nominations, even anonymous ones, via link to a unique URL– or webpage – where the nominator will be able to track their request. Suitable projects will be posted on a website with estimated work times, photos of work, clean sites and – most crucially – stories about the offenders’ experiences.
• Slide 5: The Execution. This is the business model bit. We talked about getting an app developer and hosting the devices through cloud servers. We wanted a clean, modern and professional website capable of handling live maps of projects. People could search on the map for projects in their patch, or zoom in on any part of the Trust and click on tags to reveal photos and links to stories.
• Slide 6: Next Steps. We talked about market research and publicity. We talked about our strong partnerships with police and local authorities, other Trusts who would support us and help us spread the word. We started allocating specific amounts of money to each bit.

For us,  the most important part of the app was the opportunity to engage with the public as they followed progress online on the work sites they had suggested. We imagine a map-based tapestry of local stories – stories the public could play a part in, stories about sorting out issues in people’s neighbourhoods, stories about the reintegration and rehabilitation of offenders.
We must have done OK because we got through to the final and now we are back in Southampton. Ten teams are there and there’s a genuine sense of collaboration that has been there since the beginning. Of course everyone wants funding, but there’s no overt sense of competitiveness.
The judge returns and tells us four of the ten ideas will get funding. Three prizes of £25k are announced: Groundwork’s Green Space Mapper, Ideal for All’s Shout Crime app and Sustaination. He hasn’t said our name yet, but there’s one prize left – £40k funding… It’s us!
Now the real work starts. Firstly, we get a prototype app, some cloud server and the backend of the website built. Then some testing and market research. There’s work to do, but we’ve got funding and technical support from Ordnance Survey, the backing of our Chief Executive and the words of the riot report ringing in our ears, so watch out for developments.