Technology choices

It's a great moment when after all the hard work you win some funding for your Idea. So, having won the Geovation Challenge we were faced with the, very welcome, problem of "Where do we go from here?" Well, to try and answer that from a technology viewpoint, here's some thoughts:

• You can create a good solution if you can leverage existing technology and experience; as long as you don't fall into the "everything looks like a nail" trap. 
• It's better to be able to produce something quickly - especially if you're doing service design than to go through a big software cycle only to find you've built a "dinosaur".
• It's good if you can "loosely-couple" the solution. If you make it out of blocks, you can replace them if you have scale issues, find a better way, need additional abilities etc without destroying everything.

Now as a Trust, although things like developing on mobile devices is a new area, we do know about data and that's pretty much the place to start: at the back end.

Databases - where does stuff go? 

In the sphere of databases there are a few clear trends: a) SQL system like Oracle or MySQL, b) Non-SQL systems like Hadoop and CouchDb. In fact there is a third: XML native databases. These are a form of no-sql, but are also structured.
Now we use this technology a fair bit at work as Offender information is too loosely structured to hold easily in SQL, but it is nicely modelled in XML which is also pretty flexible about changes. We use the eXist Open Source XML database at work which has a built-in XQuery language at there is no reason not to keep using that: it fits the problem and we have experience. So, the first decision is easy, i.e. we hold each project as a separate piece of XML, like so:

<document use="geovation" version="1.0">
<author>geovation_test</author>
<id>216</id>
<title>Queen St, Sandwell, West Bromwich.
</title>
<created iso="2012-11-21Z">Wed, 21 Nov 2012</created>
<status>in-progress</status>
<position>
<lat>52.519651</lat>
<lng>-1.989832</lng>
<address>Queen St, Sandwell, West Bromwich, West Midlands B70, UK</address></position>
<description>Fly tipping near the canal.</description>
<image>image/15e47c2a-7594-47f7-b96e-b88f1a3f3fd3_big.jpg</image>
<comment>This looks like a great project - let's do it!</comment></document> 

This is the first cut, with fields for position, description, status and somewhere to put images. For efficiency, we've put the image uploaded by the author on the file system, rather than in the database.

We manipulate this data with a language called XQuery. This can not only query the data, but is also a complete language in it's own right. This means we can write an entire application in just XQuery - it saves a lot of time over traditional languages which rely on embedded SQL to talk to the database! As if that isn't enough, we also get thrown in for free XForms via XSLTForms and a nice declarative way to create the web interface for the administrators to work with the projects.

So, we have a database, we can create, save, query projects. Aren't we done?

Search - set the data free!

We have a database and code, but at the end of the day this isn't a closed project; not only our code, but as far as possible our data is Open. What we want from the solution is a system that allows anyone to include the project data in a web-site, mashup, application or whatever. That gives us a few issues:

1. Scale. We don't know how successful this will be.
2. Security. We need to separate the internal project with maybe private text, from whatever data we expose to the outside world. We want to avoid log-ins and api-keys if possible.
3. Geo-search, e.g. "I'm here, what projects are nearby" is a key requirement. This is possible, but isn't really well supported in eXist at the moment - so lets not "treat it like a nail"

We don't really want to construct a whole Search API ourselves, if we can use someone else's to do the job so that leads us to Apache SOLR. SOLR is a very fast search engine built on top of Lucene. It has good scalability, a well written REST api, good geo-search,  and can spit out results in XML, JSON and a few other formats at will. So, what we'll do is this:

1. Projects get created in eXist as XML
2. When they go live, all the fields we want to expose get copied to SOLR - easy.
3. If the project alters, we keep SOLR in step

Now we have a fast search engine with a well-documented API and all the data we wont to expose. No need for extra security, logins etc. 

Pub/Sub Getting the message out.

The last piece of the puzzle we need is how to keep app users up to date with project changes. Admittedly, we could just keep a list on each phone and scan the database occasionally. That's a bit problematic as a solution though and doesn't scale well at all: imagine going to the post office to get a letter and waiting whilst they scoured the place for it. Now imagine a thousand people doing it at the same time. What you need is a pigeon-hole with your letter in it.
That's what pub/sub does. It consists of a Broker, that acts as the post office. The apps on the phone all subscribe to a topic for each project the user has created and any they are following, e.g. /projects/P0121 . When something changes in the project, the server publishes the change to the broker.
The next time the app connects, it gets any messages published for those topics - simple.

We're using a nice light protocol for this called MQTT  though we haven't decided on a broker yet. Having said that, mosquito seems very good!

Where we are now.

So, this is where we are at the moment, an application in three bits:

• a database
• a search engine
• a status messenger

all working together.

This is probably not the last iteration of this system, and once we've built it, we'll probably tweak it quite a bit. But, hopefully, the ideas of loose coupling, quick development and leveraging our strengths will prove themselves in the real world.

Service Design

Community Payback Visibility App – Designing the Service

If you had asked me a year ago what service design was, I would have struggled to guess the answer. In my ignorance, I think I would have been quite sceptical. Probably, arrogantly, I would have said it must be one of those ooshy-mooshy things people talk about in continental cafes.

But I would have been wrong... Well, possibly not about the the cafes, but certainly about the ooshy-mooshiness.

As we pitched our app idea at the GeoVation Challenge, we were introduced to the concept of service design and began to understand the benefits it would bring.

Wikipedia describes service design as “the activity of planning and organizing (sic) people, infrastructure, communication and material components of a service in order to improve its quality and the interaction between service provider and customers.”

In other words, if you are planning to provide a service for people, make it relevant, make it useful, make it responsive,  make it friendly. And don’t think you can do all that without speaking to anyone. You don’t know it all and you will not have thought of everything. Test it, prototype it, talk to users, listen carefully to the feedback. All the time, you will be informing the design.

The principles sound obvious when you think about it, but it’s incredibly easy to find examples of appallingly badly designed services – where the very people expected to use the service are tearing their hair out at how difficult it is to use, how user-unfriendly it is.

I’m sure you can think of at least one occasion when you have had bad service from a company or a public authority. Perhaps the interminable loop of automated phone options that never seems to put you in touch with the people you need to talk to. Maybe the ludicrously inept and inattentive bar person who keeps you waiting for hours to buy a round of drinks.

We don't want our app to redirect you to an auto-moron. Nor do we want it to keep you waiting for hours while it sees to something else. We want our app to be used. We don’t expect people to get beside themselves with excitement, but we don’t want them to be tearing their hair out when they try and use it.

We want the experience to be positive, so people will use it again and will recommend it to friends.

We want our staff to appreciate the design of the internal response bit. We think the actual staff who will be expected to respond to nominations are the best people to design how it works. We don’t pretend to have all the answers. We don’t want to go away, hide and design the backend then dump it on the staff and say, “There you go!” Inevitably, we will have ignored some crucial element that they will point out, by which time it will be too late.

But at the same time, we are not service design experts. We are a data analyst, an IT architect and an operational manager in Community Payback.

David and Sean

So we asked Nonon to help. Sean and David, two service design experts, came on board and agreed to work with us and help us… well, design the service!

After an initial meeting with the project team, Sean and David put together the prototyping bootcamp – a day spent with the actual staff who will be expected to respond to the nominations – working through the principles of service design and how to prototype a service.

We focussed on those elements of the project that we (as a group) thought were the most important to test. We began to put together a blueprint, mapping the user journey, identifying the touchpoints, highlighting the priority areas for prototyping.

When you get service design wrong, or don’t pay attention to it, you take a massive chance. 

You may think you know how everyone would want to engage.

How can you be certain that the customer experience will be positive? They may well get annoyed.

And if they do, you will probably pay the price in terms of continued use or recommendations to friends.

So don't be a sceptic or a know-all. Be smart and involve the users - front and back - in the design, right from the very beginning. Make use of the experts. Or just ignore service design... at your peril!

The GeoVation Challenge

Today we have a really great post from Jason Davies of Staffordshire and West Midlands Probation Trust who won the ‘How can we transform Neighbourhoods in Britain together?’ GeoVation Challenge with the idea for Community Payback Visibility.  Jason’s post is a great way to get a feel for what happens at a GeoVation Camp and the nail-biting climax of the GeoVation Showcase back in June:
Community Payback is unpaid work carried out by offenders on community service and the idea is for the public to nominate sites for Community Payback via an app and track the progress.  The government is currently undertaking a review of the probation service and is encouraging probation trusts to be innovative in responding to fundamental change.

It’s Wednesday afternoon, mid-June and we’re back in Southampton. It’s the final of the GeoVation Challenge. The judges have retired to their chambers. We’ve made our case and it’s out of our hands, but the nerves are jangling now.
This is the culmination of a Staffordshire & West Midlands Probation Trust bid for some GeoVation funding. Ordnance Survey run the GeoVation Challenge with development funding awarded to the best and most innovative ways of combining maps and data to benefit local communities.
We want to address the lack of public awareness in community sentences and feelings of disconnection between the public and authority – a sense of distance from decision-making by developing a mobile phone app to make it easier and more likely for people to nominate sites for Community Payback. We would exploit the rise in smartphones and harness the camera and GPS applications to make it happen. The reward would be far greater visibility of the unpaid work that offenders do to improve their local communities.
Part of our inspiration for the idea came from the Idaho Department of Fish and Game. The IDFG have developed an app that lets drivers report roadkill on Idaho’s roads by taking a picture with their mobile phone and sending the geo-tagged photo in for analysis. Elk and moose lovers, rejoice -this app helps boost survival rates near busy Idaho roads.
We posted the idea on the GeoVation website and waited. 74 other ideas had been submitted, so we were delighted to be shortlisted as one of 20 invited to the GeoVation Camp in May.
We got a small team together to represent the Trust: Mark from IT, Craig from Community Payback and me. We knew we’d have to pitch to the assembled audience and judges, so we got some slides made and put down a few words.
The weekend was challenging, but rewarding and fun. Until Sunday afternoon. On Sunday afternoon it got a bit tense, a bit intense! Sunday afternoon was pecha kucha.
Roughly translated from Japanese as “chit chat”, it’s really anything but. Two minutes. Six slides. 20 seconds per slide. Auto-timed PowerPoint. No room for waffle.

• Slide 1: Intro. 20 seconds to explain to an audience of non-criminal justice people who we are, what we do and what on earth is Community Payback anyway?
• Slide 2: The Problem. We quote from the final report into last summer’s riots. “People want to be involved in improving their areas and more communities should nominate projects for Community Payback… Probation Trusts should publish clearly accessible data on the outcomes of community sentences.”
• Slide 3: The Solution (Part 1). We want to develop a mobile app that members of the public could download for free. The app will let them take a geo-tagged photo of a site they want offenders to work on. OrdnanceSurvey maps on the phone will display the location to the user, who will be able to manually adjust it to give pinpoint accuracy. The app will send the photo automatically to us and we will assess the site for suitability.
• Slide 4: The Solution (Part 2). We will respond to all nominations, even anonymous ones, via link to a unique URL– or webpage – where the nominator will be able to track their request. Suitable projects will be posted on a website with estimated work times, photos of work, clean sites and – most crucially – stories about the offenders’ experiences.
• Slide 5: The Execution. This is the business model bit. We talked about getting an app developer and hosting the devices through cloud servers. We wanted a clean, modern and professional website capable of handling live maps of projects. People could search on the map for projects in their patch, or zoom in on any part of the Trust and click on tags to reveal photos and links to stories.
• Slide 6: Next Steps. We talked about market research and publicity. We talked about our strong partnerships with police and local authorities, other Trusts who would support us and help us spread the word. We started allocating specific amounts of money to each bit.

For us,  the most important part of the app was the opportunity to engage with the public as they followed progress online on the work sites they had suggested. We imagine a map-based tapestry of local stories – stories the public could play a part in, stories about sorting out issues in people’s neighbourhoods, stories about the reintegration and rehabilitation of offenders.
We must have done OK because we got through to the final and now we are back in Southampton. Ten teams are there and there’s a genuine sense of collaboration that has been there since the beginning. Of course everyone wants funding, but there’s no overt sense of competitiveness.
The judge returns and tells us four of the ten ideas will get funding. Three prizes of £25k are announced: Groundwork’s Green Space Mapper, Ideal for All’s Shout Crime app and Sustaination. He hasn’t said our name yet, but there’s one prize left – £40k funding… It’s us!
Now the real work starts. Firstly, we get a prototype app, some cloud server and the backend of the website built. Then some testing and market research. There’s work to do, but we’ve got funding and technical support from Ordnance Survey, the backing of our Chief Executive and the words of the riot report ringing in our ears, so watch out for developments.