Monitoring and Evaluation is broken. Let's really break it.

One of the sad truths that emerged at the Technology Salon on ICTs and M&E was that failure in development is rarely about the project performance, but about winning the next contract. This means that monitoring and evaluation is less about tracking and improving progress towards social change and more about weaving an advertising pitch.

This is not for a lack of frameworks, tools, mapping measurements against a theory of change, or even the need for more real-time data in development. It is about incentives. What is incentivized at the macro level is getting big numbers on the board and nice clean upwardly-trending graph lines. Micro-level incentives for filing reports to fill out the monitoring side of things focus on report filing as a requirement for salary payments or other basic carrot/stick-driven models. Neither of these actually encourage accurate, honest data, yet only with that accurate data can we remotely hope to tweak models and make improvements.

So, let's break monitoring apart from evaluation.

Development using SMS, not SMS4Dev

Cross-posted at

Where the last SMS4D Technology Salon reminded us of the unique gift of mobile technologies to be based where there impact will be, The Cloudy SMS4D Salon really drove home mobile as a multifunctional tool whose true impact is tied more to the usage than the technology itself. While we gathered to discuss SMS4D, we really talked about heath reporting and outreach, education, and community-building through knowledge management and sharing. It just so happened that these health projects were using SMS codes to report longitudinal child health statistics.

Data gathering in health, and even knowing when to gather data, is a huge burden, often relying on community health workers doing the healthcare version of the T&V system of the agricultural extension world. Waiting around for a planned infrastructure is hopeless, but working with the more incremental nature of mobile can improve reporting rates and reduce errors -- "utter chaos works everywhere" being the best quote of this tech salon. Childcount builds on existing SMS reporting to enable community health workers to rapidly register children, note any symptoms or diseases they might have, improve patient tracking (and thereby reducing duplication), and schedule immunizations and outreach. The SMS "encoding" builds off of a simple and familiar paper form, which is handy for training (but less useful than a mango tree, as we'll see). The runner-up quote from this Salon dealt with discussion around the potential risk of intentionally fabricated data -- "humans are awful at falsifying data" -- digitizing and quick, auditable reporting exposes both errors and lies.

RapidResponse Overview from Matt Berg on Vimeo.

Winning the award for innovative ideas in mHealth was the HappyPill project -- instead of boing old SMS, HappyPills uses "flashing" - where you call a number and hang up immediately to "ping" someone. Usually, flashing is just a free way to ask someone to call you back, or you can sometimes work out extensive codes -- one missed call is just saying hi, two is call me back, three means an emergency, etc.. HappyPills takes this basic, essentially binary interaction and applies it to help improve adherence rates for prescription regimens. A medical center can send out flashes to their patients, and the patients are reminded to take their pills and would then flash back to signal that they took their medicine. It's naturally not foolproof, but hugely more cost effective (almost cost-free) in comparison with sending a community health worker out to the patient on a motorcycle to witness their pill-taking.


It turns out that people are not just willing, but economically motivated and excited to use (and pay for) basic SMS-based services to improve their numeracy and literacy skills, improving their ability to communicate cheaply over their phones as well as better navigate market prices. In these low-technology communities, Tostan's Jokko Initiative is creating a curriculum to enable this via SMS, and they have also come up with an amazingly simple methodology to introduce people to menu systems using a mango tree metaphor which gracefully transitions from the concrete (planning a climbing route on a real tree to get to a specific mango) to the semi-concrete (the same, on a diagram of a tree), to the abstract (the tree diagram becomes the menu diagram, the mango a specific function). Anyone who thinks that is basic has never shown their grandparents a new shiny piece of technology, or had their entire worldview of user interface challenged by someone physically pointing a mouse at a screen).

Patatat is an early-stage solution which puts SMS into the role of a community town hall/newsletter/email list. It removes not only the normal geographic barriers that a listserv gets around, but also infrastructure barriers, so (for example) farmers across a region or the world can share knowledge around their crops without relying on the grid and hardwired phones/Internet to do so. This also centralizes costs to one "host" and minimizes it to the community, so a farmer could send one SMS (free to receive, costs to send), and the host would re-broadcast it to the entire "community." With Twitter already showing that it can (technically) report earthquakes faster than the earthquake itself spreads, this rebroadcasting tool also has clear applications in emergency announcements, citizen journalism and a myriad of other fields.

So, was this technology salon about technology, or was it about development projects? Sure, all of the projects discussed at the salon happened to use server and cloud-based SMS technologies. They also probably use paper, transportation, and people. That the technology is now moving from the focus of a project to being a (cool, exciting, powerful, still new-and-shiny) tool in the toolbox is truly heartwarming. It means it is maturing into a cross-sector role and not into another silo (sorry, a "cylinder of excellence" in the parlance of our times).


Put your technology where your solution will be

The Technology Salon on SMS4D covered a lot of ground in a few hours, but the reverberating sentiment was the power of mobile technology at the local/regional level. Part of this is a bit of sour-grapes with the hard challenges of scaling mobile solutions globally, which is as much a problem of cross-provider functionality as it is capacity. The value however is in reminding us that development solutions - while they may be globally replicable - are rooted locally.

The Technology Salon went through an inspiring round of implementations and use cases of on-the-ground efforts using texts in cross-sector development situations. Microfinance solutions, tying the payments to the notifications via mpayment were the purview of CreditSMS, lowering the costs of each loan by dramatically reducing transactional costs, allowing MFI account managers to deal with the exceptions (late/missedpayments) instead of burning time tracking payments to records and managing each interaction. mHealth, a favorite topic of Tech Salons had use cases in using SMS to replace timely and costly travel to report medicine stock levels and local disease trends, but also mobile-centric medical records management and remote, low-cost diagnostics tools, all using SMS:Medic.


Mobiles Vs Computers

Cell Phone in rural India
Are Mobile Phones the Winner?

February's Technology Salon was on the (false) dichotomy of mobiles versus computers in development. Thankfully due to the high caliber of all the attendees, we were able to establish and move quickly past the problem that so often plagues the actual projects and "real world" debates - which is better? Some people will claim mobile phones are better due to their low barriers to entry, but then you see low-cost computing and netbooks providing that same promise to computers. Others will argue that you'll never write a school paper on a cell phone.

The reality is, the entire frame of this argument is off on every possible angle.

First, there are clear cases where one technology is better suited to a task than another. I'd no more write long papers on a cell phone than I would carry around a laptop to use as a personal communications device. However there's a large chunk of tasks where either tool will suffice, and which "should" be used is more a factor of the local conditions than the features of any one technology.

Secondly -- and more importantly -- this discussion is tool-centric. We have a hammer (two, in this case) and are going around the development landscape searching for nails we can drive home, and it's a race between the two hammers to see who can hit the most nails. This is inherently the wrong way to apply ICT in development.

We shouldn't be arguing about mobiles vs computers, or even OLPC XOs vs Intel ClassMates, or Windows vs Linux, we should be arguing about specific problems in development, what tools could help, how, and for what costs (training time, implementation and infrastucture gotchas, as well as equipment costs).