What is it about indicators that makes people wince when you mention the term and opt for root canal treatment in preference? Why is measurement so unpopular? Is it psychological because they are so binary, you either achieve or fail? Is it because they box you in and pin you down? Indicators are beloved of technocrats because of the way that they can align practice with policy but they are loathed by practitioners. Recent conversation with the delivery pilot networks have informed this blog. But any opinions are mine alone.
Anyone working with urban policies, especially in a European setting is familiar with words like output, outcome, impact and result. Although we are familiar with them we might be challenged to actually define them. The European Commission’s General Direction (DG) Regio in its never ending quest to make our life easier has suggested that we abandon half these terms and focus on just two: output and result.
We recently completed a study for DG Regio (and urban) called the European Union (EU) 50 urban cases (you can google this if I am boring you). The final report was co-authored with Laura Colini and several well-known Urbact experts did case studies including Sally Kneeshaw, Ivan Tosics, Paul Soto, Fernando Barreiro. We concluded that even cities that had delivered recognised good practices while most knew what they had directly produced with their projects in terms of outputs, few or none knew what they had achieved in terms of results.
The same is more or less true of URBACT cities (several of which were in the Urban 50). We know how to count outputs whether these are square metres of workspace, businesses advised, or unemployed people helped into work. These are tangible outputs of projects. We are less comfortable in framing the objectives that we are trying to achieve and specifying what result would constitute success.
Interference
Cities in URBACT networks may be producing tangible outputs in their day job, but in their URBACT network their main product is a local action plan. Specifying what the result of such an action plan is less concrete than with a conventional project. As Tako Popma (external evaluator for the transfer and delivery pilots) emphasised in a recent meeting of the pilots in London – we need to be measuring the interference effect of URBACT. Interference can be translated as URBACT added value.
We know this when we see it. It is the Science or Technology Park which instead of being half full of non-science based businesses a year after the mayor cut the tape is nearly full of relevant companies with a high knowledge content. This is because the URBACT action plan insisted on an integrated approach where the main investment in bricks and mortar was complemented by the support for a triple helix between the city, the knowledge based universities and research centres and the SMEs themselves. Training had been ramped up, teaching company schemes which placed a graduate in key businesses were started and venture capital had been attracted to the city.
Visceral Reactions
All of this sounds fine but it does not get us past the visceral reaction of even experienced URBACT experts and cities when the word indicator is mentioned. It is not clear whether this is more an allergic reaction, the sort of face you make when sucking on a lemon or the equivalent of food poisoning. Suffice to say, that most would obviously prefer to leave the room than discuss it. Though paradoxically once they start it is hard to stop them.
The explanation for this is that most people’s experience of indicators is extrinsic. They are externally imposed requirements of funding programmes. Like all externally imposed cultures it is resented and resisted. Just as any oppressed people knows more about their colonisers than their colonisers know about them so with indicators. We have seen how they can be abused and falsified because we were the fakers. Only the high priests of evaluation units believe that just a tad more specificity here and some improved measurement there can fix the problem. Instead the currency of measurement has been devalued as if gold had been turned into a baser metal. We need a reset whereby indicators are owned by their users and co-produced with others (in the case of URBACT with the lead expert).
It is hard to turn round this culture. But we need to do so. Not because the European Commission has asked us to. But because we need to have a better sense of the result we are trying to achieve.
Losing Weight
Let us take an everyday example of indicators. For a combination of reasons many people in Europe think that they are too fat. Statistics support this view and the growth of childhood obesity is particularly worrying. Recent evidence suggests that the problem is more caused by what people eat (too much sugar and maybe fats) and not so much caused by lack of exercise. Children eat far more sugar than they used to while having about as much exercise. When you ask someone what they want to achieve (in relation to body form) they normally slip straight from the results ‘I want my body to look lovely’ or ‘I want to look healthy’ to the indicator ‘I want to lose 5 kilos’. Anyone who has tried to lose weight will know that getting fitter often adds to the kilos (muscle weighs more than fat) and that there is no machine apart from a looking glass with magical powers that will tell you how lovely you look. In some young and not so young the inputs, outputs result and the perception of the result become confused and eating disorders take hold.
The food industry naturally wants us to go on believing that it is lack of exercise and not their processed food that are the cause. Meanwhile the body image industry Photoshop images of young women (and increasingly men), advocates for size zero and pillories any model, singer or other celeb who dares to step out of line by eating and looking normal. The latest Facebook storm over photos of women eating on tubes and metros is a good example of how we have internalised these messages.
We confuse the output with the results. Over-determining the indicator can lead to some unintended consequences. We are not getting any thinner or healthier as a result.
By Peter Ramsden, URBACT Thematic Pole Manager. Find him on Twitter!
well, what a fresh and crystal clear reflection on indicators this is ! it opens up the window to the immense ocean of assessing serendipity and its wagons of clouds full of evaluation methodologies; because the real thing behind indicators of course is the good performance of public money, meaning the less we spent, the better ! isn’t is so?
An excellent approach on the “indicators” issue, so conflictive, so badly faced by practioners of urban development as Peter points. Finally, indicators should be the tool that ensures our own responsdbility. It permits to know if public policies are achieving their predicted goals or nots. It deals, so, with decision making in the public sector. In the private sector, there’s no problem: indicators are markets.
An great approach on the “indicators” issue, so conflictive, so badly faced by practioners of urban development as Peter points. Finally, indicators should be the tool that ensures our own responsibility