I have been thinking a great deal lately about the difference between data and evidence, particularly when we use either of those words to talk about change. We collect an enormous amount of data in education and development work. We count participants, workshops, courses, resources, institutions, countries, downloads and completions. We can tell you how many people were trained, how many were reached and, apparently, how many were “sensitised”. I confess that “people sensitised” may be one of the silliest things I have ever read in a results report. What exactly is a sensitised person? Someone encountered an idea. Fine. What did they understand, what did they do with it and, most importantly, what changed?
Somewhere between collecting data and reporting results, we have developed a tendency to treat the existence of a number as evidence that change occurred. It is not. If 500 teachers participated in training on the use of open educational resources, I have useful data. I know an intervention took place and, assuming the records are reliable, that 500 teachers participated. What I do not know is whether they learnt anything, used what they learnt, changed their teaching practice or ultimately improved learners’ access to appropriate resources. The number 500 cannot answer all those questions simply because it happens to be available.
This is the distinction between data and evidence. Data tells us something that was recorded or observed. It becomes evidence when we use it to answer a particular question or support a particular claim. An attendance register can be excellent evidence that people attended a workshop. It is not evidence that their behaviour changed afterwards. A list of participating institutions demonstrates programme reach, but not necessarily institutional change. A policy document demonstrates that a policy exists, but not that it is being implemented. So perhaps the starting question should not be, “What data do we have?” but rather, “What are we claiming changed?” Only then can we ask what evidence would reasonably allow us to make that claim.
This has also made me look differently at the familiar results chain of inputs, activities, outputs, outcomes and impact. We tend to represent it as a tidy progression of boxes and arrows, but I find it more useful to think of it as a series of increasingly ambitious claims. We provided the resources. We delivered the activity. People participated. They learnt something. They changed what they did. Their organisation changed how it operated. Those changes lasted and perhaps eventually contributed to something changing at the level of a sector or system. Each statement takes us further away from what the intervention directly did and towards what happened as a consequence, and each requires different evidence.
This is where results chain reach becomes useful: how far along that chain can the available evidence legitimately take us? A programme may have excellent evidence at activity and output level. It can account for every workshop, resource, participant and institution involved. But perhaps nobody followed participants afterwards. Perhaps no one investigated whether institutions changed their practices. Perhaps there was no baseline, or the indicators were designed to count participation rather than capture changes in behaviour or capability. The theory of change may continue all the way to sustainable institutional or systemic transformation while the evidence stops somewhere around the attendance sheet.
That does not mean the intervention failed. It means we cannot tell, and evaluation needs to be comfortable saying that. “We cannot tell” is not the same as “nothing happened”. It is a finding about the limits of the evidence. If an organisation consistently makes claims about long-term institutional or systemic change while consistently collecting information only about activities and outputs, there is a mismatch between what it says it is trying to change and what its evidence systems are capable of demonstrating.
This matters particularly when programmes operate at different levels. Evidence that an individual participated in training does not demonstrate that an institution changed. Evidence that an institution adopted a new practice does not automatically demonstrate systemic change. Even a national policy may demonstrate formal adoption without telling us much about implementation. Evidence does not automatically travel between levels simply because those levels appear together in a theory of change. We have to investigate change at the level at which we claim it occurred.
The further we travel along the results chain, the messier the story also becomes. Individuals, institutions and systems are influenced by many things beyond a single intervention. Training, leadership, funding, policy, technology, colleagues, political decisions and other development partners can all contribute to what eventually happens. At that point, “we caused this” becomes increasingly difficult to defend. The more useful question is often: What did we contribute to this change, and how do we know? That is why evidence of change is rarely one number. It is usually assembled from different sources that allow us to test whether the story we are telling is credible.
I also think we underestimate what missing evidence can tell us. If a programme claims sustainability but monitoring ends when the funding does, that is useful information. If it repeatedly reports how many people were reached but cannot tell us what happened afterwards, that tells us something too. The absence of evidence may reveal a gap between the change an organisation wants to create and what its monitoring and evaluation systems were designed to see.
None of this makes output data unimportant. We need to know what was done, where, with whom and at what scale. The problem begins when we ask those data to carry claims they were never designed to support. A workshop is an activity. Attendance demonstrates reach. Learning is a result. Changed behaviour is another. Institutionalisation is another. Sustained systemic change is something else again. They may belong to the same story, but they are not interchangeable simply because we can draw arrows between them.
Perhaps, then, the better question is not simply, “What results do the data show?” but “What are we claiming changed, what evidence supports that claim, and how far along the results chain does our evidence actually reach?”
There will almost always be data. We are very good at producing it. The harder task is establishing whether it constitutes evidence of change. Because if all we can tell you at the end is how many people were “sensitised”, we may know considerably less than the number suggests.