I wrote this piece in 2024 at Automattic, when I moved into employee engagement. I had enjoyed discovering evidence-based practice via my CIPD studies and wrote the following to introduce this to the company.

Introduction: lies, damn lies, and statistics

It’s a fact that higher ice cream consumption correlates with increased rates of drowning.

Common sense tells us that this can’t possibly be correct and the two things can’t be related. But they do correlate: there’s just no causal relationship directly between them. In fact, it’s a third party, the hot weather that causes both statistics to rise: when it’s hot, people buy more ice cream, and they go swimming more. This is often referred to as “correlation is not causation“.

For instance, it’s also a fact that Nicolas Cage’s film career correlates with drowning rates.

Another way statistics can mislead us is when we’re missing an ‘invisible’ factor. Take this graph of rates of left handedness:

The graph seems to suggest a huge epidemic of left-handedness in the first half of the twentieth century, and it would be tempting to take that at face value and start looking for biological or genetic factors of change…

However, left-handedness is a characteristic that has been historically suppressed and considered societally unacceptable, including left-handed people being forced to use their right hand. This is likely to have led to this factor being under-reported until, over time, it gradually became acceptable to admit to being left-handed. So, while the graph may technically represent accurately the data that was collected, we need to critically evaluate the reliability of that data and what other stories it may tell. (This graph has popped up recently on social media as it’s been cleverly used by the LGBTQ+ community to illustrate how social acceptability impacts statistical analysis of reported human behaviour.)

So now that we’ve established that we can’t take evidence at face value, and that it needs a whole lot of thought, context and bias-busting before we use it, let’s take a look at how to embed solid evidence skills into our project design.

What is evidence-based practice?

It’s an established approach to using evidence to inform decisions (not just using evidence to back up something you already think).

Evidence-based practice is in use across multiple disciplines including medicine, education and policing, as well as in management (principally championed via the CEBMa). The CEBMa’s definition is:

“Evidence-based practice … is about making decisions through the conscientious, explicit and judicious use of the best available evidence from multiple sources … to increase the likelihood of a favourable outcome”

— David Sackett

So, we are talking about taking practical actions using evidence we’ve gathered objectively and systematically (not cherry-picking) – evidence where we’ll critique the quality and provenance – we’ll use the best available evidence (rather than seeking proof, which we’re unlikely to find) – and we’ll use multiple sources and types of evidence.

So where do we start?

A simple linear framework

When we’re designing a project and want to do it in an evidence based way, we can start with a linear framework.

Can we start by identifying the problem that we’re setting out to solve with our project?

Even if we think we do know what problem we want to solve, there are going to be unknowns:

• Do we know what is causing the problem?
• Is it a problem that affects everyone equally?
• Does everyone agree it’s a problem? In the same way?
• What’s at risk if we do nothing?
• Who else do we need to involve?
• What are our options for tackling the problem?
• What resource and capacity do we have to tackle it?
• What are the pros and cons of the different possible solutions?
• How will we know if we’ve been successful in solving it?

How we answer those questions is by finding evidence.

The four sources of evidence

• Professional expertise – one’s own knowledge, or perhaps one’s colleagues’ knowledge, gained through training or experience;
• Scientific literature – articles, books, studies, journals, external best practice. Great sources of these are Consensus.app and Google scholar;
• Organisational data – outcomes of past projects, metrics, Looker data, survey results (highly dependent on your area of work)
• Stakeholder views – feedback, conversations with customers and partners

Do you see any challenges in getting hold of any specific type of data in your area of work? Is there one particular type of evidence that your team usually relies on, to the detriment of others? Can you see any opportunities to bring in other types of data?

Remember that you may already be using evidence without even realising it. Something has informed you that there’s a problem your project needs to solve. What type of evidence was that? Can you back that source of evidence up via any of the others? Or perhaps another evidence source may contradict or add nuance. It’s all good either way!

So now that we’ve got a bunch of evidence, what do we do?

The six steps of evidence-based practice

1. Asking: Translating a practical issue or problem into an answerable question Thinking critically, asking good questions that are specific and researchable (aka it’s possible to find the answer); What is the problem? Why will our idea work? What are the success factors? Who is it effective for? What outcomes do our stakeholders need to see? What do we not know that we need to know?
2. Acquiring: Systematically searching and retrieving the evidence Get the research evidence: expertise, literature, data, viewpoints.
3. Appraising: Critically judging the trustworthiness and relevance of the evidence Critical thinking – does this mean what it seems to mean – is it relevant to our context – is it from a reputable source – is there an agenda behind it?
4. Aggregating: Weighing and pulling together the evidence Collecting evidence from different sources. Avoid overload – you can have too much data – synthesise.
5. Applying: Incorporating the evidence in the decision-making process Go from research back to expertise – “how can I apply this in A8C” Diagnose – data review – more research and synthesis to draw up recommendations – ‘road test’ the recommendations.
6. Assessing: Evaluating the outcome of the decision taken Get feedback (more data) Was it successful? Why/not? What did we learn? Design something else / a follow-up.

Avoiding analysis paralysis

How can we avoid getting hopelessly bogged down during this phase?

• Keep moving through the framework. As long as we keep zoomed out on answering the specific questions (the unknowns) in order to define and solve our problem, we can keep focused.
• Remember we’re just seeking ‘best’ evidence, not proof. It’s likely not possible to be 100% correct and to have explored every route, and it’s unlikely a cost/benefit analysis would justify us getting lost in endless research. We’re looking for the best evidence from the common sense point of view. We can adopt a growth mindset – it’s ok to learn, and even fail. If this project goes wrong, we can learn from why. We should aim for mastery through learning, not ‘performing’ a perfect project.
• Keep talking with stakeholders to keep focused and accountable (perhaps remembering the ‘we’, not ‘I’, of community). If third parties know which question we’re aiming to answer right now, and that we’re going to deliver the evidence then move on, that can prevent us from going down evidence rabbit holes.

Problems, questions, more problems, more questions…

It might be clear by now that often when we ask one question, another crops up. Or, when you’re investigating one possible route, you find contradictory evidence and have to go back a step. Or, when you’re exploring possible solutions, you need more evidence, then more unknowns arrive, then more evidence is needed.

What if the problem we thought we were trying to solve, turns out not to be a problem at all but we find a totally different problem along the way?!?

Rather than the simple linear format, we could be looking more at this…

Which can turn into this…

To help us avoid getting stuck, how about a non-linear framework or two.

Listen, Understand, Action (Ask, Aha, Act)

• Ask: Capture the problem and the questions / unknowns
• Aha!: Analyse the evidence to gain understanding of how to make impact
• Act: Take what you have learned, make a decision and take action (which might be more evidence-gathering)
• Ask: Capture more feedback and measure your progress

…and keep going!

The cyclical format is a much more realistic image of the evidence-based process, since every time we answer one question or find one piece of evidence, new questions arise and we need new evidence. We might need to go round several times before we even start to design a solution or action to the problem we’ve defined – and even then, we need to analyse what happened when we took action and plan next steps.

The beauty of the cycle is that it builds in review, and that it’s ok to make mistakes, because we are listening to feedback, understanding, and modifying, to design better and more effectively.

…ahahahaha! …another evidence-question cycle

If Ask Aha Act may be slightly too simple, then the below opens things out in more detail.

Here things are broken down into two clear parts:

• What’s the issue?
  • Ask – answerable questions to identify the issue.
  • Acquire – Gather evidence from multiple sources.
  • Appraise – Rate the trustworthiness and relevance of evidence.
  • Aggregate – the most trustworthy and relevant evidence.
  • Apply – Use the evidence to clarify the issue.
  • Assess – Reflect on the process and decision outcome.
• What can we do about it?
  • Ask – questions to identify intervention options to deal with the issue, and their costs/benefits.
  • Acquire – Gather evidence to answer these questions.
  • Appraise – Rate the trustworthiness and relevance of this evidence.
  • Aggregate – the most relevant evidence.
  • Apply – Determine which interventions are most likely to address the issue cost-effectively.
  • Assess – Reflect on the process and outcome.

The overarching step after completing both parts is to “IMPLEMENT INTERVENTION AND PLAN EVALUATION.” Also known as… doing the thing!

Now that we’ve gone through this process, the intervention we’re making has all the chances we can give it of having a favourable outcome.

It’s like planting a seed. If we lack information (we watered it correctly, but we used the wrong soil; or we took a guess on what temperature to keep it) the plant may not grow, or not grow very well.

If we’ve taken care to first find out, then apply, the right lighting, temperature, water and soil conditions to give it the best possible chance to grow; all based on our research.. (and sometimes, there may be a failed plant or two along the way!); then we’ve given the plant every chance of growing well and healthily.

How to embed this in your team?

In our case, [at Automattic], we’d started out with a hypothesis: that x area of work was a problem area and colleagues wanted and needed us to work on a project in that area. Using the framework helped Spark team to completely change the area of work the project would tackle, once we’d interrogated our original hypothesis, and found the evidence didn’t back it up. We realised, using the four types of evidence, that we could identify a much more useful project area, with plenty of data pointing to its potential impact and clear ways to measure its success. Using a whiteboard ideation, we were then able to move quite fast to ask and answer questions to narrow down possible actions and interventions. We reached the point where we were able to assign evidence-gathering tasks to team members, all in one short workshop.