How to evaluate your sessions

Evaluation should attempt to maximise internal validity and external validity

Internal validity

Internal validity is the extent to which the design allows you to claim that a change has been made to behaviour, and that the change was brought about by your intervention.

External validity

External validity is the extent to which your findings may be generalised to other participants, in other settings, at other times, repeating the benefit of the intervention elsewhere.

When developing your evaluation design, consider:

  • What needs to be improved? (Problem analysis, informed by theory, research, observation, experience)
  • How will I know if I’ve improved it? (How should I measure it?)
  • Have I improved it (or left it unchanged, or made it worse)?
  • Have I inadvertently changed any other parameters, for better or worse? (Unintended consequences)
  • Would this work elsewhere? (And to this end, have I reported clearly what I’ve done so others may repeat the intervention?)

Post intervention evaluation

Evaluation post an intervention could take the form of the following types of questions: ‘Did you enjoy the session?’, ‘Do you think it will make a difference to the way you drive?’. It should be noted that where an intervention is followed by a measurement or observation this tells you whether the participants liked your intervention, but no more.

Pre & post intervention evaluation

A Before – After design for evaluation which measures or observes that which you want to change both before (baseline) and after your intervention tells you whether there has been a change. But it doesn’t tell you whether the change was wholly and solely brought about by your intervention. Especially if, as with a number of road safety indices currently, matters were already on a downward trend.