Evaluating with the Success Case Method

In my last post I mentioned that I prefer the Success Case Method for evaluating learning (and other) interventions to the Kirkpatrick approach. A few readers contacted me asking for information on the method and why I prefer it. Here’s a bit of both.

About the Success Case Method

The method was developed by Robert Brinkerhoff as an alternative (or supplement) to the Kirkpatrick approach and its derivatives. It is very simple and fast (which is part of it’s appeal) and goes something like this:

Step 1. Identify targeted business goals and impact expectations

Step 2. Survey a large representative sample of all participants in a program to identify high impact and low impact cases

Step 3. Analyze the survey data to identify:

  • a small group of successful participants
  • a small group unsuccessful participants

Step 4. Conduct in-depth interviews with the two selected groups to:

  • document the nature and business value of their application of learning
  • identify the performance factors that supported learning application and obstacles that prevented it.

Step 5. Document and disseminate the story

  • report impact
  • applaud successes
  • use data to educate managers and organization

The process produces two key outputs

  • In-depth stories of documented business effect that can be disseminated to a variety of audiences
  • Knowledge of factors that enhance or impede the effect of training on business results. Factors that are associated with successful application of new skills are compared and contrasted with those that impede training.

It answers practical and common questions we have about training and other initiatives:

  • What is really happening? Who’s using what, and how well? Who’s not using things as planned? What’s getting used, and what isn’t? Which people and how many are having success? Which people and how many are not?
  • What results are being achieved? What value, if any, is being realized? What goals are being met? What goals are not? Is the intervention delivering the promised and hoped for results? What unintended results are happening?
  • What is the value of the results? What sort of dollar or other value can be placed on the results? Does the program appear to be worthwhile? Is it producing results worth more than its costs? What is its return on investment? How much more value could it produce if it were working better?
  • How can it be improved? What’s helping? What’s getting in the way? What could be done to get more people to use it? How can everyone be more like those few who are most successful?

Here’s a good Brinkerhoff article from a 2005 issue of Advances in Developing Human Resources on the method. The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training

There are some important differences between Kirkpatrick Based Methods and the Success Case Method. The following table developed by Brinkerhoff differentiates the two approaches.

Why I like it

Here are five reasons:

1. Where Kirkpatrick (and Philips and others) focus on gathering proof of learning effectiveness and performance impact using primarily quantitative and statistical measures, the Success Case Method focuses on gathering compelling evidence of effectiveness and impact through qualitative methods and naturalistic data gathering. Some organizational decisions require hard proof and statistical evidence. In my experience training is not one of them. At best, training decisions are usually judgment calls using best available information at the time. Statistical proof is often overkill and causes  managers  to look at each other in amusement.  All they really need is some good evidence, some examples of where things are going well and where they aren’t. They are happy to trade statistical significance for authentic verification from real employees.

2. We spend a lot of time twisting ourselves in knots trying to isolate the effects of training from other variables that mix with skills to impact performance. Factors such as opportunity to use the skills, how the skills are supported,  consequences of using the skills and others all combine to produce performance impact. Only we are hell bent on separating these factors. Our clients (internal and external) are interested only in the performance improvement. In the end it is irrelevant to them whether it was precisely the training that produced the improvement. They simply would like some confirmation that an intervention improved performance, and when it didn’t how we can modify it and other variables to make it work. Success case method accepts that other factors are at work when it comes to impact on performance and concentrates on the impact of the overall intervention.

3. The approach can be used for any type of intervention designed to improve performance, including training, performance support systems, information solutions, communities of practice, improved feedback systems, informal and semi-structured learning initiatives and social learning initiatives.

4. Success Case Method results are documented and presented as “stories”. We have learned the power of stories for sharing knowledge in recent years. Why not use the same approach to share our evaluation results instead of the dry and weighty tombs of analysis we often produce

5. It’s fast and it’s simple and has a growing track record.

To learn more:

The Success Case Method: Find Out Quickly What’s Working and What’s No

Telling Training’s Story: Evaluation Made Simple, Credible, and Effective

High Impact Learning: Strategies For Leveraging Performance And Business Results From Training Investments

About these ads

25 thoughts on “Evaluating with the Success Case Method

  1. Well summarized and a great strategy. There are many informal learning metrics strategies being passed around that still only measure “tool activity” and not “value/result”. Success Case Method is both more manageable and insightful to really demonstrate value (my CEO doesn’t care how many people post on a SoMe platform, she cares how much those discussions helped fulfill the goals of the company).

    I coached many in applying ROI/Evaluation techniques with a maxim “Data Proves, but Stories Sell”.
    Charts and tables provide compelling data, but I find the C-suite perks up when you present specific examples to illustrate the data concretely: “Manager X” noted “Y improvement” to “key issue Z” for the first time since ….Technique used with the State of the Union. Measures and metrics shared, but to sell, they add the personal stories of individuals to illustrate the case.

    One thing about numbers vs stories I learned early in my career (I am a former/recovering market research analyst):
    Liars can figure and figures can lie.

    Too much faith in raw numbers won’t really tell you the information you could just get by getting the right stories (New Coke, anyone?)

    • David;
      Thanks for the comment. I agree, even when you present indisputable statistics, stories based actual implementation by real employees carry more weight. The success case method helps bring some rigour and a methodology to ensure biased stories aren’t just compiled to “sell” a training program.

      • I found your post a compelling read and inclined to agree with your view that the Success Case method is far more beneficial to the improvement of the client than simple statistics from the typical Kirkpatrick based methods of evaluation.

        My initial thoughts would be that the C-suite would benefit further if they would have the statistics to back up the cases provided by the Success Case method. This would then provide a more holistic insight in how to improve their business.

        Has there been a case/situation where both types evaluation methods be run in tandem during an implementation with inducing higher costs to the client or consultancy?

  2. Hey, I think you missed a whole bunch of steps there. Since this article is a few years old I suspect you hadn’t read up on the New World Kirkpatrick method but the way the Success Case method is explained here doesn’t make it sound like a good way of doing things. You’re missing a lot between Step 1 and Step 2.

    “Step 1. Identify targeted business goals and impact expectations”
    This is your only “before”.
    “Step 2. Survey a large representative sample of all participants in a program to identify high impact and low impact cases”
    And the program is evidently delivered and now ready to be analyzed?

    So, in this missed example you missed the following steps:
    Step 1b. Identify leading indicators of that the program will reach its result (Identify targeted business goals and impact expectations)
    Step 1c. Identify which critical behaviors will make sure each leading indicator will come to fruition.
    Step 1d. Identify which key organizational drivers will make sure each critical behavior gets supported in each possible way. Always remember to analyze as to why these are not occurring already. Strongly consider removing hinders, obstacles etc. instead of creating any sort of incentives and “carrots”. NB! This step is the most important one of them all and we’ve yet to even come to learning!
    Step 1e. Identify which knowledge is needed, if any, for the critical behaviors to occur after the program. This analysis is of course done in the light of which drivers are already in existence, i.e. if we’ve identified that we need 10 checklists we wouldn’t need to “train” anyone on the information in these but more in how to use the checklists themselves, if at all.
    Step 1f. Identify what ways to deliver the knowledge in the most relevant and contextually meaningful ways. Is a video enough? Is the manual easily read by its own? Is it necessary to train via simulations or role plays? Does these have to be face-to-face?

    NOW I’m ready to move to step 2 and the Success Case Method, which I admittedly like better than any quantitative smile sheet experience.

    • Hi Henrik
      Thanks for your detailed comment. Yes I was aware of the Kirkpatrick methodology including it’s recent tweaks before writing the post. The two approaches come at the issue of evaluation from different perspectives. Kirkpatrick is more quantitative and focuses more on the effectiveness of the learning program itself. The success factors method is more qualitative and targets performance system barriers to learning as part of the method. Have a read of the introductory chapters of the two sources i mention in the post to get a feel for the differences in the approaches.

      • Henrik: if you mean the effectiveness of the training program itself (does it develop target skills) then a Kirlpatrick level 2 is fine, but ultimate effectiveness on the job and in producing business impact is influenced by many more factors than the training program itself. Identifying and adressing those factors is what the success case method is particularly good at.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s