Levels of Evaluation

The Kirkpatrick model of evaluation is a 4-tier model for evaluating effectiveness of an instructional design program. Its creator, Donald L. Kirkpatrick, PhD, introduced the model in the 1950s. While some are skeptical of its effectiveness, it still remains the most widely used model in the world due to its simplicity.

http://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model

Level 1 - Reaction

This level measures customer satisfaction, engagement, and relevance. While a positive reaction doesn't ensure that learning occurred, often, a negative reaction almost always reduces the possibility that it did. In other words, how learners react to training is an important part of whether or not they are able to learn.

Types of questions one might ask to measure effectiveness of this level:

• How did participants react to the training program?
• Did it engage them?
• Did they feel the training applied to their jobs?
• Were the materials relevant?

Level 2 - Learning

This level measures learning. Specifically, it measures how well students have evolved in skills, knowledge, or attitude. Pre and post tests help evaluate learning at this level, but they are not always seen as useful in a work-type setting. Various other summative evaluations may be used. While a positive reaction doesn't ensure that learning occurred, often, a negative reaction almost always reduces the possibility that it did. In other words, how learners react to training is an important part of whether or not they are able to learn. Levels 1 and 2 are the most often evaluated levels. Many organizations do not ever reach level 3 of evaluation. If level 3 is evaluated, it is imperative that proper analysis of levels 1 and 2 have first occurred.

Level 3 - Behavior

This level measures the transfer that has occurred in a learner's behavior due to the training program. For many, this measures the true effectiveness of a learning program. Some argue that this level is difficult to assess due to a variety of factors beyond the instructional designers role. However, the purpose of learning is to impact on-the-job performance, so analysis of this level is important in knowing what changes need to be made to the instructional program in order to facilitate maximum return on investment (ROI). Some factors which impact level 3 results include managerial support of the client, organization processes which may affect a client's ability to transfer learning, and lack of follow-up by training organization after the training has concluded.

Types of questions/actions one might ask to measure effectiveness of this level:

• Are the new skills, knowledge, or attitudes being used in the learner's everyday environment?
• Are you (or the trainee) using what you (they) learned? Why or why not?
• If you needed to _____ (insert taught skill based off learning objective), how would do this?
• If you aren't ______ (using new skill based off learning objective), what would you need to begin doing so?
• At the conclusion of training, have students write a letter/(or, if virtual, an email) to themselves reflecting on what they learned and how they will apply the learning in the field. Mail the letter/follow up on the email to them months after training and contact them to get input on how they think they are doing applying the skills they set out to apply. The idea is that improving a learner's self-efficacy (belief that they can apply the learned skills), will in fact, increase the likelihood that the learner will apply the newly learned skills. This exercise allows for the learner to authentically evaluate self-efficacy at the end of training while also providing a means to follow up on that evaluation.

Level 4 - Results

This level measures the impact of training. This may include increased productivity, reduced cost, increased sales, higher profits, etc.

Types of questions/actions one might ask to measure effectiveness of this level:

• Have support calls on training topics reduced?
• Is there increased customer satisfaction?
• Is waste reduced?
• Has morale improved?

Sources:

http://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
https://www.mindtools.com/pages/article/kirkpatrick.htm
https://maketrainingstick.wordpress.com/
https://maketrainingstick.wordpress.com/2014/06/30/letter-to-self-easy-closing-activity-that-makes-training-stick/
http://maketrainingstick.com/pdfs/written-self-guidance.pdf

Bates, R. A. (2004). A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Evaluation and Program Planning, 27(3), 341-347. doi:10.1016/j.evalprogplan.2004.04.011

Curriculum Developer vs. Instructional Designer - What's the Difference?

Potato, potawto, tomato, tamatoh...right? 

Curriculum development and instructional design - those are the same things, right? Well, not necessarily. In my quest to clarify some terms in this relatively young field of Instructional Technology, I have found myself thinking a lot about the differences in these two terms. What am I?

I work as a Curriculum Developer for a large software company. Like most people, I have a job description that outlines my duties. And probably like most people, I would say that I do so much more than what is technically listed in that outline. So, I got to thinking...am I a Curriculum Developer, Instructional Designer, Trainer? WHAT AM I?!? Are these even the same things? So, off to research (Ok, ok, surf the internet) I went!

Here's what I discovered.

Curriculum Developers are professional experts in all things learning. They know a lot about learning theory, learning methods, educational best practices as well as the wide spectrum of learning content delivery media (classroom-based, online, synchronous, asynchronous, webinars, e-learning etc.) Curriculum Developers are adept at looking at the big picture, or blueprint, of an organization's learning needs and then mapping out the best path to achieving those needs. 

The ability to identify which content is necessary in reaching an organization's goal is key. The Curriculum Developer's knowledge of learning theories supports their ability to properly sequence relevant foundational concepts  and essential building blocks which are necessary for mastery of the identified content. Some Curriculum Developers also serve as subject matter experts (SMEs) and have some overlapping expertise in Instructional Design.

Instructional Designers are professional experts in the development of actual courses (pieces of the big picture). The development of courses includes content flow, typically designed using a preferred instructional design model, and specific learning activities that will be used to most efficiently achieve the learning goals of the organization If the organization has a Curriculum Developer, this is typically the content outlined in the Curriculum Developer's blueprint. When developing the actual courses, Instructional Designers are experts in understanding which instructional design model best meets the needs of their organization. Common instructional design models include: ADDIE, The Dick and Carey Model, ASSURE, and Morrison, Ross, & Kemp, but there are oh so many more to choose from! Instructional Designers often specialize in some medium of delivery (live, e-learning, or both)

The Dick and Carey Model

Role of the Dick and Carey Model in the Broader Curriculum Development Process

Typically, Curriculum Developers and Instructional Designers work closely together. They often have overlapping areas of expertise. So, while each entity has its own areas of specialty, they truly rely on each other to build an end instructional product that helps an organization most effectively and efficiently produce learning outcomes.

Sources:

https://ascendtraining.wordpress.com/2013/07/23/curriculum-development-and-instructional-design/

http://www.hastudio.us/5_CV/thesystematicdesignofinstruction.pdf

Writing to the Reader

How does one know if they are writing at a level in which their audience can understand? What is the ideal level writers should strive to "write to?" How can you increase the likelihood that coworkers and clients will read what you write?

50% of U.S. adults read below an eighth-grade level.

Most high school graduates read below 12th grade level.

Knowing these facts helps writers more effectively communicate with their audience. Just because a reader CAN read at a higher level doesn't always mean they should. As business consultant, Perry Marshall states:

"Don’t make people think. Except when you really want them to think."

Writers shouldn't make readers waste time dissecting sentences or wondering what an author's word-selection means. Brain power should be used to focus on the message. 

So, how do you figure out what level your writing is on?

Many word processing applications offer two tools which can be used to gauge appropriate reading level of materials. These tools can also be found online. These readability tools are called the Flesch Reading Ease and Flesch-Kincaid Grade Level indicators. Both values indicate the text's readability, or how easy it is for someone to read it. Both values are based on the average number of syllables per word/average number of words per sentence. 

The average Flesch Reading Ease value indicates how easy it is for a reader to understand the text. The value ranges between 60 and 70 on a 100-point scale. The higher the score, the easier it is for readers to understand documents.

The Flesh-Kincaid Grade Level's score indicates a grade level. If text receives a value of 7.5, this indicates the text is written at a level at which a 7th grader in his or her fifth month of school can comprehend. The average range is between 7.0 and 8.0. Some examples of text run through this calculation using Microsoft Word's calculator are shown below:

  

So, the next time you write content, consider your audience. A student's grade level, high school diploma, or college degree do not always indicate a student's true reading level. The purpose of text is to convey a message. The more informed one is about the true reading level of their audience, the more effective they can be in crafting that message in a way in which their audience can most effectively comprehend.

And because I know inquiring minds want to know, this blog falls right within the range of "average."

Set Office 2016 to include readability values

Sources:

https://contently.com/strategist/2015/01/28/this-surprising-reading-level-analysis-will-change-the-way-you-write/

http://literacyprojectfoundation.org/community/statistics/

http://www.clearlanguagegroup.com/readability/

http://www.officetooltips.com/word/tips/viewing_document_and_readability_statistics.html

https://www.perrymarshall.com/16572/reading-level/

https://www.usatoday.com/story/tech/news/2017/04/04/isps-can-now-collect-and-sell-your-data-what-know-internet-privacy/100015356/