Outcomes Levels, Data Source, and Process

Outcome Level

Data Source and Process

Level 1: Participation

Learner registration allows the Institute to track demographic information and participation. According to Moore’s updated framework for outcomes evaluation in the continuing professional development of physicians, the suggested data source for measurement of this level is the activity records.

Level 2:  Satisfaction

A detailed activity evaluation is required at the conclusion of every activity to measure variables related to design, implementation, and subject matter. Satisfaction of learning objectives, content, suggestions for improvement, suggestions for future activities, absence of commercial bias, and general questions are sought. According to Moore’s updated framework for outcomes evaluation, the suggested data source for measurement of this level is participant evaluations.

Level 3:  Declarative and Procedural Learning

Level 3 outcomes are captured by testing participants prior to live and online activities using content-related clinical assertion questions as well as embedding key measurement indicators within the activity. The same questions are subsequently included in the posttest. The aggregated results also act as indicators to determine whether concepts and data within the presentation result in an increase in overall knowledge. Additionally, barriers to learning are identified through aggregation of activity evaluations. According to Moore’s updated framework for outcomes evaluation, the suggested data sources for measurement of this level are pre- and post-tests.

Level 4: Competence

Case vignettes are used to measure application of knowledge discussed in each activity.  In many cases, the activity itself is case-based. Learners are tested on knowledge retention and application within each clinical scenario. This information, coupled with baseline knowledge of the subject and immediate learning related to the content, allows for Level 4 outcomes analysis. According to Moore’s updated framework for outcomes evaluation, the suggested data sources for measurement of this level are virtual observations in the educational setting.

Level 5: Performance

At the conclusion of each activity, learners are asked to provide examples of how they intend to change their performance as a result of the activity content. Examples are aggregated by the Institute. Through 45-day follow-up surveys, learners are asked whether the performance changes they identified were made. According to Moore’s updated framework for outcomes evaluation, the suggested data sources for measurement of this level are observations in the clinical setting and self-reported performance changes.