Learning-focused Syllabus Rubric

Overview

To enrich the resources for measuring the impact of educational development work, we have created a rubric to assess the degree to which a syllabus achieves a learning orientation. The rubric provides qualitative descriptions of components that distinguish learning-focused syllabi and uses a quantitative scoring system that places syllabi on a spectrum from content-focused to learning-focused. It is flexible enough to accommodate a diverse range of levels, disciplines, institutions, and learning environments yet nuanced enough to provide summative information to developers using the tool for assessment purposes and formative feedback to instructors interested in gauging the focus of their syllabi.

Our syllabus rubric was awarded the 2014 POD Network Innovation Award.

Here’s the project summary created for the annual conference in Dallas, TX: POD Innovation Award Poster. (PDF)

User Guide

To help others use the rubric effectively, we’ve created a User Guide that includes 1) relevant background information and details about validity and reliability, 2) the complete rubric, scoring directions, and a blank scoring sheet, 3) references, 4) and six reference syllabi which we’ve scored and fully annotated. Because this is a living document, check back regularly so that you have the most up-to-date version.

• Syllabus Rubric Guide - PDF (2-13-17)

• Syllabus Rubric Scoring Sheet - Excel (6-9-15)

You may use our syllabus rubric and associated resources for research purposes provided you cite the following:

• Palmer, M. S., Bach, D. J., & Streifer, A. C. (2014). Measuring the promise: A learning‐focused syllabus rubric. To improve the academy: A journal of educational development, 33 (1), 14-36.

We have scored and fully annotated six different syllabi to help users of our syllabus rubric understand the meaning of low, moderate, and strong evidence. For the norming process, we recommend users score the six reference syllabi (Appendix C) first without the aid of our scores and annotations. Then, compare scores, referring to the annotations when discrepancies exist (Appendix D).

• Appendix C – Reference Syllabi - PDF (11-24-14)

• Appendix D – Annotated Syllabi - PDF (5-31-17)

Analysis Data from UVA’s Course Design Institute

Using a one-group pretest-posttest study design, we have analyzed 54 pre-/post-CDI syllabi pairs. Figures 1-2 show pre- and post-CDI syllabi scores, respectively. Total pre-CDI scores range from 0 (2 syllabi) to 46 points (1 syllabus). The mean score was 9.4 (SD=10.0).  Forty-seven syllabi fell in the content-focused range (87%), 5 in the transitional range (9%), and 2 in the learning-focused range (4%).

Total post-CDI scores range from 12.5 to 46 points (1 syllabus). The mean score was 31.4 (SD=9.32).  Three syllabi fell in the content-focused range (6%), 21 in the transitional range (39%), and 30 in the learning-focused range (55%).

Total post-CDI scores range from 12.5 to 46 points (1 syllabus). The mean score was 31.4 (SD=9.32).  Three syllabi fell in the content-focused range (6%), 21 in the transitional range (39%), and 30 in the learning-focused range (55%).

The plot of percent gain versus percent pre-CDI syllabus score is shown in figure 3. Using our definitions of low, moderate, and high gain, six instructors fell in the low-gain, 27 in the moderate-gain, and 20 in the high-gain region. The overall percent normalized gain (i.e. the average of the instructors’ percent normalized gains; %<<g>>) was determined to be 60.4% (SD=22.4%).  This indicates that the average instructor is expected to gain 60.4% of the points possible to them regardless of their pre-CDI syllabus scores. In other words, the CDI intervention appears effective at moving the average instructor to at least the transitional range of the rubric.

As shown in Table 1, statistical significance was found for overall and criterion-level pre- and post-CDI scores. Effect sizes were measured with Pearson’s r (Sullivan & Feinn, 2012), and we found a practical significance (r: 0.2 = small, 0.5 = medium, and 0.8 = large) for all but Schedule.

Questions? Email Michael Palmer.