The effectiveness of the M.S. IDLT program is measured through the annual institutional assessment process, end of course IDEA evaluations, and a graduate feedback survey.
Annual Institutional Review
Anderson University articulates learning outcomes for all its educational programs, identifies measurements that will provide data on student achievement, gathers data on student performance, and analyzes the data in order to identify areas of strength and areas of concern in the educational program. Areas of concern are then targeted for improvement. This is done through course-level data where program outcomes are aligned as well as a capstone course where students submit a portfolio that aligns to program outcomes.
In the case of the M.S. in Instructional Design and Learning Technology, the Program Coordinator, Dr. Josh Herron, has primary responsibility for compiling learning performance data from the graduate faculty, directing the analysis of data, and creating the annual report. The report is submitted to the Dean of the College of Education and the Associate Provost, who reviews all annual reports and provides process feedback. Further details about this process are provided in the Program Effectiveness section.
The M.S. IDLT program assesses program-level outcomes using final project rubrics that contain the program outcomes aligned to each course through the Canvas Outcomes feature.
Table 2 below outlines where each programmatic outcome is assessed.
A Canvas rubric is used as shown in Figure 1 below that has outcomes tied to it for a faculty member to assess while grading the final project, which can then be pulled at the administrative level. This allows for a more authentic assessment of the program-level outcomes.
The data below, then, is from the score assigned by a faculty member on the individual outcome that was part of a final project for a selection of courses.
End of Course IDEA Evaluations
Anderson University uses the IDEA Student Ratings System (ideaedu.org) for student course evaluation. Response rates in the M.S. IDLT program are strong with a program average of 57.75% in Spring 2020.
Each course in the M.S. IDLT program has the following three predetermined IDEA learning objectives set as Important or Essential. Mean analysis for the end-of-course evaluations show that students make substantial progress on these learning objectives based on a 1-5 likert scale of No Apparent Progress (1) to Exceptional Progress (5).
Important: Gaining a basic understanding of the subject (e.g., factual knowledge, methods, principles, generalizations, theories)
Spring 2020 Mean: 4.68 (9 courses)
Important: Learning to apply course material (to improve thinking, problem solving, and decisions)
Spring 2020 Mean: 4.74 (9 courses)
Essential: Developing specific skills, competencies, and points of view needed by professionals in the field most closely related to this course
Spring 2020 Mean: 4.76 (9 courses)
Instructor and course design feedback is also solicited in these end-of-course evaluations. In Spring 2020, Excellence of Teacher and Excellence of Course Unit and IDEA Averages indicate above average quality on a 5-point scale:
Graduate Feedback Survey
The M.S. IDLT program leaders are interested in feedback from graduates as well. Along with frequent check-ins with alumni, a standard alumni survey is used. Soon after completing the program, M.S. IDLT alumni are sent a survey that asks about their program experience, including career impact, faculty, courses, and more.
Graduates from the recent cohorts (Fall 2019, Spring 2020, & Summer 2020) of the M.S. in IDLT program noted the importance of having faculty who were active instructional designers and their motive for choosing a Master of Science in Instructional Design and Learning Technology. 77% of alumni agree on some level (somewhat agree to strongly agree) that “It was important to me that my professors are practitioners in the field.” It is important to highlight that no students disagreed with the statement. Even further, 89% of alumni stated that they “Strongly Agree” with the statement that “My professors were knowledgeable about the subject matter.” Faculty members agree even more strongly that it is “important [. . .] that students in the M.S. IDLT program are taught from working practitioners in the field of instructional design.”