Why evaluation is important
Online learning is always held to a higher standard than conventional teaching. For tenure and promotion, if you are teaching online it is critical to be able to provide evidence that your outcomes are at least as successful as your classroom courses.
Online learning continues to develop. Change is constant. New tools and new approaches to teaching online are continuously being put into practice. Teaching is like golf: we strive for perfection but can never achieve it. It’s always possible to improve, and one of the best ways of doing that is through a systematic analysis of your online teaching..
What to evaluate
Quality online learning is outcomes based: Reaching the same level or better with an online course as for an equivalent face-to-face course.’ This has two quantitative critical performance indicators:
- completion rates will be at least as good if not better for the online version
- grades or measures of learning will be at least as good if not better for the online version.
On a qualitative level, I suggested one other criterion:
- quality online learning will lead to new, different and more relevant learning outcomes that are better served by online learning. This means change.
What worked and what didn’t?
Look closely at factors that may have influenced students’ ability to learn. Some of the questions to consider:
- What learning outcomes did most students struggle with?
- Were the learning outcomes clear to students?
- Was the teaching material clear and well structured?
- What topics generated good discussion and what didn’t?
- Did students draw on the course materials in their discussion forums or assignments?
- Were the students overloaded with work?
- Was it too much work for me as an instructor?
- If so, what could I do to better manage my workload (or the students’) without losing quality?
- How satisfied were the students with the course?
Evaluating from Multiple Angles
To evaluate your class design holistically, you can consider your class design from four perspectives or angles. Questions to consider when evaluating your class design from four angles are provided below. What sorts of data would you need access to in order to answer the question?
- Self Reflection.
- You could use a checklist to help guide your own self reflection upon your online teaching and your redesigned online or blended class. Did your teaching strategy achieve the outcome you were expecting? Is there something you would change next time to improve your practice? Would you recommend your approach to a colleague? Was the technology appropriate?
- Peer Feedback.
- Have a colleague look at your redesigned activity, resource, or assessment. What feedback do they offer?
- Student Feedback.
- Obtain feedback from your students through surveys, focus groups, or short minute papers where they provide quick feedback during the semester. Are students engaging effectively with your online activities or resources? Do they have any trouble with the technology? Do they feel more engaged? Take care that if questioning your students about the quality of your course or teaching during the class, that they understand that their answers will not adversely affect their grades or performance in the class in any way
- Explore the literature about the online technology you have chosen to use in your class. How does your experience compare with the reported findings in the literature? Are the outcomes similar or very different from what others have reported? If your outcomes are different, it is not necessarily negative, as your context may be different. You may even choose to write up your findings to share with others in conferences, journals, or workshops and presentations at your educational institution.
Leeward has the ecafe student evaluation system you set-up midway through your online course. Because these questionnaires are usually voluntarily completed by students after the course has ended, completion rates are often low. Low response rates tend to be heavily biased towards successful students. It is the students who struggled or dropped out that you need to hear from.
Small focus groups work better than student questionnaires using synchronous tools such as Blackboard Collaborate.
- Approach 7-8 specific students covering the full range of achievement, from drop-out to A, and conduct a one hour discussion around specific questions about the course.
- If one selected student does not want to participate, I try to find another in the same category.
At the end of a course, look at your student grades, and identify which students did well and which struggled. Go to the beginning of the course and track their online participation as far as possible. This qualitative approach will often suggest changes to the content or the way you interacted with students for the next version of the course.
While student feedback was mentioned as one of the angles of evaluating your class design, the use of surveys or focus groups relies heavily on self-reported data. While this form of student feedback is certainly useful, it is also important to obtain objective data on students’ actual engagement with your online or blended class design when possible. Advances in online technologies have led to the collection of vast amounts of data pertaining to students’ use and interaction with technology. Much the same way that a gas company can track energy consumption in different cities to help improve their infrastructure – some learning technologies that we use capture data about students’ online activity. Exploring this data can help reveal whether the intended outcomes of the class design have been met or whether changes are required. Analyzing the online behavioral data from your students’ learning processes to inform your teaching is a form of learning analytics. The Society for Learning Analytics Research defines learning analytics as: “The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”.
Many online technologies provide real-time analytics and visualizations that can help inform you of how your students are engaging with the online technologies. If your students are engaging with the technologies the way you imagined, then you have some indication that your redesigned class is leading to your intended outcome. If not, you have an opportunity to communicate with the students and obtain more feedback on why they are not using the online technology the way you had hoped. You may be able to modify your class design during the semester or you may need to wait until the next offering. Either way, objective real-time data of students’ actual use of the technology will help you gauge the impact of your redesigned class throughout the semester or term rather than waiting until the end when students complete class evaluation questionnaires.
Spend time at the end of the first online course evaluating it and making changes working with and EMC instructional designer. Concentrate mainly on ensuring completion rates and grades are at the standard you have aimed for.
Look at ways to improve the course such as new software (e.g. an e-portfolio package), or new processes (e.g. student-generated content, using mobile phones or cameras, collecting project-related data). This keeps the course ‘fresh’ and interesting.
Your goal is to enable students to learn effectively. Follow the experienced route, then gradually add and evaluate new tools and new approaches to learning as you become more experienced.
From Learning to Teach Online by Simon McIntyre, Dr Negin Mirriahi