The following is a research paper that I wrote for a graduate class on using data to improve instruction. The paper was written before schools commonly used learning management systems. The material below is still relevant; however, I now use my school’s learning management system instead of Moodle.
The appendices have been removed.
Research exists on the use of statistical data to improve instruction; however, there is considerably less agreement on what data is to be analyzed and how it is to be acquired. It is more difficult than is widely understood to acquire data that is both statistically valid and directly relevant to instruction. Results from standardized tests are often cited as reliable and relevant data sources despite a number of serious issues. Concerns include the manner in which standardized tests are administered, the knowledge and skills the tests seek to measure, and the manner in which the results are reported.
Consider first the manner in which standardized tests are administered. Variations in the testing environment call into question the assertion that standardized tests are administered in standardized environments. Out of necessity, standardized tests are most often administered in schools by teachers who are paid a stipend. School environments vary dramatically with respect to lighting, noise level, seating arrangements, the availability of climate control, and innumerable other factors. Teachers administering the tests vary with respect to their level of training, their ability to comprehend and follow detailed directions, and their commitment to adhering to testing procedures. Standardized tests are generally administered infrequently, often at the end of a period of instruction. This, and the substantial delay before results are reported, limits the ways in which the results can be used to influence the instruction of the students who produced them (as opposed to later generations).
Standardized tests are constructed to measure student mastery of many concepts at once, such as an entire year of math. Even if the results are broken down by topic, the topics are usually too general to suggest specific actions. Many standardized tests do not offer the option of returning test questions to either students or faculty. This, and all of the above, relegate standardized tests solely to the role of assessment. Without the ability to review individual questions, they have no formative educational value whatsoever. (Perhaps this last point explains the variations in teachers’ willingness to adhere to testing procedures.)
An Overview of Two Potential Solutions and Their Limitations
Using the results of existing course assessments as a dataset on which to perform statistical analysis addresses the issues with standardized tests presented above. Teachers control the content and administration of course assessments. With the techniques discussed below, they can also obtain control over the reporting of results. There are limitations on both of the approaches presented. To utilized the preferred approach, a school must be able to provide a one-to-one ratio of (internet connected) computers to students for the duration of assessment activities. To utilize the alternate approach, each student must have access to a (connected) computer for a short period sometime soon after each assessment.
There are a number of web based systems, some of which can be deployed on a very tight budget, that allow teachers to administer assessments electronically. Students logon to individual accounts on individual computers and complete existing course assessments online. Immediately after each assessment, teachers can obtain data for each question and for each student. Since the assessments are local (as opposed to state wide), each student takes the assessment in the same environment. Since the online assessments are based on existing course assessments, they measure student mastery of relevant concepts at relevant times. As mentioned above, techniques to use the results of assessments to influence instruction are already well known (and will not be repeated here). The immediate availability of reliable and relevant assessment results finally allows teachers to make use of existing techniques and facilitates the development of new ones.
If students cannot be provided with individual computers during assessment activities, an alternate approach may be employed. Web based forms, such as those offered by Google, can be used to collect data from students after they have completed paper based assessments. Students can be offered a nominal reward for entering the results of their assessments online. Although this is not as elegant as administering assessments online, it is easier to implement and has a lower infrastructure requirement. The same data as above can be collected, so all of the above benefits can still be obtained.
With both approaches, the assessments are of far more educational value than standardized tests. At the very least, no time is wasted, since the course assessments would have been administered anyway. The ability to review individual questions allows the assessments to be used in a formative way. Additional benefits of administering assessments online, that are not available with paper based assessments are discussed later. A section on benefits of online course management systems unrelated to assessment is also included.
Research Findings
Ambitious projects to move paper based activities online have a staggering failure rate. A number of references have been consulted in an attempt to avoid some of the pitfalls that have doomed such projects. The insights below have been gleaned largely from studies by Al-Jarf, the Consortium for School Networking, and Kok. A number of other resources have been consulted to a lesser extent.
All three studies strongly suggest that any software employed must be easy to use, stable, and well supported. Training must be available where needed and must be of high quality. Teachers should not be forced to use the system. Instead, it should be made readily available and allowed to spread. Successful uses should be publicized so that others can observe potential benefits. Teachers should be encouraged to use whatever combination of available tools is actually relevant to their courses, rather than every tool available in the system. Teachers who start using a system merely to aid in the distribution of materials and the collection of assignments may eventually take advantage of some of the more advanced tools. Successful uses of particular tools should be publicized.
Infrastructure and indirect costs should not be overlooked when adopting a system. Many of the available systems are offered as free software; however, this does not imply that there are no costs associated with deploying and supporting it. If connected computers are not readily available to students and teachers in the ratios required, the systems will be of limited use. The time required to setup and use such a system must be accurately estimated and communicated. It is easy to spend a significant amount of time on an activity with a relatively low return when that time is not spent in face-to-face contact with students.
Additional Benefits of Online Assessments
Administering assessments online has benefits beyond those that can be derived by statistically analyzing the results. Most systems for administering tests online include facilities for students to view both their answers and correct solutions for each question. Students can review the solutions on their own time, reducing the amount of class time devoted to the purpose. In lieu of reviewing an entire assessment in class, a teacher might request that students review the solutions online and return to class the next day with any remaining questions. Online assessments need not be limited to those that contribute to a student’s grade. The format is highly conducive to practice quizzes in which students receive immediate feedback on only the problems with which they experienced difficulty.
Some of the systems available allow for unique types of assessments that would be prohibitively time consuming if done on paper. For example, students may be able to attempt a multiple choice question repeatedly, with feedback specific to each answer. Students may be able to score other students’ work and well as example work presented by the teacher. Research suggests that students who teach others often learn themselves. Some software allows students to produce questions for others to attempt.
Additional Benefits of Online Course Management Systems
Course management systems offer benefits beyond online assessment. Most systems include tools to help students manage scheduling of assignments and assessments. This is especially valuable for students with special needs, though all students can benefit. As mentioned above, many systems facilitate the distribution and collection of materials. Systems may facilitate the assembly of student portfolios, whether intentionally, or merely by serving as a repository of student work. Many systems facilitate communication among students, allowing both online tutoring and peer group support. Some systems also encourage and simplify curriculum mapping. From a professional development standpoint, many systems can be used to persistently store materials from professional development activities. Teachers can use the systems to comment on techniques learned and implementations of them.
Moodle
Numerous systems exist to accomplish the above, one of which is Moodle. What follows is a brief explanation of some of the technical aspects of Moodle, an explanation of a subset of its capabilities, demonstrations of some of its online assessment features, and an example of the results produced. Moodle software is free in the sense that it can be obtained from moodle.org at no cost. The software runs on a web server (as opposed to a desktop computer), meaning that one must be setup or the use of one must be rented. Web hosting appropriate for a single teacher or a single medium sized school costs approximately $10 per month. Web hosting appropriate for a district costs approximately $100 per month (as of November 2008). Many web hosts provide a piece of software called Fantastico that can install Moodle with minimal effort. Several vendors, listed at moodle.com, offer fully managed installations for considerably greater cost.
Moodle provides all of the capabilities mentioned above. It provides an effective means of distributing materials, collecting assignments, and facilitating student discussions. Numerous other features are available that are beyond the scope if this project. A more exhaustive description of the system may be found at moodle.org. This project focuses on the online assessment capabilities offered by Moodle. Multiple choice, true/false, essay, matching, and several more exotic types of questions may be posed in online assessments.
As an example, the interface used to create a multiple choice question is shown in Figure 1 in Appendix A. Notice the large number of optional fields that allow the teacher to provide detailed feedback depending on the answer chosen by the student. Students’ answers to multiple choice questions are scored automatically. The interface to view students’ responses and scores is shown in Figure 2. Notice that each students score on each question is available. Figure 3 shows some of the statistical analysis of each question, as performed by Moodle. The data is also exportable so that more sophisticated analysis can be performed.
It is also possible to ask free response or short answer questions with Moodle. Consider a 2 part short answer question. Moodle’s grading interface allows each student’s responses to both parts to be stored sequentially, as most teachers would on paper. Alternatively, Moodle allows each response to part 1 to be scored, followed by each response to part 2. This discourages subjective grading and is considerably easier to accomplish online than it would be with paper. The interface for scoring student responses to a short answer question is shown in Figure 4.
References
Al-Jarf, R. (2005, November 10). Using Three Online Course Management Systems in EFL Instruction. Online Submission, (ERIC Document Reproduction Service No. ED497937) Retrieved November 6, 2008, from ERIC database.
Antonenko, P., Toy, S., Niederhauser, D., & Association for Educational Communications and Technology, W. (2004, October 1). Modular Object-Oriented Dynamic Learning Environment: What Open Source Has to Offer. Association for Educational Communications and Technology, (ERIC Document Reproduction Service No. ED485088) Retrieved November 6, 2008, from ERIC database.
Beatty, B., & Ulasewicz, C. (2006, August 1). Faculty Perspectives on Moving from Blackboard to the Moodle Learning Management System. TechTrends: Linking Research and Practice to Improve Learning, 50(4), 36-45. (ERIC Document Reproduction Service No. EJ774606) Retrieved November 6, 2008, from ERIC database.
Bradford, P., Porciello, M., Balkon, N., & Backus, D. (2007, January 1). The Blackboard Learning System: The Be All and End All in Educational Instruction?. Journal of Educational Technology Systems, 35(3), 301-314. (ERIC Document Reproduction Service No. EJ762105) Retrieved November 6, 2008, from ERIC database.
Cavus, N., & Ibrahim, D. (2007, January 1). Assessing the Success Rate of Students Using a Learning Management System Together with a Collaborative Tool in Web-Based Teaching of Programming Languages. Journal of Educational Computing Research, 36(3), 301-321. (ERIC Document Reproduction Service No. EJ760629) Retrieved November 6, 2008, from ERIC database.
Consortium for School Networking, W. (2008, January 1). CoSN K12 Open Technologies Implementation Study #3. Moodle: An Open Learning Content Management System for Schools. Consortium for School Networking, (ERIC Document Reproduction Service No. ED499881) Retrieved November 6, 2008, from ERIC database.
Kok, A. (2008, July 1). An Online Social Constructivist Tool: A Secondary School Experience in the Developing World. Online Submission, (ERIC Document Reproduction Service No. ED502082) Retrieved November 6, 2008, from ERIC database.
Uzunboylu, H., Ozdamli, F., & Ozcinar, Z. (2006, November 1). An Evaluation of Open Source Learning Management Systems According to Learners Tools. Online Submission, (ERIC Document Reproduction Service No. ED494265) Retrieved November 6, 2008, from ERIC database.