Research on the course

One component of an effective course portfolio is documentation of what students learn from the course. For this portfolio study, I have used a combination of a pre-test administered on the first day of class and a post-test administered on the last day to determine what students had learned during the semester. In these tests, I asked thirteen questions that focused mainly on what I considered to be important points that I wanted the students to learn. Questions were the same on both tests, although I added one item, to be discussed below, to the post-test.

Before providing the results and analysis, I will comment that this type of comparison has sometimes been criticized (Cook & Campbell, 1979, pp. 99–103), for example because improvements seen between the two tests could result from (a) interventions other than the one under study (the course, in this case), or (b) because the pre-test could cause subjects to pay special attention to learning the subjects treated on that test. To deal with the first point, I asked students taking the post-test to estimate what percentage of any improvement made resulted from this specific course, rather than from other sources, and as will be shown below, those estimates were generally quite high. As for the second point, students did not retain copies of the pre-test, and I think it unlikely that many remembered its questions through the semester. Furthermore, since the questions asked about the points I most wanted the students to learn, if the testing did improve learning of those points, so much the better.

Table 1 Short descriptions of questions asked in the pre- and post-tests.

Question	Brief description
1	Why stats useful?
2	Paired-t requirements
3	Most data normal?
4	"Significance" consistent?
5	Not significant = not important?
6	Not significant = just random chance?
7	Why distribution knowledge useful
8	Pattern and residuals
9	Define power
10	Bayesian analysis advantages
11	When use chi-square test?
12	Misinterpreting significance tests
13	Resampling methods
14	Percentage improvement from this course

A copy of the post-test is provided as Appendix 2; the pre-test was identical, except without the fourteenth question, which asked what percentage of improvement resulted from the course. A short description of each question is provided here as Table 1. Of those questions, the three that addressed the central focus of the course were 5, 6, and 12. On the other hand, I ended up not “covering” anything to do with Questions 4 or 11.

I scored the tests by first assigning random numbers to each student for each test, and then printing out all 40 answers (20 students ´ 2 tests) to a given question together, but sorted by those random numbers. Thus, as I scored each answer, I was blind to whether it was a pre- or post- answer, and blind to which student had supplied the answer. I also used different starting points and sequences to move through the answers for the different questions, to prevent consistent patterns of “good before bad” (or the like) from affecting my scoring consistently. I assigned scores ranging from -3 to +3, with the following meanings:

Score	Meaning
3	Really excellent
2	Pretty good
1	In the right direction, but not very good
0	Neutral. Not very informative, but not incorrect
-1	Slightly in error
-2	Definitely in error
-3	Really wrong, or even backward

(In some cases, I assigned intermediate scores like 1.5, etc., to provide more levels of distinction.)

Acknowledgment: I am grateful to Profs. Simon Brassell and Shanker Krishnan for administering the pretest and the posttest in my two sections of E538, as was required by the IU Human Subjects Committee.

Home Previous section Next section Table of contents