Main | Activity 4-1 | Activity 4-2  | Activity 4-3

ACTIVITY 4-3: pilot the assessments

In step 4 you carry out a pilot study of the first cycle of three cycles. Ideally the pilot will be conducted with target learners, but you may wish to do one first with colleagues or classmates who are not part of your development team. Consult your facilitator for input on the scope of your piloting. One of the members from your team should either observe or work alongside an individual participating in the testing and try to play the role of a peer.

Remember that throughout the piloting, you should be gaining insights that will be useful for revising the investigation, assessments, and review materials in the next phase.

Note. You are required to carry out this pilot only during your Virtual Design Center process. Consult your facilitator if you are advanced in your progress and ready to do your first cycle formally with a teacher and students. We recommend formally going through the three cycles after you complete the development of the inquiry-based classroom activity using your Virtual Design Center deliverables.

1. Administer the Pretest
Have the participants complete the pretest items. This should be done a day or two before the actual piloting. This makes it harder for pilot participants to remember the questions and, therefore, makes their performance on the posttest a better indicator of their learning.   

2. Complete the Quiz
Have the pilot participants individually complete the quiz item(s). They should do their best but understand that the quiz is not for a grade. Make sure that the participants do their best to answer the quiz, but avoid scrutinizing the answer in a way that might make the learner defensive or raise the issue of the fairness of the item. Emphasize the formative goal of enhancing understanding and save the more summative goals for the exam and the test. In your next piloting try including your inquiry-based activity. 

3. Review the Quiz
Have the participants collaboratively review their completed quiz item(s), using the answer explanation. If you are working with just a single participant, you should play the role of a classmate and pretend that you do not fully understand the solution. Here is a set of guidelines developed in a previous study for facilitating discussion. Because the quiz item and answer explanation are closely linked to activity, the discussion should give you a good idea what students learned in the investigation, while also advancing that knowledge.

4. Complete and Grade the Exam
Have the participant complete the exam item(s) and use their score as a measure of their understanding of the targeted concept.

5. Review the Exam
Have the participant use the exam answer explanation to review the exam item(s). Be sure to consider why the incorrect answer choices are wrong, particularly to the extent that they exploit common misconceptions about the topic.

6. Complete the Posttest and Examine Gains
Have the participants complete the test, and compare the test with the pretest. For the purposes of the pilot, you may share the results with the students and ask them about their answers. 

Do not expect large gains on this pilot testing! In several prior studies little or no gains on such tests were observed in the first implementation. Do not be discouraged and give up or drop the unrelated items from the test. Work on refining your investigation, assessments, and answer explanations.

Refine Alignment
Consider the alignment of the investigation, the quiz, exam items, and feedback materials in light of the items in your test item pool, and refine the alignment based on the pilot testing. The three levels of assessments represent the target content knowledge differently:

The investigations and the quizzes emphasize co-construction of knowledge, as when an expert works with colleagues. Students and teachers should negotiate the use of scientific terms and concepts in a similar way. Arguing and discussing what they are learning create an informal student-oriented accountability.

The exams emphasize understanding of how to solve new problems. You want learners to be able to use the knowledge from the activities to individually solve similar new problems. Solving new problems presented in exams reveals an understanding of the targeted concepts and support a semiformal teacher-oriented accountability.

The test emphasizes recognizing correct answers on multiple-choice items. High-stakes tests essentially compare ability to recognize loosely related associations drawn at random from a content domain. This is necessary for the criterion-oriented validity and reliability that is necessary for high-stakes tests.

Deliverable 4-3: Results of the Pilot Test

Download the results of the pilot test outline and complete your deliverable 4-3.

Below is an example of the results of a pilot test by a previous Virtual Design Center participant. Note that this group developed only a standards-based test.

Return to step 4 for debriefing.

Main | Activity 4-1 | Activity 4-2  | Activity 4-3