Main | Activity 4-1 |
4-2 | Activity 4-3
ACTIVITY 4-3: pilot the assessments
In step 4 you carry out a pilot study of
the first cycle of three
cycles. Ideally the pilot will be conducted
with target learners, but you may wish to do one first with
colleagues or classmates who are not part of your development
team. Consult your facilitator for input on the scope of your
piloting. One of the members from your team should either
observe or work alongside an individual participating in the
testing and try to play the role of a peer.
Remember that throughout the piloting, you
should be gaining insights that will be useful for revising the
investigation, assessments, and review materials in the next
Note. You are required
to carry out this pilot only during your Virtual Design Center
process. Consult your facilitator if you are advanced in your
progress and ready to do your first cycle formally with a
teacher and students. We recommend formally going through the
three cycles after you complete the development of the
inquiry-based classroom activity using your Virtual Design
1. Administer the Pretest
Have the participants complete the pretest
items. This should be done a day or two before the actual
piloting. This makes it harder for pilot participants to
remember the questions and, therefore, makes their performance
on the posttest a better indicator of their learning.
2. Complete the Quiz
Have the pilot participants individually
complete the quiz item(s). They should do their best but
understand that the quiz is not for a grade. Make sure that the
participants do their best to answer the quiz, but avoid
scrutinizing the answer in a way that might make the learner defensive or raise the issue of the fairness of the item.
Emphasize the formative goal of enhancing understanding and
save the more summative goals for the exam and the test.
In your next piloting try including your inquiry-based
3. Review the Quiz
Have the participants collaboratively
review their completed quiz item(s), using the answer
explanation. If you are working with just a single participant,
you should play the role of a classmate and pretend that you do
not fully understand the solution.
Here is a set of
guidelines developed in a previous study for facilitating
discussion. Because the quiz item and answer explanation are
closely linked to activity, the discussion should give you a
good idea what students learned in the investigation, while also
advancing that knowledge.
4. Complete and Grade the Exam
Have the participant complete the exam
item(s) and use their score as a measure of their understanding
of the targeted concept.
5. Review the Exam
Have the participant use the exam answer
explanation to review the exam item(s). Be sure to consider why
the incorrect answer choices are wrong, particularly to the
extent that they exploit common misconceptions about the topic.
6. Complete the Posttest and Examine
Have the participants complete the test,
and compare the test with the pretest. For the purposes of the
pilot, you may share the results with the students and ask them
about their answers.
Do not expect large gains on this pilot
testing! In several prior studies little or no gains on such
tests were observed in the first implementation. Do not be
discouraged and give up or drop the unrelated items from the
test. Work on refining your investigation, assessments, and
Consider the alignment of the
investigation, the quiz, exam items, and feedback materials in
light of the items in your test item pool, and refine the
alignment based on the pilot testing. The three levels of
assessments represent the target content knowledge differently:
The investigations and the quizzes
emphasize co-construction of knowledge, as when an expert works
with colleagues. Students and teachers should negotiate the use
of scientific terms and concepts in a similar way. Arguing and
discussing what they are learning create an informal
emphasize understanding of how to solve new problems. You want
learners to be able to use the knowledge from the activities to
individually solve similar new problems. Solving new problems
presented in exams reveals an understanding of the targeted
concepts and support a semiformal teacher-oriented
emphasizes recognizing correct answers on multiple-choice items.
High-stakes tests essentially compare ability to recognize
loosely related associations drawn at random from a content
domain. This is necessary for the criterion-oriented validity
and reliability that is necessary for high-stakes tests.
4-3: Results of the Pilot Test
Download the results of the
test outline and complete your deliverable
Below is an example of the results of a pilot
test by a previous Virtual Design Center participant. Note that
this group developed only a standards-based test.
Return to step 4 for debriefing.
Main | Activity 4-1 |
4-2 | Activity 4-3