AY 300 - Fall 2011: Sixth Day Lesson Plan

Today's class will discuss writing effective test questions, the design-a-demo presentations, and continue board work exercises.

• Review how to write effective exam questions.
• Continue practicing good boardwork technique.

The person on call for this week will share their section experience. Some questions include:

• What did you do?
• How did you implement your activities?
• What worked?
• What didn't work?
• What would you do differently?
• How did you assess learning?
• Did you receive any unexpected questions/reactions/etc.?
• Did anything unexpected happen?
• What were you thinking about while you were running section? Any moments of panic?

During this (and after), open the floor up for general questions and sharing about how sections are going.

Each group receives ten minutes for their proposal. The proposal itself should be ~5 minutes, allowing 5 minutes for questions and critiques.

• Break into groups of two.
• Instructors will distribute exam questions
• (5 min.) Answer and Critique your partner's questions
  • Note that evaluating others' exam questions is equally or more important as knowing how to write them, since GSIs almost always vet exams but only contribute a few questions at most.
• Ask them to evaluate question stem and distractors.
  • Stem - Is this clearly worded? Does it concern a “major topic”?
  • Distractors - Are these worth the words they're written with? Capture common misconceptions? Clearly worded?
  • Think about where students may get tripped up, if anywhere.
  • How can they be improved?
• (10 min.) Discuss/Argue your critiques with your partner. Reach a consensus on what makes an effective test question.
• (10 min.) Discuss with entire class
• (10 min.) Critique some of Alex's “proven to be good” questions.
  • Questions should be difficult because of the concepts tested, not because of tricky wording.
  • How might an undergrad read the question. How can you get in an undergrad mindset?

Main take-aways: 1. The stem of the question should be meaningful but not wordy. (e.g., More than just “The seasons… (a)(b)©(d)(e)” but less than a paragraph about the Earth and its seasons (and then five responses).)

2. Avoid double negatives. In general, write the stem of the question in the positive. Students read negative terms and forget to reverse the logic of the relations being tested.

3. Irrelevant difficult is unnecessary. What are you testing them on? If you want to probe their knowledge of subtle points, make sure it is warranted (e.g., “Is this subtlety necessary to understand a bigger theme of the course?”)?

4. All distractors (incorrect answers) should be plausible. I really liked Therese's suggestion of having someone not in the course read your responses (but not the stem). If they can pick out the correct answer (or obvious wrong answers), then you have work to do. That said…

5. All responses should have a similar tone and length.

6. Avoid “none of the above” and “all of the above”, especially when all possibilities are listed in parts (a)–(d)!

Other questions to critique, if time:

• Now critique some past exam questions
• Use old Alex/“proven to be good” questions
  • Have the practice final from the 2007 AY C10 reader
  • Have some MC questions from the back of Slater & Addams.
  • LSCI questions: http://aer.noao.edu/auth/LSCIspring2006.pdf
  • ADT 2.0: http://solar.physics.montana.edu/aae/adt/
  • More concept inventories: http://astronomy101.jpl.nasa.gov/tips/index.cfm?teachingID=32 . (Most of these don't publicize the questions online; you have to email the author to get them.)

• Much of this is on the handout we handed out for quiz questions.
• Exams exist to:
  • Evaluate student learning for University-required grading
  • Motivate students to study and understand the material
  • Allow the instructor to evaluate his/her progress educating students about the material
  • Provide feedback to students about their understanding and study habits and illustrate specific gaps in their understanding of the material

• What makes good multiple choice exam questions?
  • Not too long/wordy (neither the question nor the possible answers)
  • Not too much calculation
  • Not too tricky (i.e. there shouldn't be two extremely similar answers)
  • Relevant to important material (as opposed to really obscure/minute details)
  • Very clearly written, precise wording in both question and answers
    • E.g., 'Which best describes…' as opposed to 'What is…' or 'How does…happen'
  • Pedagogical as well as evaluative (e.g., some questions should probe common misconceptions)
  • Questions that are very easy or very difficult are OK as long as the test has questions with a variety of difficulties

• What makes bad ones?
  • Long answers!
  • Excessive use of 'all of the above' (some people say any use of 'all/none of the above' is a bad thing)
  • Questions that can be solved without knowledge of the material (usually because of the use of too many blatantly wrong or “funny” possible answers)
  • Multiple potentially correct answers (usually from vague questions or possible answers)

• Non-multiple choice questions (very similar to quizzes)
  • Types of questions:
    • Matching
    • Fill-in-the-table/blank
    • Simple calculations
    • Short answers and paragraph/free responses
    • Diagrams, plots, graphs
  • Most of the same points discussed above, and for quizzes, apply here:
    • Questions should be clear, easy to read, and unambiguous
    • Questions should be relevant to the material presented and emphasized (do not test on obscure passages of the textbook)
    • For high-value questions, allow for partial credit
    • Make the questions easy to grade! Don't give students the opportunity to 'core-dump' for a problem: be very specific about what you're looking for in free response type questions.
  • Can be be slightly more time-consuming than quiz questions, depending on the overall length of the exam.