AY 300 - Fall 2012: Fifth Day Lesson Plan

Today's class will discuss writing effective test questions, designing worksheets, and begin board work exercises.

• Review how to write effective exam questions.
• Continue practicing good boardwork technique.

The person on call for this week will share their section experience. Some questions include:

• What did you do?
• How did you implement your activities?
• What worked?
• What didn't work?
• What would you do differently?
• How did you assess learning?
• Did you receive any unexpected questions/reactions/etc.?
• Did anything unexpected happen?
• What were you thinking about while you were running section? Any moments of panic?

During this (and after), open the floor up for general questions and sharing about how sections are going.

Part 1 (15 min)

• Briefly discuss Bloom's taxonomy of educational objectives Bloom's Taxonomy
• Have students look over the questions on the phases of the moon and discuss which objective they test Effective multiple choice questions

Part 2 (25 min)

• Have everyone take 5 minutes to write an exam question for their class
• Break into groups of two.
• Instructors will distribute exam questions
• (5 min.) Answer and Critique your partner's questions
  • Note that evaluating others' exam questions is equally or more important as knowing how to write them, since GSIs almost always vet exams but only contribute a few questions at most.
• Ask them to evaluate question stem and distractors.
  • Stem - Is this clearly worded? Does it concern a “major topic”?
  • Distractors - Are these worth the words they're written with? Capture common misconceptions? Clearly worded?
  • Think about where students may get tripped up, if anywhere.
  • How can they be improved?
• (5 min.) Discuss/Argue your critiques with your partner. Reach a consensus on what makes an effective test question.
• (10 min.) Discuss with entire class; cover points in effective multiple choice questions slides.

Main take-aways: 1. The stem of the question should be meaningful but not wordy. (e.g., More than just “The seasons… (a)(b)©(d)(e)” but less than a paragraph about the Earth and its seasons (and then five responses).)

2. Avoid double negatives. In general, write the stem of the question in the positive. Students read negative terms and forget to reverse the logic of the relations being tested.

3. Irrelevant difficult is unnecessary. What are you testing them on? If you want to probe their knowledge of subtle points, make sure it is warranted (e.g., “Is this subtlety necessary to understand a bigger theme of the course?”)?

4. All distractors (incorrect answers) should be plausible. I really liked Therese's suggestion of having someone not in the course read your responses (but not the stem). If they can pick out the correct answer (or obvious wrong answers), then you have work to do. That said…

5. All responses should have a similar tone and length.

6. Avoid “none of the above” and “all of the above”, especially when all possibilities are listed in parts (a)–(d)!

Other questions to critique, if time:

• Now critique some past exam questions
• Use old Alex/“proven to be good” questions
  • Have the practice final from the 2007 AY C10 reader
  • Have some MC questions from the back of Slater & Addams.
  • LSCI questions: http://aer.noao.edu/auth/LSCIspring2006.pdf
  • ADT 2.0: http://solar.physics.montana.edu/aae/adt/
  • More concept inventories: http://astronomy101.jpl.nasa.gov/tips/index.cfm?teachingID=32 . (Most of these don't publicize the questions online; you have to email the author to get them.)

• Much of this is on the handout we handed out for quiz questions.
• Exams exist to:
  • Evaluate student learning for University-required grading
  • Motivate students to study and understand the material
  • Allow the instructor to evaluate his/her progress educating students about the material
  • Provide feedback to students about their understanding and study habits and illustrate specific gaps in their understanding of the material

• What makes good multiple choice exam questions?
  • Not too long/wordy (neither the question nor the possible answers)
  • Not too much calculation
  • Not too tricky (i.e. there shouldn't be two extremely similar answers)
  • Relevant to important material (as opposed to really obscure/minute details)
  • Very clearly written, precise wording in both question and answers
    • E.g., 'Which best describes…' as opposed to 'What is…' or 'How does…happen'
  • Pedagogical as well as evaluative (e.g., some questions should probe common misconceptions)
  • Questions that are very easy or very difficult are OK as long as the test has questions with a variety of difficulties

• What makes bad ones?
  • Long answers!
  • Excessive use of 'all of the above' (some people say any use of 'all/none of the above' is a bad thing)
  • Questions that can be solved without knowledge of the material (usually because of the use of too many blatantly wrong or “funny” possible answers)
  • Multiple potentially correct answers (usually from vague questions or possible answers)

• Non-multiple choice questions (very similar to quizzes)
  • Types of questions:
    • Matching
    • Fill-in-the-table/blank
    • Simple calculations
    • Short answers and paragraph/free responses
    • Diagrams, plots, graphs
  • Most of the same points discussed above, and for quizzes, apply here:
    • Questions should be clear, easy to read, and unambiguous
    • Questions should be relevant to the material presented and emphasized (do not test on obscure passages of the textbook)
    • For high-value questions, allow for partial credit
    • Make the questions easy to grade! Don't give students the opportunity to 'core-dump' for a problem: be very specific about what you're looking for in free response type questions.
  • Can be be slightly more time-consuming than quiz questions, depending on the overall length of the exam.

So if group work is so great, how do you facilitate it? Well made worksheets are a good way. Here are some examples:

• Planet Wars
• In Soviet Russia, Astronomy Does You!
• The Celestial Sphere: Lost at Sea

Each small group will be given a different worksheet. Discuss what makes the worksheet effective for stimulating group work, what could be improved upon, etc. We will share with the class. In the (perhaps near) future, we will return to worksheets and discuss ways of writing your own.

Some info for you:

Remind everyone that they're required to contribute their “best original worksheet” (or a significantly improved one) to the EBRB.

• Explore the topics discussed in lecture, focusing on the most important points. The aim is to make the students use an idea after merely having it told to them.
• allow students to examine their understanding by doing sample questions - but without the stress of being graded and with the assistance of groupmates
• serve as a reference after the student leaves the classroom
• Beyond the above, they can also:
  • explore new topics not covered in lecture in detail that are interesting, cutting edge or fun!
  • serve as a fill-in-the-blank 'note sheet' for a more lecture-style section
  • simply provide an incentive to experiment with a particularly involved demo activity

Worksheets are intended to get students collaborating in groups, which has been proven a great way to get them to learn the material in depth. How do we get students collaborating the way we want?

• Students need to believe that their work is going to be evaluated in some way – not necessarily in a grade, but that somehow the effort they put into the sheet (or fail to put into it) will be considered.
• You need to use good worksheets! Easier said than done, of course.
• Mixing up the makeup (and possibly the size) of the groups in your section can help change things up and get students to concentrate on the worksheets and not get as off-topic by talking to their usual groupmates (i.e. friends).
• You should stress early on in the semester (basically from the first time you do any kind of worksheet) that the information in the worksheets is often a restatement of material from lecture and some of it might be on a test (the magic phrase).
• In fact, if the information is important enough to warrant lecture-time and a worksheet in section, it must be important.
• Also stress the fact that section and worksheets and demos are supposed to the more fun, hands-on aspect of the class (as compared to lecture).

• Interesting/important topic (everyday experiences are always good, as are topics that students are always interested in: Mars, aliens, comets/meteors, Big Bang, black holes, etc.)
• A variety of types of questions
  • Answer format: multiple choice, fill in the blank, calculation, read a graph, free/paragraph response, plot data, draw a schematic, “discuss with your groupmates”, etc.
  • Presentation/group format: individuals, group, as a class, demo-related, etc.
  • Applies both within a worksheet and week-to-week
• Active participation (get students up and moving and talking, writing on the boards, presenting to the class, etc.)
• Promoting the idea of group activity (e.g., difficult or tedious for a single student to do, but can be naturally broken into independent components for different individuals)
• Separate activities/examples for separate groups to compare/contrast at the end of section (e.g. build up an H-R diagram)
• Be visually engaging: make your worksheets look nice (include pictures, picture captions, fill-in-the-blanks, etc. but don't make them too cluttered)
  • On just about every one of Dan Perley's worksheets he includes a nerdy, science-related comic.
• Relate concepts to commonsense notions and everyday experiences (if possible)
• Use of real or realistic data and images, instead of clearly fake examples, as long as it doesn't excessively complicate the activity
• Symbols are defined and used consistently with elsewhere in the course
• Be relevant to homework/exam questions (so students feel it was worthwhile to them)
• Be creative. Non-standard worksheets can work well. Try things that aren't discussed here - and let us know if they work!

• Too much calculation (many students will not bring calculators to section, even if you remind them constantly, and some will not be able to do exponents or scientific notation!)
• Too many mundane tasks (students will quickly lose interest if the questions are just “do this calculation, then this one, then this one” without ever seeing the big picture until the end or not at all)
• Covering esoteric or “boring” topics or topics that are too advanced to do much with at the Ay 10 level
• Too long: students can't finish the critical parts by the end of section and it's difficult to interrupt them to review in the last 10 minutes. If students learn that their worksheets won't be gone over every single time, they'll lose the motivation to actually do them
• Too short: some groups finish the activity and become bored (as a precaution, consider adding extra 'bonus questions' at the end you may or may not cover)
  • Answers to bonus questions can be posted on your website or given out via email or at office hours or TALC if you'd like.
• Sloppy wording (however, the standards are much less strict than for exams, quizzes, or homeworks)
• Rehashed, basic, dull questions more appropriate for homeworks or exams: take advantage of the group format – you can push your students' abilities since there are many of them working together on the worksheet!
• Not relevant: doesn't build understanding of the most important subjects – which, let's not kid ourselves, are “the things that will be on the final.” (If you think that the questions on the final are not addressing the most important subjects in the course, the thing to do is fix the final.)

• Divide into groups, hand out a sample worksheet to each group. Groups go over worksheets, present to rest of class:
  • Summarize content of worksheet
  • What they think its good and bad points are. Remember the metrics: will this get students talking to one another? Do you think it's the right difficulty? Will there be a good way to review the answers?
  • Hand out copies of the worksheet in question to the other groups while it's being discussed.

• Make sure you're intimately familiar with the worksheet before section. Work everything out yourself (and make a key).
• Use caution if simply printing old EBRB worksheets. They may refer to obsolete parts of the text, may have subtle errors, or the questions might be slightly different than you remember from just giving it a quick read. If using a worksheet you didn't write yourself, always work it out completely to be sure it's bug-free.
• Give some background, not just questions. Let them know why the activity is interesting and/or relevant to them. (Of course, you can also just to this verbally.)
• Start with a really easy and obvious question (and preferably an interesting one). Don't let the students get stuck immediately and stop caring. If they're led into starting the worksheet with a simple question or two, they become 'invested' in the activity and are more likely to stick with it.
• Leaving minimal empty space on the worksheet encourages use of the board. However, it also makes it harder for students to write their answers down and discourages individual participation.
• For large sections, give plenty of written guidance for how to solve the problems (you can't help everyone at once). For small sections, it's possible to go as far as completely omitting the directions.
• Always save time to go through the worksheet as a class! Each member of the class need to have some feedback on their answers (even if it's just you quickly running through the problems) or they will feel the activity petered out into meaninglessness. Be firm about ending the activity on time.
• The main points:
  • Engagement - Encourage students to engage actively and look at the material in new ways different from how it was presented in lecture
  • Relevance - Students want more than just an enrichment activity: be sure it's really benefiting them
  • Familiarity - Customize your worksheets to your section; as with just about everything in section, put your own personality into it
  • Fun - Worksheets are the single biggest outlet for creativity that you have as a GSI; have fun with it and you'll almost certainly get you students to actually want to do the worksheet.