AY 375 - Fall 2018: Fourth Day Lesson Plan

1. Deeper conceptual knowledge CAN be probed with multiple choice questions, but writing effective questions takes time and practice.
2. Especially when writing free response questions, it can be useful to develop a grading rubric for each question as a way of ensuring that your questions are specific, clear, and not testing the same concept over and over again.

(5 minutes) Recap and Why Group Work?

1. Group work appeals to many learning styles. Group work provides a sense of shared purpose that can increase motivation.
2. Group work introduces students to the insights and values of their peers.
3. Life after college will involve group work.
4. Listening to lecture and taking notes will carry the students only so far in their development. Learning cannot be passive. Group work engages students in the learning and thinking process.
5. We (as college instructors) should be encouraging and developing students' ability to do higher-order thinking.

(5 minutes) Every person/pair is given a different type of group work. Pairs predict what the group work entails and then discusses the advantages/disadvantages of the type once they read its description.

(10 minutes) Jigsaw Group Work

(5 minutes) Tips for Group Work

1. Be sure to introduce the activity with crystal clear instructions. Ambiguity leads to either poor group work or individuals going off and doing their thing.
2. The quality of the group work depends sensitively on the activity and questions asked. We encourage open-ended questions and questions that actually involve group discussion. The focus on problem solving results in individual working; new strategies are needed for this, like:
   1. Only hand out one worksheet per group.
   2. Have the students write their answers on a large sheet of paper, work entirely at one of the whiteboards, or have some sort of whiteboard at each table.
   3. Anything else?
3. Good group work activities take time, often more time than just lecturing. However, the added work results in added gains for the students.
4. A “Q&A” part of section can involve a lot of peer learning, if you get good at enabling the students to answer each other's questions. This requires more sophistication than just asking the smartest student to say the right answer; you have to ask the question in a way such that all of the students have a chance to grapple with the question initially posed.
5. “I paid all this $$ to be taught by professors and graduate students, not listen to classmates who don't know as much.” Let students know the benefits of group work. They will resist at first, but proper use of group work will show the students they are learning just as much (usually more) than if you were lecturing.
6. “Students don't like working in groups.” Students are used to working individually. Or students might fear that some group members will not pull their weight. Again, explaining the rational for group work is key, as well as providing checks for students who do not contribute.
7. Get feedback often.

Suggestions to your students (adapted from McKeachie):

1. Be sure everyone contributes to discussions and to tasks. You each have something unique to contribute. Know that you both have something to learn from others and to teach others.
2. Don't jump to conclusions too quickly. Be sure that minority ideas are considered.
3. Don't assume consensus because no one has opposed an idea of offered an alternative. Check agreement with each group member verbally, not just by a vote.
4. Set goals—immediate, intermediate, and long-term—but don't be afraid to change them as you progress. (These should be obvious in sections.)
5. For bigger multi-part tasks: Allocate tasks to be done. Be sure that each person knows what he or she is to do. Check this before beginning.

Types of Group Work

1. The Interactive Lecture
   • The “biggest” form of group work, where the whole class works as one big group (and you're a group member).
   • Can be used with worksheets and/or demos.
   • Continuously call on a variety of students to explain answers. If you don't want to call on individual students (cold calling), you might call on particular groups (“This group, what do you think?”)
   • This style tends to keep all students engaged and on task.
2. Concept Mapping
   • A concept map illustrates the connection between terms, ideas, or concepts, which creates higher-level learning. Concepts and terms are written in bubbles and lines are drawn connecting related concepts. With each line, the relation is identified.
   • Students in groups can be given a partially completed concept map and a list of terms that they need to fill in the blanks with. Alternatively, students can work at filling in the relations between various concepts.
3. Jigsaw Projects
   • Each group contributes to a specific part of the assignment. When members have completed their task, all groups shuffle so that one person from each original group is in each new group. Each person then shares their answer and explanation with the rest of the group.
   • Requires EVERY person in section to be responsible for knowing why the answer is what it is.
   • Good way of covering an entire worksheet worth of questions in a short amount of time.
   • Make sure you assess that groups understand why their answer is what it is.
4. KWL
   • Stands for “what I Know, what I Want to know, and what I Learned. It happens in three parts.
     • (Part 1) To introduce a new topic, have the students list what they know about the topic before you start discussion. Collect these lists or have them share.
     • (Part 2) Using these lists, you can modify the remainder of the section to address misconceptions and erroneous understanding. Run section employing whatever demos, activities, etc. you want to use.
     • (Part 3) At the end of the unit, have students list what they have learned. You might ask them to identify the three most important concepts, answer some questions, or just free-write. Collect these lists or have them share.
5. Choreographed Group Tasks
   • Example: Suppose you hand out a worksheet that has six questions. Go through the first two questions of a worksheet on the board (with varying amounts of feedback from students). Then have students work on the next two questions (which are similar but different to the earlier questions) in groups. Then have the class explain to you how to solve question 5 (and do so on the board). Ask for a volunteer to do question 6.
   • Students learn in different ways, so variety is a good thing.
6. Activity Stations
   • Break the class into a few stations (3 is ideal), where one station deals with one aspect of what you want to cover in section that day.
   • Have the class break into groups of three, one for each station.
   • Each group spends 15 minutes at each station, then rotates.
   • Each station could be either a demo, hands-on activity, some worksheet questions.
   • Instructor must be very careful with timing so they can make it around to each group every 15 minutes to assess.
7. Open-ended Questions / Case Studies (e.g., Think Like an Astronomer)
   • One thing that makes discussion difficult in science courses is that most of our questions have a single “correct” answer. Asking open-ended questions can encourage students to think about how concepts fit together.
   • Requires that the question is at the appropriate level of the class. Takes more time to prepare.
   • Anything that allows for interpretation is ripe for discussion.
   • e.g., You have a sealed box (of doughnuts) in the front of the class. Have students in groups device experiments to determine what is in the box (without opening it). Then tie this into how astronomers might detect dark matter, etc.
   • e.g., Give each group a budget and a catalog that includes costs of telescopes, mirrors, equipment, launching into space, etc. Have them come up with a plan to build a telescope at some particular wavelength (having to weigh whether it is in space, what resolution it will have, etc.).
   • e.g., Have students reproduce the thought process of famous astronomers and scientists (e.g., Hubble's discover of other galaxies and the realization of the size of the universe).
8. Send-A-Problem
   • Have each group try to solve a different problem related to material covered in section/lecture.
   • Each group them gives their problem and suggested solution to a different group, which then evaluates the solution and offers corrections.
   • That group then gives their altered solution to another group, who provides the final evaluation.
   • Good for lengthy 7a/7b type problems or problems involving multiple steps. Has groups practice group thinking and comparing/discriminating among multiple solutions.

1. (5 min) Individually, come up with BOTH a multiple choice and free response question for the class you're teaching for and write them down on a piece of paper. Remember to think about what learning objectives they test and what level(s) of learning they probe.
2. (5 min) Trade your questions with a partner. Have them attempt to answer the questions or at least determine what learning objectives were being probed, what level of learning it targeted, and what would have constituted as an “acceptable answer.” Once you have gone over each other's questions, discuss in your small groups what you liked about the question and offer improvements to flesh it out further. Some questions to consider:
   1. For both:
      • Is the wording clear?
      • What concepts are being tested?
   2. For the multiple choice:
      • Is there clearly only one correct answer?
      • Are there any obvious throw-away answers?
      • Are you able to rule out any response because of the wording alone (i.e., are there hints in the structure)?
   3. For the free response:
      • Does part B test the small conceptual/procedural knowledge as part A?
      • What if students cannot solve part A? What does that imply for part B?
      • What sort of responses might students give under the pressures of an exam setting?
3. (5 min) Come back as a class and discuss.
   • Did you learn anything surprising?
   • Is this easy? (Unfortunately, NO!)
   • What part of question writing did you find the most difficult?
4. (25 min) Go through question example slides as a class exercise.
   • MC Summary
     • Test what you teach and teach what you test!
     • Write short, clear questions and solutions.
     • All answers should be of a similar tone and length.
     • Avoid throw-aways, double negatives, etc.
     • Be sure to not suggestively word your responses.
     • Exams should have a variety of difficult and easy questions. Some easier questions at the start of the exam can enhance motivation.
   • FR Summary
     • Test what you teach and teach what you test!
     • Write clear prompts. Be explicit about what you want students to provide (no core dumps).
     • Multiple parts should test multiple ideas, not the same idea again and again.
     • Solutions should require novel ideas, not a summary of material in the prompt.
   • Reminder about timing: always take your own quiz/test/exams. Your students will take 2-3x longer than you will.

Some notes on multiple choice questions:

Despite their outward appearance, these questions are actually inherently nonobjective. Grading an essay exam is subjective to the personal feelings of the grader, compared to running a Scantron through a machine. Grading written problems falls somewhere between the two. This is only partially correct: “grading” a Scantron is completely objective, but the subjective aspect of multiple choice questions comes in the creation of the item (the question), the response (the correct answer), and the distractors (the incorrect choices). If everyone in the class was to write a question about the Doppler Effect, we would see a range of different questions and a range of ideas probed. That is subjective.

The ultimate goal of testing is to measure what the students actually understand, and the process of interpreting the meaning of a student's response to a MC test is a subjective one. There are three major issues behind writing these sort of questions:

1. the physical format and layout of the question
2. the conceptual hierarchy of the questions
3. the statistical item analysis

Below are some guidelines for each of these items.

You are testing the students' understanding, not their reading ability. Long passages of text cause slow readers to skim and often miss details. Questions should not include strings of prepositions, parenthetical statements, or extended clarifications. Misinterpretation is impossible to completely predict, but concise, clear questions can do a lot to minimize the chance of students misreading the question or the response. For example,

You forget that the star Betelgeuse is a red giant and apply the method of magnitudes to determine its distance. The true distance to Betelgeuse is actually… (a) shorter than you calculated, (b) the same distance that you previously calculated, © farther than you calculated.

Is short, to the point, and clear. You might be tempted to elaborate on small points that are not the main conceptual item that is being tested, but care must be taken. For example,

You forget that the star Betelgeuse is a red giant (a very luminous star in the top right of the HR diagram with relatively low surface temperature) and apply the method of spectroscopic parallax—a comparison of the star's apparent magnitude, estimated from the HR diagram, and its absolute magnitude—to determine its distance from Earth, which can be considered the same as its distance to the Sun because the Earth-Sun distance is negligible given the scales involved. The true distance from Earth to Betelgeuse is actually…

In an attempt to be completely clear, the stem has become more difficult for most students to understand!

Over the years, students have learned that when novice faculty include choices such as “all of the above”, these are frequently the correct answer. It's easier as a test writer to write correct statements than to come up with plausible sounding incorrect statements. Also, students have also learned that the longer answers are usually the correct answers. You can avoid these situations by making sure your choices are all of similar length, contain a similar amount of scientific vocabulary, and ensure that an equal number of choices (A), (B), etc. are correct on the overall test.

Consider

The thermo nuclear reactions in a stellar core are the result of (A) fission, (B) fusion.

This tests what? Unfortunately, it tests word association. A student can get by without knowing what fusion means. Students learn to adopt a strategy of memorizing definitions and words, rather than understanding concepts. As another example,

The Monotillation of Traxoline (attributed to Judy Lanier) It is very important that you learn about traxoline. Traxoline is a new form of zionter. It is monotilled in Ceristanna. The Ceristannians gristerlate large amounts of fevon and then bracter it to quasel traxoline. Traxoline may well be one of our most lukized snezlaus in the future because of our zionter lescelidge.

Directions: Answer the following questions in complete sentences. Be sure to use your best handwriting. (1.) What is traxoline? (2.) Where is traxoline monotilled? (3.) How is traxoline quaselled? (4.) Why is it important to know about traxoline?

Notice how easy it is to get 100% without understanding a single thing about the passage! Pay attention to wording.

Less applicable for GSIs, but if particular questions are frequently missed, it is necessary to probe whether they are missed because of a lack of clarity, or because the question is testing difficult concepts. If the former, revise!

See below for some general bullet points on what to look out for.

Some notes on free response questions:

These notes are written with “Astro C10 quizzes” in mind, but the ideas are generally applicable.

• These points are valid for both quizzes and exams:
  • Test the material emphasized - Exams should reflect the fact that students should know the big concepts really well, as opposed to knowing a bunch of smaller concepts only peripherally.
  • Keep questions short and to the point - Students should spend the majority of their test time thinking and answering/writing, NOT reading.
  • Edit questions for clarity - Clear questions tend to be shorter. If anything is ambiguous, it confuses and slows down students and makes it harder for you to grade it fairly. Have someone else take your exam to give some feedback. If you're taking an exam for someone else, be critical and think about possible ambiguities.
  • Don't write a long test - Keep it concise, to the point, and clear! The rule of thumb is your average student will take double or triple the time it takes a GSI to complete the exam. Also, 90% of your students should finish the exam completely in the allotted time.
• Quizzes vs. Homeworks
  • They're quite similar in their construction and type of questions.
  • The main difference is that quizzes should be shorter and have easier questions, since students have much less time to work on quizzes and must work on them alone.
• Quizzes are:
  • short
  • usually given in section
  • questions are of exam difficulty (i.e. easier than homework questions)
  • Not too in-depth or calculation-based. Some light calculation might be OK, but take care. (Not everyone will remember a calculator no matter how many times you remind them, and many people will be petrified of the idea of computing things on their own.)
  • usually only cover recent material (i.e. the past 2-3 weeks)
• Quizzes are used:
  • by the GSI to gauge each students' understanding of recent material individually (as opposed to homeworks which can be done in groups)
  • by the student to gauge their own understanding of recent material and get a feel for what a college level intro science course non-Scantron exam will look like and what level of understanding they are expected to have for the exams
  • by the prof to get a grade early in the semester that's more important than a single homework, but isn't the big production that an exam is (you might not have covered enough material for a full exam)
• What makes good ones?
  • not too long (both in length of individual questions and number of questions)
  • not too hard
  • not too easy (shoot for a variety of difficulties in questions)
  • relevant to recent material
  • varied in the types of questions (multiple choice, fill in the blank, calculation/mathematical, read a graph, free/paragraph response)
  • unambiguous with easy-to-read questions
  • not mathematically demanding - questions should probably not require a calculator and should definitely not include extensive tedious calculations
  • representative of same knowledge required for exams
  • gradable for partial credit (not simply binary right/wrong like Scantron exams)
  • specific about what you're looking for in free response type questions: Don't give students the opportunity to 'core-dump' for a problem, it wastes their time spewing forth useless information and makes your life tougher when you have to grade the mess.
  • quick to grade (this makes your life much easier and helps the grading be more fair for all of your students)
• Quizzes are meant to be relatively low stress (especially compared to full exams)

General notes from previous years in convenient bullet-point form:

• Exams exist to:
  • Evaluate student learning for University-required grading
  • Motivate students to study and understand the material
  • Allow the instructor to evaluate his/her progress educating students about the material
  • Provide feedback to students about their understanding and study habits and illustrate specific gaps in their understanding of the material

• What makes good multiple choice exam questions?
  • Not too long/wordy (neither the question nor the possible answers)
  • Not too much calculation
  • Not too tricky (i.e. there shouldn't be two extremely similar answers)
  • Relevant to important material (as opposed to really obscure/minute details)
  • Very clearly written, precise wording in both question and answers
    • E.g., 'Which best describes…' as opposed to 'What is…' or 'How does…happen'
  • Pedagogical as well as evaluative (e.g., some questions should probe common misconceptions)
  • Questions that are very easy or very difficult are OK as long as the test has questions with a variety of difficulties

• What makes bad ones?
  • Long answers!
  • Excessive use of 'all of the above' (some people say any use of 'all/none of the above' is a bad thing)
  • Questions that can be solved without knowledge of the material (usually because of the use of too many blatantly wrong or “funny” possible answers)
  • Multiple potentially correct answers (usually from vague questions or possible answers)

• Non-multiple choice questions
  • Types of questions:
    • Matching
    • Fill-in-the-table/blank
    • Simple calculations
    • Short answers and paragraph/free responses
    • Diagrams, plots, graphs
  • Most of the same points discussed above apply here:
    • Questions should be clear, easy to read, and unambiguous
    • Questions should be relevant to the material presented and emphasized (do not test on obscure passages of the textbook)
    • For high-value questions, allow for partial credit
    • Make the questions easy to grade! Don't give students the opportunity to 'core-dump' for a problem: be very specific about what you're looking for in these questions.
  • Can be more time-consuming than MC questions, depending on the overall length of the exam.

1. Peer visits are assigned. Meet up for discussion with both the person who visited you and the person you visited by 9/19. Bring a completed Peer Visitation Worksheet to class on 9/19.
2. Draft a full length quiz (20 minutes in length, 50 total points) for the class you are teaching. Bring TWO copies to class next week.