Wednesday, August 26, 2015

Top 10 stupid mistakes in design of Multiple Choice questions

Some online learning designers write their assessment items first, as they can match the assessment to the objectives or competences before being distracted by the detail in content. This avoids the trap of writing test items that simply test atomic facts and words from the presented text.
Many test items are quite simply not fit for purpose, as they don’t really assess, test the wrong thing or are so badly designed that they mislead or annoy learners. Bad test item writers tend to simply pluck out key words from the text and test the recognition or the meaning of those terms. Good test item writers, and this is a skill that few subject matter experts possess, understand good design and the need to test deeper understanding.
Here’s a few rules to follow if you’re new to this game:
1. Test understanding, not facts
What’s easy is to simply extract all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested. What’s hard is to write questions that really do test the learner’s actual knowledge and ability to apply that knowledge. Professor Dylan Wiliam, calls great questions ‘hinge questions’, as they literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly but the principle applies just as well to online learning. Here’s an example:

The ball sitting on a table is not moving. It’s not moving because:
A. No forces are pushing or pulling on the ball.
B. Gravity is pulling down, but the table is in the way.
C. The table pushes up with the same force that gravity pulls down.
D. Gravity is holding it on to the table.
E. There’s a force inside the ball keeping it from rolling off the table.
This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’ and those who have not. There are forces at play, two in fact, so C is correct. The other options introduce common misconceptions about the absence of forces in stationary objects (A), single forces (B & D) or inner forces (E).
There are four components that make great questions:
Make them THINK
Give helpful FEEDBACK
A really bad question will allow the learner to guess the right answer or simply test recognition from a list. A good question will make the learner think, a great question will make them look away, even close their eyes to recall what they know, do something in their head and move towards the answer. It will push the learner.
Let’s try this:
A bat and ball cost £1.10.
The bat costs one pound more than the ball.
How much does the ball cost?
This question really is a test of a learner’s grasp of ‘ratios’ in maths, it does a great job, as it really requires you to know how to apply a principle in mathematical thinking. The answer is not 10p as many choose. Think about it. If the ball was 10p, the bat, if it were a pound more, would be £1.10p, making a total of £1.20. The right answer is 5p. If the ball was 5p, the bat, if it were a pound more, would be £1.05p, making a total of £1.10. That’s a great question.
Remember also, that a great question also needs great feedback. In fact, it is better to call these, not questions, but ‘test items’ as you need, not only the question but the feedback. So let’s think about explanatory feedback for this question, that allows the learner to learn from the question.
A bat and ball cost £1.10.
The bat costs one pound more than the ball.
How much does the ball cost?
20p    ->  Try again.
15p    ->  Try again.   
10p    ->  Try again.
5p      ->  Correct.
This doesn’t really give any help to the learner who has jumped to the wrong conclusion or hasn’t thought deeply enough about the question. We need to provide HELPFUL feedback.
A bat and ball cost £1.10.
The bat costs one pound more than the ball.
How much does the ball cost?
20p    ->  20p+120p=140p  Try again.
15p    ->  15p+115p=130p  Try again.   
10p    ->  10p+110p=120p  Try again.
5p      ->  Correct. 5p+105p=110p
The feedback here tries to unpack the nature of the problem for the learner and gives an explanation as to why they are wrong.
Now let’s add another touch, that human element. Make it CONVERSATIONAL.  In conversation, you wouldn’t ask a question like this and simply feedback the statement ‘10p+110p=120p Try again.’ You’d make it a bit more conversational and friendly.
20p    ->  If  the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Try again.
15p    ->  If  the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Try again.
10p    ->  If  the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Try again.
5p    ->  Correct. If  the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p
Now be a little more POSITIVE:
20p    ->  Sorry. Most people get this wrong on first attempt. Let’s see why this is wrong. If the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
15p    ->  Sorry. Most people get this wrong on first attempt. If  the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
10p    ->  Most people get this wrong first time around and choose this answer. If  the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.
5p    ->  Well done. If the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p.
You get the idea. Make test items meaningful, not mundane, and be helpful, fulsome and conversational in the feedback. This type of approach to test items will make the learner feel good and propel them forward, rather than act as barriers or make them feel like failures.
2. Binary Choice language
We don’t ask each other questions in normal language using the terms True and false, so consider more natural terms such as Yes/No, Agree/Disagree
Margaret Thatcher was the only British woman Prime Minister.
􏰁 Yes 􏰁 No      or       􏰁 Agree  􏰁 Disagree
3. Binary choice mistakes
Binary choice questions are often too long or turn into complex tests of logic by having two or more ideas expressed. 
Using NOT in a binary choice simply leads to a logic test, so best avoided. Finally, and I have seen this, asking the user to try again is stupid. 

4.  Edit out unnecessary words in options
How many earths would fit into the sun?
approximately 1000
approximately 10,000
approximately 100,000
approximately 1,000,000
This is a good question, a real test of your knowledge of the relative sizes of the earth and sun. However, take out the repeated word of each of the options, as there is no need to make the learner go to that extra effort in reading the same word five times.
Approximately how many earths would fit into the sun?
And with meaningful feedback….
1000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
10,000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
100,000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.
1,000,000 ->  That’s right. In fact, you could fit 1.3 million Earths inside the Sun! It’s huge.
5. Make question grammatically agree with all answers
In golf, a one under par is a:



The word ‘a’ agrees with two of the options but not the other two. It’s grammatically wrong, as well as almost giving away the right answer!
What do you call one under par in golf?


6. Avoid negative questions
Avoid having NOT in the question, (Which of the following is not…..) unless they really are testing a competence than involves a common error or misconception. The ‘not’ can be easily missed by the learner, so capitalize if used. These questions often simply reinforce irrelevant negative information, rather than testing much that is useful.
7. Avoid making the longest option the answer
Go through the assessment items and click on just the longest option. You’d be surprised to see how often these are weighted towards the correct answer, as designers often write this first, then come up with easier, shorter distractors. Poundstone showed that this was common, even in high stakes tests. Options must be roughly the same length, of similar grammar and random in order of presentation.
8. Avoid ‘All of the above’ and ‘None of the above’
These are often imprecise tests and you don’t know what actually misled the learner if they get it wrong. They are also easy to write, so there is a tendency for poor test writers to have these as their ‘right’ answer, a fact shown by Poundstone’s analysis of thousands of test items.
9. All options should be believable.
Answers that look obviously wrong are pointless or even worse, patronizing and condescending. These are two questions from BBC Bitesize:
What is magma?
A chocolate ice cream
Molten rock
Bubbles of gas

What are the tiny air sacs in the lungs called?
I rest my case. A lesser but similar error is to make one or more of the options too obviously right or wrong.
10. Test your test items
First test your test items with an expert, namely an experienced test-item writer. It is a skill that takes some time to master. Just because you know your subject, does not mean that you know how to assess and write good test items. Second, try the questions out with real target learners and ask them to ‘voice’ what they’re thinking. They invariable say things like ‘I don’t understand this question… What do you mean by…”. Don’t argue, just change it.
Questions encourage, not just the acquisition of learning but also critical thinking. Questions stimulate curiosity. Questions can intrigue and pull the learner towards exploring a subject in more detail. Good questions can be learning experiences in themselves not just test items. They can be used to stimulate thought but also, through good feedback, explain to the learner why they have a misconception, even act as a way of getting a point across. Questions drive learners forward. Good questions diagnose strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know. Finally, questions can assess and determine whether a learner is competent against certain criteria.
Questions really do matter in learning, that’s why it is important to design them so that they are fit for purpose. Professional examination bodies all too frequently write questions that are poorly designed and in some cases, unbelievably, impossible to answer. Experts in their subjects, often write poor test items, as it is difficult to put yourself in the shoes of a novice and assessment skills are different from subject matter expertise. In online learning, even professional vendors often produce badly designed test items, due to a lack of interactive design expertise. In many ways the test items are more important than the content.

 Subscribe to RSS

Friday, August 14, 2015

7 ways women are ignored in education from Jane Roland Martin

Jane Roland Martin is a remarkable woman as she widened up educational theory in line with those such as Dewey and Illich. yet her real contribution is in the many issues around women in education. Her views are inclusive but above all expansive.
1. Women ignored
She saw the role of women as a largely unexamined issue in education. Women had for centuries been excluded from access to education but even recently, ignored in teaching, schooling, higher education.
2, New sources
As women have been denied access to education, excluded from education, educational institutions and the vehicles of educational theorising, she recommends that we must look to wider sources for the views, in magazines, pamphlets, more general literature and the wider media landscape.
3. The educational ideal ignores women
The definition of education itself, Martin claims, has neglected women. Work by women and work about women are often ignored, with the role of women in education hugely under researched and understood. This, she thinks, distorts and misrepresents reality. The very idea of an educated person or educational ideal must take women into account. 
4. Spectator knowledge
Martin excavated, in an anthropological manner, hidden assumptions and problems within education, especially the curriculum of fixed cultural subjects and knowledge, as well as male cognitive perspectives, at the expense of other disciplines and real-world knowledge and skills. She saw curricula and institutions, that focus largely on ‘spectator’ knowledge, as promoting a one-sided view of education. In particular she came to focus on gender through feminism.
5. Education to include domestic context
For Martin, the school as home should be far more relevant to life outside of school, especially the domestic environment of the home. It needs to draw back from a curriculum that focuses largely on ‘spectator’ knowledge. The role of mothering, issues around the ghettoisation of women on professions such as teaching and nursing, as well as the insensitivity to women’s often difficult role at work and at home, led her to a radical stance on taking action across a broad front to rebalance the educational system, in response to this analysis. 
6. Multiple Educational Agency
Yet the education debate so often focuses solely on schools and schooling. Dewey and many others have downplayed the role of school in the development of young people. For Martin this narrow and obsessive emphasis on schools and schooling may result in cultural miseducation. Parents, family, community, church, youth groups, sports clubs, the media, internet and other institutions, even the education of animals, have played roles in education long before schools and schooling were common. Martin has an expansive view of learning as a process of Multiple Educational Agency. To focus just on schools, which are notoriously difficult to change, is to ignore the very many other opportunities to educate
7. Learning happens everywhere
Above all she reconceptualises education as something that is all-pervasive, often informal, through many agents in many different contexts. The idea that education is a fundamental aspect of life that lasts for the whole of one's life underpins her entire re-evaluation.
Critics have claimed that her work is so general that it is difficult to either object or test. Others claim that her widening education to include, for example animals, is taking the inclusivity idea too far. Others, while agreeing with Martin’s descriptive work, have argued that Martin’s prescriptive recommendations around the reconstitution of education are either impractical, simply reinforce inequalities or replace them with new ones. Her work certainly aroused debate and passions.
Martin may not be regarded as the only or even leading voice of feminism in education but her reputation as a dogged researcher, determined to re-establish historical wrongs and implement future rights is unarguable. The re-evaluation of Plato, Rousseau, James, Wollstonecraft, Montessori, Beecher and Gilman is one contribution. Stimulating feminist and gender debate and research is another. Widening the view of education beyond schooling is another. Above all she reconceptualises education as something that is all-pervasive, often informal, through many agents in many different contexts. This was Martin’s great achievement, simply opening the world of education up to a much wider set of perspectives around gender, women and what it is to educate and be an educated person.
Martin, J. R. (1985). Reclaiming a Conversation: The Ideal of the Educated Woman. New Haven: Yale University Press.
Martin, J. R. (1992). The Schoolhome: Rethinking Schools for Changing Families. Cambridge, MA: Harvard University Press, 1992.
Martin, J. R. (1994). Changing the Educational Landscape: Philosophy, Women, and Curriculum. New York: Routledge, 1994.
Martin, J. R. (2000). Coming of Age in Academe: Rekindling Women's Hopes and Reforming the Academy. New York: Routledge.
Martin, J. R. (2002). Cultural Miseducation: In Search of a Democratic Solution. New York: Teachers College Press.

Martin, J. R. (2011). Education Reconfigured: Routledge.

 Subscribe to RSS

Wednesday, August 12, 2015

Kohlberg's moralising and its revival - character education

The Nazis and the Holocaust led Lawrence Kohlberg to focus on the moral dimension of education. Drawing on the Socratic interest in values and virtue, and John Dewey’s view of education as the development of the individual, he saw education not as moral instruction but as the development of moral judgement and behaviour. His inspiration was Piaget’s stages of development, which he applied to the moral development of children and adults. His theoretical work was matched by practical recommendations around the concept of a ‘just community’. These could be schools, professions, social groups, even prisons.
Three types of educational theory
For Kohlberg there are three movements in educational thinking:
1. Romanticism
2. Cultural transmission
3. Progessivism
Romanticism’s formative figure is Rousseau and this movement sees the child as a natural learner, with institutions that often inhibit their progress. Cultural transmission, the transfer of knowledge and values from one generation to the next, attempts to preserves cultural capital. He saw focus on the psychological aspects of learning, especially behaviourism, as well as the use of technology, as typical of this movement. Progressivism, exemplified by William James and John Dewy, sees education as an important contributor to society, it’s cultural and democratic dimensions. Kohlberg was a ‘progressive’.
Six stages of moral development
Building on Piaget’s (now discredited) stages of development:
Level 1 - Preconventional
1. Punishment and obedience orientation
2. Instrumental relativist orientation
Children think and behave egoistically, acting on potential consequences, such as punishment. This self-interest can then develop into a more instrumental outlook, where you see how others may help you promote your own interests, as in receiving rewards for good behaviour.
Level 2 - Conventional
3. Interpersonal concordance orientation
4. Society maintaining orientation
Adolescents mature into this stage by obeying society’s rules but without much reasoned reflection. Wanting to be liked or respected by others makes one behave in ways in which groups approve. Recognition of the ‘good’ and ‘bad in relation to adherence to the law also emerges.
Level 3 - Postconventional
5. Social contract orientation
6. Universal ethical principles
At this level, individuals use their own ethical principles to make judgement and do not simply adhere to external norms. There is a recognition of diversity of moral perspectives and the resolution of moral issue through democracy or other mechanisms of agreement. This may move on to higher levels of abstraction about moral principles, such as justice and rights, where individuals see themselves as like others in mutually agreed action.
Individuals move through these stages, none are skipped, we hardy ever go back, we can hold back but not accelerate stage development. He used Piaget’s notion of changing schemata, mental constructs that make sense of experience, that determine the limits of moral reasoning. New experiences are either assimilated (integrated without major change) or accommodated (new schemata created). 
New stages are more complex and high-level. Cognitive disequilibrium forces change as new experiences cause cognitive conflict. The conflict is resolved by the creation of new cognitive schemata.
In addition to drawing upon theoretical ideas from Piaget and John Rawls, he researched the hypothetical stages using Moral Judgement Interviews (MJI) where moral problems are presented and the reasoning, not conclusions, studied. Interviews were conducted every three years over twenty years and, he claims, confirmed his six-stage theory. Further research across forty countries also conformed, he claimed, its cross-cultural validity.
These six stages of moral development were highly influential and teachers were encouraged to use teaching tactics appropriate to these stages, curriculum recommendations were made and a real movement emerged around staged moral development.
The basic idea that moral reasoning lies at the root of or plays the primary role in moral behavior is rejected by many intuitionists. Many came to see Kohlberg’s interpretation as no more than that, the ‘interpretation’ of basically intuitive moral judgment. Subsequent research showed that both Paiget’s and Kohlberg’s stages were wrong, so wrong that the very idea of the Kohlberg framework had to be adjusted. However, the adjustments proved so extreme that the framework had to be abandoned in favour of other possible approaches. Elliot Turiel thought that the moral development process was also massively fuelled by social convention. There was an amusing interlude when Carol Gillighan took a huge gender swiped at Kohlberg in her book "In a Different Voice: Psychological Theory and Women's Development" (1982). She made the reasonable point that his research had only involved males and that Kohlberg was simply reinforcing stereotypical male character traits. She had a point but simply replaced Kohlberg with another set of character traits around a morality of care. It had all gone to pot. There is also the tricky issue that multiple stages can be observed in the same individuals.
Kohlberg’s developmental psychology, as applied to the moral sphere, brought an important, dynamic dimension into moral education. Yet, like many staged processes, it proved to be too rigid and has crumbled somewhat due to subsequent research. The role of institutional education in the teaching of ethics remains problematic, as religious pressures and the roles of other agents, such as families, peer-groups and the media play important roles.
Morph into character teaching
More recently the moral issue has changed into n interest in ‘character’ education. In practice, deep, political roots of moral and character education really lie in conservative worries about cultural and moral decline. Every older generation has its views that the world is going to the dogs and that we must bring back some golden age of high character (usually theirs) to tame this feral, new, non-conformist generation.
In the US the character education movement is often pushed by conservative and religious sources that see the creep of liberal values as equivalent to moral decline. The religious lobby, in particular, has been successful in pushing this agenda. The most recent Federal example was G. W. Bush, who saw it as essential component of educational policy. In the UK we have an entirely different, and hugely influential stream of thought that comes originally from Thomas Arnold and the public school system. Let’s call it the ‘playing fields of Eton’ complex, but anyone who has experience in the UK system knows exactly what this is. With a hugely disproportionate number of politicians and civil servants coming from a public-school background, this tradition is stronger in the UK than almost any other comparable nation. There is constant pressure to see the state system as dysfunctional and if we could only take some of the magic dust from the private schools and scatter it down upon the teachers in the state system, all would be well.
Character as a subject
The Dfe talks of “the teaching of character as a separate subject”. However, there is no evidence for this at all. In fact there is plenty of evidence to show that this type of teaching has no effect whatsoever. The teaching of character and values, if they can be ‘taught’ at all is a bold claim.
Let’s start with the big one, certainly the one with the biggest title, ‘Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children’ a report from the Social and Character Development Research Program  (2010). It looked at seven SACD programs and 20 student and school outcomes, all on social and character development and concluded that school-based character education programs produce no measurable improvements in student behaviour or academic performance. This was an astonishing result from a large and well designed piece of research. In fact there are no peer-reviewed studies that support the idea that character teaching has a positive, measurable effect.
Character as conformity
Character and conformity are easily confused. Far from shaping ‘character’ in schools, we should be doing the opposite and encouraging students to question these norms and become autonomous learners, able to distinguish between moral inculcation, based on assumed social norms, from more open tolerant approaches to education. It is by no means clear that the character traits of teachers is the right model or that teachers know what character traits are and how to teach them.
Character and schools
In fact, character education has been a feature of many totalitarian, religious and repressive systems, as character is moulded to match particular ideologies. In the US this of often a route for conservative, religious education. In China, the Confucian system, which is strongly character driven, pushes students towards a highly conformist, non-critical form of rote learning. In Islamic states a strictly conformist and literal form of the Koran is used to shape character. Private schools with a narrow socio-economic group is likely to promote character in terms of that group. In truth, unless a school system is truly secular (and arguably even then) character education reflects the cultural norms of that school. In the UK, with the rise of faith schools, this has already caused considerable problems.

When a politician talks about ‘character’, my heart sinks. It’s like Jimmy Saville taking a line on sex education – he’s an expert of sorts, just the wrong sort. Politicians love to meddle in educational practice, in a way they would never in say, medical practice. That’s because they think of themselves as ideals and whatever ‘they’ experienced in education must be good for the rest of us. This explains their obvious disengagement from the voters and blindness when it comes to judgments on the role of character in education, even the world at large. Let’s put this rather odd ‘C-word’ back where it came from, in the files marked ‘bad theory’, ‘old-school thinking’ and ‘political conceit’.

 Subscribe to RSS