Showing posts sorted by relevance for query dale. Sort by date Show all posts
Showing posts sorted by relevance for query dale. Sort by date Show all posts

Sunday, January 07, 2018

Astonishing fake in education and training - the graph that never was

I have seen this in presentations by the CEO of a large online learning company, Vice-Chancellor of a University, Deloitte’s Bersin, and in innumerable keynotes and talks over many years. It’s a sure sign that the speaker has no real background in learning theory and is basically winging it. Still a staple in education and training, especially in 'train the trainer' and teaching courses, a quick glance is enough to be suspicious.
Dales’s cone



The whole mess has its origins in a book by Edgar Dale way back in 1946. There he listed things from the most abstract to the most concrete: Still pictures, Visual symbols, Verbal symbols, Radio recordings, Motion pictures, Exhibits, Field trips, Demonstrations, Dramatic participation, Contrived experiences, Purposeful experience and Direct. In the second edition (1954) he added Dramatised experiences through Television and in the third edition, and heavily influenced by Bruner (1966), he added enactive, iconic symbolic.
But let’s not blame Dale. He admitted that it is NOT based on any research, only a simple intuitive model and he did NOT include any numbers. It was, in fact, simply a gradated model to show the concreteness of different audio-visual media. Dale warned against taking all of this too seriously, as a ranked or hierarchical order. Which is exactly what everyone did. He actually listed the misconceptions in his 1969 third edition p128-134. So the first act of fakery was to take a simple model, ignore its original purpose, and the authors warnings, and use it for other ends.
Add fake numbers
First up, why would anyone with a modicum of sense believe a graph with such rounded numbers? Any study that produces a series of results bang on units of ten would seem highly suspicious to someone with the most basic knowledge of statistics. The answer, of course, is that people are gullible, especially to messages that appeal to their intuitive beliefs, no matter how wrong. The graph almost induces confirmation bias. In any case, these numbers are senseless unless you have a definition of what you mean by learning and the nature of the content. Of course, there was no measurement – the numbers were made up.
Add Fake Author
At this point the graph has quite simply been sexed up by adding a seemingly genuine citation from an academic and Journal. This is a real paper, about self-generated explanations, but has nothing to with the fake histogram. The lead author of the cited study, Dr. Chi of the University of Pittsburgh, a leading expert on ‘expertise’, when contacted by Will Thalheimer, who uncovered the deception, said, "I don't recognize this graph at all. So the citation is definitely wrong; since it's not my graph." Serious looking histograms can look scientific, especially when supported by bogus academic credentials.
Add new categories
The fourth bit of fakery was to add ‘teaching others’ to the end, topping it up to, you guessed it – 90%. You can see what’s happening here – flatter teachers and teaching, and they’ll believe anything. They also added the ‘Lecture’ category on at the front – and curiously CD-ROM! In fact, the histogram has appeared in many different forms, simply altered to suit the presenter's point in a book or course. This is from Josh Bersin’s book on Blended Learning. It is easy to see how the meme gets transmitted when consultants tout it around in published books. Bersin was bought by Deloittes. What happens here is that Dales original concept is turned from a description of media into the prescription of methods.
The Colored Pyramid
The third bit of fakery, was to sex it up with colour and shape, going back to Dale’s pyramid but with the fake numbers and new categories added. It is a cunning switch, to make it look like that other caricature of human nature, Maslow’s hierarchy of needs. It suffers from the same simplistic idiocy that Maslow’s pyramid does – that complex and very different things lie in a linear sequence one after the other. It is essentially a series of category mistakes, as it takes very different things and assumes they all have the same output – learning. In fact, learning is a complex thing, not a single output. A good lecture may be highly motivating, there are semantic tasks that are well suited to reading and reflection, discussion groups may be useless when struggling with deep and complex semantic problems and so on. Of course, the coloured pyramid makes it look more vivid and real, all too easy to slot into one of those awful 'train the trainer' or 'teacher training' courses that love simplistic bromides.
Conclusion
What’s damning is that this image and variations of the data have been circulating in thousands of PowerPoints, articles and books since the 60s. Investigations of these graphs by Kinnamon (2002) found dozens of references to these numbers in reports and promotional material. Michael Molenda (2003), did a similar job. Their investigations found that the percentages have even been modified to suit the presenter’s needs. This is a sorry tale of how a simple model published with lots of original caveats can morph into a meme that actually lies about the author, the numbers, adds categories and is uncritically adopted by educators and trainers.
PS
Much of this comes from the wonderful Will Thalhemer’s original work. I wanted to give it an extra and more structured spin on the development of the fakery.
Bibliography
Bruner, J. (1966), Toward a Theory of Instruction
Dale, E (1946), Audiovisual methods in teaching
Dale, E (1954), Audiovisual methods in teaching
Dale, E (1969), Audiovisual methods in teaching
Kinnamon, J. C. (2002). Personal communication, October 25
Kovalchick, A and Dawson,K (2004), Education and Technology
Thalheimer, W Blogpost

Molenda, M. H. (2003). Personal communications

Monday, July 13, 2020

Bogus pyramids: Learning methods, Maslow and Bloom

Pyramids seem to attract a lot of attention among conspiracy theorists and peddlers of books about alien mysteries and new age nonsense. Similarly, they pop up in education and training. It is rare to see an educational or training theory course or PowerPoint without a pyramid, preferably with rainbow colours. 
Indeed, once you catch on to the pyramid scheme trick, you see them everywhere. If there’s a theory to be peddled, usually simplistic and overly-hierarchical, ignoring the real complexities of the theory, someone will have turned it into a pyramid. There are Leadership pyramids (tons), Pyramids of Success, Competency Pyramids, Values Pyramids, even Inverted Pyramids to show reversed hierarchies. This is not to say that such images are always wrong. They can do a passable job in getting the broad brush-strokes of a theory across. The problem is that they haul in all sorts of preconceptions and fictions with them, like progression and hierarchy. Rarely do they actually represent the complexity of a theory or reality.
Let’s take just three, perhaps the most commonly used pyramids to unpack the problem. One is a fake (Learning methods),  One, Maslow’s Hierarchy of needs is badly researched, simplistic and has two versions. One, Bloom’s hierarchy, also has two versions and both are misleading.

The pyramid that never was

I have seen this in presentations by the CEO of a large online learning company, Vice-Chancellor of a University, Deloitte’s Bersin, and in innumerable keynotes and talks over many years. It is a sure sign that the speaker has no real background in learning theory and is winging it. Still a staple in education and training, especially in 'train the trainer' and teaching courses, a quick glance is enough to be suspicious.

The whole mess has its origins in a book by Edgar Dale way back in 1946. There he listed things from the most abstract to the most concrete: Still pictures, Visual symbols, Verbal symbols, Radio recordings, Motion pictures, Exhibits, Field trips, Demonstrations, Dramatic participation, Contrived experiences, Purposeful experiences and Direct. In the second edition (1954) he added Dramatised experiences through Television and in the third edition, heavily influenced by Bruner (1966), he added enactive, iconic and symbolic.
But let’s not blame Dale. He admitted that it was not based on any research, only a simple intuitive model and he did not include any numbers. It was, in fact, simply a gradated model to show the concreteness of different audio-visual media. Dale warned against taking all of this too seriously, as a ranked or hierarchical order. Which is exactly what everyone did. He actually listed the misconceptions in his 1969 third edition p128-134. So the first act of fakery was to take a simple model, ignore its original purpose, and the authors warnings, and use it for other ends.

Add fake numbers

The cone was flipped on its side and turned into a serious looking histogram. First up, why would anyone with a modicum of sense believe a graph with such rounded numbers? Any study that produces a series of results bang on units of ten would seem highly suspicious to someone with the most basic knowledge of statistics. The answer, of course, is that people are gullible, especially to messages that appeal to their intuitive beliefs, no matter how wrong. The graph almost induces confirmation bias. In any case, these numbers are senseless unless you have a definition of what you mean by learning and the nature of the content. Of course, there was no measurement – the numbers were bogus.

Add Fake Author



At this point the graph was quite simply sexed up by add some academic seasoning, a seemingly genuine citation from an academic and Journal. This is a real paper, about self-generated explanations, but has nothing to with the fake histogram. The lead author of the cited study, Dr. Chi of the University of Pittsburgh, a leading expert on ‘expertise’, when contacted by Will Thalheimer, who uncovered the deception, said, "I don't recognize this graph at all. So the citation is definitely wrong; since it's not my graph." Serious looking histograms can look scientific, especially when supported by bogus academic credentials.

Add new categories

The fourth bit of fakery was to add ‘teaching others’ to the end, topping it up to, you guessed it – 90%. You can see what’s happening here, flatter teachers and teaching, and they are more likely to buy it. They also added the ‘Lecture’ category on at the front, to reel in academics. In fact, the histogram has appeared in many different forms, simply altered to suit the presenter's point in a book or course. This one is from Josh Bersin’s book on Blended Learning. It is easy to see how the meme gets transmitted when consultants tout it around in published books, courses and PowerPoints. What happened here was that Dale’s original cone concept went through several levels of fakery, to turn it from a description of media, to the prescription of methods.

Final coloured pyramid


The next bit of fakery, was to go full technicolour and turn it into a pyramid. Going back to Dale’s pyramid but with the fake numbers and new categories added, it was a cunning switch. The odd that complex and very different things lie in a linear sequence one after the other. It is essentially a series of category mistakes, as it takes very different things and assumes they all have the same output – learning. In fact, learning is a complex thing, not a single output. A good lecture may be highly motivating, there are semantic tasks that are well suited to reading and reflection, discussion groups may be useless when struggling with deep and complex semantic problems like maths and so on. Of course, the coloured pyramid makes it look more vivid and real, all too easy to slot in, as a simplistic bromide, to a lazy 'train the trainer' or 'teacher training' courses.
What is damning is that this image and variations of the data have been circulating in thousands of PowerPoints, articles and books since the 60s. Will Thalhemer’s original work did much to uncover this fakery. Investigations of these graphs by Kinnamon (2002) found dozens of references to these numbers in reports and promotional material. Michael Molenda (2003), did a similar job. Their investigations found that the percentages have often been modified to suit the presenter’s needs. Just search on Google for The Learning Pyramid and click ‘Images’ and you’ll find page after page of variations. This is a sorry tale of how a simple model published with lots of original caveats can morph into a meme that actually lies about the author, the numbers, adds categories and is uncritically adopted by educators and trainers.

Maslow’s misleading pyramids

Maslow (1908 - 1970) is almost synonymous with his hierarchy of needs. The author is invariably mentioned in the same breath as his hierarchy, which is just as invariably displayed as a pyramid.  It has been a staple for decades as a first step in educational theory, teacher training, train the trainer and management courses. Yet few who put it centre-stage in their PowerPoints, realise that Maslow never created this coloured pyramid. It never appeared in any of his published works. Fewer still question its validity.
His basic concept of prioritising needs was first published in 1943, in his paper A Theory of Human Motivation (1943) in the Psychological review. Then, based on a rather odd, selective and cursory analysis of successful people, his full-milk, hierarchical theory was published in his 1954 book Motivation and Personality.  It was here that he pared back learning to a hierarchy of basic human needs and desires. His aim was to uncover the precursors to what motivates people to learn.
So how did we end up with a pyramid? There were several sleights of hand here. Douglas McGregor, at MIT’s Sloane School of Management took Maslow’s general ideas and applied them to management. In doing so, he simplified and distilled Maslow’s ideas, removing much of the subtlety. Note that he did not invent the pyramid, that came later with Keith Davis, who in his book on management, represented Maslow’s ideas as a right-angles triangle, with a pinnacle. That was the first step, the second was in the psychologist Charles McDermid’s article How money motivated men, where the pyramid first appeared. Note that there was a rather odd method to his madness, as he thought this was a way of pushing profits. 
To be fair to Maslow, although oddly researched, with celebrity sources, his work was far more nuanced than the pyramid suggests. His focus was very much on the concepts of self-actualisation and transcendence, and whatever one thinks about those as concepts or theories, they are not magic projections off the top of a primitive pyramid.


The bottom four are all ‘deficit’ or ‘D-needs’. As each of these needs are satisfied, he individual reaches homeostasis and the feeling or need to fulfill that D-need stops, so that one can proceed to the next.
If they are not present, you will feel their absence and yearn for them. When each is satisfied, you reach a state of homeostasis where the yearning stops. It is simple and has an intuitive appeal.
The last, self-actualization, does not involve homeostasis, but once felt is always there. Maslow saw this as applying to only a few people, whose basic four levels are satisfied, leaving them free to look beyond their deficit needs. He claimed to have used a qualitative technique called ‘biographical analysis’ where he looked at high achievers and found that they enjoyed solitude, close relationships with a few rather than many, autonomy and resist social norms. Spontaneity, simplicity and respect for others were other characteristics.
What is rarely known is that it was discovered that Maslow, in 1970, the year of his death, changed his original thoughts, developed in the 1950s, to a more complex model of needs. In unpublished work, ‘knowing and understanding' and 'aesthetic' were discussed. This upgraded version of his theory was largely ignored, as the earlier model had become so deeply embedded in teacher and trainer training courses. Anyone familiar with epistemology and aesthetics will immediately see the problem. Both are notoriously difficult to define. In any case, Maslow never actually published the first or updated motivational model as pyramids. It never appeared in any of his published or unpublished writings. 

Although hugely influential, his work was never tested experimentally and when it was, from the 70s onwards, was found wanting. Empirical evidence showed no real evidence in terms of a strict hierarchy, nor the categories, as defined by Maslow. 
His ‘biographical analysis’ was armchair research. based on a self-selected group of just 18 people and in itself is defined in terms of a set of subjective criteria. . His list of 18 self-actualized people included Einstein, Mother Teresa, Gandhi, Beethoven, Lincoln, Eleanor Roosevelt and, of course - Abraham Maslow! He only included two women in his list of self-actualised people, bizarrely, the anti-Semitic Elanor Roosevelt and nun Mother Teresa. It is hard to imagine that a full theory of human nature could be built upon such strange and extreme examples. One thing that helped Maslow become the guru of motivation was that his books were very popular and he was part of a sixties movement that helped promote his views. He was friends with people like Timothy Leary (Maslow’s daughter Ellen was Leary’s research assistant). Linda Sargent Wood writes in A More Perfect Union about the rise of these holistic, utopian visions for humanity. She shows that these were largely populist, and not research or evidence, based movements. 
Kenrick et al. (2010) found huge variance in Maslow's so-called self-actualised people, showing divergence rather than convergence. As there is no control group, this is simply circular. Self-actualised people are selected by him, then used as evidence for self-actualisation.  The self-actualisation theory is therefore largely subjective. 
An even weaker aspect of the theory is its strict hierarchy. It is clear that the higher needs can be fulfilled before the lower needs are satisfied, so the theory has been repeatedly falsified. There are many counter-examples and indeed. For example, creativity can atrophy and die on the back of success. Maslow himself felt that the lines were not that clear. In short, subsequent research has shown that his hierarchy is crude, as needs are pursued non-hierarchically, often in parallel. A different set of people could be chosen to prove that self-actualization was the result of, say, trauma or poverty (Van Gogh etc.). 
Subsequent research shows that real needs and motivations do not fall into this neat pyramid or hierarchy. For example Rutledge (2011) claims that complex social connections are needed for all of the assumed levels in the hierarchy and that it falls apart when this is taken into account. Typical of the many studies that show the irrelevance of the hierarchy is Tay and Diener (2011), where 60865 people from 123 countries were questioned on Maslow's needs. It showed that the hierarchy was quite simply wrong. The problem is that the reductive nature of the analysis ignores the much messier nature of motivation. In general, the whole idea has been abandoned in modern motivational theory. 
His hierarchy is often hauled into training programmes, without any real understanding of why and whether the theory is indeed correct, beyond some simple truisms. Its appeal seems to lie in the endless repetition of a clear and simply coloured pyramid, rather than any evidence or sophistication in terms of human nature. In fact, it is a hopeless caricature of human nature, one that is best avoided.
Indeed, apart from being fossilised as a component in bad teacher-training and train the trainer courses, it is hard to see how it has any real relevance to what teachers, trainers, lecturers or instructors actually do when they teach. It is clear that having somewhere to live, food to eat, friends, and feeling safe are important but not in the hierarchical or developmental way they are presented. As a teacher, or manager, one can recognize the basic needs of a person without recourse to the pyramid structure at all. Indeed, it may be misleading. Most sets of indicators for the wellbeing of children are more complex, sophisticated and do not fall into a simple hierarchy. There are many such schemas at international (UNICEF) and national levels. They rarely bear much resemblance to Maslow’s hierarchy.
Maslow has been almost omnipresent in education and training. However, it is not clear that his theory has had any real effect other than encouraging people to look at others as human beings, rather than subject to some instrumental manipulation. This is an entry from Maslow's own journal in 1962, “My motivation theory was published 20 years ago, & in all that time nobody repeated it, or tested it, or really analyzed it or criticized it. They just used it, swallowed it whole with only the most minor modifications”. He was right. It is not a hierarchy, was not tested and as a theory of human nature it is simplistic and banal. To argue that the theory, although wrong, is useful, as it promotes a holistic view of learners, employees, managers etc. is to open the door to any subjective theory that fits a prejudice. It seems to live on, not because it is validated, real or useful, largely, perhaps, because of the colourful triangle that looks great as a PowerPoint slide. It’s a false meme.
As if to confirm its status as meme not theory, more recently, the pyramid became an internet meme, when some wag added ‘Wi-Fi!’ to the bottom of the pyramid. Many other joke needs have since appeared.

This accidentally confirmed the truth, that the pyramid is actually a bit of a joke.

Bloom’s fictional pyramid

Bloom’s taxonomy

Benjamin Bloom (1913-1999) is famous for the most commonly used taxonomy in education and training. It includes three overlapping domains:

Cognitive (knowledge)
Psychomotor (skills)
Affective (attitude)

Bloom published this taxonomy in Taxonomy of educational objectives: The classification of educational goals. Yet it was not wholly Bloom’s work. The book was edited by him but the taxonomy came out of a series of conferences and the work of a committee of colleagues at the University of Chicago. These were published as three ‘handbooks’, in other words practical texts that bridged theory and practice.
In the 2001, 45 years later, Bloom’s student Lorin Anderson revised Bloom's taxonomy in A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. Note that Bloom is neither an author nor contributor. The theory is expanded as psychology had shifted away from behaviourism towards cognitive psychology, so more recent theory informed the update. It is a complex framework that integrates knowledge with cognitive activities in learning. Far from being a simplistic pyramid, it lays out four dimensions; what is taught, how it taught, how it is assessed, and their alignment. It is a holistic and interconnected work. Importantly, in neither the original text by Bloom nor its revision by Anderson, did a pyramid appear.
It was never meant to be pure progress from lower to higher, it was more multidimensional. It is a practical, organizational structure more correctly represented in two dimensional tables, which is how they first appeared, so that one could match different dimensions. There is increasing spectrum of complexity not a hierarchy of value. Unfortunately, some like to reduce his work to word lists or represent the taxonomies as pyramids suggesting, both implicitly and sometimes explicitly, that knowledge, at the bottom, is inferior. It was devised to assist teachers and trainers to classify educational goals and plan and evaluate learning experiences. Unfortunately, the pyramid is about as far as most people get. They rarely dig deeper. Yet the actual theory and practical recommendations are lost in the pyramid representation, which not only simplifies but misrepresents the original theory and its updates.
This is how Bloom’s taxonomy commonly appears:

Yet, here it is straight from the original author’s mouth:
“The triangle does not appear in either (original or revised Bloom’s) Taxonomy. The triangular representation was quite likely designed by someone as part of a presentation made to educational practitioners.”
Lorin W. Anderson
Note: Lorin W. Anderson, recommends Daniel Willingham's diagram, as it puts knowledge in an overall and foundational position without invoking the hierarchy, progression and apex.




The pyramid is worrying, as a sophisticated theory and its even more sophisticated revision, were essentially bastardized to represent a hierarchy, in which one had to progress from one stage to another, whereas learning is far more complex with different cognitive processes interacting and intertwined with each other. This is not to say that there are no dependencies. There is a strong argument for seeing knowledge as foundational in these taxonomies.
Bloom published his now famous taxonomy of learning in 1956. Few realise that this taxonomy is now over 60 years old. There have been lots of taxonomies since then that slice and dice, many variations on existing categories. Indeed we've had dozens of taxonomies that sliced and diced in all sorts of ways. We've had Biggs, Wills, Bateson, Belbin and dozens more. We seem to have got stuck in the Bloom taxonomy. To be fair there have been good revisions, taking the knowledge nouns and unpacking them into verbs but the problem with taxonomies is their attempt to pin down the complexity of cognition in a list of simple categories. In practice, learning doesn’t fall into these neat divisions. It is a much more complex and messier set of cognitive processes, so attention has shifted to how learning meshes with memory and techniques that improve organisation, chunking, encoding, practice and recall.
Another danger is that instructionalists, like Gagne, take these taxonomies and attempt to design learning that matches these categories, destroying much of the more useful approaches which an understanding of brain science brings; such as cognitive overload, working memory limitations, top-down processing and so on. Learning theory has moved on in terms of a more detailed understanding of memory, which has put everything on a more empirical and scientific basis.

Conclusion

These three pyramids have turned into pyramid schemes, repeated, adapted, amplified and unquestioningly adopted. It gives you the illusion of science to sell your product, usually a talk, lecture or course. Most who use three pyramids, and there are many more, are completely unaware of the fakery, lack or research, variations and updates by the original authors or bogus representations of hierarchy and progression. They are images that become fossilized in books, handouts, PowerPoints. Social media now amplifies them, as a brightly coloured image acts as clickbait on many a post. 
The allure of the pyramid is its geometric simplicity, representation of hierarchy and suggestion of progression. Add some colour and it appeals to PowerPoint driven education and training. More than this, the pyramid suggests progression towards the peak as the apex of learning. It promotes a view that things at the bottom and less worthy than things at the top, which plays into the narratives that some educators like, such as 21st century skills. All three misrepresentations play this game, which is why they have proved popular. People like shortcuts, especially when they fit their views of the world.
What keeps all of this going is ‘intuition’, the feeling that it seems right. There are germs of truth, even some reasonable claims in all three of these pyramids, some more than others. Intuition then trumps reason, as you buy into the simplicity of the image, its hierarchy and progression. The sun progresses across the sky, so you assume that it goes round the earth, when, in fact things are a lot more complicated. In fact, the opposite is the truth. In all three cases; the fiction, armchair theory and reasonable taxonomy, the image becomes the established theory, when it is actually a fake, a caricature or a misleading representation.
This intuition is fueled by something else – confirmation bias. If you are a teacher, lecturer or professional in the learning field, a theory and image that puts you at the pinnacle of its presentation is flattering. In the learning pyramid, you become the end point, the goal in all this, as ‘teaching’ is the ultimate learning activity. In Maslow’s hierarchy, self-actualisation suggests that education is some sort of holy grail for the soul. In Bloom’s taxonomy, you reach a peak of creativity. It puts you at the very top, beyond reproach. Oddly, the reality is that it shows the learning game to be full of fake, poorly researched and simplistic theories and tools.
In the end, a combination of faked imagery and projected extra meaning by both the presenters and viewers of such images is a potent and saleable mix. Sadly, it is quality assured, taught and assessed. Practitioners become qualified on the back of bogus theory and practice. These three pyramids alone account for a massive and incalculable amount of wasted time and cost in education and training, not only of teachers but learners. Don’t be dazzled by colour and geometry. Question what is given and assumed as research and fact.
 “Finally, after 40+ years in the business, I am greatly dismayed that many educators get their information from oral presentations and secondary (and in some cases tertiary) sources. This practice tends to result in passing along half-truths and misinterpretations.”
Lorin W. Anderson
We can do no better than quote the above warning by the author of the revised Bloom’s Taxonomy, on the danger of visual misrepresentations of theory.

Bibliography

Bruner, J. (1966), Toward a Theory of Instruction
Dale, E (1946), Audiovisual methods in teaching
Dale, E (1954), Audiovisual methods in teaching
Dale, E (1969), Audiovisual methods in teaching
Kinnamon, J. C. (2002). Personal communication, October 25
Kovalchick, A and Dawson,K (2004), Education and Technology
Molenda, M. H. (2003). Personal communications
Maslow, A. (1954). Motivation and personality. New York: Harper.
Maslow, A. (1971). The farther reaches of human nature. New York: The Viking Press.
Maslow, A., & Lowery, R. (Ed.). (1998). Toward a psychology of being (3rd ed.). New York: Wiley & Sons.
Davis, K., 1957. Human relations in business. McGraw-Hill.
McDermid, C.D., 1960. How money motivates men. Business horizons, 3(4), pp.93-100.
Kenrick, D. T., Neuberg, S. L., Griskevicius, V., Becker, D. V., & Schaller, M. (2010). Goal-Driven Cognition and Functional Behavior The Fundamental-Motives Framework. Current Directions in Psychological Science, 19(1), 63-67.
Rutledge P. (2011) SocialNetworks: What Maslow Misses. Psychology Today
Tay, L., & Diener, E. (2011). Needs and subjective well-being around the world. Journal of Personality and Social Psychology, 101(2), 354.
Wahba, A; Bridgewell, L (1976). "Maslow reconsidered: A review of research on the need hierarchy theory". Organizational Behavior and Human Performance (15): 212–240.
Wood L.S. (2010) A More Perfect Union. Oxford University press.
Bloom, B.S. (Ed.) (1956). Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. Longmans, Green.
Anderson, Lorin W.; Krathwohl, David R., eds. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman. 
Blog by Lorin W. Anderson (2017) https://larryferlazzo.edublogs.org/2017/04/15/guest-post-from-lorin-w-anderson-co-author-of-the-revised-blooms-taxonomy/
Guskey, T. R. (2005). Benjamin S. Bloom: Portraits of an educator. Lanham, MD: Rowman & Littlefield Education.

Thursday, November 07, 2013

When Big Data goes bad: 6 epic fails

Data, in the wrong hands, whether malicious, manipulative or naïve can be downright dangerous. Indeed, when big data goes bad it can be lethal. Unfortunately the learning game is no stranger to both the abuse of data. Here’s six examples showing seven species of ‘bad data’.
1. Data subtraction: Ken Robinson
Don’t let the selective graphical representation of data, destroy the integrity of the data. A good example of blatant data editing is the memorable ‘ritalin’ image used by Sir Ken Robinson in his TED talk at 3.47. This image is taken from its RSA animation.
Compare Robinson’s graph with the true source.
His has no legend and he’s recalibrated states to look as if there’s zero prescriptions. To understand this data you have to look at its source to understand that the white areas represent states that did NOT participate in the study or did not have reported prescription data. It’s a distortion, an exaggeration to help make a point that the data doesn’t really support
In fact, much of what passes for fact in Sir Ken Robinson’s TED talks are not supported by any research or data whatsoever.
2. Data addition: Bogus learning theory
Ever seen this graph, or one like it? It used to be a staple in education, training and teacher training courses. Only one problem - it’s bogus.
A quick glance is enough to be suspicious. Any study that produces a series of results bang on units of ten would seem highly suspicious to someone with the most basic knowledge of statistics.
But it’s worse than nonsense, the lead author of the cited study, Dr. Chi of the University of Pittsburgh, a leading expert on ‘expertise’, when contacted by Will Thalheimer, who uncovered the deception, said, "I don't recognize this graph at all. So the citation is definitely wrong; since it's not my graph." What’s worse is that this image and variations of the data have been circulating in thousands of PowerPoints, articles and books since the 60s.
Further investigations of these graphs by Kinnamon ((2002) in Personal communication, October 25) found dozens of references to these numbers in reports and promotional material. Michael Molenda ( (2003) Personal communications, February and March) did a similar job. Their investigations found that the percentages have even been modified to suit the presenter’s needs. 
The one here is from Bersin (recently bought by Deloitte). Categories have even been added to make a point (e.g. that teaching is the most effective method of learning).
The root of the problem is an image by Edgar Dale’s depiction of types of learning from the abstract to the concrete. He has no numbers on his ‘cone of experience’ and regarded it as a visual metaphor implying no hierarchy at all.
Serious looking histograms can look scientific, especially when supported by bogus academic research. They create the illusion of good data. This is one of the most famous examples of not ’Big’ but ‘Bad’ data in the history of learning.
3. Claims beyond the data – University League Tables
University league tables are used by politicians, Universities, parents and students. But they contain a dark, dirty, data secret. They claim to rank universities but, astonishingly, tell you absolutely nothing about ‘teaching’. They often claim to have ‘measures’ on teaching, but they actually draw their data from proxies, such as employment and research activity and use nothing but indirect measures to measure teaching.
The Times rankings are a case in point. They claim that their ranking scores include teaching. In fact, only 30% is based on teaching but they use NO direct metrics. The proxies include student/staff ratios (which is skewed by how much research is done) and, even more absurdly, the ratio of PhDs to BAs. It is therefore a self-fulfilling table, where the elite Universities are bound to rise to the top. There is no direct measurement of face to face tome, lecture attendance or student satisfaction.
4. Skewed data - PISA
Like the real Leaning Tower of PISA, the OECD PISA results are built on flimsy foundations and are seriously skewed. Nevertheless, they have become a major international attraction for educators, and regularly spark off annual educational ‘international arms’ races.
Both left and right now use the ‘sputnik’ myth, translated into the ‘Chinese competitiveness’ myth, to chase their own agendas – more state funding or more privatisation. This is a shame, as the last thing we need is yet another dysfunctional , deficit debate in education. Nations have different approaches to education, different demographic and social mixes and different agendas.
The problems in the data are extreme as PISA compares apples and oranges. PISA is seriously flawed because of the huge differences in demographics, socio-economic ranges and linguistic diversity within the tested nations. There are many skews in the data, including the selction of one flagship city (Shanghai) to compare against entire nations. Immigration skews include numbers of immigrants, effect of selective immigration, migration towards English speaking nations, and first-generation language issues. There’s also the issue of taking longer to read irregular languages and selectivity in the curriculum.
Sven de Kreiner Danish statistician says PISA is not reliable at all. In the reading tests 28 questions were supposed to be equally difficult in every country. PISA has failed here as differential item functioning - items with different degrees of difficulty in different countries - are common. In fact he couldn't find any that worked without bias. Items are more difficult in some countries. He used his analysis to show that the UK moves up to 8 or down to 36. PISA assumes that DIF has been eliminated but not one single item is the same across the 56 countries
Politicians and activists distort PISA to meet their own ends. They cherry pick results and recommendations, ignoring the detail. Finland is famously quoted by the right as a high performing PISA country. Yet, it is a small, homogeneous country with no streaming, high levels of vocational education, no substantial class divisions and no private schools. Facts curiously ignored by PISA supporters.
5. Faked data
Eysenck worked with Cyril Burt at the University of London, the man responsible for the introduction of the standardised 11+ exam in the UK, enshrined in the 1944 Butler Education Act, an examination that, incredibly, still exists in parts of the UK. Burt was subsequently discredited for publishing largely in a journal that he himself edited, falsifying, not only the data upon which he based his work, but also co-workers on the research. To be precise, Burt's correlation coefficients on IQs in his twin studies were the same to three decimal places, across articles, despite the fact that  new data had been added twice twice to the sample of twins. Leslie Hearnshaw, Burt’s friend and official biographer, claimed that most of Burt's data after World War II were fraudulent or unreliable.
This is just one of many standardised tests that have become common in education but many believe that tests of this type serve little useful purpose and are unnecessary, even socially divisive. Many argue that standard tests have led to a culture of constant summative testing, which has become a destructive force in education, demotivating and acting as an end-point and filter, rather than a useful mark of success. Narrow academic assessment has become almost an obsession in some countries, fuelled by international pressure from PISA.
6. Dirty data deeds
One example of data gathering in education stands out as truly evil. In 1939, the CEO of IBM, Thomas Watson, flew across the Atlantic to meet Hitler. The meeting resulted in the Nazis leasing the mechanical equivalent of a Learning Management System (LMS). Data was stored as holes in punch cards to record details of people including their skills, race and sexual inclination and used daily throughout the 12 year Reich. . It was a vital piece of apparatus used in the Final Solution, to execute the very categories stored on the apparently innocent cards - Jews, Gypsies, the disabled and homosexuals, as documented in the book IBM and the Holocaust by Edwin Black. They were also use to organise slave labour and trains to the concentration camps.
This is not the first time the state has recoded educational details to keep tabs on potential dissent. It was common in the Stasi infused East Germany. I shared a room at University with someone who became a Stasi spy in the UK and have taken some interest in their methods. Perhaps the most meticulous storing of data ever taken by a state, right down to smells in jars, from clothing and towels placed under interviewees during interrogation. The idea was that dissenters could be found by dogs, when necessary.
Conclusion
I am a fan of Big Data in education, even though it’s really closer to ‘large or little’ sets of data. However, we must be wary of data when it is used to exaggerated claims through addition or subtraction or spearhead prescriptive programmes and extreme testing. I am appalled at the way politicians and educators take up PISA, PIAC and OECD data, with little or no detailed examination of their assumptions or relative values and use it to shape prescriptive policies that do more harm than good. Big Data in the hands of little brains is downright dangerous.