Sunday, January 23, 2005

Everybody's an Expert: When Cognitive Science Goes Public

I have heard many linguists complain, in Rodney Dangerfield voices, that they can't get no respect. A popular belief among the general public (and even among many scientists) is that just about everybody learns to talk at a pretty young age, so language can't be that hard. All of those alphas, thetas, and X-bar thingies seem pretty silly when you think language is really simple. The same is unfortunately true of cognitive science in general. Try to explain to your grandmother that you study how the visual system recognizes and individuates objects, and she'll probably say something like, "I look at a table, and see it's a table. What don't you understand about that?" In other words, people think, perceive, and speak everyday, and have for as long as they can remember, so they figure they've got a pretty good handle on how it works. As it turns out, many (if not most) of our conscious intuitions about our own thought processes are pretty damn wrong, but that can't be helped. We simply don't have access to the relevant information, so we're essentially creating stories to explain the information we do have. That's pretty much what our conscious minds do. And when we actually study cognition empirically, to get at the information we can't get introspectively, it turns out that thinking is a pretty complex thing.

You're probably wondering, what brought all this on. Well, I'll tell you what brought all this on. This did! Now, I assume that Todd Zywicki is a fairly intelligent guy. He's on the faculty at what, for all I know, is a good law school. Yet, it appears even this career intellectual isn't immune to the, "I know how my mind works" syndrome when it comes to evaluating cognitive science research. If you follow that link, you will find one of the most intellectually vapid critiques of a scientific research program ever. Zywicki demonstrates quite well that he knows nothing about IAT or implicit attitude research, writing nonsense like:
Is it really plausible that my impression of Bill and Hillary is driven more by whether I have a messy desk than my personal perception that Bill Clinton is a liar and Hillary Clinton is a megalomaniac and opportunist?
No, it's not, Todd, but that's now what the IAT is about. The IAT is about stable attitudes that may not be measurable through explicit measures (which is not to say that they are never measurable explicitly, another misunderstanding Todd seems to have), not ephemeral attitudes that are based on things as irrelevant as how organized your desk is at the moment. I don't know if such attitudes exist, but if they do, the IAT was not designed to discover them.

Now, I'm no fan of the IAT (as I've said before), but my problems with it are methodological. Other implicit tests that avoid the IAT's methodological problems (e.g., evaluative movement assessment) have also shown that implicit attitudes (or evaluations, or valences, or whatever you want to call them) do in fact exist, can be measured, and correlate quite nicely with several behaviors. Now, neither the IAT nor EMA can tell us where these evaluations come from, but that's not their goal. The fact remains that a lot of rigorous research has demonstrated the existence of such attitudes or evaluations, and Todd's disbelief in them doesn't amount to anything like a critique of that research.

Yet, Todd thinks, so he assumes he's qualified to talk about thinking. I wonder, if a physicist had conducted some research that produced a finding that Todd disagreed with, but Todd had not bothered to read up on the actual research, would he write a post calling it absurd? I mean, Todd has existed in the physical universe for as long as he can remember (I'm assuming, of course, that Todd hasn't had any major bouts of psychosis in the past), so he should be able to comment on any findings that concern that universe, right?

My real suspicion is not that Todd doesn't like the IAT. He knows shit about it, so how could he not like it? What I really think is that Todd doesn't like the idea that much of his mental life goes on below the level of his awareness. That's not surprising. I'd bet that a lot of people dislike the idea of that. So, Todd lashes out at a methodology, an area of research, and an entire scientific field, because he's uncomfortable with the truth. I just have to keep reminding myself that this is one of the hazards of my chosen field. When you study something people already think they're experts on, you're going to have to deal with stupidity sometimes. I suppose I should thank Todd for reminding me of that.

(Link to Zywicki's post via Universal Acid, who also thinks the criticisms are stupid.)

UPDATE: In an update at the end of his post, Zywicki responds to my criticisms, and those of Andrew at Universal Acid. I'll ignore the fact that he endorses evolutionary psychology (which is never a good sign), and focus on the remarks that are actually relevant. He writes:
I think the study of cognition and unconscious reasoning is very useful and explains much. I just think that it is important in studying this, as with everything else, that we remain aware of the limitations of the work and, in particular, make sure that the conclusions and implications we draw are actually supported by what the experiments are actually calibrated to test.
Of course, this is what IAT researchers, and others in the field are already doing, and nowhere does Zywicki provide evidence that they are not "Aware of the limitations of the work" or "[making] sure that the conclusions and implications we draw are actually supported by what the experiments are actually calibrated to test." Of course, as I've noted, there may be problems with inferences of "prejudice" from the IAT, but there is still a debate going on about that, and since Zywicki neither references that debate or any of the evidence, his criticisms seem misguided.

Zywicki also confirms my suspicion that his real beef is with the idea that many of his attitudes may have unconscious sources, as he writes:
Clearly many of my beliefs and actions are motivated primarily by my subconscious, equally clearly to me many of my other beliefs and actions are motivated primarily by my conscious, and most is in-between. I recognize that my love for my family or the Pittsburgh Steelers is heavily rooted in my subconscious mind; but I also find it much more likely that my slight preference for Bill versus Hillary Clinton has a lot more to do with my conscious.
The problem is that what research like that on IAT, EMA, goals and evaluations, source monitoring, conscious will, and automaticity has demonstrated is that Zywicki is in no position to determine the extent to which the sources of his attitudes are conscious or unconscious. He can, of course, consciously access his attitudes towards Bill and Hillary, but their sources may (in fact probably do) allude him. That he insists on denying this, without reference to any evidence, only exacerbates my frustration with yet another non-expert who feels qualified to speak with authority on issues he knows little or nothing about.

Which brings me to Zywicki's dumbest remark of all, the one with which he justifies criticizing the work despite his ignorance of it. He compares IAT to astrology, writing:
Do I have to be an expert in the "science" of horoscope reading in order to reject the proposition that "the stars" are in control of my life? I think not.
No, Todd, but you should at least be aware of the differences between science (which IAT research is, and you've offered no reasoned argument to the contrary) and astrology, before comparing the two.

UPDATE II: Zywicki responds again, this time in an entirely new post (once again, via Andrew of Universal Acid, who happens to have added an update to his original post as well). The gist of Zywicki's argument, which he originally used in an email to me, is that if most of the sources of our attitudes are not available to awareness, then it's likely that his (and my) attitudes toward the IAT are not available to awareness, and therefore there's no reason for me, or Andrew, or anyone else to try to convince him to change his mind. In making this argument, Zywicki once again indicates a complete ignorance of cognitive psychology, and I promise you that because he insists on commenting out of complete ignorance, this will be my last attempt to address his "arguments." To do so, I will clarify a few things:

1.) Just because we are aware of, or consciously guiding, our reasoning on a particular topic, doesn't mean that our reasoning cannot be influenced by unconscious processes, associations, and attitudes (valences). In fact, the information of which we are consciously aware is largely the product of unconscious processes which produce a coherent and generally verbalizable (except, of course, in the case of some aspects of visual consciousness) output. Put differently, the information of which we are consciously aware has been polished by unconscious processes, which have largely determined their content.
2.) Just because we are not aware of our cognitive processes and representations (in part or in their entirety), doesn't mean that information of which we are aware (e.g., arguments or facts that others communicate to us) can't influence those processes or representations. In most cases, complex conceptual information must be attended to be processed, and that means we're probably going to be aware of that information, at least while we're encoding it (i.e., while it's present in working memory).
3.) Just because we are consciously aware of our attitudes, behaviors, etc., and perhaps even have some conscious idea of our reasons for them (or consciously reason about them), doesn't mean that the ultimate attitudes, behaviors, etc. are in fact the product of conscious thought. As Dan Wegner has shown over and over again, our inferences of conscious will or forethought are often mistaken.
4.) Just because our ideas are the products of unconscious processes doesn't make them any less rational. If we had evolved to have a largely irrational cognitive system that is supplemented by a fairly small, rational component, we wouldn't have lasted very long. In fact, I would argue that in many cases, the unconscious processes are more likely to be rational (in the behavioral sense, which means something like "optimal") than those that are influenced by conscious thought. Consider a basketball player who is shooting free throws. Most of the time, he does so "unthinkingly," which is to say, utilizing wholly automatic and largely unconscious processes. Occasionally (especially if he's missed a few in a row) however, he will start to "think" about the shots, and reason about how to shoot free throws consciously. His accuracy will almost certainly decrease when he does this, because the automatic processes have been honed through years of practice, and all the conscious reasoning does is interrupt those processes. Zywicki is likely to object that this example isn't quite like his well-reasoned, conscious beliefs about the Clintons, and he'd be right, to some degree, but he'd also be missing the larger point. The unconscious mind isn't some Freudian playground in which nefarious drives and forces produce beliefs and attitudes willy nilly, with no justification. The unconscious mind is dominated by processes that have been honed through millions of years of evolution, and years of life experience, to produce actionable, adaptive outputs.
5.) While it's understandable that Zywicki would hold an antiquated view of the mind, since he's clearly unacquainted with any psychology literature whatsoever, it is not clear why he holds one particular misconception, namely his belief that the IAT only measures unconscious attitudes. Nowhere on the IAT site, or in any of the various publications on IAT in the peer reviewed literature, is this claim made. The word "implicit" in "Implicit Attitude Test" denotes the nature of the test, and not necessarily that of the attitudes it is supposed to be measuring. If Zywicki had actually read the Project Implicit website, he would have learned this, and perhaps even learned that the use of implicit tests is necessitated by the problems involved in getting people to report their attitudes when those attitudes may be socially undesirable. It is true that the IAT people tend to believe that some, and perhaps many attitudes are implicit, and the empirical evidence supports that belief, but critiquing that belief does not count as a criticism of the IAT.

10 comments:

Anonymous said...

What did you expect? The only thing that these guys can accept from psychological research is dubious IQ stuff and sloppy evo-psycho thinking. That is, everything that may legitimate their felt superiority. Otherwise, it's all garbage to them. They simply can't accept stuff like the IAT, because they have to protect their belief in free will and in a (chosen) immaterial soul. Try to imagine how Allport's "The Nature of Prejudice" was received in those circles, back in the 50's...
Noticed the authoritarian way in which he insists to associate "anarchy" to "order" in a simple, plain semantic task? We should attach SCR wires to those guys when they pass such tests, how they must hate to see their own biases online, right before their eyes, their ANS must be trumpetting. 

Posted by onclepsycho

Anonymous said...

I agree with you on everything except for one thing... My name is Andrew, not Anthony (of Universal Acid). :) I've also responded to Zywicki's response on my original postBy the way, Zywicki has a new post up. 

Posted by Andrew

Anonymous said...

Oops, sorry about that Andrew. I was going from memory, and as I can tell you from extensive personal and scientific experience, memories, particularly mine, suck. I've corrected it. 

Posted by Chris

Anonymous said...

By the way, the "edited" comments in the new posts are from my email to him. 

Posted by Chris

Anonymous said...

What I find irritating about project implicit are the frivolous political assumptions it makes. Project implicit claims 1) that people are subconsciously racist, homophobic, sexist, etc. and 2) that this is politically significant. Project implicit has rather shamelessly courted media and popular attention, leading me to suspect that the project sees itself as "educating" the American public on their moral and political turpitude. I think that there is a strong distinction between attitudes that respondents will reveal candidly to researchers and those that they will not. The former, I would argue, are far, far more politically significant than the latter. Why is this? Because biases openly expressed are more likely to be openly discussed in the political arena and are more likely to become the basis for political action. Groups of people are not likely to organize sustained, successful political action on bases they are too embarrassed to reveal to each other or the outside world. There is a distinction between unspoken individual biases and successful political action, in other words.

For example, I am sure that a similar test would find that the American public is subconsciously anti-Semitic, anti-Catholic, anti-Mormon, and anti-Jehovah's Witness. Would this make a political difference? Heck no. Yet Project Implicit has chosen to only test for biases which it perceives as politically significant.

As an example of why I don't think these types of tests should be used for political purposes, you often post on this blog about your liberal preferences on political issues. These, I am sure, are motivated by subconscious views of how the world should operate. You are likely, for example, to be reflexively distrustful of political programs proposed by evangelical Christians and "free-market" libertarians, regardless of the merits of those programs. Your dislike of Zwycki, right down to your mispelling of his name as "Zywiki" like "Wiki," the computer program, stems as much from your political opposition to his views and other subconscious biases, as what you perceive as his ignorance. Are your political views totally absurd because they are motivated by subconscious bias? If you answer that question in the negative, than you have rejected the principal claim made by project implicit.  

Posted by Noah

Anonymous said...

Noah, you make good points. While I disagree with the idea that implicit attitudes don't have implications (the EMA study I linked actually demonstrates one such implication: people who are undecided will end up voting in accordance with their implicit attitudes), I do agree that their political implications are no greater than those of explicitly expressed attitudes. That's cetainly not inconsistent with the claims that the Project Implicit people make on the website or in their peer reviewed papers.

I also agree that the project implicit people have brought this on themselves, by making their test public. It's inevitable that people will misunderstand complex concepts that aren't sufficiently explained, and while Zywicki's comments ignore even what they do say on their website, many of the other criticisms I have seen have been based on misunderstandings that the project implicit people do nothing to avoid.

The great thing about science is that these things are debated not on blogs by people who haven't read the literature, but in the scientific literature, by people who are actually beholding to the empirical evidence. There is a debate going on about the political, social, and even personal implications of implicit attitudes, how they affect behavior (which is not entirely clear, yet), etc. These things are being debated by deriving predictions from the various positions, and testing them. It may turn out that implicit attitudes affect our behavior in profound ways (it's certainly not inconsistent with any data we have so far), but it may also turn out that they are less important than those of which we are aware. Or perhaps they're both equally impotant. All of these are possible, and dismissing any of them before the data is in is unscientific, and unintellectual. Zywicki's not participating in that debate. He's not even offering reasoned arguments. He's just dismissing things out of ignorance. That's just stupid.

Finally, I agree that the IAT, and better implicit measures (including the go/no-go version of the IAT, and EMA) should not be used for political purposes. We don't know enough about what's going on to use them for such purposes yet. I think Project Implicit is designed more for shock and awe than with real political goals in mind, and I haven't really seen Greenwald or the others try to derive political views or policies from them, but if others think it can be used for such purposes, they're sadly mistaken. I made that point, all be it directly, in my first post on the IAT. 

Posted by Chris

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

p.s., I fixed the spelling of his name. Thanks for pointing that out. 

Posted by Chris

Anonymous said...

Noah,

1. How ironic that you yourself have misspelled Zywicki's name as Zwycki.

2. It's simply not true that covert biases are politically unimportant:

a. covert biases significantly impact people's lives. Suppose you are a black man: when you go shopping, the clerk follows you suspiciously; when you walk with your black friends at night, people cross to the other side of the road or clutch their bags more tightly; taxi drivers refuse to pick you up at 3am. Some of this is overt racism, but some is also completely unintentional covert bias. Yet who will deny that this is all part of the institutional racism that destroys the dignity of African-Americans? And that this racism is still an important force for political action on both sides?

b. unspoken biases can very easily form the basis for political action. Witness the Republican Party's "Southern Strategy," which features coded appeals to Southern white racism. No one need openly speak of racism for racism to form a basis for collective political action.

3. Project Implicit is not arguing that all political views motivated by subconscious bias are absurd, and the fact that you think this is its "principal claim" suggests that you still don't understand what Chris has been saying about the cognitive unconscious. See point #4 above, "Just because our ideas are the products of unconscious processes doesn't make them any less rational."  

Posted by Andrew

Anonymous said...

The cold winter, does not seem too for wedding. Most recently, the cold one after another,

hangzhou air temperature pelter, let many wedding in late November bride is due a single wear

gauze or dress, will feel cold, wear too much and feel very bloated. For the bride, the winter is

the biggest test how wedding in temperature and balance between poise.

Don't be too upset, actually this season only fees cheap Wedding Dresses, marriage can create a different

character "winter wedding", also more memorable.

You can use the glittering and translucent white fairy tale, cheap cell phones
,the artistic conception to dress up oneself's wedding, choose blue, green, white ice cold tonal,

decorous atmosphere to create beautiful, Can borrow snow machine and bubble machine build indoor

romantic atmosphere, snowflake, feathers, Christmas tree,cheap cocktail

dresses
, even is the element such as silver crystal, can add to your winter wedding dreamy

colour, cheap

jewelry
you can even in a pile of snowman YingBinChu lovely, guests, we must take it and will

soon be well.

On New Year's hottest r306c SONY Ericssoncheap cell phones

wholesale
, r306 first win in shape, give a person the feeling of professional and heavy. Its

design clamshell has hidden qualities. Although the low-end market positioning, introduction, SONY

Ericsson R series launched two must let those radio enthusiasts. And there are cheap ones, let the

public mobile phonewholesale of delight

The word "shanzhai" is very popular in China. It is industry phenomenon of imitation, high quality

and low price. Moreover, cell phones are the

representative of them.The global financial crisis led to a significant shri10:25 2009-12-26 nkage

of assets for people. "china wholesale" demonstrate the

might. most of them would like buy the cheap cell phonesnow .

Through google search engine, when you put in "discount cell
phones", you will find more than 15,000 pieces of related information.Cell phone china has been

concerned in the world. Many

international sellers can get a considerable profit by ways of wholesale cell phones from China

and switch selling. So it is a very

good job on cell phones wholesale in the current cosplay costumes

financial crisis.