College, Critical Thinking, and the Null Hypothesis

Posted on by

According to a WSJ story, as described by Newsweek,

The Journal found that at about half of schools, large groups of seniors scored at basic or below-basic levels. According to a rubric, that means they can generally read documents and communicate to readers but can’t make a cohesive argument or interpret evidence.

The WSJ story itself says that

At some of the most prestigious flagship universities, test results indicate the average graduate shows little or no improvement in critical thinking over four years.

Sounds like another win for the null hypothesis.

21 thoughts on “College, Critical Thinking, and the Null Hypothesis”

  1. I wouldn’t be so hasty to score a win on this one.

    The trouble is that the the CLA is a negative-stakes test. So it’s a really useless methodology and a bad, unreliable measurement.

    In fact, this is so obviously true, that if anyone really cared about getting accurate measurements, they would create some positive incentive, at least something like “cum laude” latin honors on the transcript for high improvement or high scores. The fact that most colleges require the test, but none of them create positive incentives for good effort and performance, is revealing.

    I can attest to this fact from personal experience, as I took the CLA upon entering and exiting my undergraduate university. In fact, I couldn’t get my transcripts without doing so, which makes sense, because otherwise I wouldn’t have even showed up.

    There was zero incentive to put in any effort, and indeed, some positive incentive to answer C’s all the way down and write the shortest possible essays in order to turn the thing in quickly and enjoy the rest of ones afternoon. I never even received my scores even just to satisfy my curiosity, but even if I had received them, it’s not like I could put them on a resume or use them on applications to graduate schools.

    I figure that most of the serious students put in a tiny bit of effort mostly out of personality and habit, and scored in a way correlated with general intelligence. And average intelligence of the student population and proportion of serious students already correlates with selectivity, prestige, and average SAT/ACT scores, so really, what’s the point?

    The moral of this story is GIGO. That doesn’t take away from the null hypothesis, but neither does it support it.

    • Still, the test should be measuring something if the incentives are the same freshman year and senior year. The cohort that rushes the test freshman year would roughly equal the number that rush it senior year, leaving us measuring the improvement of conscientious test takers.

      • So, they have no clue how to run a test, but they might do everything else right? 😉 ; )

      • That’s a good point, but there are some problems.

        First, it assumes that a measurement between two zero stakes tests is of equal validity to one between two high stakes tests. That may be true, but I’d like to see some evidence. In High School we have the PSAT and later the SAT, which both have high stakes, especially for the smarter kids with the National Merit problem. Obviously we see a big improvement in scores, and it’d be worth a dissertation project for someone to figure out some clever way of comparing that to zero-stakes testing that takes place at the same times.

        But second, the problem with that analysis is that it’s holding motivation constant per individual, despite big differences in test conditions. I agree it’s plausible that on average personality traits related to testing are probably pretty stable over a college career, however:

        1. Senioritis is a thing.

        2. There is a big difference between taking a test at the beginning of a semester, refreshed, with no other testing or projects going on, and the end of the year right after finals.

        3. Freshman coming out of High School high stakes testing like SAT’s, ACT’s, AP’s, etc. are more likely to believe (even erroneously) that tests of critical reasoning at the beginning of their college career – with scores that go in their “permanent record” might actually mean something and have some influence on their reputation, or the way they are treated in college, perhaps making them eligible to get into honors or other special programs. By the end of senior year, those speculations are clearly out of the realm of possibility.

        Bottom Line: I just can’t take claims based on CLA results very seriously. I’d give them some credit – they are better than nothing – but only a little. And this is coming from someone who is convinced by the null hypothesis and generally agrees with Bryan Caplan’s analysis of education.

        • But does not the fact they aren’t even trying to do it properly raise the status of the result when they attempted to do it incorrectly?

          If they aren’t measuring how can they possibly be improving it?

  2. I suspect Handle is correct about the no-stakes issue.

    Taking any test requires effort. Anyone who has taught undergraduates will have noted that many of them tend to conserve effort–it’s hard to tell why. “Conservation of Effort” may arise from being overwhelmed, from being lazy, from not having good drive and diligence, from sleep deprivation, exhaustion, poor self-regulation, or just from having their own priorities.

    I don’t think we are really going to have good evidence until there are two high stakes tests.

    Test Number One at the beginning of the freshman year.

    Test Number Two at the end of the senior year.

    Option: Dropouts and never-attended college can take at end of imputed freshman and senior year (had they attended and had they finished). Probably will need age cut off–no testing after age of 30? Unless you were air dropped from Somalia / Nigeria / Peru / Papua New Guinea into USA at age 29.

    Diligence and effort on both tests could be encouraged with cash awards, from a hedge fund manager or internet magnate, if not from government.

    Cash Awards for highest score at each and every school.
    Cash Awards for most improved between first and second
    Cash Grant to School with highest rate of improvement
    Cash Grant to School with higher percentage sitting for test.

    Option for publicly listing test score on transcript and campus newspaper / internet site. One more fact, just like playing high school football or making Eagle Scout.

    Basically make excellent test results worth enough money that some / many students will expend effort.

    The more the tests are worth, the less credible the lazy student’s claim of “I could have gotten a better score, but winning $x amount of money for a day’s diligent test taking just wasn’t worth the effort.”

    Problems:

    1. we have enough of a mandarinate already.
    2. Unintended consequences are sure to emerge.

    If I recall correctly, somewhere in _The Bell Curve_ Herrnstein and Murray called for such a test–maybe just taken once by all 17 year olds, with cash award from line in federal budget.

  3. Sounds like another win for the null hypothesis.

    Then why do the consumers of pay more for a degree? I work in an office that has a lot of jobs that don’t require a college degree but I have seen many of these positions require a degree over the years. Also, I have noticed the candidates with a degree tend to get the position during interviews than non-college degree. (In my experience I learned a lot from college on better writing. I still not that good but much better.)

    In theory, it does seem like business firms could become ‘Branch Rickey’ (creator of the Major League minor league system) where they get a lot of talent and train them for the majors. But these firms today are following more of the NFL in terms of college, draft and high school training.

      • I suspect there are SEVEN huge issues here:
        1) I see the primary problem here is Charles Murray does not have clue how labor demand works. For an economic libertarian he sure sounds like he does not know the office environment. Charles has some weird belief that if people stop getting college degrees than employers would stop paying more for employees. (I think IQ test are limited here and can’t large companies develop their own aptitude tests?) Also I don’t see the economic libertarians pushing their kids to college.
        2) How come the best employers tend offer tuition support for degrees?
        3) In terms of labor supply, Murray vastly underestimates the issues of job loss and getting rehired. Having college degree helps tremendously with getting hired again at decent pay level.
        4) Given the economic reality of Rust Belt towns, it sure seems like a career of a two year stint in the military and then working at the factory is not a good choice for the future.
        5) I think the problem of economy 2017 is labor supply is now in short supply for growing regional businesses. (Conor Sen is documenting this reality.) And the reason labor supply is limited is the birth rates were lower in the 1990s and young people don’t just want a job anymore. And Charles Murray and other economic libertarians don’t like this reality that young people are focuses on college and career and not just working.
        6) I believe what Charles Murray and Bryan Caplan truly hate is the late marriage and small families. This limits point 5 but in the case of Charles Murray has weakened the influence of religion on the population.
        7) The worst thing of very economic libertarians they sure seem like they desire the return of Victorian economic system. Or Charles Murray wants the return of company towns of 1900. These conditions don’t exist because the population did not like them anymore.

        • I like Murray for his clarity of exposition.

          He may have half-baked ideas and pet hobbyhorses, just like anybody else.

          = – =- = – =

          I tried replying here once–something happened so I’m re-entering a comment–a bit different this time.

          The issue is not just “office work” but all sorts of skilled trades, project-based work.

          Collin you raise a lot of good points above–many things are “an empirical question” in the end.

        • can’t large companies develop their own aptitude tests?

          They can develop their own tests but they can’t USE them for hiring, promotion, etc. until they have given them to a large enough group of people and then followed those people long enough to prove to the feds that the tests are actually “predictive.”

          So, realistically, no.

          • That sounds about right.

            It’s my sense that the Armed Services Vocational Aptitude Battery (ASVAB) is highly “g” loaded and serves as a proxy for an IQ test, as well as subcategories such as spatial relations and mechanical aptitute. ASVAB is used in part for allocating soldiers to positions and specialties. Perhaps it is also used to disqualify those who appear, at first inspection, to be “as dumb as a stump.”

            I took it on a lark in high school and did well–into my 40s I occasionaly would hear from recruiters. Otherwise the existence of the test and its role would probably be unknown to me.

            As I understand the issue, the problem is in part that the private sector cannot administer anything similar and then use rather like the way that the military uses the ASVAB. Duke v. Griggs Power made it illegal to do so.

            My guess is that barring exams, businesses look at anyone who has won a “tournament.” They use certain tournaments for proxies. (Getting into Yale, for example). Kevin Carey’s blog first drew this point to my attention–the existence of tournaments.

  4. I’ve always found this anecdote from Joseph Epstein during an Uncommon Knowledge interview to be insightful as to the fact that critical thinking, actual study, is not rewarded by the professors except by happenstance.

    The system looks for gameshow answers, whereas true learning is brooding, full of blind alleys and relentless. Learning to do well in the gameshow environment of the university does not promote critical thought or learning. The latter may come later, although student exposure to real scholars rather than striving academics is rarer now so they encounter fewer examples of critical thinking in practice.

    https://www.youtube.com/watch?v=JF2eJSHKKd0&feature=youtu.be&t=1049&ab_channel=HooverInstitution

  5. So, either Ho gets a point or people don’t know how to do a study gets a point. I think I’ll give them both a point.

  6. How much evidence do we have that critical thinking is a valid construct? I have certainly seen the term used in a variety of ways. If the test is measuring a phantom, you might expect unclear results.

  7. As a teacher, I’m struggling to figure out the difference between the potential of the student and the performance. The performance of many students is depressing, no doubt. But because they are at school by compulsion, many of them will not demonstrate their capability. Right? I sure hope so. I’ve created an underachievement index to try to explain the extent to which they are slacking off. The interesting thing is to see how they perform in their last year of school, when it really matters. For my small yr 12 economics class, I found a correlation of 0.9 between their year 9 standardised test scores for literacy (a general ability test) and their assessment performance in year 12. This would support the null hypothesis if it were a bigger sample.

  8. In general, I’m a believer in the null hypothesis – but this one doesn’t support it.

    Yes, students at top schools don’t have critical thinking skills – but there is no evidence that anyone is trying to teach critical thinking. In fact, you could make a strong argument that many schools are striving for the opposite outcome.

Comments are closed.