A reader points me to a piece by Dale C. Farran and Mark W. Lipsey, which studies the long-term effects of the TNVPK pre-kindergarten program.
As is evident, pre-K and control children started the pre-K year at virtually identical levels. The TNVPK children were substantially ahead of the control group children at the end of the pre-K year (age 5 in the graph). By the end of kindergarten (age 6 in the graph), the control children had caught up to the TNVPK children, and there were no longer significant differences between them on any achievement measures. The same result was obtained at the end of first grade using two composite achievement measures (the second created with the addition of two more WJIII subtests appropriate for the later grades). In second grade, however, the groups began to diverge with the TNVPK children scoring lower than the control children on most of the measures. The differences were significant on both achievement composite measures and on the math subtests. Differences favoring the control persisted through the end of third grade.
The null hypothesis is that educational interventions make no difference. Technically, the last two sentences suggest that the null hypothesis is rejected here. The intervention of sending kids to pre-K made their outcomes worse in a statistically significant way.
Read the whole article. This will create some cognitive dissonance for progressives who have faith in universal pre-K and also believe in using rigorous social science to guide policy.
And some cognitive dissonance for James Heckman. He argues that measurements at third grade are noisy, but lifetime outcomes favor pre-school education. Pointer from Mark Thoma. I think Heckman is really reaching.
Another case of nothing that can be measured is important and nothing important can be measured.
Again, kids and parents get graded every single day and some days twice. So don’t tell anyone.
The solution is to put the power in the hands of people who care about quality and are close to the situation (i.e., parents), not in the hands of teachers’ unions.
Why bring teachers’ unions into this? That’s crazy.
The reason the state–not unions–requires education is precisely because so many low income parents are profoundly uninterested in school, period, quality or not. Urban school districts spend thousands of dollars a week hunting down truant kids who don’t go to school. Absenteeism is a huge problem for k-6 because of uncaring parents. After 6th grade, the truant kids themselves are uninterested, whether the parents are or not, and the parents have no control.
So leaving it in the parents hands is, forgive me, a moronic idea. The parents who do care are interested not so much in school quality, but peer quality.
That is, by the way, why so many progressives push the idea of integration. Peer quality won’t so much improve education outcomes as it will reduce class disruptions and improve the likelihood of a kid behaving long enough for information to sink in.
However, the parents of the desired peers want nothing to do with this idea because they, too, understand the importance of peer quality.
What probably would work is *not* more choice, but less choice, particularly for the problematic peers. Instead of having charters for motivated kids, have charters for the kids who aren’t controllable, who deliberately upset classrooms or don’t care. Give them charter school that’s basically bootcamp, making it a very undesirable place. Then the marginal kids will be more likely to behave, making schools full of poor kids more manageable.
But unions? I mean, what the hell. It’s bizarre how many people think they are responsible for school outcomes, one way or the other.
Maybe bootcamp for mid to older kids but what about kinesthetic learning for the younger disrupters? Expecting kids to be at the same point exactly when their development slopes are steepest and most divergent is insanity. The only thing we know for certain is that is wrong.
And if you physically remove people what you are diung most of all is just waving the white flag in surrender to signaling.
This is what the null hypothesis implies. We are spending unbelievable resources doing things and we have no idea what we are doing.
Unions: as long as we have no clue what we are doing I am not going to blame anything. But do unions not impede experimentation in general and any changes that are perceived as negatively impacting its members? If not, then they aren’t doing their job.
Look, one of the top priorities of the teacher organizations is to advocate for the teachers to dialogue with administrators/mgt regarding all issues in their workplace. That said, there is frequently a bias toward protecting negotiated rights and benefits. Many of these are good for the schools. It is not a fact that teacher organizations are against innovation. In fact I would challenge anyone to point out a profession where continuous change is greater than in education and where employees have regularly embraced and initiated experimentation and change.
As with any relationship, employee/employer relationships in education have many examples of both good and bad faith interactions. There is a damned good reason that employees have representation before employees. That is the only thing beyond good faith and good intentions of the employer to counterbalance a power relationship.
On the whole, teachers and their professional representatives act responsibly and in the interest of both their students and the teachers.
I’m totally on board with the null hypothesis. What I’m saying is that unions are just unrelated, and that parents don’t make good choices, which hurt the kids and society at large–at least, that’s the rationale behind public ed.
I am then pointing out the push for integration is basically to “dilute” the poor behavers and mitigate the damage of a bad peer set. Because whether we have proof or not, parents will always demand the best possible peer group for their kids.
I think this is a comment to the previous post, “Solution Disconnected from Problem.” Is there some way to move it there?
Since Brown the story has been that ‘dilution and integration’ would be the key to close the achievement gap, but in the many dozens of mandated experiments across the country, nothing ever panned out. Busing didn’t work, so this time, instead of moving the kids away from their neighborhoods, we’ll try moving the kids into different neighborhoods. I doubt the result will be much different.
There is a little evidence for non-academic ‘character’ benefits such as less disciplinary trouble, better behavioral habits, attendance, and higher graduation rates. But it’s not a slam-dunk case by any means.
The trouble is that even if it’s better for the integrated kid, ‘peer group effects’ might be negative sum overall if “one bad apple spoils the bunch”, peer-group benefits require extreme concentration, or if the number of problem children can hit a critical mass of, say, 20% or something. Whatever they say, people certainly act as if these things are all true, when their own kids are on the line.
So, in the progressive political language, the analysis is the familiar one of inequality and redistribution.
But the trouble is that you can’t just soak the rich like you can with normal redistribution or paying for government programs. Because when it comes to this kind of ‘social capital’ the unit of account is a child, and there aren’t enough rich kids to spread around. And there may not be enough ‘high social capital / positive peer-group effect’ kids to spread around in general, at any SES level – it’s not 1954 anymore. Or even 1984.
That’s going to make it awfully hard to enact any program based on democratic legitimacy, without some court or Civil Rights division in some department forcing it on people, like they seem to be doing with the AFFH regulation and regarding ‘affordable housing’ set-asides in zoning regulations (a good recent example is what happened to Westchester County in New York)..
The reason is that you have no choice but to soak the middle class in what is a clear net loss for them, like with Medicaid. At least with Social Security, ACA, Medicare, and some other programs, the idea is that “yes, you pay, but everyone is eligible to benefit, including you maybe one day.”
Not so with redistributing kids to spread the wealth of peer effects. This is more, “They probably gain but you probably lose,” which is politically tough, especially if the expected value of the latter is higher than the former.
Spreading the bad kids around will reduce disruption? Where ya bin? Progressive la-la land? All it takes is one loony kid who hates the lesson and wants to be the center of attention. Why not lump all of the idiots in one classroom and let the rest of the kids learn?
Ask any teacher what really makes the difference in class performance…..the answer always leads back to the home. Are the parents (or parent) rein forcing the learning daily, or are they too busy, tuned out, or absent? You cant mandate good parents. Show me two parents that never take a day off or a play off when it comes to the training they give their kids, and I’ll show you a successful citizen. Show train their kids and half that dont, I can easily pick out the successful kids and the unsuccessful kids.
Interested to see a translation to plan english. What does this mean?
When progressives talk about choice, they berate it as if it is an intervention and bring up a lot of legit problems that would atend a grand intervention. When we talk about it, we view it as a movement away from grand intervention and view it more in terms of opening doors to marginal tweaks.
I am curious whether the result of worse outcomes in 2nd and 3rd grade will replicate and, if so, what the explanation is. I suspect that one possibility is “burnout.”
According to the linked article, “TNVPK [Tennessee Voluntary Prekindergarten] is a full‐day prekindergarten program for four‐year‐old children expected to enter kindergarten the following school year and whose family income qualified for free or reduced price lunch.”
So the kids in the TNVPK have had an extra year of full day academic day care–of sitting still, following the teachers’ directions, being expected to learn various things (and often failing). By the end of second grade, they may be rather tired of it. Especially if they come from a non-academic background (which is true of most poor non-immigrant children).
More schooling leading to less learning is certainly contrary to conventional wisdom. It is also contrary to various other studies. Part of that may be better study design. Unlike in many educational studies, where the experimental group had applied for the intervention (and may be supposed to be more interested in education) and the control group hadn’t, the control group here was children who had applied but hadn’t won the lottery to get in.
Another problem with many studies is that people who are not doing well drop out of the experimental group, skewing the results in a positive direction. Farran and Lipsey report that, “More than 90% of the sample remained in the study across the four years.”
In fact, there have been very few studies of pre-K effectiveness that go beyond a few years. This, alas, is true of most education research.
I also wonder if the pre-K programs in some other studies were boutique programs that do not scale. This was certainly the case with the Perry Preschool, which is almost 50 years old now and whose exceptional results, as far as I know, have never been duplicated.
Fans of Finland will point out that Finnish kids don’t start real academics until they are seven, and then get superior results on international tests when they are in their high school years.
Coda: I read to the end of the linked article before I hit Post Comment and found this. Seems my suspicion is hardly original. “State programs that are not careful to protect the instructional environment for 4-year-olds may find the children burning out in the early grades from too much repetition of the same content and instructional format. Rather than building enthusiasm for learning, confidence in their abilities and a foundational understanding of literacy and math, the programs may only be teaching children how to behave in school, an enthusiasm that fades with repeated exposure.”
Maybe they just get a short term boost to sitting still which helps initially but is irrelevant later outcomes.
When my kid entered kindergarten I realized it was sit-still-and-listen bootcamp and the grading was based largely on this skill. First and foremost schools need kids who can be managed 30 to 1 by teachers sitting in front of a corral. Otherwise the system doesn’t work. This is murder on a young 5 year old (kindergarten signaling). But once that is achieved the actual instruction begins.
This is another case of tolerating large but self-terminating programs as preferable to trying to debate the conclusions of small studies. That’s as close as we can get to prediction-market-style ‘policy impact betting’.
So, sure, everybody in some big district gets free pre-K for a few years of study. Then we measure the results, and if it doesn’t measure up to what the proponents claimed would happen as detailed in the legislation itself, then it self-terminates and provides very good evidence for everyone else.
If the stated threshold impact is low, then the legislation reveals itself to be a joke at the outset – “All this money and effort when only a 0.2 standard deviation improvement is expected. And why do the scholars not expect any more improvement than that?” If the threshold is high, then if the claims are exaggerated or bogus, the program will die by its own hand.
Same could be done for Chetty. Oh, the integration into high-mobility zip-code 12345 didn’t actually increase mobility among the study group? Auto-terminate the whole program, no matter how large or disruptive.
The point is to avoid government by baloney sandwich. The best way to suppress the sandwich is to make people personally liable for errors in their claims. Second to that, it is make their preferred policy agenda self-reversing should they exaggerate its purported impact.
I like to say, from now on, every day is an experiment. But it is only science if we record the results.
My guess here is the optimal solution for kids is to have a married couple with a single income with minimal pre-K government solutions. These universal Pre-K studies have a secondary function in which child care during the day is provided.
Even as a progressive, I tend to think most pre-K schooling has value for 4 -5 hours a week for kids. All they need learn is the basics and being away from Parents for a short while.
Heckman data concluding there is significant lifetime benefit and ROI to pre-k education are driven primarily by incarceration rates as adults. Many who have looked at the data are skeptical that not attending pre-k actually impacts incarceration rate that dramatically 20-30 years later. Without these incarceration rates attributed to not attending pre-k and the resulting high cost of imprisonment the data doesn’t support his conclusion in the same way.
When I looked at it the causal mechanism it seemed to me to be due to bad parenting which meant any time away from the parents benefitted the kids. There are cheaper and more effective ways of reducing the,effects of bad parents.
20 – 30 years later?? Try 10.
A quick google search uncovers a lot of studies that just show the opposite–some preschool programs are effective. Not just the Perry preschool, which was one of the costlier ones.
Those studies tend to have improper control groups or to be short-term. No one argues that kids who go to pre-school because “their parents care about education” do better than kids who don’t. Nor does anyone argue that there are short-term gains.
This research suggests that when you compare apples to apples over a four-year time period, the gains eventually don’t just fade out; they reverse.
Not having read the research, I wonder if the samples measured to whether low income demographic groups, when measured within that demographic, showed any improvement in tests over a control group of low income children. In other words, maybe low income children who have the early childhood education perform better than without it. Do low income children still do better with the pre-k intervention than those who do not even if they exhibit a gap with the population at large from higher income children?
This research compared two groups of children, all of whom were eligible for free or reduced lunch. They had to be to get into the Tennessee Voluntary Prekindergarten. A parent or guardian also had to apply. There were more applicants than spaces, so applicants were accepted by a lottery. The control group here were the kids who lost the lottery. The researchers compared the two groups on various measures and found them to be very similar.
Thanks.
Apples to apples.
I wonder if the explanation might be that the pre-K kids get discouraged because the no-pre-K kids catch up to them so fast. The study says the pre-K kids are ahead in Fall of kindergarten and then equality is reached by the end of first grade. It says the teachers notice bad attitudes by the pre-K kids in spring of first grade, *before* the performance lag. Then in second and third grade, teh pre-K kids do worse, but their attitudes are the same as the no-pre-K kids. I wish they’d asked the children what they thought about their situations.
“I wish they’d asked the children what they thought about their situations.”
Our results surprisingly show, contrary to expectations, that “recess is too short” is the cause of this longitudinal attitudinal disadvantage.