| Home |
| About Us |
| the Review |
| Links |
| Contact Us |
Censorship-Braggadocio Files
Economists and others who should know better.
-
Skip to: The Physicists of the Social Sciences
Skip to: They should know better.
Skip to: Until economists do it,....
Skip to: Until think tankers do it,....
Skip to: From some, you expect it.
Skip to: Sometimes, there's simply no explanation for it.
Economists discover testing (a century after everyone else [yet, somehow, still first])
Educational standards and standardized tests have existed for millennia. Psychologists first developed the “scientific” standardized test over a century ago and they, along with program evaluators and education practitioners, have conducted hundreds of thousands of studies with or about them since.
Nonetheless, over the past decade, a number of economists have proposed that none among these scores of psychologists ever thought to study the various effects of various types of educational testing. The world needed the unique insight of economists to think of conducting such clever studies.
If you are willing to believe what is writ above, you should have no trouble believing the claims writ below.
Selected quotations on standardized testing, listed in reverse chronological order:
“Given what a decade of research tells us about test-based accountability it seems reasonable to think about policy changes... Here is my vision....
— Ladd, Helen F. Education Week, January 23, 2008, p. 27.
“As the first large urban school district to introduce a comprehensive accountability system, [in
the 1990s!] Chicago provides an exceptional case study of the effects of high-stakes testing— a
reform strategy that will become omnipresent as the No Child Left Behind Act is implemented
nationwide.
— Jacob, Brian A. (2003). High stakes in Chicago. Education Next. 1(66).
“...it is important to keep in mind the limited body of data on the subject. We are just getting
started in terms of solid research on standards, testing and accountability”
— Tom Loveless, of the Brookings Institution, as quoted in "New Report Confirms....," U.S.
Congress, Committee on Education and the Workforce, press release, February 11, 2003.
"...most studies of school-based accountability do not utilize individual students’ data and thus
cannot examine many outcomes of interest or investigate how effects vary across students."
— Jacob, B. A. (2002). Accountability, incentives and behavior: The impact of high-stakes
testing in the Chicago Public Schools, 2.
“‘Most of the evidence is unpublished at this point’ and the answers that exist are ‘partial’ at
best.”
— [as reported by] Olson, L. (2002, June 19). [quoting Erik A. Hanushek] Accountability studies
find mixed impact on achievement. Education Week. 13.
"Despite its increasing popularity within education, there is little empirical evidence on test-based
accountability (also referred to as high-stakes testing)."
— Jacob, B. A. (2002). Accountability, incentives and behavior: The impact of high-stakes
testing in the Chicago Public Schools, 2.
"While research on the outcomes of accountability systems is growing rapidly, it still represents a
young and highly selective body of work."
— Hanushek, Erik A. & Margaret E. Raymond. (2002, June 9–11). Lessons about the design of state accountability systems. Paper prepared for conference: Taking Account of Accountability:
Assessing Policy and Politics. Harvard University.
—Hanushek, Eric A. & M. E. Raymond. (2002, June). Improving Educational Quality: How Best
to Evaluate Our Schools? Paper prepared for conference: Education in the 21st Century: Meeting
the Challenges of a Changing World. Federal Reserve Bank of Boston.
— Hanushek, Eric A. & M. E. Raymond. (2003). Lessons about the design of state accountability
systems. in P. E. Peterson & M. R. West, (Eds.). No Child Left Behind? The politics and practice
of accountability (pp. 126–151), edited by Paul E. Peterson and Martin R. West, Washington,
DC: Brookings Institution.
"There has been little investigation of whether the purported benefits of these policies (of
standardized grade promotion testing)-in the form of increased achievement on standardized tests-actually occur. ...We know very little about whether the introduction of high-stakes testing,
particularly when combined with extra resources and with school accountability measures, will
increase achievement on standardized tests for all students prior to the promotional gate (both
those who are promoted as well as those who may later be retained."
— Roderick, Melissa, Brian Jacob & Anthony Bryk. (2002). The impact of high-stakes testing in Chicago on
student achievement in the promotional gate grades. Educational Evaluation and Policy Analysis,
24(4):333–357.
"In 1996, [our city's schools] became one of the first large, urban school districts to implement
high-stakes testing, introducing a comprehensive accountability program that incorporated
incentives for both students and teachers."
— Roderick, M., B. Jacob & A. Bryk (2002). The impact of high-stakes testing in Chicago on
student achievement in the promotional gate grades. Educational Evaluation and Policy Analysis,
24(4), 333–357.
“In 1996, Chicago began a national trend when it coupled a new school-level accountability
program with an accountability initiative with high-stakes consequences for students. ...Over the
past five years, virtually every major school system and many states...have instituted elements of
[our city's] policy."
— Roderick, M., B. Jacob & A. Bryk (2002). The impact of high-stakes testing in Chicago on
student achievement in the promotional gate grades. Educational Evaluation and Policy Analysis,
24(4), 333–357.
"Nearly 20 years later, the debate surrounding MCT [minimum competency tests] remains much
the same, consisting primarily of opinion and speculation.... A lack of solid empirical research has
allowed the controversy to continue unchecked by evidence or experience.... This paper...."
— Jacob, B. A. (2001, Fall). Getting Tough? Educational Evaluation and Policy Analysis.
"The lack of empirical research on the achievement effects of mandatory graduation exams is
striking, particularly in light of their growing popularity across the nation. The few studies that
have examined the impact of MCT on student achievement... "
— Jacob, B. A. (2001, Fall). Getting Tough? Educational Evaluation and Policy Analysis.
“...the evidence on graduation exams and achievement is limited and mixed, ....”
— Jacob, B. A. (2001, Fall). Getting Tough? Educational Evaluation and Policy Analysis, 334.
“…disinterested appraisals of the research on the predictive validity of test scores conclude that
there is only a very modest connection between test scores and productivity ratings by
supervisors. Indeed, an overall summary of the potential economic gains from using test scores
for employment selection suggests that the economic claims of industrial psychologists are flawed
and highly exaggerated.”
— Levin, Henry (2000). High-stakes testing and economic productivity, in Gary Orfield & M.
Kornhaber, (Eds.). Raising standards or raising barriers? Inequality and high-stakes testing in
public education. New York: Century Foundation.
"Given the widespread interest in school-based recognition and reward programs, it is surprising
how little evaluation has been done of their impacts."
— Ladd, Helen F. (1999) The Dallas School Accountability and Incentive Program: An evaluation of
its impacts on student outcomes. Economics of Education Review.
"...several states and a few local districts have introduced school-based incentive programs. This
paper provides one of the few evaluations of the effects of such programs on student outcomes."
— Ladd, H. F. (1999). The Dallas School Accountability and Incentive Program: An evaluation
of its impacts on student outcomes" Economics of Education Review.
Those who work in testing/accountability research have no excuse for dissing a century's worth of research literature on the effects of testing. They do it either because they do not make the effort to read the research literature, or they do not wish to.
"What of the impact of accountability on other segments of the student population—traditionally higher performing students? ...The average student? ...there is no obvious accountability mechanism for the "average student. There is little research on this issue"
— Herman, Joan (2007, October). Accountability and assessment: Is public interest in K-12 education being served? CRESST Report 728
"Research to date makes clear that score gains achieved under high-stakes conditions should not be accepted at face value. ...policymakers embarking on an effort to create a more effective system of ...accountability must face uncertainty about how well alternatives will function in practice, and should be prepared for a period of evaluation and mid-course correction.
—
Koretz, Daniel (2005, June). Alignment, High Stakes, and the Inflation of Test Scores, CRESST Report 655
"Thus, even in a well-aligned system, policymakers still face the challenge of designing
educational accountability systems that create the right mix of incentives—
incentives that will maximize real gains in student performance, minimize score
inflation, and generate other desirable changes in educational practice.
This is a challenge in part because of a shortage of relevant experience and research..."
—
Koretz, Daniel (2005, June). Alignment, High Stakes, and the Inflation of Test Scores, CRESST Report 655
"
Research has yet to clarify how variations in the performance targets set for schools affect the incentives faced by teachers and the resulting validity of score gains."
—
Koretz, Daniel (2005, June). Alignment, High Stakes, and the Inflation of Test Scores, CRESST Report 655
"In terms of research, the jury is still out
"
— Koretz, Daniel (2005, June). Alignment, High Stakes, and the Inflation of Test Scores, CRESST Report 655
"Despite the clear appeal of assessment-based accountability and the widespread use of this approach, the development of assessments that are aligned with content standards and for which there is solid evidence of validity and
reliability is a challenging endeavor. Alignment of an assessment with the content standards that it is intended to measure is critical if the assessment is to buttress rather than undermine the standards. Too little attention has been given to the evaluation of the alignment of assessments and standards."
— Linn, Robert L. (2005, April). Issues in the Design of Accountability Systems
CRESST Report 650
"As with all the previously-mentioned consequences [of the use of standardized testing] the evidence on this outcome is just beginning to come in. Because all research on the consequences of high-stakes tests is comparatively recent, the evidence on positive and negative consequences is
necessarily skimpy."
— Cizek, Gregory J. (2001, September). More unintended consequences of high-stakes testing?
Educational measurement: Issues and practice, 7.
"On nearly every front, Mehrens... concluded that: the evidence for a test's influence on either
curricular content or instructional processes is not totally clear; the evidence regarding the effects
of large-scale assessments on teacher motivation... is sketchy; and with respect to assessment
impacts on the affect of students, we are again in a subarea where there is not a great deal of
empirical evidence."
— Cizek, G. J. (2001, September). More unintended consequences of high-stakes testing?
Educational measurement: Issues and practice, 2, 3.
"...perhaps the most troubling aspect of the current debates is the almost total omission of any
serious articulation or consideration of the /positive/ consequences of high-stakes testing."
— Cizek, G. J. (2001, September). More unintended consequences of high-stakes testing?
Educational measurement: Issues and practice, 3.
“...all of the researchers interviewed [including Daniel Koretz and folk from FairTest] agreed with FairTest’s contention that research evidence
supporting the use of high-stakes tests as a means of improving schools is thin.”
— [as reported by] Viadero, Debra (1998, January 28). FairTest report questions reliance on high-stakes testing by states. Education Week.
“Despite the long history of assessment-based accountability, hard evidence about its effects is
surprisingly sparse, and the little evidence that is available is not encouraging. ...The large positive
effects assumed by advocates...are often not substantiated by hard evidence....”
— Koretz, Daniel M. (1996). Using student assessments for educational accountability, in E.A.
Hanushek & D.W. Jorgenson, Eds. Improving America’s schools: The role of incentives.
Washington, D.C.: National Academy Press.
“Although much has been written on achievement motivation per se, there has been surprisingly
little empirical research on the effects of different motivation conditions on test performance.
Before examining the paucity of research on the relationship of motivation and test
performance....”
— Kiplinger, V.L. & Linn, Robert L. (1993, March 3). Raising the stakes of test administration: The
impact on student performance on NAEP. CSE Technical Report 360. National Center for
Research on Evaluation, Standards, and Student Testing.
Until economists do it, it hasn't been done.
Lest one believe that economists' selective research memory applies only to the topic of testing....
"Few studies link principal attributes directly to student achievement,... This report includes new empirical research...." "Despite the large amount of attention that has been paid recently to understanding the determinants of educational outcomes, knowledge of the causal effect of the most fundamental input in the education production function - student study time and effort - has remained virtually non-existent. In this paper..." "Despite the growing debate and the thousands of under prepared students who enter college each year, there is almost no research on the impact of remediation on student outcomes. This project addresses this critical issue..." "...approximately one-third of entering postsecondary students require remedial or developmental work before entering college-level courses. However, little is known about the causal impact of remediation on student outcomes. ...This project..." "While high standards have been advocated by policy-makers, business groups, and teacher
unions, very little is known about their effects on outcomes. Most of the existing research on
standards is theoretical, generally finding that standards have mixed effects on students.
However, very little empirical work has to date been completed on this topic. This paper
provides the first empirical evidence on the effects of grading standards, measured at the teacher
level."
“There has been very little previous empirical analysis of teacher cheating. …Our paper represents
the first systematic attempt to (1) identify the overall prevalence of teacher cheating empirically
and (2) analyze the factors that predict cheating.” Until think tankers do it, it hasn't been done. "Despite the importance of arbitration, the process has largely escaped either scholarly or journalistic attention." "Little empirical evidence supports or refutes the existence of a causal link between standards and
enhanced student learning. ...we found few empirical studies of the impact of any of the types of
standards on schools and students."
"In contrast to the literature on the mostly deleterious and unintended effects of high-stakes testing, which is substantial and largely indisputable, research on the relationship between high-stakes testing and its intended impact on achievement is sparse.
"Despite all the positive and negative consequences of testing that have been discussed over the past few years, one of the most important questions regarding testing remains largely unanswered: Does test-based accountability result in increased student learning?
"Despite the widespread belief that state-mandated testing—standards-based or otherwise—contributes to educational improvement at the local level, evidence to support this claim has yet to be established.
"This shortage of empirical investigations means that the hopes of policy-makers and the public that more tests will somehow lead to better teaching or more learning rest on largely unvalidated assumptions.
"The validity of increasing the use and importance of (state-mandated) tests in order to improve the schools is a long step further in the unknown. In sixty years of vast international research on school testing, the policy of emphasizing test performance in order to improve education has never been validated.
Sometimes, there's simply no explanation for it. "Policy needs to affect what actually happens between teachers and students in specific classrooms. But we have little direct evidence about what teachers do in those rooms and how it affects students' learning. ...judgments are never tested against other observations of the student or against more complex assessments of a student's effectiveness in real-life settings. Neither, for the most part, are assessments aligned to any well-defined conception of how and in what order, steps, or stages knowledge and skills are acquired over time and with instruction."
"Proficiency also implies a level of competence that would provide substantial prospects of success when applying a skill ...in further study... However, none of the measures used in state assessments (or even in NAEP) has any direct empirical validation for such an interpretation of current proficiency levels."
— Goldhaber, D. (2007, December 4). Principal compensation - More research needed on a promising reform. Center for American Progress, University of Washington.
— Stinebrickner, T. R. & Stinebrickner, R. (2007). The causal effect of studying on academic performance. NBER Working Paper No. 13341.
— Bettinger, E. P. & Long, B. T. (2005). Addressing the needs of under-prepared students in higher education: Does college remediation work? NBER Working Paper No. 11325.
— Bettinger, E. P. & Long, B. T. (2005). Addressing the needs of under-prepared students in higher education: Does college remediation work? NBER Working Paper No. 11325.
— Figlio, David N. & M. E. Lucas. (2000). Do high grading standards affect student performance?”
NBER Working Paper No. W7985, National Bureau of Economic Research.
— Jacob, Brian A. & Levitt, S. D. (n.d.) Rotten apples: An empirical investigation of the prevalence
and predictors of teacher cheating.
— Hess, Frederick M. & Kelley, Andrew P. (2006). "Scapegoat, Albatross, or What?" (p.85) In J. Hannaway & A. Rotherham, Collective Bargaining in Education.
— Nave, B., E. Miech, & F. Mosteller. (2000, October). A lapse in standards: Linking standards-based reform with student achievement. Phi Delta Kappan.
— Nichols, Sharon L.(2007). High-stakes testing: Does it increase achievement? Journal of Applied School Psychology, 23(2), 47-64.
— Jones, Brett D. & Egley, Robert J. (2007, Spring). Learning to take tests or learning for understanding? Teachers' beliefs about test-based accountability. The Educational Forum, 71, 232-234.
— Cimbricz, Sandra (2002, January 9). State-mandated testing and teachers' beliefs and practice. Education Policy Analysis Archives, 10(2). Retrieved February 22, 2008 from http://epaa.asu.edu/epaa/v10n2.html.
— Zancanella, D. (1992). The influence of state-mandated testing on teachers of literature. Educational Evaluation and Policy Analysis, 14(3), 283-295.
— Stake, R. E. & Rugg, D. (1991). Impact on the classroom. In R. E. Stake (Ed.), Advances in program evaluation: Vol. 1. Effects of mandated assessment on teaching (xix-xxii). Greenwich, CT: JAI Press Ltd.
— Mosher, F., Fuhrman, S. H., & Cohen, D. K. (2007, Fall). The research that policy needs. Educational Horizons.
— Mosher, F., Fuhrman, S. H., & Cohen, D. K. (2007, Fall). The research that policy needs. Educational Horizons.
"Third Education Group" is a registered trademark.