In recent years, we have seen an explosion of research aimed at judging the effectiveness of initiatives including charters, vouchers, class size, and various curricula. On the whole, this effort has not given us definitive answers. One could hope that the problem is lack of good data and that as better and more complete data are collected the results will become more useful for making decisions. The underlying assumption of much present research is that there are some elements that lead to success and these can be identified with the right experimental design. For example, if element X promotes success, we should be able to see that, if we hold all elements constant and measure the relationship between the amount of X and success.
But this is not the only possible model. Perhaps X promotes success in some situations, has no effect in others, and has a detrimental effect in a third, depending on other factors in the school. So a certain class size may be critical in some cases and irrelevant in others.
By analogy, let’s imagine that the Austro-Hungarian empire decided to improve the design of clocks and sent out a team to carefully measure the properties of all clocks in the empire to establish relationships between the characteristics of the clocks–gearing, length of pendulum, materials, etc.–and their accuracy. What they would likely find is that there are no consistent relationships because the right gear, for instance, depends on all the other parts of the design.
Accepting a similar model for schools, it no longer makes sense to compare charter schools to traditional public schools. Some will be better; others will be worse. The differences between individual charter schools will be far greater than the average differences between the two types of schools. The lack of a substantial overall difference does not mean, however, that chartering can be dispensed with. For the successful charter school the autonomy granted by a charter may be a critical part of its success.
Thus rather than trying to measure the effect of individual properties in isolation, it may be more productive to identify successful schools, analyze the factors they need for success, and then ask whether the model can be replicated.