Quality Control? How School Performance Varies Within American Cities

School quality doesn’t vary only between cities; it varies within cities. All parents know this, of course, and seek to enroll their children in the best schools. But parents face different scenarios in different cities.

In some cities, it doesn’t really matter which public school one attends; variation in quality is limited. This can be good news if most schools are decent—the case in Arlington, Virginia, for example. But it can be bad news in cities where most schools are dreadful—the case in Birmingham, Alabama. For parents in those school districts, the only way to improve their children’s situation is to move elsewhere or enroll their kids in private school.

In other cities, such as Washington, D.C., median quality is low, but there are high-quality exceptions. Parents in such school districts face the task of doing what it takes to enroll their children in the limited number of decent schools. And there are cities, such as San Francisco, where median quality is relatively high but where there are many low performers, too. Parents in those school districts should avoid having their children, in effect, draw the short straw.

This paper quantifies such variation in elementary and middle school quality within 68 of the largest U.S. public school districts, which collectively serve about 7.8 million students. It also examines whether cities that see students with certain demographics (specifically, low-income and nonwhite) concentrated in certain schools experience higher variation in school quality. The findings, especially those in the comprehensive Appendix table, can serve as a guide for parents, as well as an admonition to school officials in cities where quality is consistently low or where the quality gap between the best and worst schools is wide.

DOWNLOAD PDF

Urban public school districts in the U.S. vary in quality. Educational policies, distribution of resources, and the challenges that students bring to the classroom differ substantially across localities. Little surprise, then, that academic outcomes—such as high school graduation rates and average school performance on standardized tests—fluctuate widely, too.

When comparing the quality of schools across localities, measures of average student and school performance are the appropriate first-order concern. But average performance doesn’t fully characterize the quality of an urban school system. Most students don’t attend a city’s average public school. Some students attend better schools, while others attend worse schools.

That school quality varies within any particular city is well known. But previous research has not established the degree to which such variation in school quality—within a given school district—differs between cities. Is the spread between bad and good schools substantially larger in some cities than in others? This report investigates the question by comparing the variation in the quality of public elementary and middle schools across 68 of the largest public school districts in the U.S. (see Appendix for comprehensive results).

The variation in the quality of a city’s schools (“city” and “school district” will be used interchangeably) has important implications for parents seeking access to an appropriate school for their child, and it speaks directly to whether a city is providing equitable access to educational quality for residents. For instance, securing a spot at a decent public school is less difficult in a city where all schools perform relatively similarly (such as Arlington) than in a city with comparably decent average school quality but much wider variation in quality (such as San Francisco).

In this paper, I find that the variation in school quality is fairly similar in America’s largest school districts, with a large plurality having a gap of 15–20 percentage points between good and bad schools (as defined below). Several cities, however, including New York and San Francisco, exhibit substantially wider school-quality gaps.

I also examine the relationship between variation in school quality and student demographics. Overall, I find no strong correlation between variation in quality and the proportion of a school district’s students who are low-income or nonwhite. However, cities where low-income and nonwhite students are more heavily concentrated within particular schools (such as Portland, Oregon) do exhibit larger variation in quality.

I measure and compare the variation in school quality in each of the 68 member cities of the Council for Great City Schools, an advocacy group, for which the necessary data are available. The focus is on these large school districts because each has enough schools to calculate meaningful measures of their quality dispersion. I utilize school performance and demographic data developed for the purpose of ranking school performance available on the SchoolGrades.org website*.[1]

SchoolGrades reports test-score information for each public elementary and middle school in the U.S., including traditional public, magnet, and charter schools.[2] I use an “adjusted overall proficiency” score as my measure of school quality.[3] Doing so allows for cross-state comparisons between schools and accounts for the economic profile of students in the school.

When calculating an adjusted overall proficiency score, SchoolGrades first takes the average percentage of the school’s students who scored proficient, or better, on statewide math and reading exams. Because statewide tests and proficiency standards vary, SchoolGrades adjusts a school’s score to align it with a uniform national standard. To do this, it takes the difference between the average percentage of students who score proficient, or better, on the respective statewide exam and the percentage scoring proficient, or better, on the National Assessment of Educational Progress (NAEP), a test given by the federal government to representative samples of students in each state.

When assessing a school’s performance, it is important to account for the socioeconomic profile of its students. Lower-income students face challenges that affect their academic performance but are independent of the school’s quality. To account for this, SchoolGrades adjusts each school’s score to represent the predicted percentage of students who would score proficient, or better (according to NAEP standards), if the school had the same proportion of students eligible for free, or reduced-price, lunch—a common measure for determining whether a student is low-income—as the average U.S. school. (Adjusting for students’ socioeconomic background ends up increasing the quality score for schools with large low-income populations, and it ends up reducing the score for schools that serve primarily higher-income students.)

The SchoolGrades scores that I use are, of course, an imperfect proxy for a school’s quality: we ask schools to do more than increase math and reading test scores, and there may well be other factors that are outside a school’s control but nevertheless affect test scores. Because of these limitations, parents and policymakers should not use the SchoolGrades scores, or any other school evaluation measure, in isolation to make enrollment or policy decisions. Nonetheless, the measure provides a meaningful approximation for comparing school performance—holding constant the challenges that students bring to the school—across states.

I also utilized SchoolGrades data on the percentage of students in each school who are eligible for free, or reduced-price, lunch, supplementing such numbers with U.S. Department of Education data on the number and proportion of a school’s students who are nonwhite. I also calculated a Gini coefficient to measure the dispersion of low-income and nonwhite students across a city’s schools.[4]

Figure 1 illustrates two of the scenarios described in the executive summary—scenario 2: low average quality, low variation; and scenario 3: low average quality, high variation. Blue squares represent schools in the Santa Ana, California, school district. Red triangles represent schools in the Washington, D.C., school district. The vertical axis represents the adjusted percentage of students in a school who are proficient or better. The horizontal axis represents a school’s proficiency rank within its district. The dashed red and blue lines show the 25th and 75th percentiles for school proficiency in the respective cities (henceforth, a proxy for “good” and “bad” schools within a district). The variation in school quality in each district is measured by the gap between the district’s 25th and 75th percentile lines.[5]

Overall, school performance is quite similar in the two cities. About 29% of students in the average D.C. school score proficient, or better, compared with about 25% of students in the average Santa Ana school. In both school districts, about 23% of students score proficient, or above, in the median school (the 50th percentile).

But the variation in school quality within these districts differs meaningfully. In Santa Ana, in the 75th percentile school, the share of students who score proficient, or better, is only 9 percentage points higher than in the 25th percentile school. Yet in D.C., the “proficiency gap” is 26 percentage points.

Given that the average performance and the median performance in the two cities are so similar—and that nearly all parents want to send their kids to as high-quality a school as possible—should parents and policymakers prefer the school-quality distribution in D.C. or in Santa Ana? The answer depends on how one weighs the trade-off between the chance to attend a high-performing school against the risk of ending up at a low-performing school.

Economic theory suggests that when choosing between two lotteries with the same average (i.e., expected value), a person who is at all wary of risk would prefer Santa Ana because it exhibits the smaller variation in potential values. But the situation is more complicated when comparing districts with a wider gap in the performance of the average school. And it is complicated further when we consider that not all parents have the same access to better schools in a city.

How policymakers and parents should weigh the average and spread of school performance in a city is an issue for further research. But the first step in any such analysis is to assess the extent to which the variation in school quality differs across cities.

The dots on Figure 2 illustrate the difference in the adjusted percentage proficient at the 75th and 25th percentiles for all 68 school districts. Cities that exhibit the least variation in school quality are highest on the vertical axis. In 44% of cities (30 of 68), the difference in the quality score for the 75th and 25th percentile schools is 15–20 percentage points. In 12 cities, the quality gap is less than 15 percentage points. And in 11 cities, it is more than 25 percentage points.

Variation in school quality is greatest in cities such as New York, San Francisco, and Newark (NJ), where the proficiency gap exceeds 25 percentage points. Variation in school quality is lowest in cities such as Birmingham, Rochester (NY), and Santa Ana, where the proficiency gap is less than 10 percentage points.

Figure 3 illustrates the relationship between a city’s variation in quality and the quality of its median school. The 68 circles represent cities—the bigger the circle, the greater the number of schools in the city. The vertical axis is the adjusted percentage of students who are proficient, or better, at the city’s median school. The horizontal axis is the difference in the adjusted percentage of students proficient between the 75th and 25th percentile schools.

The vertical and horizontal red lines separate the cities into four quadrants that demonstrate their position relative to the medians—for both variation in quality and median quality—for the 68 cities. The 15 in the top left quadrant have relatively high median school quality but relatively low variation in school quality. The 15 in the bottom right quadrant have high variation in quality but low median quality. The 19 in the top right quadrant have both relatively high median quality and high variation in quality. And the 19 in the bottom left quadrant have both low variation in quality and low median quality.

The dashed regression line illustrates a positive correlation such that, on average, cities with higher variation in school quality tend to have higher median quality.6 But the relationship between the median and the spread of school quality is not deterministic—some school districts with relatively high median quality experience lower variation in quality, and vice versa. The relationship between median quality and spread of quality is also nonlinear. The relationship is steepest for school districts with low median proficiency,7 and it flattens considerably for better school districts, such that there is little relationship starting for districts with about 30% of students scoring proficient or better. If the four cities with the lowest quality variation (Birmingham; Rochester; Santa Ana; and Aurora, Colorado) are excluded, there is not even a clear relationship between variation and median quality.

Figure 3 also shows that school-quality variation is not determined by the number of schools in a city. Small, medium, and large cities are found in all quadrants.

What factors might account for the large gap in school quality between different cities? Data limitations restrict our ability to answer this question. For instance, the large number of selective traditional public schools in New York City likely contributes to its status as the major school district with the largest difference between the 75th and 25th percentile schools. But we can’t analyze this particular factor because not all 68 cities report such data.

There are, however, data for all 68 cities on two demographic factors: the proportion of students who are low-income (as determined by their eligibility for free, or reduced-price, lunch) and who are nonwhite. (Note: The analysis below is descriptive, not causal—i.e., it reveals only the extent to which cities with certain characteristics tend to have certain variations in school quality.)[8]

Are school districts with higher proportions of low-income and nonwhite students more likely to have a wider variation in school quality? Figure 4 illustrates the relationship between school-quality variation (y-axis) and the proportion of students who are eligible for free, or reduced-price, lunch (x-axis). Figure 5 does the same for nonwhite students. (In both figures, dots represent cities while the sloping red line is a regression line that illustrates the relationship between the two variables.) Though the variation in school quality tends to be lower as the proportion of low-income and nonwhite students rises, the relationship is not strong, as indicated by the dispersion of dots.[9]

In addition to differences in the proportion of nonwhite and low-income students, the extent to which such students are distributed across schools varies across cities. School districts where low-income and minority students are heavily concentrated in specific schools might exhibit larger variation in school quality, for at least two reasons.

First, cities that do not equally distribute such students might also inequitably allocate resources between schools—directly, via spending per pupil, and indirectly, via, say, allocating better or worse teachers. Second, low-income students who attend schools with heavier concentrations of higher-income students often have access to more high-performing peers, which can have a positive impact on academic outcomes.

Figures 6 and 7 illustrate the relationship between school-quality variation in a city and the Gini coefficient for the distribution of, respectively, low-income and nonwhite students in a city. (The higher the Gini coefficient, the greater the student concentration in specific schools.) Both figures show that cities where low-income (such as Minneapolis) or nonwhite students (such as Pinellas County, Florida) are heavily concentrated in specific schools tend to exhibit greater variation in school quality.[10]

In 44% of America’s largest school districts (30 of 68), the quality gap between schools at the 75th and 25th percentiles was 15–20 percentage points. However, in 18% of cities (12 of 68), the quality gap was less than 15 percentage points. And in 16% of cities (10 of 68), the gap was more than 25 percentage points.

As for the relationship between variation in school quality and a district’s proportion of students who are low-income or nonwhite, no strong correlation exists. However, cities where low-income and nonwhite students are more heavily concentrated within particular schools do exhibit a larger variation in school quality.

More broadly, the Appendix shows that overall school quality in most large U.S. public school districts remains dismal. In only nine districts (13% of the total) were at least 40% of students proficient (after adjusting for family income), or better, at the median school; and in a mere two districts, more than 50% of students were proficient, or better, at the median school. Of these nine districts, just three had a variation in school quality of less than 20 percentage points—and none had a variation of less than 15 percentage points.

In short, few of America’s largest public school districts offer consistently decent elementary and middle schools. But motivated parents in low-quality districts should not despair entirely: in 38 districts (43% of the total), at least 40% of students were proficient, or better, at the 75th percentile school. In other words, in many school districts that are of low quality overall, there are pockets of decent schools.

*This report describes School Grades, a project that Manhattan Institute instituted in 2015 and ended at the end of 2019, as the development of websites from other organizations, particularly Great Schools, specifically dedicated to national school data have appeared.

See Appendix and Endnotes in PDF