Methodology - Quality Counts '99

Read a detailed description of the analysis.

Methodology

ducation Week compiled data on more than 75 indicators across five categories. Within the resources category, we assigned separate grades for the adequacy, allocation, and equity of school spending. We did not grade states in the student-achievement category. To arrive at a grade in each category, we assigned points to each column of data or information. For columns that contain information on state policies, we assigned letter grades based on whether the state had adopted such policies, and then granted points according to the following scale: An A was worth 100 points; a B, 85 points; a C, 75 points; a D, 65 points; and an F, 50 points. For instance, a state that has a charter school law received an A, worth 100 points, while one that has no such law received an F, for 50 points.

For columns that contain numerical data, such as percentages, we assigned points based on one of two methods:

If the benchmark for a perfect grade was 100, as in the column on the percent of teachers with degrees in the subjects they teach, we simply assigned the same number of points as the percentage.

If the benchmark for an A was different from 100, we assigned letter grades according to a scale. For instance, states that allocate 70 percent of their education dollars for instruction got an A for 100 points. Each scale is individually described below.

After obtaining the number of points for each column, we calculated the overall grade for each state in each category using a formula that gives greater weight to certain topics. The formula for each category yields an overall number grade for each state, from 0 to 100. We then assigned overall letter grades based on the following scale: 93 to 100 percent = A; 90 to 92 percent = A-minus; 87 to 89 percent = B-plus; 83 to 86 percent = B; 80 to 82 percent = B-minus; 77 to 79 percent = C-plus; 73 to 76 percent = C; 70 to 72 percent = C-minus; 67 to 69 percent = D-plus; 63 to 66 percent = D; 60 to 62 percent = D-minus; below 60 percent = F.

Here, in greater detail, is how we graded the states in each category:

All Students Achieving at High Levels

Education Week ranked the states by the percent of students who scored at or above the "proficient" level on each of four National Assessment of Educational Progress exams.

Academic Standards, Assessments, and Accountability

Weighting: The five columns related to standards together count for 50 percent of the overall grade. The assessments columns count for 30 percent of the grade.

State exit tests geared to the 10th grade level and state participation in the most recent NAEP test each count for 2.5 percent of the grade. The five school accountability indicators together count for the remaining 15 percent of the state's grade.

Standards: In the first column, states that have adopted standards in the four core subjects--English, mathematics, science, and social studies--received an A.

States that have adopted standards in two or three subjects received a B; one subject, a C; still developing standards, a D; no standards and no plans to develop them, an F.

To grade the clarity and specificity of standards, we relied on a report by the American Federation of Teachers, "Making Standards Matter, 1998." The AFT rated state standards in the four core subjects at each of three levels--elementary, middle, and high school--for a total of 12 separate ratings. To convert the ratings into a number grade, we divided the total number of ratings for which a state met the AFT'S criteria by 12.

Assessments: States that incorporate performance tasks or portfolios in their assessment programs earned an A. States that use a criterion-referenced, multiple-choice test aligned to their standards that does not include performance tasks received a B.

States with a criterion-referenced test that is not aligned earned a C. States that assess student performance with only a norm-referenced, multiple-choice test received a D. Those that have no uniform statewide assessment earned an F. States that also use a direct assessment of writing ability earned an additional 3 points.

States that measure student achievement against standards in all four core subjects earned an A; three subjects, a B; two subjects, a C; one subject, a D. States that do not have assessments aligned to standards in any subject received an F.

Accountability: States that have an exit exam pegged to 10th grade standards or higher earned an A. States that will use such tests in the future earned a C. States that have no requirement and no firm plans to implement one earned an F.

States earned an A for each component of a school accountability system that is in place and an F for each component that is lacking.

States that participated in the most recent National Assessment of Educational Progress received an A. The rest received an F.

Efforts To Improve Teacher Quality

Weighting: The indicators for performance-based licensure make up 40 percent of the total grade. The percent of secondary teachers assigned in-field is worth 20 percent. The three professional-development indicators are together worth another 20 percent. Finally, five teacher education indicators constitute the remaining 20 percent of a state's overall grade.

Performance-Based Licensure: States that have adopted standards describing what new teachers should know and be able to do earned an A. States that have adopted such standards but use them solely to approve education school programs, or that rely on education schools to assess whether students have met the standards, earned a C. States that have no standards at all earned an F.

We could not determine the quality or rigor of the written tests states require for teachers to earn a license. So we based our grades on the kinds of knowledge states test. States that have tests of both subject-matter knowledge and teaching, or pedagogical, knowledge earned an A. Those that have only a subject-matter test earned a B. Those that require only a test of teaching knowledge earned a C. States that require tests of basic skills or general knowledge earned a D. And those that do not require any assessment for a teaching license received an F.

States that require all new teachers to participate in an induction program that includes mentoring by an experienced teacher, and fully fund the program, received an A. States that provide a program for a fraction of their new teachers, or that require it for all new teachers but do not fund the program, earned a C. States that do not have an induction program to support new teachers received an F.

States that have an assessment of classroom-teaching ability tied to licensure earned an A. States that require districts to assess new teachers' classroom performance but still tie those evaluations to licensure also received an A. States that do not assess the classroom performance of new teachers or that do not tie the assessments to licensure earned an F.

States earned an A for providing at least one incentive for teachers to earn certification from the National Board for Professional Teaching Standards, either licensure portability, licensure renewal, fee supports, or pay supplements. States that offer no such incentives earned a C. Because less than 1 percent of teachers in any state are board-certified, we did not grade states on their numbers of nationally certified teachers.

Teaching In-Field: States earned the same number of points as their percent of secondary teachers with degrees in the subjects they teach. For example, if 81 percent of teachers hold degrees in the subjects they teach, the state earned 81 points for this column.

Professional Development: States earned an A if they require districts to set aside days in the school calendar for professional development, or if they require teachers to accumulate hours or credits in professional development to retain or renew their certificates.

States also earned an A for providing professional-development opportunities at the state level, such as conferences or workshops. Finally, states earned an A for providing money to districts for local professional-development activities.

States that indirectly provide money and do not specifically earmark dollars for professional development did not receive credit. For each of the three elements a state does not provide, it earned an F.

Teacher Education: States earned an A if they require an academic major, rather than an education major, for secondary teachers to earn a license.

Those that require a number of credit hours in the subject to be taught, roughly equivalent to a major in the subject, earned a C. States that don't require academic majors for secondary certification earned an F.

States that require teacher-preparation programs to be aligned with state K-12 academic standards received an A. States that do not earned an F.

States earned the same number of points as the percentage of prospective teachers who graduated from institutions accredited by the National Council for Accreditation of Teacher Education.

States that require teacher-training institutions to include field experiences in K-12 schools prior to student teaching earned an A. States that have no such requirement, but where state officials told us that all their teacher-training programs do it anyway, earned a C. The rest received an F.

States received an A if they require teacher-preparation programs to incorporate student teaching. States that have no such requirement, but where the programs require the candidates to student-teach, earned a C. The rest earned an F.

School Climate Conducive to Learning

Weighting: Class-size indicators account for 35 percent of the overall grade; student-engagement indicators, 20 percent; parent-involvement indicators, 20 percent; and school-autonomy indicators, the final 25 percent.

The two charter school indicators together are worth one-quarter of the autonomy total; combined, they carry equal weight with the other three indicators of school autonomy.

All other columns were given the same number of points as the percentages in them.

States where students can choose to enroll in any school in the state received an A. States with limited public-school-choice policies--where students can choose from schools in their own district, or where it is voluntary for districts to have local choice programs--received a C. States with no public school choice earned an F. States that allow charter schools earned an A; those that do not earned an F. States with strong charter school laws, as rated by the Center for Education Reform, earned an A; those with weak laws, an F. States that require or permit site-based management of schools and that grant waivers of education regulations earned an A. Those that do not received an F.

Resources: Adequacy, Allocation, and Equity

Weighting: We gave equal weight to each of the three columns under adequacy, and calculated separate grades for allocation and equity.

Adequacy: We adjusted each state's per-pupil expenditure, or PPE, using the "geographic cost-of-education index" from the National Center for Education Statistics. For 1999, we used a benchmark of $7,369, slightly higher than the previous year, to account for 2.3 percent inflation. We divided each state's adjusted PPE by $7,369 to obtain the number of points it earned out of 100.

We calculated the percent change in inflation-adjusted education spending per student by subtracting each state's inflation-adjusted 1987 PPE from its 1997 PPE and dividing the difference by the inflation-adjusted 1987 PPE. We gave 100 percent to states that increased per-student spending by at least 20 percent over inflation; 85 percent, or a B, to states that raised spending 15 to 19 percent over inflation; 75 percent, or a C, to states that raised spending 10 to 14 percent over inflation; 65 percent, or a D, to states that raised spending 5 to 9 percent over inflation; and 50 percent to states that raised spending 0 to 5 percent over inflation. We gave a zero to states that did not raise spending enough to keep up with inflation.

We calculated the percent of total taxable resources spent on education by dividing the combination of a state's local and state-level education revenues for 1995-96 by its gross state product for 1995. We used 5 percent of state wealth as our benchmark for a perfect score. We divided the percent of a state's wealth spent on education by that benchmark to assign each state a grade on this indicator.

Allocation: Percent of annual expenditures spent on instruction refers to spending directly related to the interaction between teachers and students, such as teacher salaries and classroom supplies. We used the following grading benchmarks: 70 percent or greater is an A; 69 to 69.9 percent is an A-minus; 68 to 68.9 percent is a B-plus; 66 to 67.9 percent is a B; 65 to 65.9 percent is a B-minus; 64 to 64.9 percent is a C-plus; 62 to 63.9 percent is a C; 61 to 61.9 percent is a C-minus; 60 to 60.9 percent is a D-plus; 58 to 59.9 percent is a D; 57 to 57.9 percent is a D-minus; less than 57 percent is an F.

Equity: The relative inequity among districts in spending per student was calculated by Management Analysis and Planning Inc., or MAP, for Quality Counts using the U.S. Census Bureau's F-33 database. The measure, known as a coefficient of variation, summarizes how widely spending across a state's districts varies from the average per-pupil spending within a state. We adjusted each district's spending to account for its poor and special education students and the differing costs of hiring teachers and purchasing supplies. We excluded districts with fewer than 200 students from our calculations and assigned special weights to nonunified districts.

We used the following grading benchmarks: 1 to 3.9 percent variation is an A; 4 to 4.9 percent variation, an A-minus; 5 to 5.9 percent variation, a B-plus; 6 to 8.9 percent variation, a B; 9 to 9.9 percent variation, a B-minus; 10 to 10.9 percent variation, a C-plus; 11 to 13.9 percent variation, a C; 14 to 14.9 percent variation, a C-minus; 15 to 15.9 percent variation, a D-plus; 16 to 18.9 percent variation, a D; 19 to 19.9 percent variation, a D-minus; 20 percent or greater variation, an F. A detailed, step-by-step description of the analysis may be obtained on our World Wide Web site at www.edweek.org/sreports/qc99/states/indicators/in-map.htm.

Vol. 18, number 17, page 123