DOWNLOAD: School Grades (excel document)

This analysis of school report cards was prepared by Michael Markowitz and distributed to the CEC2 meeting in September.

Send feedback to witzeroo@yahoo.com

Bottom Line: I believe the DOE's school "Progress Report" system is fatally flawed; it yields random results. I will explain why below.

In fairness to DOE, the made numerous changes to the system since last year, and would likely say the below analysis is not fairly apples-apples. Unfortunately, they did not re-run last year's raw scores through this year's formulae. But DOE has said these changes were in response to input from CECs, amongst other sources, and that I take exception to, given they did the opposite of what was requested.

e.g., speaking at a CEC2 meeting last year at which Jim Liebman was the featured speaker, I asked for a reduction in the "progress" score weight from 55%, and an inclusion of some quantification of teacher-assigned grades. Instead, we got an increase to 60% -- and a decrease in the "performance" metric, from 30% to 25%. Still 85% dependent on high-stakes tests. Environment stayed steady at 15%. Opportunity for Additional Credit was increased to a maximum of 15% (total 115% possible).

And still no teacher input. Scantron machines know our kids better, evidently.

Please bear in mind that while it would be interesting to see the schools compared to a standard standard... the peer group cohorting (which can also change from year to year) and the related data-norming are wild cards I can't unwind.

Thought experiment #1: Which is the better indicator of school quality: a "B" at Harvard, or an "A" at the local junior college?

For those of you who weren't at the CEC2 meeting last night, based on the attached analysis showing absolutely zero predictive ability of 2008 grades based on 2007 grades, I called the DOE system a "random number generator."

If within a given year, the bottom line grade is a crapshoot, due in great part to the 60% weight on the "progress" metric, the change from one year to the next is a crapshoot even moreso.

Thought experiment #2: If you got a "A" on a test one year and an "B" the next, should your transcript say "D"? Conversely, if you went from a "D" on a test to a "C", do you deserve an overall "A"? And if you went from a "D" to an "A" on the DOE system, did you most recently test at A, B, C, or D?

Note further that $14 million in bonuses (albeit from private donations) was given to principals -- some in failing schools slated for closure -- based on this system. [See NYT article]

That money could have built 200 school seats -- cash -- 2,000 or more if financed. Or hired say 280 teachers -- enough to teach 7,000 kids for a year. Or restored an untold number of enrichment programs, etc., facing the 2.5% budget cut axe.

NY Times on subject: "More New York Schools Get A's"

(* Easy when there's no objective standard. It's called "grade inflation" where I come from. Last year, DOE gave 22% of the schools A's. This year, 45% got A's. Oddly, the score cutoff only moved from 64 to 60. In a random world, a school's chance of an A just doubled, regardless of its prior grade.)

Per the NYT article, "[Jim Liebman] said it was the only school-accountability system in the country where the results could not be predicted by poverty or race, since all results are adjusted based on demographic peer performance. He also said that, analyzing the data, school size and class size do not appear to be important factors affecting progress on test scores." (Emphasis added.)

And this admitted masking of the REAL achievement gap is a good thing? Note that this statement stands in stark contrast to DOE's own disaggregated "Performance Index" metrics -- by race, ethnicity, and special ed status -- at least for CSD2. It is also true of a RANDOM-izing system.

Fun quiz:"Grading Schools in NYC: A Pop Quiz"

"Statistics are used much like a drunk uses a lamppost: for support, not illumination."
Vin Scully

Attached is the "School "Progress Report" Grades spreadsheet

* (Note that amongst those seven are three underperforming schools, two with obvious special ed impacts: M047 American Sign Language elem and M347 ASL middle; and M407 Collab MS)

* New column W, reflecting what the grades might reasonably be if ONLY the "Performance Category" score was used, averaged over two years, normed around "B". Have a look.

*(Note that even this contains carry-over "peer group" impacts. There is no way I can disaggregate the public data to put all schools on ONE peer-free grading spectrum. IMHO, we are not doing underachieving schools and achivement gap students a favor by NOT comparing them to a standard standard.)

* Comparison of "Performance Only" grades to DOE's 2008 grades

* Revised notes and comments; formatted for legal paper

Feedback welcome.