Affiliated with PerformanceAssessment.Org
Contact us at firstname.lastname@example.org
NYCDOE Chief Accountability Officer James Liebman recently responded in writing to several questions from CPAC (Chancellor’s Parent Advisory Council). Disappointingly, his answers were often vague and incomplete.
Below are CPAC’s questions (in italics), Liebman’s responses, and TOFT’s own commentary [bold and in brackets].
1. New York City has more testing than anywhere else in the country and you are telling me that we need more tests to tell us how our schools and children are doing? Why haven't you already gotten all you need to know from the tests our children already take? Why give more tests to get that information? How about putting the money already spent on all these tests and evaluations, including ECLAS, into real teaching?
Liebman: Assessment is a key part of real teaching. It enables teachers to know whether all students are learning the skills and content being taught, and if not, to identify which children need what additional attention on what particular skills. They also help teachers communicate that information to parents, so parents can help. The annual tests you mention don't serve these functions. Assessments provide real time information that can be used immediately to respond to each student's needs.
[TOFT: True, assessment is certainly integral to teaching. Teachers assess their students everyday through their performance on classroom work, assignments, and class tests. They are already pursuing the channels available to them for students who appear to need special services. To improve their teaching, teachers and schools need opportunities for real professional development, not mandated assessments and tests every 6-8 weeks that will further narrow the curriculum and increase time taken for test prep.]
2. New York City has now had standardized testing for more than forty years. In recent years, the testing has become high stakes -- that is, a single score determines promotion, graduation and entrance into screened schools. Yet, the city's graduation rate has not improved under Mayor Bloomberg and now stands at an abysmal 38.9 percent. African American and Latino children fare even worse. What evidence do you have that even more testing as mandated by this new district policy will prevent further erosion of graduation rates?
Liebman: There is extensive evidence that periodic assessments of the sort we are providing contribute to important gains in student achievement. See for example, http://www.asu.edu/copp/morrison/LatinEd.pdf
[TOFT: Mr. Liebman has not answered the question. He does not even address the graduation rate issue. He also says there is “extensive evidence” that periodic assessments lead to gains in student achievement, yet he cites only one study, which TOFT has already extensively critiqued.]
3. According to your responses on the DOE website to the Accountability Initiative FAQ (frequently asked questions), the standard DOE assessment system will include DIBELS for grades K through 2. DIBELS, for those who may not be familiar, is a highly controversial assessment system that requires children to decode as quickly as possible lists of such nonsense words as “paj,” “vuv,” “zub,” and “oc” without regard to comprehension. Why do you believe this is an appropriate assessment? What research studies can you provide that support the DOE's choice?
Liebman: Dibels has been very effectively piloted as part of a set of assessment choices along with ECLAS-2 in 100 NYCDOE schools. Your description of it is incomplete. DIBELS measures many things besides the phonetic word fragments you mention.
[TOFT: Again, Mr. Liebman has not answered the question. He provides NO research evidence that DIBELS is an appropriate assessment. He says that it has been “effectively piloted” in 100 schools, but provides NO documentation to back this statement.
In contrast, nationally and internationally renowned reading authority Ken Goodman, Professor Emeritus from the Department of Language, Reading and Culture at the University of Arizona and past president of the International Reading Association and the National Conference on Language and Literacy decries DIBELS as "a set of silly tests" that "misrepresent pupils" and "demean teachers." You can read more about DIBELS in the new book, “Examining DIBELS: What it is and What it Does.”]
4. The FAQ describes the conditions under which the DOE system can be altered or substituted; it states that these alternatives must be "comparable to the default system to differentiate and compare different students' literacy and math skills and sub-skills and to trace progress through the school year.” Would you provide detailed information on how the DOE envisions such an alternative?
Liebman: To exercise this option, schools complete a one-page application that describes their alternative assessment system and demonstrates that the alternative is as rigorous as the DOE’s basic periodic assessment system and aligns with state standards. Applications are submitted to the Accountability Office. The DOE is committed to supporting schools to facilitate both the customization and design-your-own options, and is already working with groups of schools toward that end.
[TOFT: We commend the DOE for providing schools with an option to design their own assessments. However, we remain wary of how this option will play out. We also question how many schools will truly have the capacity, time and energy to exercise this option.]
5. Who created the formal assessments that you are planning on using and where were these field-tested?
Liebman: As noted the ECLASS-DIBELS assessments have been field tested in NYC's 100 Reading First Schools. DOE is designing the 3-8 and 9-12 assessments and will field test them in 2006-07 in the Empowerment Schools (which chose to take part in the assessments next year) before making the assessments available to the remaining NYC schools in 2007-08.
[TOFT: Aside from the controversy surrounding DIBELS, described above, we have heard many complaints that the administration of DIBELS and ECLAS is extremely time-consuming, even with the promise of hand-held devices to speed up the process. We want teaching, not testing!!
For the older students, the jury is still out as to what the tests will be since the 3-8 and 9-12 tests have not even been designed yet. However, considering the errors that have plagued DOE-generated test materials in the past, we remain wary. For example, The New York Times reported on January 18, 2006 that the NYCDOE prepared an error-ridden answer sheet for the state 7th grade reading test (“For Students, Figuring Out Answer Sheet Was True Test”).]
6. As is evident from the DOE's web site, the number of tests for which schools are responsible is overwhelming. Parents have frequently expressed concern about the amount of time spent prepping, giving and grading tests. How will teachers find time to fulfill the new testing requirements including entering the data and interpreting the results and still have time to teach? What specific details can you provide?
Liebman: The assessments we are providing will substantially decrease the amount of time required for the ECLAS and DIBELS assessments in use in our primary grades. Assessing how well children are doing at the beginning of the year and how well they are responding to the curriculum throughout the year is not an add-on to, or a distraction from, teaching. It is a core component of teaching. Many teachers and principals in NYC are demonstrating that now every day, and we are being attentive to their successes and methods in designing these assessments.
[TOFT: Mr. Liebman not only evades the question, but he makes the assumption that the additional assessments will give us more valuable information than currently provided by teachers. Though the new hand-held devices MAY cut down on the time required to administer the ECLAS and DIBELS tests in the primary grades, it is still too much, especially as the new mandates demand that these time-consuming tests be given every 6-8 weeks.
Mr. Liebman also assumes that the new periodic assessments will not be considered a distraction from teaching. Unfortunately, he has ignored the research on periodic assessments, including the study on Latino schools he cited earlier. That study found that if a school is not COMMITTED to an assessment program, it will not spend the time and energy analyzing and utilizing the data. And, as yet another study (Ingram, Seashore Louis, & Schroeder, 2004) discovered, even teachers who were committed to their continuous assessment systems found that they had neither the time nor energy to devote to data analysis.]
7. The DOE has proposed a new grading system for NYC schools in which every school will receive a grade of A, B, C, D or F. What research is the Department basing this new system on? In particular what evidence is there that this system helps the lowest performing schools to improve?
Liebman: Clear goals for academic improvement and measures of whether those goals are being met are widely associated with achievement gains in the research.
[TOFT: Mr. Liebman does not answer the question nor does he provide any research to support the DOE’s new policy. He does not provide any evidence that a school grading system helps low-performing schools. In contrast, a study by Jones & Egley (2004) found that after four years of testing in Florida, where schools are rated, 80% of teachers surveyed believed the state tests were not taking the schools in the right direction, primarily because the tests were neither accurate nor being used properly, teachers felt forced to narrow the curriculum, teach to the test and engage in test prep, and there was too much pressure on students and teachers.]
8. If a principal's future employment hinges on the school's letter grade, what is to prevent schools from trying to keep out or get rid of ELL or special ed students or those with poor grades and uneven attendance records?
Liebman: The Progress Reports reward schools for getting good results for ELL and Special Ed Students and Students at Proficiency Levels 1 and 2. They also hold schools accountable for educating all students admitted to the school, whatever problems the students thereafter encounter. The Progress Reports are directly responsive to the problems you rightly note.
[TOFT: Unfortunately, Mr. Liebman does not address the very real possibility that some schools may not want to chance their future test scores by admitting students who do not look promising. Nor does he address the very real phenomenon of “pushing out” failing students. (See The New York Times, “To Cut Failure Rate, Schools Shed Students,” July 31, 2003.)]
9. Research shows that top down mandated policies do not lead to effective school reform. Indeed the Chancellor's often cited study " Why Some Schools with Latino Children Beat the Odds and others Don't" warns against using both district imposed assessments and linking those assessments to a system of punishments. With those findings in mind, how can the DOE justify this new accountability approach?
Liebman: The period assessments we are providing are not linked to any system of punishments. They are separate from the Progress Reports.
[TOFT: Mr. Liebman does not address the link between top-down mandated policies and failed reforms. Though he insists that the periodic assessments are not linked to any system of punishments, he fails to address the role the periodic assessments will likely serve in preparing students for the state tests, which are to be used in grading schools.]
10. The DOE, mostly through Lori Mei and Princeton Review, repeatedly stated that the Princeton Review Interim Assessments (now defunct) were not supposed to be used for promotion decisions. Nevertheless, we know through teachers, administrators, and parents that they were. The DOE states the new accountability tests will not be used that way, but how will we know for sure?
Liebman: The Princeton Review "interim Assessments" are not defunct. They will continue during the 2006-07 school year while the new Periodic Assessments are being designed. Princeton Review will also deliver the periodic assessments next year in the Empowerment Schools for grades 3-8 on a pilot basis, while we design the system for citywide use in 2006-07. The new assessments will not be used for the high-stakes purposes you list.
[TOFT: Mr. Liebman does not address this very serious issue: the Princeton Review “Interim Assessments” WERE used, according to several sources, to make promotion decisions. Before Mayor Bloomberg decided to end so-called social promotion, the city’s 3rd, 5th and 7th grade tests were NOT used to make promotion decisions either. Now they are. Parents rightfully remain anxious that the new interim assessments will eventually be used for high-stakes purposes as well. They need a guarantee, especially as these test results will be in a student’s permanent record.]
11. At a District 3 President's Council mtg. in the past year Dep. Mayor Dennis Walcott stated that the Princeton Review Int. Assessments were being dropped "because we found they weren't aligned with the curriculum.” A District 3 principal had warned parents about exactly this two years earlier, when the tests were first introduced. How can we be sure these new tests will be aligned with the curriculum?
Liebman: We are explicitly designing the assessments to align with curriculum, and providing alternatives and customization options for exactly this reason, with the recognition that not all schools teach the same things in the same order.
[TOFT: We are both concerned that the tests will become THE curriculum and that classes will be further reduced to test prep. The DOE’s track record does not inspire confidence, and by the time we find out what is REALLY happening in the classroom, our children will already have been affected.]
12. If schools can choose their own tests, how many variations on the tests will there be?
Liebman: As many as there are schools that choose to customize or design their own.
[TOFT: We commend the DOE for its stated flexibility and hope that it will honor the proposed alternatives.]
13. Who will be designing and writing these tests? Who will be grading these tests?
Liebman: We are designing them, and will determine the best partners, if any, to assist us in delivering them.
[TOFT: The DOE is asking parents to trust its judgment. Given past history, many parents have concerns.]
14. The Princeton Review tests were said to provide a "snapshot" of what children knew at a given time. There were reports of children -- including very able ones -- saying the tests were asking things they had not yet been taught. Yet these tests were graded, results handed out at teacher/parent conferences, and promotion-in-doubt letters sent based on them. Will these tests also be asking children things they have not yet been taught?
Liebman: No. We are explicitly designing the assessments to avoid the issues you mention.
[TOFT: We will not know for sure until the tests have been developed and administered to our children.]
15. Various school districts throughout the country are using Educational Management Organizations (EMOs) to track accountability for them. Is the New York City DOE using an EMO? If so, what company is the EMO contract with? Is it Edison Schools?
Liebman: NYC is not using an EMO.
[TOFT: It will be important to stay on top of this possibility.]
Goodman, K. (2006). Examining DIBELS: What it is and what it does. Brandon, VT: Vermont Society for the Study of Education.
Herszenhorn, D. (January 18, 2006). For students, figuring out answer sheet was true test. The New York Times (Section B, p. 1).
Ingram, I., Seashore Louis, K. R., & Schroeder, R. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. New York: TC Record.
Jones, B. D., & Egley, R. J. (2004). Voices from the frontlines: Teachers’ perceptions of high-stakes testing. Education Policy Analysis Archives, 12 (39). Retrieved 8/23/04 from http://epaa.asu.edu/epaa/v12n39/.
Lewin, T. & Medina, J. (July 31, 2003). To cut failure rate, schools shed students. The New York Times (Section A, p. 1).
|produced by Naava Katz Design|