Published: November 19, 2019
On October 11, 2019, the La Canada Unified School District sent out a district-wide email to all parents (“An announcement from LCUSD regarding 2018-19 State test scores”) touting its results on the 2019 California Assessment of Student Performance and Progress (CAASPP):
The state of California publicly released results for the 2019 California Assessment of Student Performance and Progress (CAASPP) yesterday and once again, La Cañada Unified School District scored superbly, showing outstanding academic performance and growth. Students in grades three through eight as well as grade eleven, take this annual computer-adaptive assessment in English Language Arts and Mathematics. Scale scores, which provide a common reference over the years, indicate that La Cañada Unified scored in first, second or third place statewide in all grade levels tested, both in English Language Arts and Math. Eighth graders ranked first in Math statewide. Similarly in English, eighth and eleventh graders were in the top spots. As a district, scale scores show that LCUSD is the second highest in California in both English and Mathematics.
As in past years, the La Canada Outlook followed the announcement with a front-page, headline story a few days later:
[Source: La Canada Flintridge Outlook, Vol 22 No. 42, Thursday, October 17, 2019 ]
Though the district’s performance on the state-wide standardized test measure was outstanding, it was overstated. The district’s claim that “La Cañada Unified scored in first, second or third place statewide in all grade levels tested, both in English Language Arts and Math” was not accurate. The Outlook’s claims that “La Canada High School 7/8 ranked No. 1 in English for 8th graders’ scale score — its 7th graders’ marks were the second-highest — while LCHS 11th-graders’ tally also was the state’s best” were demonstrably false. Here are the district’s claimed rankings versus its schools’ actual rankings as calculated by La Canada Math Parents (LCMP) from state data:
For context, there were 1,030 public school districts in California during the 2018-19 school year, about 6,000 public elementary schools (and district level programs in the elementary grades) and about 2,600 high schools (and district level programs in the high school grades) participating in the CAASPP program. Of those, the state did not report a mean scale score for roughly 10-20% of the grade level cohorts because 10 or fewer students took the CAASPP exam in that school/grade. The California Department of Education does not report CAASPP scores for schools when 10 or fewer students take the exam out of student privacy concerns. Thus the actual ranks above are out of thousands of public schools.
So What is Really Going on?
The problem with the district’s top 3 claims is that they fail to mention that the district is only comparing its district average scores against a limited number of other Unified districts of “like” character, where “like” was not defined either in the email or in the accompanying press stories. As La Canada Math Parents (LCMP) discovered in an investigation in 2018, there are five major types of public school districts in California, only one of which is Unified:
-
- Unified School Districts generally include elementary, middle and high schools covering all pre-college grades K through 12.
- Elementary School Districts generally include schools covering grades K through 8.
- High School Districts generally include high schools (i.e. grades 9 through 12.)
- Joint Districts – “Joint” denotes that the district includes territory from more than one county. There are joint elementary, joint high school, and joint union districts of both types.
- Union Districts – “Union” denotes that the district was formed from two or more districts. There are both union elementary and union high school districts. A “joint union” district means it was formed from two or more districts, and includes territory from more than one county.
[Source: http://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=EDC§ionNum=86]
By narrowing the comparison to only other Unified school districts of “like” kind, vast numbers of other schools were omitted from the comparison. As LCMP stated last year in its analysis, “Since only 37.7% of school districts in California are Unified or Joint Unified districts, thousands of public schools were omitted from the comparison.” Schools in the five major different types of districts are indistinguishable from each other on the outside — their students are held to the same state education standards, take the same standardized tests, and use the same types of curricula. A district is an administrative unit for governing multiple schools within a region, and local educational agencies and governments have different reasons for organizing their districts among the five major types.
The district could have avoided the deceptive claim if it fully stated the scope of its comparison as other high performing districts like South Pasadena Unified School District did when announcing its CAASPP performance in 2018:
“2018 CAASPP Place South Pasadena Unified in Top Five California K-12 Unified School Districts“
[Source: South Pasadena Review, Oct. 2, 2018.]
If the district wants to claim it is tops in the state, then it should either compare LCUSD schools against all public schools in the state, or state the constraints of its claim as South Pasadena USD did.
The district presented more details on how it arrived at its claims in a presentation given to the LCUSD Governing Board on October 29, 2019 (agenda here, audio transcript here.) LCUSD Director of Program and Services Karen Hurley gave the presentation on “California Assessment of Student Performance and Progress” in which she showed the ranking of LCUSD to “like” districts in Mathematics and English Language Arts CAASPP mean scale scores for all grades tested. Though not defined on her slides (downloadable here), Hurley defined “like” districts during the meeting as “Unified school districts that we have historically compared ourselves to in the past.”
LCUSD Governing Board Member Dan Jeffries asked a clarifying question about the limited scope of the comparison, “Is there a chance we are not comparing to someone who is doing really, really well? …Is it possible there is some district out there we are not even aware of — (like in) San Diego County — like they are outperforming everyone else? We would know that wouldn’t we?”
Hurley answered that she scanned all other schools in Los Angeles County and LCUSD was “at the top,” but did not look statewide. She further defended the decision to limit the scope of comparison by saying that her predecessor Lindi Dreibelbis-Arthur had done a lot of research in coming up with the comparison cohort and chosen unified districts that had similar demographics, size, and socioeconomic status. While this may have been true at the time, it is not true today and the omission of over a thousand California public school districts and thousands of schools from the comparison is misleading.
Comparing Against All Districts
LCUSD Board Member Jeffries’ questions in the Oct. 29, 2019 meeting about the narrow scope of the comparison are insightful because an analysis of ALL public schools and districts indeed reveals other non Unified districts and hundreds of schools that are outperforming LCUSD’s students in grade level cohorts in mathematics and English language arts.
LCMP downloaded the raw data sets from the California Department of Education’s website for the 2019 test and re-ran the district’s comparisons against all districts and schools. Shown below are the district’s comparisons against what it calls “like districts” along with comparisons against all public schools and districts in California.
All district tables are taken from Hurley’s presentation slides (downloadable here) used at the the Oct. 29, 2019 LCUSD Governing Board meeting. LCMP tables are generated from California Department of Education research data located here:
https://caaspp-elpac.cde.ca.gov/caaspp/ResearchFileList?ps=true&lstTestYear=2019
CAASPP Mean Math Scale Scores – Districts
First, here is what the district claimed in justifying its “top 3 in all grades” ranking in mathematics:
[Hurley, K., presentation to LCUSD Governing Board on Oct. 29, 2019, p.15]
Note that all La Canada grade-level cohorts are in the top 3 with 8th grade 1st according to the district.
Since CAASPP scale score ranges are computed differently in each grade, scale scores should not be added, compared or averaged across grades as Hurley did in the above slide in the column labeled “All Met & Exceeded” and the Outlook then subsequently reprinted in this table:
Hurley compounded her error by failing to weight the mean scale scores by grade. To avoid this problematic approach, LCMP normalized the scores in each grade by taking the rank order of the districts’ math mean scale score in each grade and then averaging the grade level ranks to arrive at a district-level rank. Here are the results for CAASPP mean math scores at the district level:
As can be seen, only LCUSD’s 8th graders ranked in the top 3 statewide against other districts, and were not 1st, rather 3rd. The other grades ranked no higher than 4th to a low of 17th. Overall, the district’s average rating placed it 8th among all California districts. The other districts above La Canada USD were Elementary, Union, and Union High School districts — all omitted from the district’s comparison.
CAASPP Mean ELA Scale Scores – Districts
Here is what the district presented in justifying its claim of “top 3 in all grades” in English Language Arts (ELA):
[Hurley, K., presentation to LCUSD Governing Board on Oct. 29, 2019, p.14]
Note that the table shows LCUSD 8th and 11th graders as 1st in the state. Here are the results for CAASPP mean ELA scores at the district level compared against all public school districts in the state, averaging the district’s constituent grade-level ranks based on ELA mean scale score:
LCUSD as a district did much better in ELA than in mathematics when compared to all districts in California and can rightfully be called a “top 3” district in CAASPP ELA performance.
Comparing Rankings Against All Schools
As discussed in the article “When Top 3 Isn’t Top 3” from LCMP’s January 2019 newsletter, LCUSD also benefits from being a “boutique” district — it is unusually small with just one high school, whereas 94% of California public high schools are in districts that include two or more high schools. The small size of the district confers advantages such as lower administrative overhead, the ability for schools to coordinate more easily, and the ability for the district to act quicker than larger districts. Since almost all high-performing California public high schools reside in districts with two or more high schools, LCHS and LCHS 7/8 appear better when comparing them by district mean scores than when comparing by school mean scores. High-performing high schools in districts with more than one high school appear lower in district rankings because their scores are averaged with other lower performing high schools in the same district. For this reason, when evaluating the district’s middle and high schools by looking at their statewide rankings as LCUSD does every year, the district should compare LCHS and LCHS 7/8 test scores against other schools’ test scores, not against averaged district scores. Even at the elementary level with three elementary schools, evaluating LCUSD elementary schools by the district’s average scores can mask potential issues at individual schools as readers will soon see.
Are LCHS 7/8 8th Graders Really Tops in the State in Math?
Recall that the October 17, 2019 Outlook headline story on LCUSD’s CAASPP results claimed “In math, LCHS 7/8 placed first in 8th grade and second in 7th-grade scale scores.” Below is the listing of the top 30 public schools in California ranked by 8th grade CAASPP math mean scale score:
Click on the graphic for a larger view of the same table. As can be seen, LCHS 7/8’s 8th graders ranked 26th in the state, not 1st. There were 2,670 public schools with 8th grade CAASPP mean scale scores in the 2019 CAASPP test data so 26th is still remarkably high — placing LCHS 7/8’s 8th graders in the 99th percentile statewide. So why the need to exaggerate LCHS 7/8’s 8th grade performance?
How About Other Grades?
Shown below is a table of how all of the district’s schools did by grade level and by CAASPP test (i.e. ELA and Math) compared to how the district claimed they did when looking at district grade-level mean scores. Note that there are more public elementary schools than middle/intermediate schools, than high schools so we show both the ordinal rank as well as the percentile within that grade for better context, along with the number of schools in each grade with a score:
The lowest ranking class for each subject among all grades in LCUSD schools is highlighted in the red cells and the highest in light green. As can be plainly seen, no cohort in any grade in any LCUSD school scored among the top 3 schools in California.
To see which schools ranked above LCE, PCR, PCY, LCHS 7/8, and LCHS to begin to answer Board member Jeffries’ question about finding other districts that are truly outperforming everyone else, LCMP generated the tables of top 30 schools (plus LCUSD’s schools since they fall outside the top 30 in almost every elementary grade cohort and in 7th grade) in each grade, first by CAASPP Mean ELA scores:
Top 30 California Pubic Schools by CAASPP Mean ELA Scale Score (by Grade):
And below are the top 30 schools in California ranked by CAASPP Mean Math Scale score in each grade:
Top 30 California Public Schools by CAASPP Mean Math Scale Score (by Grade):
Click on a thumbnail to see the table at full resolution.
What’s Going On at PCY?
When the percentile ranking of the district’s schools in math are plotted, there is a noticeable difference in performance at PCY compared to LCE and PCR:
The difference is still visible in the percentile ranking of the district’s schools in ELA, with the exception of PCY’s 3rd and 6th graders:
Note that the middle and high school grades and all PCR and LCE grades perform in or very close to the top two percentiles. PCY 4th graders, on the other hand, scored in only the 92.9th percentile in ELA, and 95.1st percentile in Mathematics. Thus there are literally hundreds of other schools whose 4th graders scored higher than PCY, a far cry from the top 3 headline touted by district.
The Bottom Line
The entire LCUSD community prides itself on its academic achievements. Community members are consistently told LCUSD is home to high achieving schools when compared to other public schools in California. However, cherry picking data to show our schools in a more favorable light than they actually are misleads parents, students, teachers, administrators and the rest of the community. The district should be justifiably proud of its students achievements. It need not tarnish the students’ accomplishments by exaggerating its state rankings. Furthermore, as evidenced by the line of questioning by LCUSD Board members Dan Jeffries and Joe Radabaugh at its October 29, 2019 meeting, limiting the scope of comparison deprives the district of the opportunity to see how other higher-performing schools and districts are doing, and how those lessons learned can help inform LCUSD’s drive for continual improvement.