FSAA Standard Setting Rule Development Workshop: Rule 6A-1.09430: Establishing Achievement Level Standards for Florida Standards Alternate Assessments
Webinar Transcript
February 20, 2017

(Slide 1) 

(Slide 2) My name is Vince Verges, and I’m the Assistant Deputy Commissioner of Accountability and Measurement at the Florida Department of Education. My official title is Assessment Director, and I’d like to walk you through this rule development workshop for the Florida Standards Assessment, Alternate Assessment, achievement level standard setting process.

This webinar is being record, and at the end of this presentation, there’s a link to the PowerPoint where you can have this PowerPoint as well as your feedback forms. I will go through this and provide some guidance as well as ways you can provide your own feedback, as well as talk through the process that we use to recommend these new standards.

So the purpose of the workshop is to express our intent to develop the rule for consideration by the State Board and legislators that establishes achievement level standards for the Florida Standards Alternate Assessment Performance Task. We want to obtain input from you, the audience, to be considered by the Commissioner in determining the recommend achievement levels that we’re going to present to the Florida legislature for their review and then to the State Board of Education for action at a May meeting.

We’ll provide you an overview of the alternate assessment, we’ll talk about the achievement level standard setting process, the recommendations from the educator and articulation panels, what those panels are and who those folks are, we’ll let you review the impact data that is what affect will these recommended cut scores have on students achievement, the percentage of students who are proficient, and we’ll ask for feedback from you and we’ll tell you how to do that as I mentioned.

But first of all, an overview of the Florida Standards Alternate Assessment. And as we go, you’ll have the opportunity to provide any questions in our chat box. I’ll stop at various places during the presentation when we get to natural stopping points. And if you have a question for me, please type it in the chat box, and I’ll answer them and they’ll be there for your answer as well as on the webinar recording for other folks to see what the question was and what the answers were.

So, first of all, the FSAA overview. The FSAA is, consists of two separate parts: the Performance Task, which is much like the old Florida Alternate Assessment, the assessment that most students who take the alternate assessment take. But we have a new component of the Florida Standards Alternate Assessment, the Datafolio. For the standard setting process, for the Datafolio, will take place during this summer and late this fall. And the standard setting process that we went through this past week was only dealing with the Performance Task, and that’s the grades and subjects that were talking about during this workshop.

So what exactly is the FSAA Performance Task, who’s assessed on the Performance Task, and what content is assessed on the Performance Task? We’ll go through that over these next few slides.

As you can see in this chart, we’ll talk about all these grades and subjects: ELA, Mathematics, Science, Algebra 1 EOC, Geometry EOC, and Biology 1 EOC. Other grade and subjects will be assessed this year. But for the purpose of this standard setting, it is just these grades and subjects that we’ll be discussing today. In ELA, Mathematics, Science, and EOCs, these are also the same assessments that students take on the Florida Standards Assessment.

Approximately 25,000 students take the alternate assessment each year in Florida, and this assessment is designed for students with significant cognitive disabilities. Students Individual Education Plan (IEP) teams responsible for determining whether students with disabilities should be assessed in the administration of the general standardized assessments or the FSAA. The IEP teams use the Assessment Participation Checklist to make that determination.

The assessment measures the student achievement of the Florida Standards Access Points. The Access Points link is there on the screen for you. Also, assesses students’ knowledge in Science on the Next Generation Sunshine State Standards Access Points. The FSAA Performance Task was implemented for the first time in Spring 2016, which is what we call a baseline administration. This is a transition from the last generation of alternate assessments, the FAA, which many folks are familiar with.

Stand by just one moment. We’re just making sure the slides that you’re seeing are the ones that I’m walking through. Okay.

(Slide 10) Alright. For those of you who are familiar with the Florida Standards Assessment Performance Task, this is, as this suggests, a task-based assessment. Teachers administer to students individually, via paper, so students see a paper-based test, and a certified teacher or other licensed professional who has worked extensively with the student, can also administer the test. And this person needs to be trained on the assessment procedures.

During the test, students select an answer to a question from three response options that are represented by pictures paired with text or numbers, symbols, words, and so forth. Students use whatever they use as their primary mode of communication. It’s important to note that the FSAA Performance Task is untimed.

(Slide 11) I also want to note that the assessments are designed for what’s called Universal Design for Learning. We also provide accommodated materials. And Universal Design is utilized during all development to ensure that equal access is available to all items for all students, whether or not they have a disability in addition to the significant cognitive disability that has them assigned to taking the FSAA. We also provide different formats and make other adjustments to ensure that a student with a disability gets the accommodations that he or she needs, such as braille or tactile materials. They may get a one-sided booklet to avoid the least amount of confusion. They may be able to use object replacement, such as cut-outs. And ASL is available, that’s American Sign Language that’s performed by the test administrators familiar with the student.

(Slide 12) As I mentioned before, students take a paper-based version of the assessment. That’s put before the student. And sometime after the test has been administered, the teacher or some other qualified administrator inputs the student responses into a system. So that computer interface is not done real-time.

(Slide 13) As I mentioned, here are the standards that we assess on the alternate assessment. For English Language Arts and Mathematics, we use the Florida Standards Access Points and Essential Understandings. Again, more about the Access Points and Essential Understandings is provided on the links that I gave earlier. And we also assess the Next Generation Sunshine State Standards Access Points in Science and, for this year, Social Studies, which you’ll see in standard setting in future years. So, we did not have any Social Studies assessments in this round of standard setting, so we won’t be discussing that today.

(Slide 14) So with that, I’ll pause. Are any questions? Checking if there are any chat questions. And if there are not, we’ll go on to the Standard Setting Process Overview.

(Slide 15) And so, and important question to ask, is why do we need standards, and we’ll talk about different kinds of standards. We want to identify or define what students know or are able to do. We want to make sure we have clear expectations for students, parents, and teachers. And we use these standards and the assessments and results that follow to improve teaching and learning.

(Slide 16) I mentioned that there are different types of standards, and the most important one, the foundational one, ones, are the content standards. Those content standards define exactly what students should know and be able to do. So the Sunshine State Standards Access Points were used for the old FAA. And we have, we do mention here, the Next Generation Sunshine State Standards were assessed on the FAA. But it’s still true for the Science standards; we use the Next Generation Sunshine State Standards. And then of course the Florida Standards Access Points that we have for ELA and Mathematics.

What we’re talking about here, and why it’s in italics, are the achievement standards. This is how much content knowledge a student is required to demonstrate. As I mentioned, the achievement level standards or performance standards, or you’ll hear me use the term cut scores later, are required for graduation so a student can meet their requirements by having a satisfactory score on the Access Algebra 1, the Grade 10 ELA assessment. 

And then accountability standards, such as any school grading criteria or annual measurable objectives. It is important to note here that how the Florida Standards Alternate Assessment scores will be used in school grading is yet to be determined. So we’re not sure if that’s going to happen or how it’s going to happen, but it is a possibility that FSAA results may be used in school accountability at some point. 

(Slide 17) So the textbook definition of what standard setting is, is it’s a process of deriving the levels of performance on a professional or educational assessment, by which we can classify students into certain categories. So, again, that’s the textbook definition. 

We use test scores, like other states, to group students into meaningful achievement levels. And standard setting is where we’re going to draw the line. So what’s the line that separates a level 1 student and a level 2 student, a level 2 from a 3, or a 3 from a 4, in the various achievement levels? We’re required to do this when implementing new standards or new assessments.

(Slide 18) It’s important to note that Standard Setting is what we call aspirational. So the process is based on determining what all students should know and need to do and not what a particular class and set of students or a group of students in the school know or are able to do. So the goal in Setting Standards is for all students across the state who take this assessment, not just students in certain groups or areas of the state or certain schools.

(Slide 19) You heard me mention four achievement levels. What’s different about the Florida Standards Alternate Assessment from FSA or even the old alternate assessment is that we’re using 4 achievement levels. And how we describe the general rule of what a student in each level is, is what we call the Achievement Level Policy Definition. So, you can see here level 1: students who are in level 1 do not demonstrate an adequate level of success with the Florida Standards. While in level 2, students are demonstrating a limited level of success. At level 3, we move from limited to satisfactory level of success. That’s the important level: 3 and above is what we’re hoping to see students in. And students at level 4 will demonstrate an above satisfactory level of success. 

So, again, these policy definitions need to apply to all grades and subjects. Whether it’s ELA, Mathematics, Science, and Social Studies will all start with these achievement level policy definitions before we get into the actual skill and knowledge that we expect students to know in each stated subject. 

(Slide 20) Here’s a representation of what I was speaking about earlier on the last two slides. When we have four achievement levels, we need to look at three cut points. Let me go back here. These level 3, as we mentioned before, indicate a satisfactory performance on each assessments. So when you see horizontal arrows—the down point arrows—those are the individual cut points where we’re looking to set those cut scores to separate students into each of the levels. 

(Slide 21) It’s a very complex process that we’ve used for all of our other assessments, and we’re using again for the Florida Standards Alternate Assessment. Whereby actively have the achievement policy definitions written, we’ve had achievement level descriptions drafted and reviewed by a panel of educations. And we had the educator panel this past week that actually set—recommended the achievement level cut scores. An articulation panel, subgroup of the educator panel from ELA and Mathematics, set additional—made additional recommendations. 

We’re at this step now workshop. This workshop which is happening live as well as the recording will be available along with a feedback form to be made public for educators and any other stakeholder to provide their input. After we collect all that information, the Commissioner will make her own recommendations to the State Board of Education, and while—or before the state board gives their approval, we’ll have a 90-day legislator review process whereby legislators and legislative staff can look at the recommendations and provide their own input. Then it all culminates in a May State Board of Education review. I’ll talk a little bit more about the stages after we look at this slide here. 

(Slide 22) I mentioned that we’ve done this before. The first time we did this way back in 1998. I actually served on the high school Mathematics panel for Grade 10 when the FCAT mathematics were first used as graduation requirements. 2008 is another interesting one to look at: that’s when the last set of alternate assessment achievement levels. And we’ve done it just about every year since. So this is not a new process to us although it may be new to many of you in the field, and it gets—it’s been nine years, almost nine years, since we’ve done it for an alternate assessment. 

(Slide 23) I mentioned the steps along the way and I’ll talk about when these rules happened and what they were. So we had the educator panel reviewed the achievement level descriptions back in December. The educator and articulation panel for Standard Setting were last week in Orlando. Today is our public workshop with a legislative review taking place for the rest of this month through May until the anticipated State Board of Education approval at a presentation at the May 16th State Board meeting

(Slide 24) Diving a little more in-depth on these panels, the achievement level description panel that was in December that was held over two days with 24 panelists. Those panelists reviewed the draft achievement level descriptions to look at all the exact skills, the required skills and knowledge, specifically what they should know and what they should be able to achieve for each graded subject. 

The ALD is the link between the content and the standards and the achievement standard that were set last week. And, again they were set by the policy definitions that guided the writing of the achievement level descriptions. 

(Slide 25) Last week’s meeting, the educator panel was held over three days with three rounds of standard setting which I’ll talk about in the next few slides. There were approximately 100 panelists in 11 rooms of about 7-10 panelist per room, setting standards all together. They recommended cut scores based primarily on the content of the standards although at some point along the way the educator panelists, and I’ll talk about when that was, had access and viewed impact data. That was after the first two rounds of judgment. 

(Slide 26) Next is the articulation panel which took place on Friday of last week. We had two panelists, each a special educator and a content educator from each educator panel. We’ll show you the names of those articulation panelists for ELA and Mathematics. They reviewed the educator panel recommendations for ELA, mathematics and recommended adjustments based on that. Worked with approximately 14 panelists. It’s important to note that once the Science and EOC panelists were done—since we didn’t articulate through the EOC or for grades 5/8 Science—we do not use an articulation panel for those grades and subjects; therefore, those were the final recommendations that you’ll see in the next few slides that were provided to you. The educator panels in these grades that I have here on the slides, ELA grades 3-10 and Math 3-8, those preliminary results then get reviewed by our articulation panelists. We’ll see both sets of recommendations, there are two steps along the way for those graded subjects. 

(Slide 27) The standard setting method that we used last week is new to many folks in the state but it’s not new for the standard setting processes. It’s the Body of Work Method. So, this is where panelist look at actual student work—this is Florida student work—and they make judgment regarding the achievement level to which the student work most closely corresponds. They did all their work largely based on the student work samples which were profiles. These profiles are based on actual FSA student score profiles from Spring 2016. And those profiles represent a typical pattern of item scores for students with various ability levels. Panelists classify each profile into the achievement level in which they felt that entire profile belonged. 

(Slide 28) Why do we use this method? Well, this method allowed panelists to actually see samples with actual student work to make the determinations, Body of Work method is really useful for complex assessments which, yeah, the Florida Standards Alternate Assessment-Performance Task certainly qualifies as a complex assessment. It’s been used in the past successfully in standard setting on similar assessment. It’s been used in other states and in Florida’s previous alternate assessment. So again, it’s new to many folks who have been involved in the standard setting process in Florida, but it’s not new to the field of standard setting, and it’s been used in Florida previously on the old alternate assessment. Consequently, since it’s got a good grounding in science, and it’s been used successfully; it’s certainly resulted in defensible achievement level standards in case anybody ever questioned the process by which the cut scores are set. 

(Slide 29) I mention those profiles. We’ll talk a little bit more about them here. Again, it was a primary tool that the panelists used. We have a set of 45 profiles per grade and content that was divided in the panelists. And each profile represent a typical pattern for students at that ability level throughout the 45 profiles. The profiles were ordered based on the ability level of students, from lowest to the highest ability. I have a graphic representation of that coming up in the next couple slides. All the items that were used in the profiles were operational and that means that they contributed to student scores in spring of 2016.

(Slide 30) I mentioned that visual representation. So here’s an example of the first 10 of the 45 profiles. You see the number there on the left, 1 through 10. The panelists have rounds, 3 rounds of judgments. And you can see the rows across the right from 1 to 16 represent the 16 items, and you can see—for the first profile you had a student that didn’t score anything, a low ability student that got 1 point on the Task 7, 1 point on 13. And as you see—if you go through the 10 profiles all the way down to the tenth profile, you see the student, the student for whom that profile is, has a couple of zeros but lots of 1s, couple of 2s, you have a 3 there. So that student—that profile demonstrates a little bit higher ability. And those profiles continue so on and so forth for the 45th profile where the students are getting most of the points. You still might see some zeros in the 45th profile, but you’re going to see mostly the higher score points. You’ll see a good number of 3s there.

(Slide 31) I mentioned the round, and here’s what happened in each of those rounds—how the Body of Works method worked. There was a practice round where panelists had a rating form and they had three profiles to start off with. And they were practicing how to categorize the students into one of the four achievement levels. After the practice round they began with the lowest ability profiles; panelists, by themselves, each independently classified each of the profiles into what they thought would be the achievement level for that profile.

At the beginning of round 2, they discussed the results of the 1st round judgments with the entire group. And after they had some discussion back and forth about why each panelist saw each profile in each achievement level, they, again, independently revised their judgments as appropriate based on their discussions.

During round 3, they saw the 2nd round judgments, but they also had a chance to see how many students would be classified in each achievement level in 2016 had those cut scores been in place for that round and at that time. So they could see what percent of the students were level 1, what percent of those students would have been level 2, 3, and 4. And then they made their final round of judgments.

So, the articulation panel on the Friday meeting—that one day meeting—the panelist reviewed how the standards appeared across the grades so they were able to see not just their own panel’s results; they got to see what all the other panels in their subject area for those grades judged as well. And they also got to see the impact data. So they not only saw the across-grades performance, but they saw the impact data across grades and see how many students would be in each achievement level. They also got to see the gender breakdowns. How did males versus females do versus the overall performance? And they also got to see how ethnicity, the major ethnicities, performed. And you’ll see some of those examples as we go through as I show the results.

(Slide 32) I mentioned the cut points. I mention the cut points there, and what we’re looking for is something we call the “just barely” achievement level descriptions. So, when considering where students fall in each achievement level, we’re really interested in those students who are just barely reaching into that standard. It may not be typical of a student from each achievement level, although, just barely the student does reach the standard, and I’ll give you an example of a level four student. So, you think of a level four student being a very high ability student. And you think, okay, for a level four maybe the student gets everything right. But that’s not necessarily the case. There may be cases, as I mentioned with the 45th profile, there might be some zeros. So the question is, to get into a level four, how many zeros are ok or how many ones are okay? So at some point, the panels make a judgment–the panelists make a judgment. What’s just good enough to say that this student is above satisfactory and not just merely satisfactory? And that same thing applies with the just barely level 2 and just barely level 3. What is it that makes this student just satisfactory? Or where they demonstrate, it’s inadequate, but they’re demonstrating more than just the bare minimums for level 2. So again, that “just barely description” is very important in the process. That gets you to where those red arrows are; what’s the minimum score you need to get into that level?

(Slide 33) We’ll talk for a little bit about who these educator panelists were, and we’ll look at their results and what they did. 

(Slide 34) As I mentioned, the educator panelists, not only did they do those steps that I mentioned before, of course, they got a general orientation. We need the slides that you’ve already seen and that you’ll see again. The panelists saw them in the orientation. They had an overview of the assessment and we talked about the standard setting process with them as well. Then as I’ve kind of alluded to, I’ve mentioned, panelists reviewed the tests in their grade level and the content area. They looked at the achievement level descriptions. They reviewed the profile and rater sheet, and then they completed the training round that I mentioned of the profile classifications. And then again, they looked through all achievement levels, all of the 45 profiles, and they set those standards in 3 rounds of making their own recommendations. 
And it’s important to note the educators were not expected to reach consensus. They gave their own individual judgments. We didn’t expect the panelists to all come to a consensus, to come to an agreement on where they thought those cut scores should be within those profiles.

(Slide 35) I think it’s also important that everybody understands who served on this panel. What did they look like overall? What was the male versus female breakdown, ethnicity, the district size, and so forth? Because it’s important to recognize that you want a panel that reflects the wide diversity of the state. Again, gender, ethnicity, district size, geographic location within the state, and so forth. The percentage that you see may not add up to 100 due to rounding. 

(Slide 36) So here’s how all the panels were broken down. For example, in ELA you can see we had grades 3 and 4 grouped together in one panel of either. Grades 5 and 6 were grouped together with either panelists. Seven and 8, grades 7 and 8, were grouped together with eight panelists; and grades 9 and 10 were grouped together. They had nine panelists. We had a total of 33 panelists for ELA.

Now I’ll read the rest of the slide and you can see how Mathematics, Science, and the EOCs were grouped together. And you may think about this grouping when you review the results of the educator panelists. You’ll see how much similarity or how much difference there was in those grades that were grouped together into single panels. Again, this PowerPoint will be available for you to download so you can study it without me lingering too long on a slide. You’ll have this information for you to refer back to.

(Slide 37) Gender. Again, I have the breakdown of the grades and the panelists, EOC (end-of-course) number of panelists with percent of the total. Female. You will see again, education is a fairly female-heavy profession, and so while male-female-ethnicity split may not represent the demographics of the greater public report, it certainly represents the demographics of what we see in the field of education. There are a few instances of where panelists, for whatever reason, did not provide a gender, maybe did not provide ethnicity, or didn’t provide location. So you’ll see some not-provided. Hopefully it will be a limited number, but you will see that in some of the breakdowns. 

(Slide 38) Here’s your gender, for example, for Mathematics. Again, you have this to refer to so it kind of follows the pattern the pattern you saw in ELA with lots of females. 

(Slide 39) Science. 

(Slide 40) Gender breakdown for the EOC panelists.

(Slide 41) The ethnicity breakdown. And you can see this from the students as well, you have white, black, and Hispanic. This is kind of an order by larger groups to less populous groups. You will see, this will be reverse when we show ethnicity breakout by the grade level subject, and I’ll explain how those are broken out. There’s ELA. 

(Slide 42) There’s Mathematics. 

(Slide 43) Science. 

(Slide 44) EOCs.

(Slide 45) And then district size. Very large, that would be your Orange, Miami-Dade so far. Large might be your Lee. Then Levy, we’ll call Levy in small. And we have “not provided.” 

(Slide 46) District size for Mathematics.

(Slide 47) Science, and just seven panelist so it’s hard to represent the entire district, entire state with just seven panelists. 

(Slide 48) And EOCs.

(Slide 49) Our impact data. So I’ve mentioned this before, our impact data showed the percent of student that would reach each achievement level in each grade and subject. Again this was introduced in round three after the panelists had made judgments in their first two rounds based solely on content. The impact data was there to inform the panelists’ recommendations but it did not necessarily determine those recommendations. In the end, the educator panelists’ recommendations were driven by the content of what was assessed.

(Slide 50) The slides that follow will show the impact data for all the grades and subjects as well as achievement level recommendations for grade 5 and 8 Science and EOC assessments that were made by the educator panels. And this impact data is going to help you inform your judgments. And, as a reminder, the educator panelists’ feedback is primarily based on content and it’s up to others including you as stakeholders to make those policy judgments. And then for grades 3-10 ELA, and 3-8 Mathematics, the educator panel’s recommendations were based on content as compared to the recommendations of our articulation panelist recommendations, which do take into consideration the impact data.

(Slide 51) I’ll linger on the first few slides to make sure we have a clear explanation of where it is that you’re looking at; then when the slides are similar, I’ll move through those a little more quickly. So it’s important to notice to note that we’re recommending a scale for the grade level assessments. That’s Science grades 5 and 8, ELA, and Mathematics; grades 3-8 Mathematics and 3-10 ELA. The scale goes from 540 to 660 with 600 as the mean scale score. 

What you see in this chart, and you’ll see some other charts like it later, you have the minimum-the lowest scale score on the new FSAA scale of 580 is the lowest scale score a student needs to get to be classified into level 2. Again, this is with the educator panel recommendations in Science. A student would need to get no lower than a 599 to be classified in a level 3. And a score of 616 to be classified into level 4. That’s a similar pattern you see for, again, the same panelists served on grades 5 and 8; you see a similar pattern here to the level 2 cut score, the level 3 cut score, and the level 4 cut score. 

So I refer to these as cut scores and all the slides where we see these diamonds; this orange-ish color is the level 2, green is level 3, and blue is level 4. And it’s also important to note here, the panelists never saw these scale scores. They made all their recommendations, and had all their discussions solely with respect to the profiles. So they only spoke in terms of the profiles on a scale of 1-45. They did not see these scale scores.

(Slide 52) Here’s a representation of what we have for the percentage of students in each achievement level. And again, I’ll linger on this slide then I’ll move more quickly through them when we get to the other slides that show the same information. So here you see that in Grade 5 Science you have something less than 20% of the students in level 1. The percentage is between 20 and nearly 50 in level 2. Between about 50 and to almost 80%, that group constitutes the level 3 students. And above that is the level 4 students. And in comparison you can see that there are somewhat fewer level 4 students and more level 3 students. You can see here that it takes a little bit more to get to a level 3 than it did down here. So here’s your level 2. So you have a lot more 2 students and about the same number of level 1 students.

(Slide 53) Here’s a tabular representation of the slide I just showed you. This shows you the exact percentage of students in each achievement level. So grades 5 and 8. Grade 5 here you had 17.7% in level 1, 29% in level 2, 31.3% in level 3, and 22% in level 4. And again when we get to other slides that show the same information, I’ll move a little bit more quickly through those.

(Slide 54) We broke it down, as I mentioned, into ethnicity and gender. So here’s the breakdown for grade 5 impact data for all students, then the percentages of male and female students that would be in each achievement level, and you see both the graphic representation, the bar graph as well as the table representation of that information that I’ve shown on previous slides. 

(Slide 55) Then there’s the grade 5 Science impact data. Here’s the grade 5 impact data by ethnicity and you’ll see here it’s arranged alphabetically. So you see these for the rest of the grades and subjects, alphabetically with American Indian, Asians then Black or African Americans, Hispanic-Latino, Native Hawaiian or Other, Two or More Races, Unknown, because the student data may not be complete for ethnicity, and then White or Caucasian. Again, it’s alphabetical; it’s not by percentage of students taking the test. Again, I’ll just take an example here. Look at all students. You’ve got 22% of the students in level 4 and 31.3% of all students in level 3 for grade 5 Science. And if you look at Black students, you see Black students somewhat outperformed the rest of the students in the state in level 4 and level 3 on the test while White students in this particular grade subject slightly underperformed the state average on the Florida Alternate Assessment at 20.6% and 26.8%. And again, I’ll move more quickly. Here you have the graphic and table representation for that data.

(Slide 56) Similarly for grade 8 you can see the comparison for all students, and male and female. Again with bar graph and table. 

(Slide 57) Science grade 8 impact data by ethnicity; and that trend I mention with Black students and then White or Caucasian students holds true there for grade 8. 

(Slide 58) Then the end-of-course exams. Again, I won’t linger on this. End-of-Course tests all had a scale from 725 to 875 with a mean of 800. So a 725-875 with a mean of 800. And you can see the minimum cut scores needed to get into level 2, 3, and 4 for Algebra 1, Biology, and Geometry. Again this presentation is there for you to look at, at your leisure. 

(Slide 59) Here’s the percentage of students that will be in each achievement level based on the educator panel recommendations for those three subjects: Algebra 1, Biology, and Geometry. 

(Slide 60) Here’s the percentage of students in each achievement level for those grade and subjects, or for those subjects. You can see quite a few more students in level 3 and 4 in Biology as compared to either Algebra 1 or Geometry. These were each separate committees. 

(Slide 61) The Algebra 1 impact data by gender

(Slide 62) By ethnicity. 

(Slide 63) Biology impact data by gender.

(Slide 64) Ethnicity.

(Slide 65) And lastly for the EOCs, Geometry by gender.

(Slide 66) And by ethnicity.

(Slide 67) This is an important table I think everybody should pay attention to. If you look at the percentage of students in level 3 and above, again, satisfactory is what we’re looking for; so you can see that column there, Level 3 and Above. If these recommendations were to go forward, you’d have 53% of the students in grade 5, 51% in grade 8, 47% in Algebra 1, 66% at 3 and above (Biology), and 53% 3 and above in Geometry, if these were to hold as they are. Now remember there is no articulation panel so this is what, based on, until we get your feedback, this is what the Commissioner will see as far as recommendations from the field, again, until she hears from other stakeholders.

(Slide 68) Now again, I won’t linger on these longer. It’s most important for the ELA and Mathematics grade level tests to look at the articulation panel but I wanted to provide this as context as you make recommendations. You’ll be making recommendations on where you might like to see adjustments for cut scores. For example, as I’ve mentioned, scale score range for the grade level assessments is 540 to 660 with a mean of 600. I’ll point out the difference here between grade 7 ELA. We see here the lowest for a level 3 is a 595, and compare that to a 611. If it’s on the same scale, it doesn’t necessarily need to be the same across grades and subjects, but you see a 611 is the lowest score the educator panel recommended for ELA. So 595 to 611; you’ll see that as you look at later slides. Again, there’s other analyses and comparisons you can make, but I wanted to point that one out. 

(Slide 69) And if you had those educator panelists’ recommendations in place with that kind of distribution for students in each achievement level, you can see it varies quite a bit across grades. Maybe not so much at level 1, but you see a lot of variation in level 2 and 3.

(Slide 70) You see the percentages. Again, it’s the articulation panel we want to get to, but this is just there for comparison.

(Slide 71) Then the impact data for; we do it all for gender.

(Slide 72) And ethnicity for each of the grades. 

(Slides 73-74) Again, I’ll move quickly through these. It’s the articulation panel that we want to focus on.

(Slide 75) Grade 5 gender.

(Slide 76) And ethnicity.

(Slide 77) Grade 6 gender.

(Slide 78) And ethnicity.

(Slide 79 Grade 7 gender. 

(Slide 80) And ethnicity. And a bit of a change there if we look at the African American and Caucasian; we see a little bit of a shift there from earlier. But again, you can look at all the changes across that ethnicity and gender.

(Slide 81) Here’s grade 8 gender.

(Slide 82) And ethnicity.

(Slide 83) Grade 9 gender.

(Slide 84) And ethnicity.

(Slide 85) And Grade 10 gender.

(Slide 86) And ethnicity.

(Slide 87) The Mathematics cut scores; a little bit less variability across grades within each of the achievement levels.

(Slide 88) A little less variability in the educator panel’s recommendations.

(Slide 89) There’s the table that shows the data we have in the previous slide that was in the graph.

(Slide 90) And then gender and ethnicity we will go 3 through 8. Gender.

(Slide 91) And ethnicity for Grade 3.

(Slide 92) Gender.

(Slide 93) And ethnicity for Grade 4.

(Slide 94) Gender Grade 5.

(Slide 95) And ethnicity.

(Slide 96) Grade 6 gender.

(Slide 97) And ethnicity.

(Slide 98) Grade 7 gender.

(Slide 99) And ethnicity.

(Slide 100) And lastly, Grade 8 gender.

(Slide 101) And ethnicity.

(Slide 102) Here I’ve mentioned the variability between grade 5 and 7 ELA. You’ll see here in the table you have for level 3 and above for ELA. Again this is educator panel; you range from 31% level 3 and above in grade 5 all the way to 65% three and above at grade 7. And then a little bit less variability here. It ranges from 40 to 58%; so about an 18% spread in there between 3 and 8.

(Slide 103) And now our articulation panel. 

(Slide 104) As I mentioned that we only did the articulation panel for ELA and Mathematics, grades 3-10 ELA and 3-8 Mathematics. And as I mentioned, all of these educators that you see here served on the educator panel, and they represented their grade band on the panel.

(Slide 105) So we kind of have some differences but really there are a lot of similarities. So educator panel was comprised of individuals with content and special education expertise, which were also on the articulation panel. Both the educators and the articulation panel understand what’s expected of Florida students. But the educator panel focused mostly on content while making recommendations. The articulation panel, a big difference was, as I mentioned previously, they got to examine achievement level distributions across all grades and subjects,all grades 3-10 and grades 3-8. And they consider policy when making their recommendations. And they looked at consequences, so what would it mean, what might happen, if we let our recommendations stand. 

(Slide 106) The articulation panel looked at what students should know and be able to do, of course, and do they, their question was, did the educator panel, is their recommendation consistent with expectations for student achievement? And then looking at the population and the assessment, is the impact data reasonable? 

(Slide 107) So again, I’ve mentioned this previously, the educator panel did not have to reach consensus, but we asked the articulation panel to all reach consensus on any adjustments that were made in the profiles, those one to 45 profiles. So if there were modifications, the facilitator was able to pretty quickly provide new impact data to show what would happen with those proposed modifications. And then the panelists discussed the rationales for the changes. And we did tell the panelists that their recommendations were going to be published and considered by you, the general public, by the Commissioner, and by the State Board of Education in determining the final achievement level standards.

(Slide 108) So here’s the articulation panel for ELA. And if you recall the educator panel, there’s a lot of variation across the level 2, level 3, and level 4 cuts. After the articulation panel looked at it, there was a lot more consistency. Look at my example was Grade 7; that level 3 cut score, the minimum acceptable score for a level 3, was 598, and here it’s a 602. You remember previously there was a good bit of spread between those two. And again, you can make yourself make those comparisons because you’ll have this PowerPoint to look at. And again, reminder that the scale is from 540 to 660.

(Slide 109) There’s the breakdown by achievement level using that data. 

(Slide 110) And there’s a tabular representation of what’s in that graph as we have in those previous slides, and again this is for ELA. 

(Slide 111) And then impact data. So this is much like the previous slides, but this is the articulation. This is the impact data based on the articulation panel recommendations that you saw on the previous slides. 

(Slide 112) There’s by ethnicity for Grade 3. Again, I can move quickly through these and you can have these to refer to. 

(Slide 113) Grade 4 by Gender.

(Slide 114) Grade 4 by ethnicity. 

(Slide 115) Grade 5 by gender. 

(Slide 116) Grade 5 by ethnicity. 

(Slide 117) Grade 6 by gender.

(Slide 118) Grade 6 by ethnicity.

(Slide 119) Seven by gender.

(Slide 120) Grade 7 ELA by ethnicity. 

(Slide 121) Grade 8 by gender. 

(Slide 122) ELA Grade 8 ethnicity. 

(Slide 123) Grade 9 gender. 

(Slide 124) And ethnicity.

(Slide 125) Grade 10 by gender.

(Slide 126) And by ethnicity.

(Slide 127) The Mathematics recommendation; they were already pretty smooth. The articulation panel made some additional adjustments in some places. I will take this time to talk about when you make your recommendations here as we get closer to the end. If you have a recommendation that you would like to see more students in a certain level, I’ll say grade 5 Mathematics for an example, 601. If you think there should be more students in level 3, you might suggest a cut score that’s lower than 601. So you want more students in level 3, you want to lower the bar a bit and make it a bit easier to get in a level 3, you can make a recommendation that we reduce this by a few points. 

Now, it may vary, and that’s why you have the educator panel recommendations handy, as well as the articulation panel. You can see the adjustments that were made through the educator panel and the articulation panel and the effect that had on the percent of students in each achievement level, and the effect it had on gender and ethnicity breakdown, the impact there. 

So, again, you can use those as a rough proxy to make a recommendation by what you might like to see the cut score. Or when you put the feedback on the form, you may say you’d like to see a certain percentage of students at level 3 and above, or a certain ethnicity issue or impact to be addressed by a certain way, and you can make a recommendation that way. So you can recommend to adjust a cut score up or down, or you can recommend a percentage of ethnicity or impact that you might see by achievement level or by gender, and make a recommendation about what percentage you would like to see as a consideration. And it will be helpful if you provided a rationale for your recommendation as well. Give us a recommendation of a cut score to adjust, or a percentage to adjust, and the rationale for that cut or for that change.

(Slide 128) And here’s the, again, the impact by achievement level for Mathematics for the articulation panel. 

(Slide 129) The table representation that we’ve seen previously; this is for Mathematics articulation panel. 

(Slide 130) Grade 3 by gender.

(Slide 131) By ethnicity.

(Slide 132) Grade 4 gender.

(Slide 133) And ethnicity. 

(Slide 134) Grade 5 by gender.

(Slide 135) And ethnicity.

(Slide 136) Grade 6 gender.

(Slide 137) And ethnicity.

(Slide 138) Grade 7 gender.

(Slide 139) And ethnicity.

(Slide 140) Grade 8 gender.

(Slide 141) And ethnicity. 

(Slide 142) And finally, this is after articulation. You can see the percentage of students that are 3 and above, and again, also 2 and above. We haven’t talked about that much, and those students who are just in level 4. Which you can see here after articulation, a much closer spread. We have a low of 47%, 3 and above in grade 6. And we go as high as 58% in grade 7 at 3 and above. An 11 percentage point difference across the grades there. And Mathematics you can see we have a low of 47% and we go as high as 55% in grade 8 level 3 and above. And it’s based on the articulation panel.

(Slide 143) So we have some slides here that follow. It says FSAA recommendations as compared to FAA. But we also, we’ve added some information about FSA, so we’ll make a change to this presentation so when people pull it down they’ll see that recommendations for FSAA, for FAA, and FSA. 

(Slide 144) So here, if you look at Reading versus ELA, you see a column here. In 2008 when we first had standards on the Florida Alternate Assessment, again they were different achievement levels, but at some point, how it was scored and reported, we had 69% of grade 3 students that were considered satisfactory or above. So 69% satisfactory or above. You compare that for ELA grade 3 on the FSA, that’s the statewide assessment the first year it was given, we had 53% that were satisfactory. And then based on the articulation panel’s recommendations as they stand now, you have 49% of the students with the satisfactory or above. 

So, I won’t read the rest of these, though I’ll have you compare across to look at where the comparisons were for FAA. You’ll see they are mostly, they were in the high sixties to low seventies, up to 70%. FSA ranges in the low to mid-fifties, and FSAA ranges from around 49%, or high forties to high fifties. And again this is something that is perfectly reasonable for you to consider in making your recommendations based on this or based on cut scores.

So there’s your Reading/ELA. 

(Slide 145) There’s Mathematics, and you see from the mid- to high-sixties for the old Florida Alternate Assessment. From somewhere as low as 45% in grade 8, which was in 2015. We had many of the students who took the grade 8 assessment who will have normally taken the Algebra 1 EOC. So you have a lot of 8th graders taking the Algebra 1 EOC who did not take the grade 8 assessment. Therefore, that’s why you see that percentage a bit lower there. But otherwise in the mid-anywhere from 50 to 59% satisfactory and above. And then FSAA based on the standard setting panelists’ recommendations you go from anywhere from 47% to 55% satisfactory.

(Slide 146) And here’s a FAA, FSA, AND FSAA percent satisfactory. Actually in Science it should be the old NGSSS in Science, we’ll add those as well. But you see there again, high-sixties, low-seventies for the old alternate assessment. We did not have a Biology 1 alternate assessment back in 2008, nor did we have an Algebra 1, nor Geometry EOC. So you see N/A there. The Science scores from the first year of the NGSSS Science. And you see the Florida Standards Alternate Assessment grades 5 and 8 at 53% and 51%. 66% three and above for Biology. 47% for FSAA Algebra 1, and 53% for Geometry. And you can make your recommendations based on that. And again we’ll those Science grades 5, 8, and 11 back from when those were; we didn’t have a column there.

(Slide 147) The Commissioner is going to consider the recommendations of the educator panel, again mostly for Science and the EOCs. She’ll also be looking at what the educator panel recommended for the grade-level assessments. She’ll look at what the articulation panel recommended for ELA grades 3 through 10, and Math 3 through 8; the input from the public, including all the stakeholders, including educators and parents, before she makes her recommendations.

Then the Commissioner’s recommendations will be provided to the legislature for the statutorily-required 90-day review period. And again, we expect these to be approved some time at the State Board of Education meeting so when scored are reported in June, the first week on June, they will include the Commissioner’s recommendation, or the State Board of Education’s approved cut scores, on the students’ reports and in the file the districts will receive.

(Slide 148) This rule, including the new PowerPoint, I’ll be presenting to the State Board of Education for adoption in May. The State Board will look at the educator and articulation panel recommendations; any public input we gather between now and the State Board meeting, again all the way into May; along with the Commissioner’s recommendation. At which point the Board will adopt those new achievement level standards.

(Slide 149) We definitely appreciate your interest, whether you’ve been with us live or you’ll viewing the recording of this. We invite you to visit the standard setting page to provide your feedback. There’s a link to the document there, and this PowerPoint is also linked there. Again, we’ll have a slightly revised version up sometime tomorrow that addresses some of the thing we discussed here. You can click on the feedback form there under the Florida Standards Alternate Assessment and provide your input. We really appreciate hearing what you have to say, whether it’s a change to increase or decrease cut scores, or whether you are comfortable with these cut scores. So any feedback is appreciated and accepted. 

Thank you. I see we have no comments, no questions. Good Afternoon.

(Slide 150)