Advancing Global Competency: Assessing Student Learning of Twenty-First-Century Skills
The Alliance for Excellent Education and Asia Society Invite You to Attend a Webinar on
Advancing Global Competency: Assessing Student Learning of Twenty-First-Century Skills
Laura Hamilton, PhD, Senior Behavior Scientist,
Mariana Haynes, PhD, Senior Fellow,
Alliance for Excellent Education
Anthony Jackson, PhD, Vice President for Education,
Gen Ling Chang, Associate Director, Toronto District School Board (Ontario, Canada)
Please join the Alliance for Excellent Education and Asia Society for a webinar on measuring twenty-first-century skills—the central focus of a RAND report presented at the October 2013 meeting of the Global Cities Education Network in Singapore.
The Asia Society organized the Global Cities Education Network, which works to identify internationally informed practices in urban school systems in North America and Asia to transform learning and achieve equity.
This webinar will focus on the design of “state-of-the-art” measures of student proficiency in critical thinking, problem solving, and related twenty-first-century skills essential to succeeding in a technologically advanced, interconnected global economy. The panelists will explore the challenges educators face in eliciting and assessing student outcomes beyond simple academic content knowledge. They will examine what this means for designing curricula and models of professional development that provide teachers with the knowledge and skills to help all students attain high-level cognitive and interpersonal skills. Panelists will also address questions submitted by webinar viewers from across the country.
Register and submit questions for the webinar using the form below.
Please direct questions concerning the webinar to firstname.lastname@example.org. If you are unable to view the webinar live, an archived version will be available at https://all4ed.org/webinars within one to two business days after the event airs.
This webinar is made possible with support from
The Alliance for Excellent Education is a Washington, DC–based national policy and advocacy organization dedicated to ensuring that all students, particularly those traditionally underserved, graduate from high school ready for success in college, work, and citizenship. www.all4ed.org.
Asia Society is the leading educational organization dedicated to promoting mutual understanding and strengthening partnerships among peoples, leaders, and institutions of Asia and the United States in a global context. Across the fields of arts, business, culture, education, and policy, the Society provides insight, generates ideas, and promotes collaboration to address present challenges and create a shared future. www.asiasociety.org
>>> Hello. I am Mariana Hayes senior fellow for the alliance of excellent education, a non-profit policy and advocacy organization based here in Washington, D.C. the alliance and excellence society have a webinar for 21 the century skills, the central focus of the rand report presented at the October 2013 meeting of the global city’s education network in Singapore. The Asia society organized the global cities education network which works to identify internationally informed practices to transform learning and achieve equity.
We’ll learn about all of this work in a moment, but before we begin, I would like to thank the MetLife foundation for its generous support of this webinar and also to thank our panelists for joining us today. Today’s panelists will explore of state-of-the-art measures of students’ proficiency in critical thinking, problem solving and effective communication skills. These deeper learning outcomes are considered essential to succeed in a knowledge-based global economy, and our guests will address what educators will consider in designing or selecting to improve instruction or evaluate the impact of their system. So what types of assessments provide valid information regarding the cognitive and interpersonal skills students need to succeed in college and career? How can educators use information from these measures to improve practice and learning in the classroom?
So let’s meet our guests. With us in the studio today is Dr. Anthony Jackson, the vice president for education at Asia society. We strive to enable all students to graduate from high school, prepared for college, for work in the global economy and for 21st century global citizenship. Over the past nine years — nine years, tony has led the development of Asia society’s international study schools network and he also leads Asia society’s partnership for global learning, a national membership network of practitioners and policymakers dedicated to integrating knowledge about high-performing systems into the American education system.
Next in our studio is Dr. Laura Hamilton. Laura is a senior behavior scientist with Rand Corporation. She’s a professor at the rand graduate school and associate professor at university of Pittsburgh’s learning sciences and policy program. Do I have all of that right? >> Yes. >> okay.
Finally, joining us remotely from Ontario, Canada, gen ling chang. She serves as the associate director for the Toronto district school board. Previously, she was responsible for teaching, learning curriculum across the district in 580 schools, serving over 200,000 students. She’s also worked with 60 of 72 district school boards in the student achievement division in the ministry of education in Ontario.
So like most our webcast, this is an interactive affair and we relish your participation. If you would like to ask questions of our webinar guests please do so using the number below the window and we’ll turn to your questions from time to time throughout this webinar. Also if you’re on twitter, we encourage you to tweet throughout the webinar using the hash tag that you will see in the left corner of the window.
Let me set the stage for the webinar. Globalization of the economy and constant innovations in technology are posing new challenges for innovations and societies alike and competition in the worldwide marketplace. Young Americans must be proficient thinkers, problem solvers and effective communicators. By 2018, approximately two-thirds of all jobs in the U.S. will require some post-secondary education. Many students leave high school, diploma in hand, but are unprepared for a post-secondary education. Roughly 40% of all college students and 60% of students at community colleges are not college-ready and must take at least one remedial course to advance to credit-bearing course work. So policy discussions of late focus on how do you teach and assess these 21st century skills. We are also learning a great deal about how the U.S. compares to other advanced nations on international assessments such as the program for international student assessment or PISA which is administered to 15-year-olds in 34 nations of the organization for economic cooperation and development. PISA measures student’ ability to apply deep knowledge and complex thinking to real world problems. These kinds of skills are reliable predictors of economic and social well-being for both the individual and in the aggregate for the society at large.
So in response, 46 states and the District of Columbia are in the process of implementing the common core state standards or other college and career-ready standards, covering more than 90% of students. Of these internationally benchmark standards base a premium. Simply on students acquiring information, but on their ability to analyze, evaluate and create new solutions and products. So how can education systems ensure that assessments provide quality information as well as inspire practices that lead to these deeper learning outcomes?
So to begin our discussion, we’re going turn now to Dr. Jackson. So tell me, Asia society launched the global city’s education network to connect these urban systems and they’re focusing on some of the most challenging elements of their systems and thou make improvements both in terms of learning and equity and you’re dealing with cities that are both in the U.S. and Asia. So can you kind of provide an overview of this initiative, the commissioning of the rand report on assessing 21st-century skills, the recent convening that took place this past month in Singapore and what’s your thinking about assessing 21st-century skills? >>
let me back up just a little bit and first say that everything we do in the education department and from the partnership for global learning is about developing students’ global competence and college and career readiness. That’s our mantra and so we define global competence the capacity and disposition to act on matters of global significance and I think it’s important to begin the discussion right away by realizing that you don’t develop global confidence as an extracurricular activity. It’s core to the study within the disciplines and the interdisciplinary kind of instruction that occurs around the core curriculum. And so what we try to do in all our work is to enable students to develop competencies in four main areas of defined global competence. One has to do with investigating the world and the ability to investigate the world, to frame and resolve problems that may have no one right answer, but really require the flexibility of mind that we want students to have and to be able to use international resources to pose solutions to those kinds of problems. We also want our students to have the capacity to have perspective. We believe it is important. For them to communicate along cultural lines and communicate in more than one language and we also think it’s very important for students to use their knowledge to take action and to mobilize others to take action. So all of this work around global competence, kind of, as I said, is a through line to all our work. In addition to the global cities education network, let me mention another network that precedes that. It’s the network of primarily urban schools called the international study schools network. This network was founded about ten years ago, in 2003. The premise here is to use global competence — the quest for global competence as the core premise of what these schools are all about and to use that not only to develop global coverage tense and to develop high achievement and this is a network of schools. 35 different schools across eight different states and 17,000 students served primarily in low income communities and kids of color and we’ve basically proven that this global focus can be the basis for high achievement. Across the board, 95% of the kids graduate on time. About 95%, and 85% go on to college so it is a network they think is showing that global competence can be the basis for high quality education. One of the key factors in that work is a system we call the graduation performance system and that is our way of focusing in on the curriculum assessment and it is a performance-based instruction system that enables teachers to reframe their course work into modules that make use of project-based instruction, informative and some of the performance assessment as a way to drive towards high achievement and actually, not unlike the kind of assessment, questions, for example, that appear on PISA, but have the opportunities to be about learning and upon driving learning particularly toward the 21st century global competencies we want them to have and this is all in a digital platform that allows teachers to be able to score work and create evidence or denote evidence of student proficiency right on the paper or the presentation itself. The kids get the feedback that allows them to learn from their work or learn from whether or not they were able to provide evidence of their competency and use that to improve their performance going forward. So that’s the ISSN, as we call it. That’s a network of urban schools. The network, as you mentioned is a network of urban school systems. It’s five systems in Asia, and five systems in north America that have come together as a kind of learning community to exchange best practices and think deeply about some of the common problems they have including assessment and try to come to common solutions.
So Hong Kong, Melbourne, Seoul, shanghai, Asia, Denver, Houston, Lexington, Seattle, and Toronto have come together at the most recent meeting in Singapore, as you mentioned and the key focus there was 21st-century skills and global competence. What do we know about instruction and in particular, what do we know about assessment in those kind of competences and to undergird that discussion we asked the rand corporation to develop a paper that essentially would give us the state of the art on some assessments, large scale, in-class, the whole kind of current landscape, if you will, of what assessment — what kind of assessment for 21st century skills out there. That’s the paper that we’ll discuss with Laura, I think. >> I am so glad you commissioned it, by the way.
>> It was great.
>> It’s a wonderful report. We are so glad you’re here with us, Laura. So you wrote this with your colleagues — let me see if I have this right, Jim foland and brian stecker and the title is measuring “21st century skills guidance for educator to the October 2013 meeting of the global cities education in Singapore.” that’s the full title, correct? >> Yes. >> The link to this report will be provided on the registration page where you will find the archived video for today’s webinar. So, Laura, tell us about state of the art in assessing these proficiencies in critical thinking, problem solving and other related 21st-century skills and if you can help us all to understand and think about how the design assessments can drive the nature and the quality of instruction.
LAURA HAMILTON >> Thanks. Yes, so I definitely want to acknowledge my colleagues Jim foland and Brian stecker who contributed to the report and also the Asia society and healing foundation for supporting the work. You know, one of the big challenges that educators are facing is trying to figure out how to incorporate these 21st century skills and competencies into their instruction. It’s been a focus of a lot of debate at the — at the district and state level and educators are really struggling to figure out how do they get this into their instruction? The latest challenge is figuring out how do you assess students’ progress in these skills and once you’ve incorporated it through teaching and in particular, there is a huge range of tools out there to do this assessment and teachers are often in a position of trying to figure out how do you select from among this set of tools to figure on the which set will work in the classroom context. So the Asia society commissioned us to write a paper that would explore some of the tools that are out there and both ones that are currently in development as well as ones that are more used on a more widespread basis. I should say the link to the paper that’s there is a draft version. A work in paper and we’ll have the final version probably in about a month, but we encourage you to take a look at the paper version if you’re interested.
>> So before I start talking about the measures I want to say what we mean by 21st century competencies. I think the term 21st centuries and 21st-century skills are something we’ve heard a lot about these days. We’ve shifted the title these days and changed the word skills to competencies in part because a lot of these things aren’t skills in terms of how we typically think of skill, but regardless, we’re really talking about all of the characteristics that students need to demonstrate in order to be ready for college and careers in the future. And I think when we talk about this; we usually mean things that go beyond the traditional academic content that we usually measure with our high stakes accountability tests.
>> So, we — in our report, sometimes we discuss what we think these skills are. Lots of different groups have published framework and characterizations from these different skills and one that we liked was the national research council which divided them into three categories and cognitive will skills and intrapersonal competencies. It doesn’t matter how much how you categorize them and our purpose here was to just lay out a range of these different competencies so we could talk about what assessment looks like and we can come up with guidelines that might be applicable across a range of these kind of attributes. So we went out to look at what different assessment tools were out there and there were quite a few that educators have to choose from. You can see here, we have listed just a sample of them and this is just the tip of the iceberg. There are lots of assessments that have been developed. Some of them have been used for years and some have been piloted and some are very much in development. We couldn’t possibly examine all of them. In the report we go into depth on about 20 different assessments and we picked them to sort of represent a range of different 21st-century competencies as well as a range of different assessment format. So what I wanted to do office on show you some examples and I’ll go through these pretty quickly, but they’re meant to illustrate the different formats that are used and the different skills that have been assessed. The first one is the mission skills assessment and this one is designed by etf and it’s for including, collaboration, motivation, creativity, time management and resilience. It produces school-level indicators rather than student level indicators and it provides information from a variety of different sources. They’re open-ended assessments as well as student responses to a situational judgment test which is what you see on the slide as an example of one of those items. We talked with teachers and school administrators in some schools that were using this and they talked about the benefits of having this kind of assessment in terms of helping them really understand how they could incorporate these kind of competencies into their instruction and how they could be more intentional about making sure that they were working toward student proficiency in these areas and they also talked about how they helped students to recognize what their strengths and weaknesses were and how to address those. >> so, the second example is the simulation of — that’s a measure of — it’s actually more of a curriculum/assessment of foreign language skills. It has the user interact with an avatar and speak and it provides responses when the student speaks. So it’s got — you’re in a situation that’s very rich in social context. There’s an intelligent computing system that is sensitive to local idioms and dialects so it gives you feedback. You know, if you’re many using the wrong phrasing. If you’re saying something that’s not quite correct it will come in and correct you. It’s a nice example of how to blend the assessment that gives frequent feedback and helps students learn.
>> the next one I will talk about is one of the assessment consortia that are aligned with the common core state standards in the U.S. and one of them is the partnership for the assessment of readiness for college and career or PARCC we think a lot about multiple choice as being an inexpensive way to administer lots of items in a short timeframe and it’s automatically scored so it’s cheap and on the other hand we have the more complex open-ended items that might require human raiders and some of the tests we’ve seen in a place like PARCC are a nice blend. So the one we’re showing you here, ask students to look at a spread sheet and drag information from tiles into equations. So it’s a little bit more complex and requires more construction on the part of the student relative to a traditional multiple choice item, but at the same time it doesn’t take a long time to administer this item and it’s automatically scored so it opens up opportunities for administering assessment in a cost effective way. >> the next one I’ll talk about is pizza. In 2015 PISA will be experimenting with a computer- — and one of the reasons why this is a nice example because it combines two of the 21st century competencies that we talked about. We talked about the list of problem solving and the collaboration and this is a way of linking the two using technology that involves having the student interact with another person who happens to be an avatar, but, you know, sort of simulating what it would be like to work with another person, bring your different sets of knowledge to bear on a problem or big you are out how to combine that knowledge together to reach a solution. One of the reasons to use an avatar as opposed to another student is that you can see what you get from another person. The next example is the college and career ready diagnostic and I’m showing this part to demonstrate that not all of these things use super nifty new computer technology. This is a kind of self-report measure. It could be administered either on computer or on pencil and paper. This one was developed by David Conley at the policy improvement center. Excuse me, I’m sorry. Allergies. It’s a pretty standard way of measuring some of the intrapersonal skills through the self-report measures. Back to the nifty computer technology and the next one I’m going to show is the echo move which is sort of immerses students in a complex problem-solving com text where they’re solving a problem related to the environment. Students can take different measurements and figure out what they want to measure. They can interact with different people to get information. So it’s really designed to simulate how a scientist might go about trying to solve this kind of a problem. The next one I’ll show is developed by ets and it’s the formulating hypothesis assessment that measures an aspect of creativity. Creativity is one of those somewhat hard to define skills and it can involve a variety of different specific abilities, but one of those is the ability to come up with explanations for things that you see that you might not quite understand. So this is an assessment that provides some information and then asks the student to identify some possible hypothesis for why they see the information they do. And then the final one I’ll mention which tony had described was the [gp] is that Asia society developed, and I think this is a nice example of, you know, an attempt to kind of blend instruction and assessment to allow students to engage in more complex problem-solving activities, but it’s different from the typical portfolio student work in that teachers are given pretty specific guidance about task development. About how to rate student work and so forth so that in the end we have a measure of student accomplishment that was somewhat consistent across students and teachers and it was more structured than your typical portfolio of student work.
Okay. So that was a quick overview of just, you know, some of the examples of out there and you can get a sense of the variety and also the challenge facing educators trying to figure out how to do this. So if you want to decide and you see a measure that seems like something you might want to adopt or maybe it uses some of the nice simulation kinds of approaches that look like they might be engaging for kids. It looks like there’s maybe some evidence that the scores are reliable and valid for the purpose that you want them for. At the same time you’ve got to think about how much time it will take not just to administer, but also how to train my teachers to use this. How does this fit into my curriculum? Do I have the necessary computer technology to implement it? And so it’s a huge set of considerations that you need to keep in mind when making this choice and that was what we were trying to accomplish with this paper to try to help people figure out what are those considerations and how do they weigh them. So the very first thing that we recommend people think about is what is the purpose? Why do you want to measure critical thinking or creativity? What do you plan to do with those scores when you get them?
So there are a number of possible purposes for these assessments and you might want to use this just as a monitoring school. You can think about the national association of education progress as an example and it’s a temperature-taking mechanism. Look at how we’re doing. You might want to use them as accountability purposes to assign words or sanctions to identify schools that might be in need of improvement to evaluate individual teachers and so forth. You might want to use them as a way of setting priorities or communicating goals. You know, the teachers accomplish a whole bunch of thing, but if we’re measuring three of them we’re sending a clear message that these are the three things we want you to prioritize. Finally, you might be adopting it to help improve teaching and learning at the classroom level and that’s the emphasis of our paper and what I’ll focus on in this presentation is how do we use these assessments to try to improve teaching and learning? So within, you know, even within that purpose there are still things we need to consider. We need to cluster these into three categories, instructional, practical. Those responsible for choosing assessments need to weigh these against one another and I’ll talk about those specific considerations.
So the first thing we recommend, un, if your goal is to use this to improve instruction and the instructional considerations are probably what you want to look at first, and there are a number of those, but I’ll highlight, too. And you want an assessment that gives actionable information to teachers. So if we’re going use this to make decisions about instruction it needs to produce information in a way that teachers can look at it and make decisions about what to do next. So that means it should be aligned with the curriculum. It should provide information with the level of detail that allows you to determine specific strengths and weaknesses and so forth, so you need to look at the reporting system and what kind of information you get back from it.
The second consideration is that for certain purposes you might want an assessment that’s authentic in terms of putting the student in a situation that mimics the real-world problem solving. That’s not to say that all of your assessments need to be that way, but for certain instructional uses, you know, you may want to consider the extent to which is this an authentic information and will it engage students and elicit their best thinking. Once you determine it serves your instructional purpose you also want to look at the technical considerations and the three main ones are reliability, validity and fairness. If you’re using tests for high stakes, we have very high standards for thinking of validity, reliability and fairness. And if you’re just using them at the classroom level for instruction, typically, you don’t worry quite as much about those things but you should worry about them some. So if the tests do not produce scores that are consistent from one administration to the next it’s probably not going to be useful for teachers. So what you really want to do is look for evidence that this particular measure has validity for the particular purpose they’re using it for. If the goal is to inform instruction you want something for which there’s evidence that it’s useful for that purpose.
And then finally, we don’t like to think about this one quite as much, but there are the practical considerations and there’s the cost, the technical requirement, how many computers do you need and what kind of broadband access do you need and how is the assessment scored and do you need to train folks to score it? This is when we start to get into tradeoffs because often as you start to meet the technical and the instructional considerations it also imposes significant burdens on the practical side.
So what we try to do in the paper was to provide some guidance to educators for thinking about how to balance these different considerations against one another so that they can select the best tools for their purpose.
So on this slide we came up with 12 sets of the guidelines and I’m not going go through each one of them in detail. As a teaser, if you want to look at paper to figure out what all of these mean, I’ll just mention a few of them. You know, we already mentioned the need to start with figuring out what the purpose is probably the first and most important, but also thinking about where this measure fits in the overall assessment strategy and keeping in mind that you probably can’t serve all purposes with one measure. There’s often a temptation to, you know, if you have an assessment that’s being used required by your state and you’re getting these scores back, you might think I can also use this for student placement purposes. I should also be encouraging my teachers to use it for decision making in the classroom, and oftentimes the assessments that are developed for one purpose don’t work well and can even have negative consequences if they were used for a purpose that they weren’t validated for. Do I have measures that maybe serve more purposes than others that might serve more formative purposes and how do they all fit together? Another one that I just think is important to mention is the idea of unintended consequences. We know from decades of research on high-stakes testing in particular that often you get negative unintended consequences when you attach stakes to certain measures, and even when you don’t attach stakes to them and just publish the results from them. So it’s important once you’ve chosen a test and put it into place to monitor how it’s working and what kind of consequences it’s producing and if you see that it’s not generating the kinds of consequences that you’d like you need to think about how do I change the administration conditions? Do I need to pick a different measure and so forth? So that’s just a sampling of the things we talked about, but taking all of these considerations into account in a systematic way is really important for maximizing the likelihood that your assessment will ultimately lead to improved teaching and learning.
>> thank you so much. Thank you. I’m going to turn now to you, tony as the commissioner of this report to talk about what you consider some of the key takeaway, at least in terms of how you’re thinking about the work you do for the issn network, the international study schools network. You work with the global cities education network. Are there things that you found that resonated in particular to what you’re trying to achieve through the Asia society?
>> I would just say one thing. This very notion of performance assessment being for the purpose of learning. I think that’s really been key to our thinking and there’s a colleague of ours in Hong Kong of what is the purpose of what you’re doing? The purpose of assessment should be to from moat an accounting of how well students learn, but also to advance their learning unto themselves. As we developed the graduation performance system that’s been utmost in our mind and to have a system that provides opportunities for teachers, for example, to use the common core standards and our set of performance objectives to frame the opportunity for students to dive deeply into problem, but at the same time, to help them develop their capacity and to help them meet the requirements of the new tests of the common core so that it’s a learning opportunity as well as an assessment opportunity at the very same time. So that is one of the things I’ve taken away is that assessments really can have a major influence on driving instruction, but it can be a form of instruction unto itself. That’s one of the takeaways that we’ve season and we try to incorporate our thinking in the work and the issn.
>> go ahead.
>> a lot of the examples I showed really do blend the instruction and the assessment into one system. So rather than thinking about let’s do the teaching and then at the end we’ll test what students learned it’s much more integrated.
>> there is always that tension in terms of what gets signaled. What’s the driver else is with the driver, so I understand the need to focus very much on the purposes for these assess wants, but maybe you can speak to the challenge of how you create this integrated coherent system that is very intentional about what gets signaled to teachers in what matters most in terms of their teaching?
>> I think one consideration for sure is thinking about what kind of stakes you will put on anything. If it’s merely publishing school-level scores in some kind of a report card or using it for teacher evaluation purposes which is a common use now, that just fundamentally changes the nature of how teachers and students interact with the assessment and how they use it, and so a key consideration is if you want this to serve formative purposes and you want this integration it’s probably not a good idea to use it for high stakes purposes.
>> any comments on that? >> I think that’s right. What we see in Asian education systems, for example, is there is never an occasion where the results that the student produced on high-stakes tests or otherwise are the sole contributor for their evaluation as teachers. It is their data point, but it isn’t such that teachers are driven to teach you because it will have consequences of their own well-being, as it were. So there’s a much greater emphasis on trying to get the tests that are tightly coupled to the syllabus and the curriculum is, so you’re teaching and assessing the same things and more and more, we’re using some of the Asian high performance systems is embedding within their examinations more of the kind of open-ended performance-based approach that allows kids to show what they know and not simply repeat facts and use formulas in a rude manner. So I think they’re signaling through their assessment system that 21st century schools are very important and we’ll put them in a curriculum in a very systematic way and we’ll assess those in a way that will allow kids to show what they know through a moral performance.
>> like an international baccalaureate program, the kind of assessments they use their courses on.
MARIANA HAYES >> very good. I will turn now to gen ling chang. She has been patiently waiting and she’s joined us from Ontario, Canada. Welcome, gen ling. Canada has been a top performer in OECD PISA for the last decade. They raised questions about how the use of assessment from Ontario may differ from the united states. So my question for you, gen ling, is how has Ontario succeeding in consistently producing highly skilled young people and what can you tell us about how Ontario thinks about the design and use of assessments?
GEN LING CHANG >> hello, everybody. I want to pick up what Laura and tony earlier said, but to contextualize it for our provincial assessment happens at fourth junk you are from the youngster’s education from kindergarten through grade 12. At grade 10 it’s a literacy test to get a secondary school diploma. So that is a provincial assessment and that provincial assessment is equipment based and there is a provincial assessment handled by the ministry of education called the — the — and they developed that provincial assessment, but with educators at the table as well. So first and foremost the designers of those assessments are not — but they do involve educators. As tony say, those assessments are never, ever used for teacher evaluation because I think there’s always attention between the purpose of assessment and we all agree that assessments are used predominantly, one to improve learning and instruction, but at the same time we use assessments for public accountability and that is more of a politician agenda, the political leader’s agenda than say it could be the educational leaders’ agenda and they’re not notably in conflict, but if there is no provincial assessment or even for ranking schools and that is one very clear message that is not happening in Ontario. We do not use them to evaluate staff. However, they do give us benchmarks of where our students are and ooh a feedback like the Toronto school board mirror back where is the district going by way of meeting provincial benchmarks of developing and standards, expectation of literacy, mathematics and a variety of dimensions. Given that that is what we all know and that’s the assessment which is pretty high stakes because it does influence the outcome of graduation at grade 12. Each year, at least in our district, we always begin the year of bringing those assessments to our teachers and it’s deconstructed so that is a bigger name of what we say is helping what Laura referred to getting teachers to understand the connection between pedagogy and assessment and learning when to assess the two and why the tools. They do not answer all of the educational questions or the needs that a teacher might identify. So when to use a tool and why to use a tool and when to use an assessment and why use an assessment is connected to tool selection and the tool selection must be connected to the relevance of the learning goals that a teacher and student may have constructed, must have alignment to the curriculum and must also be attempted to some degree to the students’ experiences where the school experiences — home experiences or community experiences that were counter to that development.
>> with that in mind, one of the things we do is to take our provincial assessment and that begin to deconstruct political mapping because there are three strengths of each curriculum. They’re a specific content right down to the granular and mapping out what other big ideas and the big concepts that those granular, tiny bits of knowledge construction that’s in the curriculum speaks to and that is an enduring, sustaining learning that we want our kids, too, but the other positive curriculum is mapping all of the action verbs that maps out to thinking and higher thinking skills. When we do that kind of deconstruction and letting teachers get into the assessment tool in the note itself they begin to become very knowledgeable. If the tests are aligned to the curriculum they give a very deep understanding of the curriculum and how to do curriculum mapping that maps on to students’ developmental benchmarks. That knowledge then feeds into the ability to design learning tasks because if elmore, dr. Richard elmore is correct, that learning past predicts performance and achievement, if you are giving worksheets every day to children and then put them into situations to not move then they’re not going to be able to perform. So we believe that the daily learning tasks in the hands of the teachers are to some very large degree will be able to predict a child or a student or a group of students will be able to perform in the variety of that texts has shown and it’s for different purposes. We have on different competencies, expectations of the test taker which is as the student. They have teachers that, and it is no longer just informed by the textbooks and resources out there, but actually having a deep understanding of the curriculum, and I think why Toronto are so keen to join the education city’s network is precisely the kind of knowledge mobilization that we get from the rand paper and the different purposes and we have the professional learning capacity with teachers and our teacher education or — of how it teaches in the school. I’m going to pause here and in case you have questions to ask.
>> I would like to comment on one thing gen ling said and that is how important it is that the classroom experiences of students prepare them for these kinds of tests. There is no disconnect. I, frankly, worry about what’s going to happen when the tests kick in because I think they’re probably going to be much more authentic tests. They’re going allow for more of the problem-solving capacity or demonstration of that capacity than for it, but are we preparing kids in the classroom for those kinds of tests and that’s really as we thought about our own graduation performances, that’s really what we’ve been trying to think about and that’s going to be the kind of capacity that kids are going have to show on these kind of new assessments on the common core. We have to be something now in the classroom to enable the teachers to help the students ready for the tests and to develop the thinking skills and respond to the novel situation that those will be asking for. >> and at the same time while we use assessments for those purposes we have to balance it back by saying the other component is one who teaches the construct and whatever assessment they use we have built capacity around what we call teacher modulation which is in the way that the faculty of the school must have some kind of common understanding or consistent understanding of benchmarks. The whole notion that dr. Douglas reeves talked about that when the grade three teacher doesn’t understand the benchmark of five or six teachers they have a problem in the schools. So we engage teachers in PLC that we take whatever the assessment they have, and bring them to the table and engage them in reading and the student’s performance is the product of assessment. Whether it’s provincial or whether the teacher has selected in the classroom, we engage them in teacher modulation to get the consistency. We also balance with the teachers working with them that part of the assessment, no matter how sophisticated or how nuance a tool can be, you cannot let go of the need to observe students at work and then we work on the interpretive framework that they use to understand, not student works and artifact, but students at work and that forms a critical component in what we have provincial report card. A report card for parents on school by school. So that is another tool that the ministry of education has set. So the first page of a report card is nothing to do with the disciplines of the subject, but actually what we call learning skills and self-regulation. The first page of the report card from grade one to grade 12 is consistent, it’s the same set of learning skills, but increasing complexities they go up the grade. So self-regulation in grade one takes on a very different form in self-regulation in grade two, in grade 12. Our collaboration in grade one take on a very different form in grade 12. In the collaboration, a child needs to demonstrate the appreciation of the diversity and what do we mean? Is it taking? Is it the ability to build on the idea. Is it the ability to see another perspective from a cultural lens? So the teacher’s documentation starting with kindergarten, teaching teachers to document learning and using interpretive frameworks to define those learning is as equal to the other component that we talk about that gets into the report card that they meet the parents three times a year that also shapes the learning and the very skills that tony is referring to that may not be captured with any kind of assessment tools. That are very important in developing the competencies we talk about that are able then to be successful and live a sustained career and livelihood and be able to be citizens of the world.
MARIANA HAYES >> this is a very powerful description of really high-quality learning, but it’s interesting in the sense that the way that gen ling has so eloquently described that you understand what has to happen collectively across schools in term of the task and how that gets defined and what you’re evaluating within that task and how you bring teachers together, I think one of the key findings from oecd findings has have been that those that are high performing are very good at creating these collaborative structures or cultures so that this is the constant work of educators together in trying to design this and as a consequence they’re much more consistent. They don’t have a huge variability across even classroom to classroom in terms of what teachers do or what they expect students to do. Somehow coming to that, that much closer intersection is where you want to go, but how we drive that, how we think about what are the policy considerations is a tough challenge, particularly as we’re transitioning from sort of one way of doing business to another.
>> yes. If you see the report card as a kind of assessment tool really because it teaches the parents and students on the performance three times a year. It is a form of assessment, and if the minister of education revised our report cards a couple of years ago and develop a policy called growing success, it is a policy document. We spent, the ministry spent a huge amount of money on professional development around that and the role of professional judgment of the teacher because that report card became a very important tool in shaping both how teachers see development and be able to be in the position to report to parents as professionals.
>> well, I think one of the things this discussion highlights is there’s been a lot of policy debate about what teachers need to do to get red for their common core standard, and I think what we’re hearing here is that we’re not just what teachers have to do and we really need to be thinking about how do we prepare our school leaders and principles to create the conditions that will support this kind of integration of assessment and instruction and what is the role of district leaders, for example. A hot of the conditions that everybody’s talked about having professional learning communities that allow teachers to get together and opportunities to review students work together and to keep thins vertically across what does communication look like from grade one to grade 12. All of that requires a leader who, you know, gives teachers the opportunities to engage in those kinds of activities so we really need to be thinking a lot about how we’re preparing educators at all levels and not just at the classroom level to do this.
>> we have a question from Rebecca in new jersey about how do parents see the need for 21st century skills and that pivotal question is really — it goes to the — looking at, you really have to come up with some decisions about what’s valued both in the community and among families and within the school even before we get to those purposes for selecting an assessment, it would seem. So any discussion about how you bring parent, families and communities into understanding what are those core competencies and what are 21st century skills and why is that meaningful? Why does my son or daughter need to develop those kinds of competencies and what does that look like?
>> I’ll bet you ling can help us with that, I’m guessing. >> how do you bring the community within Toronto around these kinds of decisions that you’re making about –
GEN LING CHANG >> we see our communities as two very key so-called school board in addition to students which would be the parents and other part is a large proportion of the population which may not be parents, but taxpayers. When we listen to them, they do understand the need for global competencies and it’s not about debating for us, and they do understand the need for learning how to learn. They do understand many of those dimensions that you can tell about the cognitive outcomes of interpersonal and they won that, and the employees and when they interview they keep saying the mismatch between the school will and what they’re seeking for and the industries in sector counsels and they have the competencies that are very much like what we are talking about and what tony has defined as global competencies. Where the disconnect is, understanding the pedagogy, understanding why teachers are doing and for example, you take mathematics and we do have tensions where parents still believe that arithmetic dominate the day and while we are moving toward a different pedagogy, that is where educators need to take a leadership role as well as principals and superintendents and the ministers of education in helping why pedagogy is changing because pedagogy is informed by scholarship, by research and by universities and so on as well as what is expected of youngsters when they have to compete for careers and jobs down the road and when they begin to see the alignment they can understand. What they fear is something’s being taught and the child is not learning the signs as needed or the mathematics as needed and I think that is the conversation that especially in Toronto we have a united nations. In one school we have children coming from 86 countries. We have children that are coming from 86 countries in the world and the parents bring very, very different experiences and expectations of the school system. So that creates tension and that does create to some degree, stress on our teachers trying to understand and help parents and the community understand what they’re doing in classroom, and as opposed to what they would hope the teach tore do that might be like. The gap is they’re thinking about 20 years ago pedagogy. >> so one of our concerns is the issue about equity, and your network has been very successful in addressing those issues and producing very different outcomes for students within your network than in more traditional schools.
>> so, and that is — that’s a focus for the global cities education network to some degree. >> equity? >> equity. >> absolutely.
>> so how are some of these members, the city, the members of the global cities education network thinking about the use of assessment in addressing that issue around equity? That’s where it shows up on our measures and that’s how we tend to think about in comparison to sub group performance and so forth, but is there another way to think about the design of assessments in relationship to improving outcomes for students that have been traditionally underserved and where there are major gaps in their performance?
>> let me just say briefly, and I’ll defer to my colleague, the expert, that actually I’ll refer to PISA, for example, we know the new one will be out December 3rd and there will be new emphasis on the rankings and one of the important things that PISA does is to the extent there is uniformity or not and the performance of different schools within the same jurisdiction and so I think what you’re seeing there is the assessment could be a very good diagnostic tool to see where those gaps are showing up and then it allows you to pose the question, why is that occurring? What are the differences in the instructional strategies or the litany of things that can make a difference in those outcomes that you need bear in mind as you think about creating the systems of support for those schools, for all schools and the particular support and go to schools in the lower end of the gap. So I think assessment in that context and that’s what I hope will have a real close look at PISA in that regard of the united states is where it’s showing and where we’re not making the investment that we need and not so much worry about where we stand in relation to Singapore and Toronto and so forth. What does it tell us about what we need to do to support equity.
>> and I think, you know, one of the goals of going to a more formative approach to assessment where you’re embedding assessment into instruction and using it to make decisions about individual kids on a day to day basis to meet the needs of the kids who are struggling a bit. With the emphasis that we’ve had on no child left behind on this country and looking at metrics like what percentage of kids are above or below that score, that’s information to work with. You know, a typical reaction to that is let’s just take my group of kids who aren’t proficient and take them into this program over here that will help them move up a little faster and instead, we really want to think about assessment that’s embed in the classroom and giving detailed information owe the teachers aren’t lumping kids into categories and able to meet their individual needs and I think some of these technology-based systems in particular that can adapt instruction automatically, freeing the teacher of the need to be teaching 20 different things at once which no human being can actually do have some promise for promoter equity, and I think it’s a different way of thinking about how you do with assessment information that has the potential to reduce the variability and bring up those kids who have been traditionally lagging behind a bit.
>> so I’ll come back to you. In terms of what you have that information the system has to respond effectively and we have had a lot of information in the united states as a result of the standards-based information that’s been available to us in any number of forms, but we had to move the needle very far in some respects in particular areas and I’m just wondering if you could share sort of how Toronto approaches this. You have certain kinds of information and what’s actionable and how does the system start to respond particularly given the diversity and the size of the school district.
>> I think if we take — we take the assessment data and say the provincial assessment data, I think we see it as two roles that has to be in sync. One is using the data and how do we change pedagogy at the classroom and school levels with the principals and teachers with the unit of organization of improvement at the school and then we begin to address in response to pedagogy. How do we bring in for example, texts for children that is more relevant to them, because it resonates with them we can develop the high-thinking skills rather than that is so foreign to them in terms of developing those reasoning skills and thinking skills, et cetera. That is at the classroom school example and so on. >> at the system level, we do engage you what we call the broad improvement plan which for us we just completed and then do a major overhaul and reduce their targets and that target is based on the Toronto school board, and it’s, I think, the only board so far that’s taken the bold and courageous step where we talk of provincial assessment based on racial groups. Based on special needs, based on gender. So we can tell ourselves as a system, which groups they’re undeserving, which group we need to pay better attention. Now that put us under a the lot of public scrutiny and accountability, but we felt there was a moral urgency for us to do that because historically they keep seeing the same sub groups not succeeding and that, in the long term will become a social issue because to us, school success and student achievement is actually about public health and public good. So we begin with this and we do offer an example of first nation’s people. The fastest-growing population in our city, the youngest growing population, but the population in a system where there is a provincial assessment and we begin to see the other root causes and we begin with the assumption that the root causes are not the families. What are the conditions that we can change? So at the system level there are services and programs that has to be designed at the system level will to support the change at the school level. As much as even including like sometimes nutrition programs including transportation and so on, and now summer programs. For example, we’re putting summer programs to the council of education where we are putting in summer programs addressed to literacy and have shown an increase of sustaining of the two and a half months for communities. The parents are so busy earning a livelihood that the children over the July and august month and they backslide if they don’t pride the programs and they begin to introduce some illiteracy with a strong component of recreational art at the same time addressing either remediation, consolidation or even sometimes reaching these and we see those yielding benefits. So this example of desegregating the data, what are root causes and what can the system put in place to make sure all youngsters will succeed because it is good for the nation. It’s not just a way to succeed and that all public schools are schools of choice. You do not need to live in a certain neighborhood to have a school of choice, to have public quality education and unless central office and system undertake those programs, school on their own beyond the point that they cannot change the equity agenda and they can’t change pedagogy, but that’s why we need both to be working in sync so that we begin to narrow the achievement gap. We do see it increasing where we desegregated the data and we show them running parallel. The increase is running in power lines rather than the lines of converging. So then they say how do we make the literacy and girls literacy achievement converge and what program do we need to put in place? Because closing the gap is to make those two lines converge and not run parallel on an uphill incline because that gap then still exists. >> I hope I answered your question.
>> we can go back to your point where you are saying the assessments are needed to illuminate. Illuminate what is really going and who is well served and who isn’t and that is a distinct purpose and I think in some ways when you start to do that you start to confuse the two in terms of the ability to use that information and to find the root causes and that’s where it gets a little murky. Do you want to say a few words about that?
>> so, it’s not something — the different purposes have different features. So the kind of tests that we use for large-scale, high-stakes that we’ve been doing at the U.S. and state level typically, you know, involve a set number of items cover a huge range of cop at the present time and give me back a score and it might not be informative and where is the student struggling and there may be a math test might have five geometry items and it’s not quite enough to get a good handle on how they’re doing in geometry and you want to supplement that and that’s why we talk about the need to talk about an assessment system. With something at the classroom level which is not high stakes for teachers and schools which provides detailed and refined information and given concerns about excessive testing times and there is a temptation to have the state test and let’s use it for all of these different purposes. May as well take full advantage of it and that can be a mistake because that doesn’t always serve those purposes very well.
>> what she says is exactly right. What we need to have a greater emphasis on in the united states is what they might call teacher’s assessment literacy and their capacity to understand assessment and use assessment, particularly more performance-based assessment with the diagnostic role, before we get to a summit of assessment to the curricular unit and there are a series and providing real time feedback and data for database decision making around who is learning and who is not learning and so forth. So that kind of use of assessment as part of a learning cycle is critical and that’s what we need to really have a great emphasis on and as we think about the teacher and professional learning.
>> it’s not always something teachers are getting in their — >> preparation. >> no. And the time element, having enough time and focusing on student work and trying to hold in your practice. >> and not making it an add-on saying you need to do this on top of everything else you’re doing, but instead integrating it into the curriculum that you’re teaching so that this is all as seamless as possible and doesn’t feel like an extra burden that I can’t figure out how to squeeze in.
MARIANA HAYNES >> we have a question from David and your neck of the woods gen ling, Canada. I guess that’s not exactly your neck of the woods. Getting 21st-century skills in the courses can be challenging because of course content issues and again, they’re feeling this tension between, how do you use these more complex measures when most of the focus is on content coverage on the acquisition of information that you’re going to be asked to provide on some measure. So how again do you sort of navigate the fundamental tensions?
GEN LING CHANG >> I think it is a fundamental tension and you have to change. We can’t have it the way we used to or it will be based on facts and formulate our use of formulas and so forth. We have to change the way in which we think about how much we want kids to learn and sort of narrow down the content and really emphasize the use of that content in the service of developing these broader sets of learning skills. So I think that’s the rub right now. So if we have assessments that continue to want to place as Laura said emphasis on a very broad range of knowledge that kids have to somehow on-y demand their assessments it will be difficult to have this problem solving-based assessment and instructional strategies. But in our work, we have tried to make a blend of having teachers to know that the kids need to be able to learn and embed that with an interesting and sort of really attention-grabbing global problem and so there is the learning of the content in the service of thinking how do I use those content and use those skills in solving an issue of world hunger and how to deal with world health issues and so forth and there are create of ways in which you can create skills and content learning together and at some point it has to be to also be authentic and grab kids’ attention and hold it because motivation to learn is a huge factor as to whether or not kids will stay in the game and pay attention. So it’s a complex issue and there are ways to chip away, but ultimately there needs to be a change in the way we need to do business in terms of what we ask kids of what we’re able to do. >> college and career readiness has been helpful. When we stop and think about what is it that students need to be able to do once they go into the workplace and post-secondary education. It becomes very clear that there are problem-solving and high-order skills that they need to be able to demonstrate and maybe we can put more emphasis on memorizing names, dates and so forth, and of course, like history. So I think that this dialogue that’s been going on about really looking ahead to what it is that we’ll prepare students for the future has been helpful in helping us rethink it. >> it’s the expansion of knowledge. How do you learn to learn? >> right. >> I want to add also — >
> this is gen ling. I’ll respond to the question from Vancouver that while we talk about teaching and teachers, one of the biggest network is teacher education. So who is responsible for teacher education and we begin to examine initial teacher education and that teachers are in the career and for the role and responsibility of the school board in our case in the professional learning. So teachers who are listening are not out there alone and they’re also talking as the role of the principal, the role of superintendent and the role of instruction leaders. What is it that they’re doing to support the change that tony is talking about. That is the pedagogy to change the way we look at curriculum and what has been learned has to be changing. Teachers need to be supported. To address the teachers learning is the function of the responsibility of the district school board to address to the superintendent and to the principals and to the instructional support and professional support, and the choices they have and the kind of different forums and they’re able to learn from blended from online and synchronous online and the face-to-face learning and school-based professional learning communities. >> the whole idea is the collective. >> the teacher education. >> taking responsibility for it.
>> the key as gen ling said, you to have a system of professional learning across an entire district. You can’t have it where some schools have the opportunity some teachers do and some teachers don’t and that’s why in a global education network, one of the first working groups we put together is on how do you take to scale high-quality professional learning city wide because that’s — >> and the fundamental challenge and the fundamental challenge that we faced. >> it sounds like fascinating work. >> it is. >> very cutting edge. You’re right on the front lines of everything.
>> so I’m sorry, we’re running out of time, but I want to thank tony, Laura and gen ling and to our audience for joining us today and as a final note as tony said on December 3rd, the alliance in partnership with other organizations will hold these and many other issues and that will be part of the official release of the results for the 2012 program of student assessment. For more information go to pisaday.org and if you missed any information from today’s webinar you can watch the archived video and it will be posted along with the video and PowerPoint slides At www.all4ed.org/webinars. Thank you very much for joining us and have a great day.
Welcome to the Alliance for Excellent Education’s Action Academy, an online learning community of education advocates. We invite you to create an account, expand your knowledge on the most pressing issues in education, and communicate with others who share your interests in education reform.
or register for Action Academy below: