Real-time collaboration and continuous action
create ideal conditions for future success.
boilerplate image

Advancing a Research Agenda for “Hard-to-Measure” 21st Century Competencies


You are cordially invited to a special webinar hosted by the Alliance for the Hewlett Foundation as a follow-up to the February 3, 2014 White House Workshop on “Hard-to-Measure” 21st Century Competencies, which was hosted by the White House Office of Science and Technology Policy.

Participants for the webinar include Marc Chun of the Hewlett Foundation, Roberto J. Rodriguez, White House Domestic Policy Council / Special Assistant to the President for Education; Brian M. Stecher, Associate Director of RAND Education; and Laura S. Hamilton, Senior Behavioral Scientist at the RAND Corporation.

Register and submit questions for the webinar using the registration form below. After registering, you will receive an email confirmation. Please check your email settings to be sure they are set to receive emails from

If you are unable to watch the webinar live, an archived version will be available at usually one or two days after the event airs.

> Hello, I’m marc Chun, education program officer at the William and Flora Hewlett education foundation. I’m your host on this webinar. Welcome and thank you for joining us today. I’m your host for this webinar on advancing a research agenda for hard to measure 21st century competencies. This webinar in hosted by the alliance for excellent education, this conversation is part of the ongoing conversation that started in February, 2014, at a white house workshop sponsored by the white house office of science and technology policies. There researchers, petitioners, measurement scientists and funders explore the current state of play and research agenda for measuring the hard-to-measure competencies. April, the rand corporation researchers facilitated a follow-on conversation at a meeting of researcher in Philadelphia. The rand team went back and interviewed many of the workshop participants. Based on that, developed a research agenda. Our goals are to present key finding from this report and to examine their implications for moving this forward together. I’m pleased to welcome Roberto Rodriguez, who serves on the white house domestic policy council as special assistant to president Obama for education, Brian Stecher, associate director of rand education. And joining us by phone is Laura Hamilton, senior behavioral scientist at the rand corporation. Brian and Laura conducted the interviews, synthesized the findings and prepared the agenda report we’ll be discussing. Please join the conversation by submitting your questions using the box below the video window. I thought I’d start by providing some context for our discussion. The 2012 report, education for life and work, set the stage for this work. The NRC panel asked what competencies are necessary for life and work, and made the case for three domains which I’ll describe in a minute. Those enable you to take what you learned and apply them or transfer them to solving new problems. The NRC recommended foundations and federal agencies should support research programs designed to fill gaps in the evidence base on teaching and assessment of these competencies and the process of transfer. In particular, they advocated for additional research around assessments in a noncognitive domain. Two weeks ago, the American institutes for research released a report based on a quasi-experimental research study that provided new evidence that adds to the established body of work that will focus on the key 21st century competencies is associated with higher test scores, increased on-time graduation rates, and admission to more selective colleges. An air research provided an exemplar about what the competencies tell us about learning. It’s noteworthy, they used indirect measures such as surveys or proxies in the key competencies. Then lastly, jobs with the future released the first in a series of research papers about these 21st century competencies. In the first paper, David Connelly, founder of the education policy improvement center, an expert on college readiness, argued that the time is right for major shift in educational assessment. Noting that in addition to standardized tests of mathematics and reading, we need to use multiple measures that can help gauge progress in learning around a broader range of crucial content and skills. What are these competencies? I’d like to turn to the NRC’s three domains 21st century competencies I mentioned a moment ago. The first domain is a cognitive competency which include among others content and being able to engage in critical thinking and problem solving. Then there are two domains of noncognitive competencies that the NRC recommended the field invest in in order to build new assessments. These are the domain of interpersonal competencies about how people react with one another which includes competencies such as communication in both written and oral form, and collaboration. Then the domain of intrapersonal competencies or managing behavior and emotions which show learning to learn and academic mindsets, for example. Of these competencies, there’s a subset that already have existing scalable measures of assessment. But there remains the hard-to-measure competencies that include oral communication, collaboration, learning to learn, and maintaining academic mindsets. And so rand took on the task of considering how we might move forward together with the research agenda for these hard-to-measure competencies. To summarize, the NRC report defined, organized, and made the case for these 21st century competencies. Air answered the call for evidence with a research study empirically demonstrating the quality and pushed the standards. The paper that jobs for the future puts out the call for new measure. The rand researchers chart a collective path forward. Before we move forward to discuss of the rand report, I’d like to have Roberto help provide a policy context for this important discussion. Roberto, to help us get started, how does this effort to dispel assessments for hard-to-measure competencies fit into the broader education reform effort?

Thanks, marc. You know, today’s conversation underscores the imperative for change in our public education system which is probably greater than it’s ever been before. You know, for years, education in our country has been rooted in memorization. Our students were expected to memorize facts and figures, the quadratic formula, and periodic table. And then that type of smattering of memorization of considered education. We need to dramatically evolve our characteristic him, our teaching and learning in our schools, to maybe better match what are the demands for knowledge and skills in today’s economy. Today we live in an innovation economy. That values important problem solving skills, important communication, critical thinking skills, and we know that we need — in order to equip our students to be successful and economy, we need to conceptualize and redesign teach and learning in our schools — teaching and learning in our schools. They a key, important takeaway from the conversation. Our schools shouldn’t be thought of as the place where students are passively receiving knowledge, passively receiving skills. Rather, as a place where knowledge and skills are ultimately student centered. The endeavor of learning is individualized and personalized for students, and that it’s applied to real-world experiences for our students. So in order to realize that, we really have three major challenges that I think stem from today’s conversation. The first is the lack of policy architecture and scale to really bring to bear better assessments of these hard-to-measure skills. Measures exist, but the challenges, how to think about those assessments, add scale, and as important complements to and contributors to the larger scale assessment practice across the states as it relates to standardized assessment and accountability. Second, we still lack a pedagogical roadmap that fosters these competencies and puts students in the driver’s seat in terms of knowledge acquisition and helping to ensure that they have a successful path in college and career. And third, there’s still a lack of knowledge, and ultimately a need to continue to build a research base on how to support teachers and foster these hard-to-measure skills. Both in and out of the classroom. We need to do more to support and provide the tools for our educators that ultimately will be able to deliberately promote these type of skills in their children’s learning.

Great, thanks a lot, Roberto.

Thank you.

I’d like to turn to Brian Stecher as well as Laura Hamilton, on the phone. And Brian, what was your process, and what did you and Laura find in your research?

Thank you very much, marc. As Roberto said, assessments are a piece, but you they’re not the complete answer. We’re going to talk about the synthesis that Laura and I did of input from about 75 researchers, assessment developers, policymakers, practitioners, and funders, to try to answer the question of what can we do next to promote the research and development of assessments of these hard-to-measure interpersonal and intrapersonal skills. I’ll divide this talk into four chunks. The first chunk is trying to answer the question what steps do you need to complete in order to develop a new measure regard will of which skill we’re talking — regardless of which skill we’re talking about. Secondly, we’ll talk about the strategic decision-making process and what we would need to do to actually initiate a new area — era of research and development around intrapersonal and interpersonal skills. Third, Laura and I are going to recommend a strategy, an organizational strategy from managing this process. And lastly, thinking about this strategy, we’ve identified a few constraints and challenges that will have to be addressed over time to ensure that the process stays on trap. Let’s begin by talking about what the steps are that go into measure development and preparation. The first thing is to get a clear definition of the con distracts that you’re interested — constructs that you’re interested in measuring. Most people think about achievement testing and reading and math, and they’re fairly familiar with curriculum designs and with standards that delineate exactly what they expect students to be able to do and what kind of tasks to perform. We don’t have as clear a set of definitions of the constructs that we’re interested in in the interpersonal and intrapersonal domain. Leadership, for example, is one such construct. But it’s complex. It involves a number of characteristics. You need a certain vision of the future. You need to be able to communicate and collaborate with others. You need to be goal oriented. And before you can design a measure, you need to decide what’s in this domain and what’s outside of the domain. Effort needs to go into defining the constructs of interest. And then measured development is driven a lot like purpose. So the second task is to get clear on what your purpose is. Are these going to be assessments used in the aggregate to sort of monitor how well a cohort of students progress through the system, or are they designed for teachers to use to inform day-to-day instructions, or might someone want to incorporate them into an accountability context? Those choices will drive how you develop and design the specific measure. Thirdly, you need to go through the process of developing measures. In the domains we’re talking about, this is not straightforward. A lot of what’s going on isn’t elicited clearly — elicited clearly in behaviors. You can’t tell that much about someone’s leadership ability or their academic mindsets by having them do a particular task in a small, concrete amount of time. Some cleverness and thoughtful not has to go into development of measures. Then you need — thoughtfulness has to go into the development of measures. Then you need to test them and make sure there’s adequate quality. That usually uses reliability in terms of scores and the scoring process, validity, and fairness to all students. And then finally you need to take a longer term view and look at what happens when measures were implemented in school. And think about the consequences in terms of the delivery of instruction, performance of students, and the interpretation of scores. These are the sort of broad things that need to happen to develop a measure. They happen roughly in that order. There’s a lot of overlap in the stages. It’s not a perfectly linear process. With that vision for what needs to happen to get the measures moving ahead, we turn to the next set of questions which are what do we do next? What’s the process of actually making the choices and moving forward? Here we’ve identified three or four major questions that have to be answered to build a strategy for creating new measures. The first one, sort of a simple notion, is where do we start. Which of the many competencies identified are the ones we ought to begin with next. And the advice and the suggestions that we got from the field, many of them congealed around this idea of educational efficacy. One of the first filters ought to be whether the competencies are thinks that we know can be developed within the school context and are of interest and importance to educators. If it’s something that kids bring with them to school, we shouldn’t put a lot of effort into measuring it unless we think that there are ways that schools can enhance the development and mastery of these comments. Once we’ve aimed the efficacy filter, we look at the adequacies and if they’re efficient or not. You look at the complexity of developing a new measure. If there are adequate measures already out there, we don’t need to invest in new ones. If we have two choices, one of which is similar to things that have already been done and might be accomplished quickly and the other of which is very different and might require a longer period of time, we can make choices about where to start. In our conversations were there those who advocated for going after the low hanging fruit. But make some successes early. And there were those who advocated for investing in the tougher nuts that have yet not been cracked by researchers working on their own. So tradeoffs here that have to be navigated. The next thing that has to be considered is what research and development activity are we going to invest in. There are really four kinds of thing that need to occur. The first is thinking about — do you do basic research to understand how a particular competency develops in an individual. What does it look like when it’s a nascent skill, what does it look like over time as you build and stretch your capacity to perform it. That’s the key to developing the measure that will be sensitive to change over time. Now there’s some basic research. You need path development to actually sit down and figure out what the situation will be in which you present to a student, how you’ll assess their response and figure out a way to score or rate it. The basic business of developing an assessment. In many cases, you will need to do some fine-tuning of measures, assess how well they perform in a pilot phase, or if they already exist, where they’re adequate for adaptation to the school context. Then finally, there are long-term questions that deal with consequences of using the practice. We northbound the past that sometimes high stakes measures can drive behaviors that are not optimum. And so it’s important to look not only at the immediate measures of reliability and validity but also to look long term at consequences. And then the last couple of sort of strategic questions to ask are how long is this going to take, how much is this going to cost. And here we have fewer models to rely on. Our answers are much less precise. I think if you look at what has happened in the development with psychological tests, educational tests, screening tests for employment, we can say that the development task is not going to be accomplished in weeks or months but it’s going to require years. It’s not going to cost thousands of dollars or hundreds of thousands of dollars, but it’s going to cost millions of dollars or tens of millions of dollars. Those kind of are a ballpark fence around what this process will take. There’s a lot of uncertainty in what I’ve said so far because these are hard-to-measure skills, after all. We haven’t had a lot of success doing it. Nevertheless, having laid out what the steps are and what the particular challenges are, Laura and I came up with what we think of as a fairly likely to be successful strategy for pursuing these goals. That’s a way to manage all of the moving parts that have to come together to make that happen. On our suggestion is a creation of a couple independent research coordinating boards that would actually not do the work. They wouldn’t design the measures or test them or judge their quality, but they would create the research and development agenda that would help to guide them. Then they would monitor the process as it evolved, looking at — excuse me, at researchers who might take on the tasks, monitoring evidence that comes in from field testing and piloting. Making decisions about how to allocate resources and which things to put first and which things to defer. Making changes in direction and ultimately helping to share and disseminate the results of this long-term effort. Now the boards can be construed in lots of different ways. We can mention maybe making one responsible for the interpersonal domain, and the other responsible for the intrapersonal domain. But before formulating them, someone would have to establish guidelines, set some management policies in place, clarified that this is a multi-year process, and they have a multi-year mandate. I think of this as a kind of idealized model in practice. There may be alternatives that we can talk about to move us forward with the overall goal of creating measures if we are unable to fully constitute a fully funded board. But there are some things about the board structure that are essential. And I think the most important one is bringing together multiple voices in a collaborative way to comment on this. That researchers, practitioners, clinicians, policymakers need to contribute to the decision business how we move ahead. Even if we did constitute the boards the way we outline them in our paper, there will still be challenges to carrying out this agenda. We’ve identified three or four of these challenges that center around the idea of staying the course and making sure there’s enough is and long enough vision to bring this about because it’s not going to happen overnight. Keeping a collaborative culture is key. And we can talk about how then how we might try to constitute this and bring together the people who are interested in this topic to make it happen.

Great. That’s briefly what’s in our report which will be available by month’s end.

Okay, that’s great. Thanks, Brian, for sharing that. We want to pivot now and have this become more of a conversation as we discuss this. We’ll bring you questions from the audience. Roberto, do you want to get us started?

Sure. First of all, thank you, Brian, for the wonderful overview. I think your comments in particular highlight the facets of the challenge from a measurement perspective in particular in terms of how we advance these skills. We also have to address the pedagogical challenges and structural challenges. This requires a real rethinking of how we prepare, development, and advance our teachers. How we organize our school structures to promote collaboration. And to provide I think one of the other important points you touched on, to provide real-world traffics and opportunities for real-world — tasks and opportunities for real-world applications for knowledge and skills that reach beyond our textbooks and into projects and project-based learning, as well as opportunities to learn outside of the classrooms. So the fortunate thing is I think there are wonderful schools that are models of deeper learning where we’ve really seen great examples of restructuring, what the teaching and learning experience can look like for students. And really redesign that around students’ interests and needs. We’ve seen some really encouraging results, particularly around greater motivation, greater engagement. So one of the things we have from a policy perspective it to think about how we can bring those types of opportunities to scale for more of our students and in more of our districts. And this directly relates to one of the challenges that the president’s put forth for us which is to really redesign American high schools so that they’re more connected to real world experiences, and that they engage young people in planning about their future. About the post-secondary opportunities for higher education for a career earlier in their educational experience.

I would comment that they need to go hand in hand. The pedagogy and measurement need to move forward simultaneously. One of the things that having the measures does is signal to teachers the importance of the skills which otherwise might get lost in the current accountability context. So I agree with you, and I think both need to be integrated, both pedagogy and new measures.

The advent of college and career-ready standards which we’re now seeing in 48 states as well as the District of Columbia teaching this fall to college and career-ready standards. Different states and different districts, obviously different points along that process. But all really working to raise expectations and raise the bar on relevance for students tamp provides a great opportunity — students. That provides a great opportunity for how we deepen and enhance measures, large scale measures as well as our school-based and performance-based measures, as well as think about new directions for pedagogy.

I’m also thinking of project-based learning and project assessments. They can think of teaching and learning at the same time. Great projects since they’re learning, you also can assess progress in terms of the learning trajectory which I think is important. There’s a couple questions that came in where folks were asking how this work can move forward given that there is a general fatigue with testing. There’s concerns about how the general backlash from the public around having a new regime of testing. I wonder if you can reflect on this given what you’ve shared, Brian.

I think a lot of the fatigue relates will to formalized external testing. And much of what people talk about when we have these discussions around these interpersonal and intrapersonal skills is testing much more tied in with classroom-based instruction and learning. And I don’t hear anyone complain that there are too many end-of-chapter tests or that there’s too — some people complain there’s too much homework. But the general notion that teachers need better measures of what — how well their students are performing is one that I think still has support. So despite the caution about more structured summative assessment, I think there’s still clearly space for formative — and within classroom assessments that look at this whole set of competencies.

I agree completely with Brian. I think we have to agree that this assessment is ultimately taking up a lot of the oxygen in the room when it comes to our conversation around education reform. And we have to acknowledge that there’s an important role for some of assessments to play from the schools accountability as well as making sure students are on track around the mastery of college and career readiness. Assessment is also an important educational practice in the classrooms. That’s where you can really seize the opportunity to measure a deeper set of knowledge and skills from a formative perspective, provide real-time information to teacher and other instructional leaders about how to really meet students where they are and help them really grow, reach their full potential, develop that academic growth mindset that we’re really seeking in terms of many of these higher order skills. So there’s a real opportunity also to think about how those assessment practices, as I said before, can contribute to and complement a large-scale assessment program. Again, I think as we begin to see some of the new assessments come on line in our schools, particularly through the smarter balanced consortia, we’re hopeful about thinking about how these practices can contribute to those.

What’s the connection between that and the call for equity? How can those come together?

I think when we ask that question, we have to really look at what the expectations are for our students in a global economy. There’s no doubt, I think when we look at an innovation economy, knowledge economy, the level of debts and problem solving and application of knowledge in collaboration, being — in being able to work together with peers to solve complex problems and to use higher order thinking on a day-to-day basis in the workplace, it demands this level of knowledge and skills. And ultimately this level of teaching and preparation. So there are, again, a number of schools that are providing that for our students. The question is do we have the will really to make sure that we are seizing this opportunity from an equity perspective for all of our students. And particularly for our students in low-income schools so they’re not being delivered a curriculum solely about memorization.

I want to make sure — we have Laura on the phone, as well. Make sure to jump in any time you’d like to.

Yes, I actually wanted to ask one thing to that which is currently a lot of our kids are — we’re very focused on equity in terms of the academic opportunities to provide kids in schools. But a lot of the kids who are developing these inter and intrapersonal competencies are doing it in extracurricular activities, things that may not be available to all students. This another is reason why it’s important for us to measure these outcomes and to emphasize them so that we recognize that there are students who aren’t currently getting these in the traditional school setting and can think about a way to make sure that those are incorporated in so that they’re getting those same opportunities that the wealthier kids might be getting out of school.

I would add that most teachers wouldn’t be able to define an academic mindset or meta cognitive assessments so their judgments are not going to be equal. One of the advantages of having assessment that’s are standardized, that are available to lots of people is that you can make comparisons. You set equal expectations for kids in different contexts.

What do we know about existing assessments to get at those competencies?

Well, there’s a real variety out there. In fact, I think one of the ways we can take — take step one toward moving this agenda forward would be to build a better repository that people could access to look at what exists. In some areas, there’s quite sophisticated measurements, some researchers have spent years developing assessments in, say, growth mindset or grit or creativity with good statistical properties and evidence behind them. Others, much less so. Much less. One of the things we’ll be doing at least on a trial basis over the next few months is trying to pull together what exists and what evidence there is about the quality of these measures at least in a first sweep of what’s out there so people can come on line, take a look, and do a better job of identifying gaps.

Okay. I think just to add to that, marc, one key challenge that we need to take on is how we can invent more of these deeper measures of learning within a standardized regimen of large scale assessment. I think ultimately we have good practices around assessing these skills. Certainly more needs to be done to develop a second generation of these skills. We have a lot that I think we can work with now. And the question becomes how do we integrate those more actively into some of the assessment regimen. And I think, you know, we’ve seen some other countries that have started to look at that, started to look at how they enhance their assessment items, instruments, and move — ultimately we need to move away from some of the lower level, fill in the bubble assessment practice that has too often been the norm. You know, in our country, particularly as we began the standardized assessment — standardized assessment, we need to advance that movement through the next generation of assessments. And this is why we’ve really invested in the administration and the development through the race to the top assessment challenge. And developing new assessments that are aligned to career-ready knowledge, but that utilize a deep measure of student learning.

Okay. And what do you think is the role of the federal government or other key stakeholders in setting the conditions in which this can move forward?

I think we have a role to play. At the federal level we can invest or enhance assembly instruments. Again — assessment instruments. Again, I feel we’ve played a constructive role in the smarter balance and like the PARCC– I think there’s more federal — consortia. I think there’s a more federal role and a role for teachers to play. Ultimately when you have an instructional team working together developing a set of performance-based assessments that can complement a state assessment program or large-scale assessment program. I think there’s a lot we can learn from that about how to really get a more sophisticated picture about the progress that our students are doing.

I worry that if we let it be ad hoc that we won’t accomplish the goal because we’re sort of set for us, when Hewlett challenged us to come together. I don’t know whether we’ll be able to get support to create the idealized boards that Laura and I talk about, but I do hope we can create some kind of a structure that would exist to continue to review and guide in effort going forward.

That would be valuable. To review and guide the development and review of practice and guide the conversation over how we drive pedagogical change around the skills.

A specific question came in perhaps title 6 fund or waivers could be used — founds or waivers could be used in supporting this work.

I think that’s a good question. You know, title 6 dollars under ESEA are dollars that support assessments. Right now those are largely supporting and sustaining the state assessment practices. We would love to see more of the embedding of these type of assessments in the existing state policy. That’s one way that I think as states recognize opportunities to embed these measures and help support districts in these measures alongside their state of standardized assessment, I think that’s one way that title 6 can support this work. We believe a lot of work can be done even without necessarily waiving federal requirements. I would argue that it’s not necessarily the federal requirements that are in the way of promulgating and promoting these types of practices. It’s really our own leadership at the federal level and also at the state and local level that needs to drive this forward. Idea of thoughts on what other stakeholders can be doing? I’ll also ask Laura to comment in a moment. I’m familiar with California which recently changed school funding and accountability formula and the new local accountability approach delineates a much broader set of competencies that students should demonstrate and defers to local school districts, the challenge of measuring them and reporting on them. There’s a great need there for the kinds of measure we were talking about, and perhaps there’s an opportunity there for some kind of collaborative development.

Question just came in, and it was about how the data could be used. Rather than considering them as separate data points which I think this competency and this competency, are there ways this which they should roll up together? How do we triangulate and make the set more meaningful? I wonder if you can reflect on that. Brian? Why don’t you kick us off, and I’ll add to —

The purpose of the assessments to reveal something about the development of competencies and individual students to make the best that they can be. And there are individual differences. So I don’t think we want a cookie cutter approach that’s everyone should reach the same profile on these 12 assessments. They can be more engaged with one kind of activity with one another. I do think we want to approach — an approach that says it’s worth noting where every student is on the slate of competencies that we value and helping them decide to pursue things that help them master as much as they want to. Laura?

You know, this is an area where we are lacking in research, and it’s one of the reasons that we, you know, are arguing for this sort of big long-term coordinated effort. Is getting a better understanding of what students’ profiles look like across these different measures. To what extent do we see there are certain competencies that tend to be correlated with one another, and to what extent are their competencies where you see dramatically different levels of performance in the same student. I think understanding that is important, but then taking that next step of thinking about, well, how might we combine these into a — you know, a single score. What kind of information is this going to be most useful for giving back to teachers, to students, to parents, and so forth. This is an area where a large-scale data collection effort is really needed.

I would just also say there’s a real potential to utilize these measures of deeper knowledge and higher order thinking in a way that is discipline specific and that is subject specific. We should challenge ourselves to know how we can measure important skills and knowledge in science, in mathematics, in reading, in — in art and other areas using these types of measures. And you know, for instance, something like project-based learning does just that where you are engaging young people in a collaborative task and application of a specific college and career-ready skills and knowledge that are part of the standards and part of the expectations for the grade level that are, you know, grade and subject specific. I think there’s a way to braid those two and do this in a way that really helps us make sure that students are not mastering a core set of knowledge.

Yeah. Excellent point, Roberto. One of the thing we don’t know is how general some of these competencies are.


Persistence in one domain doesn’t necessarily transfer to persistence in another. And why that’s the case and how much that’s the case are open questions that need further study, I think.

I think overall the key point, in previous conversations, folk have tried to separate skills from the content. It make more sense to bring them together. The only way that makes sense.


I think you have to braid these conversations together, and part of that is to have an imperative for college and career readiness along with continuous standards. There’s only so much time in a teacher’s day and a student’s time, learning time that can be focused on that. So I think with time being of the, sense, I think it’s imperative to try to measure these things together.

Uh-huh. Another question. Someone asked about the NRC’s formations, the three domains of competencies. I know you use that as a way to anchor or organize thinking around these assessments. You can reflect on that? Does that resonate with you? Are there other ways in which we might parcel the landscape of these competencies?

Well, I’m particularly not a fan of referring to these as noncognitive skills or competencies because they definitely are cognitive. And we adopted the NRC framework rather than creating a new one because it really is very thoughtful, and does a fairly good job of defining clusters in ways that seem meaningful. In the two large — there’s the achievement domain, cognitive side. And then be there are two large clusters. Within those clusters, the report that Jim Pellegrino did identify subclusters. So they’ve started to break this out in a way that might make them more attractable from the assessment. So I’m generally satisfied. I don’t think we need to engage in a lot of definitional wars about what to cause thing. Although at the point that you start to develop the measure, you have to have clarity about what you’re focusing on. So you need some clarity, but it’s not — it doesn’t seem important to me to argue a whole lot about the specific labels that get attached.

When you goat assessment, defining what students are able to demonstrate, you don’t have to have bottles about what to call it. We can see it and see what it looks like, we can find greater agreement.

I have a question, when you think about the assessments, you ticked off the different ones, Brian, which mitt seem like low hanging fruit? Which could we do more immediately because we’re close, you described how further along in the continuum.

I think that the phrase low hanging fruit was used to talk mostly about the interpersonal competencies because you can see them, evidence directly in behavior. If you want to know whether someone can communicate, you can put them in a setting where they have to communicate with another person or with a group of people. If you want to know if they can collaborate, you can give them a task to perform and observe them performing directly. So behaviors that are evidence through naturally occurable interactions are probably lower hanging than the ones that are psychological constructs that only emerge from some long-term observation of decision-making over time in different settings. So those were probably the lower hanging ones.


And in building assessments, how can we prevent the misuse of data? What if parents are concerned that we’re going to assess these and use these in inappropriate ways in terms of judging or evaluating our students?

Well, I mean, I’m always concerned about this question, and the greatest misuse I think has come from taking measurement and incorporating it into a somewhat different consequence where there are stakes attached to particular scores. One way to caution against and prevent misuse is to reduce the sent to which these things become pawns in an accountability game.

I’d also add to that and not only focus on that, but also focus on making sure that these don’t become gatekeepers to opportunity or particular gatekeepers to student-centered decisions about the courses they take and learning opportunities that they might not be afforded. I think there are important assessment practices that need to be adhered to not only in this vein but in the broader vein of both, you know, smaller scale as well as large-scale assessment. Fortunately we can rely on the national academy and other resources around appropriate use of assessments. But certainly that shouldn’t be a reason not to move forward in development of these measures. I think there’s a path forward that has appropriate use of these assessments to really guide teaching and learning for students.

One thing I’d add is a powerful assessment practice that has been used mostly in the achievement testing realm is involving students in the — in data use. Don’t think of it as something teachers or principals do. It’s an activity that students are engaged in. They have buy-ins. They help set their own goals and really understand what these — what these measures mean and can give thought to how to improve their performance on the dimensions. And that I think is away to alleviate environmental concern and other concerns that are wrought, and it’s been shown in the cumulative arena to be an effective practice in terms of promoting better use of data and improved outcomes on this measure.

There is tremendous power in providing student clarity about the expectations that they have to reach, as well as into the data behind that. You know, providing a more and greater opportunity on a periodic basis for students to know how they’re doing. How they’re progressing toward a standard, and how they’re doing at mastering that standard I think is really, you know, it’s critical to their own persistence and engagement and ultimately ownership for their learning. I think that there’s a — an important link that you just made to the conversation that I think has real potential for students.

And there’s a lot of research that suggests that engaging students in thinking, very simple step can dramatically change their behaviors. The work on growth mindsets suggests this it doesn’t take a lot to help kids understand the importance of their perspective on their own learning. Simple thing can transfer -. And a conversation around measurement and how you personally are performing can be something that has that contributing factor.

Great point. How can students, teachers, parents, what should they do to be a part of this larger conversation while sitting this agenda?

I think ultimately that’s a question we need to encourage our parents and families to become more involved and more aware of the educational opportunities and experiences that their students are afforded. Become more aware of the expectations of what their children will need to know and be able to do, to progress toward a continuum of college and career ready not, ultimately toward high school graduation. And to ask those important questions about how much — that get to the depth of the curricular experience for students. What type of opportunities are — is my child being afforded to apply what they’re learning in real-world settings. Are there opportunities to learn outside of the classroom? Are there opportunities to learn through projects, tasks, beyond what I might see in my child’s textbook when he or she bring it home each day? How are we measuring and to engage in conversation with our principals and school leaders about that. I think those are important questions for parents to ask.

I think parent would find the conversation more natural than the one about academic performance and math achievements. I mean, when I went to parents night, the things we talked about were just these sorts of personal competencies. Your kid is reliable, your child does homework on time, they’re respectful in class. Those are the things that I think parents have actually a real good sense of and can contribute to. This is something — this is not something far away that need to be brought down from the mountain and simplified for parents. I think these are fairly natural conversations that they’ve engaged in having.

You mentioned the real world applications and talked about career and world readiness. A question that came in is, are there ways this which we can assess for that, to know are the things that will help in career, really help in college, help prepare for those unexpected challenges they might face in the future?

Ultimately, I think the important set of knowledge and skills around our expectations for our students needs to be mapped to success in the post secondary system. Certainly our administration’s approach to this has been to say that we want to make sure that all of our students upon graduation from high school can be well equipped to enter higher education without the need for remediation. I think that’s one important benchmark. I think another important benchmark is what’s the set of knowledge and skills that our employers are demanding of their work force and how can we give our young people a head start and jump-start at developing that set of essential knowledge and essential skills that will help them be successful in the job market. So as you benchmark against, you know, what, for instance, human resource officers are looking for in some of our more successful companies, that’s where we begin to see that kind of set of essential skills that relate to real-world application of knowledge and problem solving come into play.

I don’t want to forget the college and careers and citizenship, the role of individuals in our egalitarian society.


A lot of these things contribute, I believe. The evidence we still need to gather, contribute to positive roles as adults, as citizens in a democracy.

I’m glad you mentioned that. The civic mission of our public education system is an important pillar here. And I think an important part of this conversation.

I wonder if you can describe what a public/private partnership could look like as we try to move this agenda. Are there examples? It could be from here or other countries, to pull off something like this.

Well, I would say I think one of the exciting partnerships that we’ve witness sudden to think about how we can bring some of our community partners, employers, and business partners more actively into the learning experience for students. Whether it’s adjunct teachers or coaches to teachers that are in particular subjects, whether that’s through internships or mentorship opportunities for our young people or other structured learning opportunities. And that can take place in a variety of high-tech settings. Can take place in — at our museums, in a number of settings throughout the community. So one of the things that our investment focused on through the youth career program is how we can strengthen some of the partnership between that are really relevant to particular sectors in our economy, through or business partners and our  high schools, and we’ve put forth a set of grants in communities across the down think about redesigning the curriculum and scope and sequence for students to provide opportunities to really gain real-world knowledge and skills alongside a very rigorous scope of college and career-ready academic work.

And how would you connect that to the assessment piece of the equation then?

Well, you know, I think it’s a matter of being able to define and develop tasks where we’re measuring how students are doing in real-world settings, using and applying the knowledge that’s part of their particular standard career and college career-ready standards. That’s often done in an informal setting but in a formal manner. There with ways of approaching it in a more standardized format. And those types of experiences and portfolios contributing to the broader overall measure. Of how students are progressing in their particular grade.

And we also turned up a couple of example of partnerships to foster research in new areas. For example, the national science foundation and the Gates foundation are engaged in partnership for agriculture development, and it could exist to shepherd this work going forward and maintain some logical ordering to thing, make decisions about how to allocate some resources. And there are other partnership that people mention to us that might be models for these coordinating boards that we hypothesize in the paper.

We’re starting to run out of time. I did want to ask you to think about — imagine ten years in the future, what might this look like? What would be your vision if we were able to successfully gather together more toward these assessments?

Well, I actually think that the role of assessment in education will be somewhat different ten years in the future. It will be much more integrated into the activities and behaviors that students engage in under the direction of their instructors. We’ll have a lower incidence of external tests. They’ll be used more for monitoring and for making broader systemwide decisions, and the day-to-day level of assessment will be keyed to developmental milestones in the content areas, and to what we understand about how students learn and develop these competencies. It will be more integrated and less identifiable as a stand-alone activity.

And I would say, too, it would be more personalized. So we’re seeing a lot of move toward personalized learning, a lot relying on technology, but trying to offer material to kids that meet them where they are and accelerates their learning so they eventually become college and career ready. And I think that these measures of other competencies can play a really important role there. We see it becoming more personalized, more individualized, especially in big group opportunity and something more tailored to the student’s needs and designed in a way that gives that student very specific detailed feedback to help them improve.

I think we know much more today than we did a decade ago about effective assembly practice or two decades ago. I’m confident that there will be a great deal of learning or research underway so that decades from now we can enter into another generation of what assessment looks like. I do believe it will be more tailored to individual students’ learning experiences. It will be more personalized and more focused and relevant to the classroom and out of classroom instruction for students. And I think our challenge is to make sure we are marrying that with the proper accountability systems that are needed so we can continue to have the important summative hook at how our students are progressing, particularly for a guarantee around equity. I think it will be a very exciting practice. I hope it will be a practice that our students are even more engaged and have greater ownership of than they do today.

All right. Great. Thank you. Well, there’s so many more questions. I’m sure the panel would like to discuss. Unfortunately, we’re just about out of time. I think the rand report gives us a starting point for this conversation, a chance to also engage you, as well, in how we move this work forward. We’re going to man to continue these conversations over the coming months. I’d like to thank Roberto Rodriguez, Brian Stecher, and Laura Hamilton for joining me and thank you, our viewers, for joining us, as well. If you missed any of today’s webinar, they’ll be archived at Thank you again for joining us, and have a great day.

Categories: Uncategorized

Action Academy

Welcome to the Alliance for Excellent Education’s Action Academy, an online learning community of education advocates. We invite you to create an account, expand your knowledge on the most pressing issues in education, and communicate with others who share your interests in education reform.

Register Now

or register for Action Academy below:

Join the Conversation

Your email is never published nor shared.

What is this?
Multiply 3 by 1 =
The simple math problem you are being asked to solve is necessary to help block spam submissions.



Every Child a Graduate. Every Child Prepared for Life.