Counting and Recounting: Assessment and the Quest for Accountability

Lee S. Shulman

When my daughter Dina returned from the first class in managerial accounting early in her MBA program, I innocently asked how it had gone. I fully expected her to describe her boredom with the rigors of accounting, since pursuing an MBA was decidedly an afterthought for my iconoclastic daughter, who already held degrees in theatre and social work.

Imagine my surprise when Dina responded that accounting was unexpectedly interesting because, she now realized, it should be understood as a form of narrative, a kind of drama. Within the ethical and technical rules of the field, the task of the accountant is to figure out which of the stories of the company should be told through the medium of its “books.” Accounting is basically about creating the plot, characters, and setting of the story. As the instructor explained to the class, “Your task is to render an account: to tell the facts of the case, the story of the condition of a company in an accurate and yet ultimately persuasive way.”

In the world of business, an account is a story told in quantitative form. It publicly documents all the income and investments that enter the company and all the products and liabilities that emerge from it, all its assets and debits, all its profits and losses. When the books balance, the account is closed: the story has been told.

When I draw our attention, as Dina did mine, to the ghosts of narrative and storytelling that stand behind the counting, measuring, and computations that lie at the heart of modern assessment in the service of educational accountability, I do not aim to undermine the credibility of assessment. I am not referring to “mere storytelling” as if narrative is a lesser form of discourse. The connections between counting and recounting are built into the etymology of these words in many languages. Thus, in German, to count is zählen and to tell (a story) is erzählen. In Hebrew, a language with utterly different roots than English or German, the verb for counting is lispor, while the word for telling is lesaper.

I believe the lesson is clear. How and what we choose to count and the manner in which we array and display our accounts is a form of narrative—legitimately, necessarily, and inevitably.

Tools for Counting and Recounting

When my teacher Benjamin Bloom led a group of university examiners in the development of the Taxonomy of Educational Objectives in the late 1940s and early 1950s, their goal was to provide a structure within which evaluators and teachers could determine which story they wished to tell about the learning of their institution’s students. They had determined that most of the instruments then in use to assess students—and thus to render them, their teachers, and their colleges accountable—were exclusively stories of the acquisition and retention of knowledge, of the students’ success in recalling facts, events, principles, and concepts they had learned in class or read in their textbooks. Bloom and his colleagues argued that this was an impoverished story, one that missed the most important aspects of the account the examiners needed to give of students’ learning.

By elaborating the cognitive outcomes of education into a taxonomy comprised of six categories—ranging from knowledge and comprehension through application, analysis, synthesis, and evaluation—Bloom and his colleagues developed a much richer array of plots and themes for the story of academic performance. A program that appeared to be achieving great success when knowledge alone was measured might look much less impressive if the “higher-order” processes were accounted for. Bloom and his associates also were committed to extending the story from the cognitive to the affective domains in order to include the development of emotions, motivations, passions, and identity.

The story told by an assessment is thus ultimately a function of the dimensions of measurement that determine the possible directions the narrative might take. So accountability requires that we take responsibility for the story we commit ourselves to telling. We must make public the rationale for choosing that story as opposed to alternative narratives. This requires that we first deliberate with our colleagues and stakeholders about the goals we set, the missions of our schools, and the elaborated conceptions of our purposes.

Only then should we defend the adequacy of the forms of measurement and documentation we employ to warrant the narratives we offer. In the case of educational accountability, we are limited in our recountings by the instruments we use to count. As my colleague Lloyd Bond regularly reminds me, “Since we can’t normally measure everything that counts, we had better remember that what will count is what we choose to measure.” Taxonomies and indicators are critical aspects of how and with what coherence and credibility these stories can be told.

Seven Pillars of Assessment for Accountability

Most of the principles I want to offer here are familiar, even venerable. The fact that they remain pertinent suggests how persistent many of the challenges of assessment remain.

1. Become explicit about the story you need to tell and the rationale for choosing it. An account is one story among the many that could be told about the quality and character of an educational experience. No instrument can claim validity, no account can earn a warrant, without a clear explanation of why this story is being told instead of others. Indeed, it should be clear what the major alternative accounts could be and why they were rejected. Any one form of assessment, however rich, is a compromise, a choice among a set of legitimate possibilities.

2. Do not think that there is a “bottom line.” An early step in the deployment of any instrument, new or old, should be a process of locating the instrument in a larger conceptual framework that explicitly stipulates what it does measure and what it does not. Since there is no real bottom line, the first obligation of the person rendering an account is to take responsibility for locating its unavoidable insufficiencies.

Moreover, judgments of validity are never a property of measuring instruments per se. Validity can only be judged when we examine assessment results in the context of a particular argument or narrative. The cardinal principle of accountability is that counting is only meaningful and useful in the context of valid recounting. Indeed, we might make a distinction between measurement and assessment in this regard, with assessment referring to the manner in which one arrays, displays, and interprets particular measurements in the service of judgments, decisions, and actions.

3. Design multiple measures. It is dangerous to permit highly consequential decisions of policy and practice to rest on the results of a single instrument, however carefully it has been field-tested and ostensibly validated.

4. Work on combining multiple measures. A set of instruments, each with its own scores, indices, and observations, will deliver on its promise only if we take on the hard task of developing rules for deciding how to display, organize, and aggregate those indicators for making decisions. Inevitably, those decisions are functions of human judgment—which is, after all, an essential element in any such process, not something to be feared or avoided.

5. Remember that high stakes corrupt. High stakes attached to assessments have a tendency to distort the educational and evaluation processes they were intended to support. This is not only because teachers and students are sorely tempted to cheat when the stakes are high. It is also because when test designers know that high stakes are involved, they have a tendency to use items less likely to be uncertain or subject to competing judgments and arguments. As the instruments are weeded of such items or sections, they gain reliability and objectivity but often at the sacrifice of validity and nuance.

The most significant feature of high-stakes assessment is this: The higher the stakes, the greater the likelihood that teachers will teach to the test. These assessments must be designed so that the tests are worth teaching to. This is not a trivial challenge. It cries out for a strategy of embeddedness.

6. Embed assessment into ongoing instruction. Assess early and assess often. In my early days in Chicago, we used to joke, “Vote early and vote often.” High-stakes assessments are likely to be used very late in the course or program where they are employed in the service of accountability. But the later the assessment, the later the knowledge of results, and the less likely it is that the assessments will yield information that can guide instruction and learning. I call these “high-stakes/low-yield” forms of assessment. They may satisfy accountability mavens but have little educative value. Instead, we should develop low-stakes/high-yield forms of assessment, much like the “running records” used by K-12 reading teachers or the routine medical history, physical examinations, or lab tests that physicians and nurses administer.

Embedded measures will necessarily be designed with a different “grain size” from those designed exclusively for external, high-stakes assessments. They will be more particular than general; more dedicated to measuring individual student progress than institutional success; repeatedly administered, with quick turnaround, rather than being single end-of-course events. This is assessment as a regular physical exam rather than as a public autopsy.

This aspect of assessment emphasizes the need for bilateral transparency. That is, the progress students are making needs to be as accessible to them as it is to teachers. Such transparency can empower students to take greater control of their own destinies. It is, after all, ultimately the student who must own her or his understanding and progress. Systems of assessment that are opaque, secretive, and slow-responding cripple students’ sense of responsibility.

7. Become an active and collaborative site for research on new forms of assessment, new technologies to support such work, and better strategies for integration of such approaches with instruction. If the use of single-instrument, high-stakes/low-yield assessment tools will undermine the most important goals and purposes of education, then those of us who design and deploy assessments have a professional and ethical responsibility to design them to contribute more positively to the quality of teaching and learning for all students. The need now is for new assessment research and development, a project that can succeed only if institutions collaborate, experiment, and open their windows so that national work can move our fields ahead.

Taking Control of the Narrative

One of the reasons Dina was so taken with the metaphor of narrative in accounting was that the careers she had pursued before her MBA program were as an actor and as a psychotherapist. During her graduate study in social work, she had been drawn to “narrative therapy” as an approach to counseling. In narrative therapy, the central idea is that each one of us is living the life of a character in a play or a novel. Some of us feel that we have a great deal of influence over the plot of the play, while others, alas, feel that they are characters in someone else’s drama. The goal of the psychotherapy is to support one’s clients in seeing the narratives they feel they are living but have no control over, and to develop strategies for becoming the authors of their own stories, able to act responsibly in the situation and exercise real agency over their lives.

In this spirit, our responsibility is to take control of the narrative. We educators must take advantage of the deep connections between counting and recounting to define the characters, the plots, the foreground, and the background for new programs of accountability that measure the efficacy of our educational initiatives. We must summon the creative energy and ambition to take advantage of the momentum (and resources) unleashed by new policies and programs. We must exploit them to initiate the long-overdue progress in assessment needed to improve the quality of learning in education.

We are obligated to recount the narratives of most interest to our key stakeholders, but we cannot be limited to those alone. We must display the evidence of teaching and learning (and their embarrassments) through the multiple legitimate narratives we create about our work and our students’ fates. We must account for higher-order understanding and critical thinking, in addition to factual knowledge and simple skills. We must tell of the development of civic responsibility, moral courage, and identity formation even when our stakeholders have not thought to ask for those to be accounted in “the books.”

Moreover, we must make the process through which we render the accounts transparent to our stakeholders. The most important of these stakeholders are our students, who need to feel a sense of agency and responsibility in this relationship as well.

The current quest for accountability creates a precious opportunity for educators to tell the full range of stories about learning and teaching. Counting and recounting can only be pursued together. Counting without narrative is meaningless. Narrative without counting is suspect. We now have an opportunity to employ the many indicators of learning and formation that we can count in the service of the most important stories we have to tell.

Adapted from the January-February 2007 issue of Change: The Magazine of Higher Learning.

 

Lee S. Shulman is president emeritus of The Carnegie Foundation for the Advancement of Teaching, having served as its 8th president (1997-2008). He is the Charles E. Ducommun Professor of Education Emeritus at Stanford University. He serves as chair of the advisory board of the Consortium for Applied Studies in Jewish Education (CASJE) www.casje.org. CASJE is dedicated to improving the quality and extensiveness of research that will be useful in understanding, guiding and evaluating the work of Jewish Education in the broadest sense. [email protected]

Dina Shulman is a qualitative market researcher in Los Angeles who engages in counting and recounting professionally. 

Return to the issue home page:
Image
HaYidion Taking Measure Fall 2015
Taking Measure
Fall 2015