Assessing the Intangible in Our Students

July 27, 2017

Michael Morgenstern for The Chronicle

As the summer winds down, many faculty members, who just a short time ago listened to inspiring speeches during commencement season, have changed their focus to assessment. The switch can be depressing, not only because faculty rarely enjoy assessment work but because of the dramatic lowering of the horizon.

Commencement speeches routinely portray the consequences of earning a baccalaureate degree in impressive terms. Regardless of the image the featured speaker crafts in the minds of the institution’s newest alumni, the degree is envisioned as not just a credential, not just proof of the acquisition of some career-relevant skills and content knowledge, but as empowerment itself. The new graduates have acquired resilience, ethics, and an overwhelming sense of civic responsibility. The speakers reflect a societal expectation of college as a way to develop and encourage our highest values.

Those of us attending the annual celebrations also hear from our institutions’ presidents. With the skills, knowledge, values, and aspirations graduates have accumulated during their college years, with gratitude to the wise counsel of their inspirational professors, our alumni will soon be changing the world, living out the lofty mission and vision of their alma mater.

As I listen to these speeches every commencement season, I could not be more proud of what my faculty colleagues and I accomplish. Another year of frustration and exhilaration in the trenches once again becomes faith that what we do changes the world for the better.

But now, just a short time later, I and other faculty members have turned to the hard work of assessment. Some of us gather together to review portfolios to assess student learning in the major. Other groups use the Association of American Colleges & Universities’ earnestly crafted rubrics to evaluate student artifacts and learn about the strengths and weaknesses of the campus general-education program.

And so the lofty becomes the quotidian. We look for evidence of skills in communication, quantitative reasoning, information literacy, and critical thinking. We discuss the results of our review, consider what we might want to work on and improve so that students can write more clearly, better calculate statistics and probability, or more clearly evince their ability to discriminate good from poor arguments, depending on which committee we were voluntold to serve on.

We might spend time together both appreciating and questioning the "Value" rubrics, which are designed to measure student learning in 16 categories; perhaps listening to someone from the institutional-research office tell us how well our students performed on the Collegiate Learning Assessment examination (CLA+), which aims to measure gains in critical thinking. And then at the end of the day we might sit in a bar with a colleague or two and wonder together if these tests and rubrics are actually measuring something meaningful.

Later, we might sit in an armchair and wonder if our students have come to understand the ideals that motivate our teaching. Do my statistics students understand that those measures of variability are signs of the perhaps inexplicable and yet marvelous differences between people? That people who fall far from the mean should not be marginalized? Do my positive-psychology students understand that a life of meaning and purpose is more than the right answer to an essay question?

As an undergraduate many years ago, I learned to distinguish Reims cathedral from Chartres in a "Gothic Art and Architecture" course. But ever since, I have contemplated the ineluctable beauty that must have overwhelmed the minds of 12th-century peasants, and the spiritual significance of light. I learned to interpret plays by Tennessee Williams and Eugene O’Neill in another elective course, but have found that their storytelling helped me be more sensitive to and more accepting of the messiness of life as I have experienced the challenges of adulthood.

And I remember, from my freshman year at college, our professor telling us that the purpose of his version of the "Problem of God" course was to make our midlife crisis easier to manage. At 18, I was puzzled by the relevance of Dag Hammarskjöld to my grandfather’s uncharacteristic new convertible. At 48, I was finally and unexpectedly grateful to that professor for improving my life.

As I contemplate assessment results, I dream of my students learning such lofty perspectives, of their transformations — and of their potential midlife revelations. And I wonder about the omissions of an accountability system in higher education that focuses solely on "measurable student learning outcomes." How could my professor "close the loop" on assessing a learning goal that requires 30 years for fruition?

Ten years ago, Richard Shavelson wrote that accreditation pressures could lead us to focus on easily formulated, standardized, and measured student outcomes and to neglect "personal and social responsibility" skills — "personal, civic, moral, social, and intercultural knowledge and actions." He warned that if we do not measure those, "they will drop from sight." If we neglect the ineffable outcomes in our efforts to understand what college is for, and what we accomplish in higher education, they could disappear from our attention, our aspirations, and eventually from our teaching.

Ten years later, regional accreditors require that we provide evidence of student performance on, at one extreme, externally specified student competencies, and at the other, simply requiring that we post evidence on our website of the success of students in meeting our intended educational goals. In either case, evidence of student achievement is required, meaning that the learning goals must be measurable — and that we have put in the resources to measure them. While there are efforts toward assessing more complex skills, there remains a wide gap between what we typically assess and what our mission statements value.

And what then of those possibly unmeasurable learning goals? Those transformations, the sources of subsequent life revelations? If our resources of time and energy are spent on measurable goals and student achievements, what happens to the ineffable outcomes? And most significantly, what are we telling the public that higher education achieves? This content, those skills. Period.

Measurable learning goals of content and skills that students acquire in college are important. But, with apologies to Wittgenstein, about those things we may not be able to measure, we cannot remain silent. Otherwise, in the public eye, institutions of higher education may become two-dimensional skill-development, content-acquisition centers. Things one can acquire via Wikipedia and on-the-job training while avoiding student-loan debt. I hope instead that we can learn to tell a story in the summer that confirms the commencement speeches we heard back in May.

Raymond J. Shaw is an associate professor of psychology at Merrimack College. He is co-editor of The Journal of Assessment and Institutional Effectiveness.