Dissecting the Classroom

Brett H. Gilley took a seat in the back of the lecture hall and laid the tools of his trade in front of him.

The first was a gridded sheet of paper, on which Mr. Gilley, a science teaching-and-learning fellow here at the University of British Columbia, wrote the course’s name. The other was the stopwatch on his cellphone, which he started.

About 100 students took their seats. Mr. Gilley had come to analyze Stuart E. Sutherland’s teaching of “Earth and Life Through Time.”

Mr. Sutherland, a professor of teaching in the department of earth, ocean, and atmospheric sciences, is known for his skill in the classroom. He’s won five teaching awards since 2002 and frequent praise in student evaluations.

“An amazing prof,” one student wrote on Rate My Professors, a website. “If nothing else, take his class just to prove to yourself that there are teachers out there who are doing it for us.”

Mr. Gilley’s presence in his colleague’s class represented something that is relatively rare in higher education, where the classroom remains a professor’s private domain, the black box at the core of the traditional educational experience.

Higher-education scholars at more than a dozen institutions are seeking to shed light on the components of good teaching by documenting precisely what happens in the classroom. Some have received grants from the National Science Foundation to study how much their colleges have adopted teaching methods shown to improve learning and to increase the number of students majoring and persisting in science, technology, engineering, and mathematics: the STEM fields. Other academics are wading into a longtime argument between critics who bemoan the state of college teaching as insufficiently interactive and traditionalists set on defending the virtues of lecturing.

As Mr. Sutherland taught his class on mass extinctions, Mr. Gilley looked at the grid, with its 25 categories of student and instructor behavior. Every two minutes, he interpreted what he saw.

Was Mr. Sutherland chiefly doing a demonstration, asking or answering questions, writing?

Mr. Gilley placed a check mark under lecturing.

What could he see the students doing? Were they discussing a clicker question, working in small groups, making presentations?

He put a check under the column labeled listening.

How would Mr. Gilley characterize the level of engagement of the students sitting in front of him? Of the 18 nearby, most were looking at Mr. Sutherland. Three were texting or looking at websites unrelated to the course. Over all, he decided, their engagement was medium to high.

As the class proceeded, Mr. Gilley wrote one checkmark after another, forming two columns down the page. One column indicated that Mr. Sutherland was lecturing, the other that his students were listening.

In the 22nd minute, Mr. Sutherland almost broke the pattern. He might have gotten a mark under the posing-a-question column when he asked his students how plausible it was that gamma rays had caused extinctions.

“What’s the evidence for this?” he asked. But then he quickly answered his own question. “Absolutely none.”

Mr. Gilley placed another checkmark under “lecturing.”

At colleges, complex questions are routinely asked and answered. Yet the question of what, exactly, is happening in the classroom is seldom approached through direct observation.

Now scholars are beginning to fan out through campuses to study the subtle dynamics of the classroom. Some are doing so for scholarly reasons, hoping that they can distill teaching to its essential parts and identify which methods produce specific types of learning. Others, like Mr. Gilley, seek to gather data that paint a picture of an individual professor’s practices, which can be compared with the norms of a discipline or type of course and then be used to improve his or her teaching.

Observers of both kinds have encountered mixed receptions to their work among colleagues. Such visits are often tied to tenure or promotion recommendations, a context that makes many observations stressful, unwelcome, and rare.

As a result, much of what is known about the state of teaching depends on students’ evaluations of professors or faculty members’ self-reports. Survey data, for example, show that nearly half of all faculty members at four-year institutions say they lecture “extensively.” Such a data point may obscure more than it reveals.

Teaching is cognitively demanding, making it difficult for professors to simultaneously reflect on how well they do it while engaged in the act itself. The result is that professors can be unreliable in reporting on their behavior. The mismatch has been described both empirically and anecdotally.

Mr. Gilley recalls observing a professor who was teaching with the Socratic method. The professor hadn’t realized until after Mr. Gilley watched him that he had called on only four of his 20 students during the entire class session. In the next class the professor made sure to involve different parts of the room and call on those who hadn’t answered yet. Fifteen students spoke up that day.

One day last January, Karen K. Inkelas, an associate professor and director of the Center for the Advanced Study of Teaching and Learning in Higher Education, at the University of Virginia, shared what she calls her “big idea” with about a dozen colleagues. They were members of her advisory council, a group of faculty members across disciplines who like to discuss and study their own teaching at Virginia.

Her idea was to get into classrooms, document what unfolded, and connect the data points from what faculty do to what students learn.

Ms. Inkelas knows that asking fellow professors to let her into their classes is a tough sell. She summarized for the council the objections she had heard and tried to refute: College teaching is idiosyncratic by discipline and course. Studying it intrudes on academic freedom. There is little consensus as to learning outcomes. And, well, it’s just obvious that teaching produces learning.

Ms. Inkelas hoped that some members of her council could become part of a “coalition of the willing,” a group of professors in the vanguard of supporting her work. She wanted their number to grow and form a critical mass across the campus, so that a wide swath of professors eventually open the doors of their classrooms to her.

It hasn’t been easy. About 20 professors have allowed her in. Other faculty members privately expressed interest in seeing her results but have been leery of offering themselves as research subjects.

Observations are a form of assessment, which faculty members often see as a fruitless exercise. They have had too many experiences, Ms. Inkelas knows, in writing stuff on a piece of paper, sending it to the accreditor, and hearing little about it again.

“Nobody likes it,” she says. “It’s like going to the dentist.”

At the end of the meeting, she asked the council members for feedback on the observation tool she showed them. On paper, it looks more like a table than a grid. A computer application is in the works.

The sheet of paper has four columns and four rows. It divides each class into five-minute increments. The observer chooses from more than 40 codes identifying what the instructor and students do, and enters the indicators by hand. There is ample room for the observer’s comments

Some of her colleagues braced for what her observations might reveal. “I don’t care how charismatic I try to be,” said Claire R. Cronmiller, a professor of biology. “I know there are students in the back playing computer games.”

Others were skeptical that observing such a tiny slice of a course would be worthwhile. “It’s like picking up War and Peace, reading one page, and deciding if it’s a good novel,” said Kirk A. Martini, an associate professor of architecture.

Still, Mr. Martini was one of the few faculty members willing to allow Ms. Inkelas to watch him teach.

Two months after the advisory-council meeting, Ms. Inkelas sat in the back of the raked lecture hall sketching the layout on paper as about 90 students in his introductory course on structural design took their seats. At five-minute intervals, she jotted down codes and scribbled notes about what she saw.

Ms. Inkelas recorded every time Mr. Martini posed a question. In the first 15 minutes, he asked 11. A few were straightforward, like when he asked his students to identify a building on campus. Most were conceptual and content-specific: How does weight affect load path? How do you “read” a truss to understand how it’s organized?

Mr. Martini stopped to listen to the answers, clarified misconceptions, and amplified ideas. Even though he was in a lecture hall, standing on stage, and speaking much of the time, he was running the class as if it were a small-group discussion.

None of the students were on laptops. Ms. Inkelas saw one student take out a cellphone but quickly put it away.

Mr. Martini shifted his students’ focus in a fluid cycle. He would project an image on the screen behind him, ask a series of questions, demonstrate using a physical model, and draw an illustration by hand on the overhead projector. The column on the left side of her paper, which reflected what Mr. Martini was doing, showed an array of codes—for demonstrations, diagrams, using diagrams, lecturing, and asking questions—with several appearing during the same five-minute interval.

After a half-hour, he displayed an image and told the students to pair up and find all of the zero-force members in a diagram on the screen, which brace those that support a structure’s load.

Ms. Inkelas watched as the students interacted with one another. A few sat quietly. “They’re not all comfortable,” she said, “but most of them are.”

After three minutes, Mr. Martini asked them to share how many zero-force members they had found. One student identified two.

Mr. Martini said there were two more. He waited, letting the silence hang as the students thought.

At a break. Ms. Inkelas shouldered her bag as she stood.

“Kirk is one of the best teachers I’ve ever observed,” she said.

Mr. Gilley is one of 20 science-and-teaching fellows at British Columbia. They have expertise in a scientific discipline and in teaching and learning. Their presence at the university is part of a project named for Carl E. Wieman, a Nobel Prize-winning physicist who advocates for changes in the teaching of science. He was on the faculty at British Columbia from 2007 until last year, when he moved to Stanford University.

The fellows consult with professors, observe them teaching, and carry out research on teaching and learning. Mr. Wieman helped develop the observation tool, but he also sees its limitations.

“There are a lot of strong psychological reasons that faculty members focus so much on what happens inside the classroom,” he says. “It really dominates the course.”

What happens in the classroom is critically important, but the same is true of what students do outside of it, he says.

Such an opinion from a supporter of classroom observation reflects the mixed feelings that many faculty members have about the practice.

Observations once enjoyed broader support than they do now. In 1992, Peter Seldin, who was a professor of management at Pace University, surveyed the leaders of faculty-development centers. They ranked 10 preferred methods for improving teaching. Classroom observations was second, behind workshops.

Twenty years later, he repeated the exercise. Observations fell to fifth place, largely, he says, because they came to be seen as resource-intensive and of limited use. At the time, they were often based on one visit by a single, untrained person.

Like most faculty-development work years ago, observations tended to be undertaken by people with humanities or social-science backgrounds. The observations often reflected the disciplinary sensibilities of those academics, says Mary Deane Sorcinelli, associate provost for faculty development at the University of Massachusetts at Amherst.

In keeping with the watchers’ disciplinary orientation, teaching was thought to be more art than science. Each class was a distinct event, as was an observation of it. And the resulting feedback rarely included quantitative data but instead involved a narrative.

As groups like the National Science Foundation and the Association of American Universities have become increasingly interested in the quality of classroom teaching, particularly for STEM courses, teaching has started to be viewed through a scientific lens, says Ms. Sorcinelli, who writes about and conducts observations. Observation tools have changed to reflect this shift.

In this view improving teaching is a matter of hypothesis and experimentation. With the help of observation, teaching can be broken into its atoms, categorized, and analyzed.

A two-minute excerpt from a class says a lot, according to this conception of teaching. While a single page from War and Peace pales as a means of appreciating the full sweep of the novel, it’s probably enough to show that Tolstoy was a great writer.

Observers like Mr. Gilley, at British Columbia, and Ms. Inkelas, at Virginia, try to avoid going to one place: inside students’ heads. They try not to guess at the cognitive processes going on around the room.

Other tools have asked whether an instructor’s questions trigger “divergent modes of thinking” or prod students to be reflective about their learning.

The tools in Vancouver and Charlottesville, however, were developed with a simple goal: identify what can be observed happening around you. That was Matthew T. Hora’s idea when he devised the Teaching Dimensions Observation Protocol, upon which the other tools are based. Mr. Hora created his protocol with Joseph J. Ferrare, a fellow researcher at the Wisconsin Center for Education Research, at the University of Wisconsin at Madison.

At a working group of researchers, convened in late 2012 by the National Science Foundation, Mr. Hora acknowledged that observing and categorizing at the minute level of detail that his tool calls for can prove challenging. “The deeper you go, the better the data,” he said, “but the harder it is to do.”

An anthropologist by training, Mr. Hora designed his tool to break down teaching to some of its smallest details. The 47 variables cover five categories, including teaching methods, pedagogical actions, interactions between students and instructors, and students’ apparent level of cognitive engagement.

Watchers categorize the class in two-minute intervals. It can take more than two days to learn how to use the protocol tool reliably, Mr. Hora says.

Drawing arbitrary lines between lecturing and methods of teaching that require students to participate more actively obscures valuable distinctions, he argues. Teaching is sophisticated and complex, and no single approach is always good or bad.

The week before he traveled to Washington for the NSF session, Mr. Hora and Amanda Oleson, who was then a graduate student in education-policy studies at Wisconsin, were in Madison. They trained two professors using a videotaped lecture in chemistry from the Massachusetts Institute of Technology.

It quickly became apparent how tricky some of the fine-grained distinctions can be. For example, his observation tool distinguishes prepared visuals from those handwritten on the spot. That matters, he says, because the latter typically signals that an instructor is going through the process of solving a problem in front of and involving students.

As the group watched the MIT professor, other questions arose. How, for example, should they categorize “humor” in the lecture? The code exists because when professors use humor—when they enter the students’ seating area, for example—it tends to command attention. Mr. Hora has defined what counts as humorous in the observation. It’s not enough for the observer to think a remark is funny. A student must laugh.

The new observation tools are starting to yield data, with researchers beginning to note patterns in disciplines’ use of teaching methods.

Mr. Hora has found, for example, that math instructors use handmade visuals far more often than instructors in biology, chemistry, geology, and physics. Mathematicians often like to work through problems in class, and the distinction would be lost if a description of their teaching was limited to lecturing.

But it may be a long time before certain methods can be said to produce specific kinds of learning gains.

Distinctions have also been observed among professors in the same discipline and type of course. In the January issue of the Journal of College Science Teaching, Mr. Hora compared two instructors of introductory biology who seemed to be teaching in the same way.

One lectured in 96 percent of the intervals observed; the other did so 75 percent of the time. “However,” Mr. Hora wrote, “it would be a mistake to characterize both instructors’ teaching styles as lecturing.”

The behaviors observed in the first instructor—the use of slides, anecdotes, humor, illustrations, and multimedia—led Mr. Hora to conclude that the quality of interactions between professor and students was “low and limited.”

The other instructor lectured, too, but used demonstrations, handwritten and prepared visuals, and interactions that engaged the students far more.

When Mr. Hora’s research subjects ask him for the data from their classes, he obliges. But he refuses to give the information to administrators who would use his observation tool for a tenure application or to judge the quality of teaching at their institution.

Any effort to improve teaching, he says, should also include both a preobservation interview of the instructor and student feedback.

Nor does Mr. Hora want his observation protocol to become a tool for what he calls the “audit culture and accountability culture.”

“We’re very hesitant to put anything out there that will be construed as ‘This is the way to measure teaching efficacy,’ " he says.

Twenty minutes after Mr. Sutherland’s class on extinctions had wrapped up at the University of British Columbia, Mr. Gilley stepped into the professor’s office to talk about what he’d seen.

He folded himself into a chair at a table.

“Basically,” Mr. Gilley said, pointing to the completed observation form, “this column here is what the students are doing.”

The checkmarks proceeded in a long line under “L,” for listening.

Ideally, he says, he likes to see checkmarks spreading across several columns on the page because it signals that the professor is changing what he or she is doing, and the students are being asked to exert themselves intellectually in different ways.

Mr. Sutherland nodded.

“Then you were lecturing for much of the time,” Mr. Gilley said.

The checkmarks for engagement, he said, were generally high. “Almost all of them were paying attention all of the time because of your beautiful accent.”

Mr. Sutherland, who is from Manchester, England, trades on his accent. Many students say how much they enjoy listening to the sound of his voice.

For Mr. Sutherland, having someone watch him in the back of the class affords him a view of students’ laptops he seldom gets.

“Any Facebookers?” he asked.

“Oh, yeah,” Mr. Gilley said.

Those were the times that the level of engagement had dipped to moderate. A few students were texting or playing games on their phones. One was shopping for a new coat.

“That seems fairly standard,” Mr. Sutherland said.

Then, gently, he arrived at his central criticism. Mr. Gilley subtly lowered his posture. His voice nearly a whisper.

“You could put so many activities in that lecture,” he said.

Mr. Sutherland had planned to be more interactive that day by including clicker questions, but technical problems had interfered. He explained that he had revised the course significantly over time. He now reserved Fridays for almost all of his classroom activities, a change that meant he had to jettison one-third of the material to make time.

“Sometimes I just want them to listen,” he said.

Mr. Gilley sympathized. Students often prefer to sit passively. “They love your lectures,” he said.

He returned to the moment when Mr. Sutherland asked rhetorically what evidence there was that gamma rays caused extinctions. Instead of debunking the theory himself, what if the students did an activity?

They would break into groups and each argue one of the theories that Mr. Sutherland had presented. The rest of them would rebut each theory. Everybody would have to learn at least one theory well enough to explain it to the others.

“That’s right. That’s good,” Mr. Sutherland said, thinking it over. “That would be a good little activity to bring into that, actually.”

Mr. Gilley has learned not to push too hard. An incremental improvement is still an improvement.

The session was almost over, but not before Mr. Gilley ribbed Mr. Sutherland about a moment at the end of class when he had posted a slide of learning goals, which showed what he expected his students to get out of the class.

The verb was the giveaway. When professors teach using a traditional, transmission-focused style, the learning goals use words like “know,” “understand,” or “describe.” To Mr. Gilley, it often means the professor is asking the least of his or her students.

Once faculty members decide that their students ought to do more in class, those verbs change. Students start being told that they must “apply,” “analyze,” “compare,” “contrast,” “create,” or “evaluate.”

On Mr. Sutherland’s slide, he asked them to “understand.”

Mr. Gilley has become a master at giving pointed feedback without sounding judgmental.

“You tried to get through it quickly so I wouldn’t notice,” he said. Both men laughed.

It was the third time that Mr. Sutherland had gone through a debriefing with the science teaching-and-learning fellows. He admits being apprehensive at first.

“You become very precious about your class,” he says. “However much you tell yourself logically these people are here to help, it’s sometimes difficult to have that extra pair of eyes in the classroom.”

See the Notes

To view these notes in full screen for easier reading, click here.