Ask business school deans what keeps them awake at night, and they’ll respond with a litany of similar concerns: raising money, retaining faculty, recruiting students, and doing well in the rankings.
Ask faculty what keeps them awake at night, and their answers will range from achieving tenure to securing grant funding. But one topic that is gaining momentum as a faculty concern is how to measure impact. It’s a complex issue with implications for individual careers as well as institutional reputations. It’s affected by funding models, the rankings race, and public perception. And it’s not even close to being solved.
Debra Shapiro, Herman Aguinis, and Elena Antonacopoulou are three professors who have written extensively about the importance of assessing faculty members’ impact with metrics that go beyond counting publications in top-ranked journals. Shapiro, the Clarice Smith Professor of Management at the University of Maryland’s Smith School of Business in College Park, is also the president-elect of the Academy of Management, whose nearly 20,000 members represent over 100 nations. Aguinis, the John F. Mee Chair of Management at Indiana University’s Kelley School of Business in Bloomington, is also president of the Iberoamerican Academy of Management. And Antonacopoulou is a professor of organizational behavior at the University of Liverpool’s Management School in the U.K., where she leads GNOSIS, a center for excellence in management research. She also is a member of both the Academy of Management and some of its affiliates: the British Academy of Management, the European Group for Organizational Studies, and the European Academy of Management.
Last fall, at the Academy of Management’s annual meeting in Orlando, Florida, they were part of a symposium titled “Scholarly Impact: A Pluralist Perspective,” where panelists discussed the strategic importance of broadening the ways faculty impact is assessed. Individually and separately, Shapiro, Aguinis, and Antonacopoulou have written on related topics that try to answer key questions: How should schools assess the intellectual contributions of faculty? And how should those metrics change in light of the current educational environment?
Shapiro presents the argument in a straightforward fashion: The scope of faculty responsibilities is expanding due to a variety of pressures (see “The Pressures and the Problems”). As professors divert energy to these new responsibilities, they have less time for research and writing activities. Yet, when it comes to tenure and promotion decisions, faculty still are being evaluated based primarily on the number of their publications in top-ranked journals.
Says Shapiro, “Ideally, the evaluation system used in schools to assess faculty performance would change to reflect the fact that research productivity is impeded by increased demands associated with teaching and other school-related needs. However, anecdotal evidence suggests that the metrics used to assess faculty performance remain unchanged. That is, most schools only consider articles published in A journals as intellectual contributions. This is true in large part because these are the only intellectual contributions measured by the Financial Times rankings.”
The tight focus on publication in top-tier journals is problematic for more than one reason. It ignores all the other demands on professors’ time, as Shapiro points out—but it also ignores all the other gifts that faculty bring to the table. Specifically, the current approach does a poor job of measuring what kind of impact faculty have on their students, their institutions, and the broader community through their research, teaching, and service.
“We’re trying to be exceptional scholars, and to cultivate conscience in our students and in our communities. That means that we are teaching, conducting research, and serving the profession and the wider community in direct and relevant ways,” says Antonacopoulou. “We must not only support learning, but also act responsibly and critically. We must be accountable for the impact we make. Fundamentally, we add value through all the practices that constitute our scholarship.”
She adds, “What matters to faculty is a recognition that what we do makes a difference, and therefore needs to be accounted for in a way that reflects the difference we make. If citations are the only way we choose to measure ourselves, we are stabbing ourselves in the foot.”
Aguinis agrees. While he stresses that “research is the pillar, the foundation” of academic purpose, it’s not enough on its own. “We need to show that our research adds value to society and to organizations and doesn’t just improve the careers of faculty,” he says. “But what do you do with research? Do you just talk about it within the Academy of Management or other professional groups? Or do you bring it into the world and use it for the greater good?”
The problem goes back to the way faculty are evaluated. “Typically we measure faculty performance by counting journal articles and how many times those articles are mentioned in other journal articles—but those are really only read by other faculty, or internal stakeholders,” says Aguinis. “If we want to reach external stakeholders, should our faculty performance metrics consider textbooks written for undergraduate, graduate, and executive education? Should we measure the extent to which practice is affected by the research we publish in journal articles? At the moment, at the majority of research-oriented business schools, textbooks do not count at all in terms of faculty performance.”
Shapiro points out another problem. “If books and book chapters count as nothing when schools conduct annual faculty reviews, then who will write them?” she asks. “My own interest in writing chapters for books, even prestigious research-based handbooks, has been diminished because I know the work will count as zero. And on the occasions I have decided to co-edit a book, I’ve had difficulty finding authors who are willing to contribute chapters because they know the work won’t count in their own schools’ evaluations of their productivity and impact. This mentality of only counting A-journal publications not only harms the learning environment of schools, but also the advancement of knowledge that is possible in management science.”
But where should schools start if they want to amend their evaluation metrics? As in any other industry coming to terms with change, administrators must begin by defining performance, says Aguinis. They need to ask key questions, he says: “What do we want faculty members to do? What are our strategic objectives? What individual behavior will show value-added activities, and what metrics will measure these activities?
“For instance,” he adds, “if we want the school to receive attention from the media, we have to consider if professors receive any kind of reward for spending an hour on the phone talking to a reporter versus working on their own papers. We cannot reward A while we hope for B. We cannot hope faculty will talk to the media when all we reward is publication of papers.”
Business faculty teach about performance and how to measure it, Aguinis points out. “We should use that knowledge to help create new systems for business schools.” He believes faculty also should use their knowledge of organizational change to help revamp the whole culture of higher education.
He says, “One thing to do is to make sure the faculty understand the new pressures facing schools, particularly budgetary pressures. But the easiest and fastest way to encourage change in any organization is to change the reward structure. The obsession with publishing in A journals is driven to a large extent by a reward system in which schools tell faculty, ‘You have to publish in these journals if you hope to get tenure or get promoted.’”
Shapiro, Aguinis, and Antonacopoulou are not alone in focusing on the impact of scholarship. Currently, the entire higher education field is considering how it can measure the impact it makes both internally and externally—that is, on practitioners, legislators, media, and society at large, not just on other scholars. “Impact” is one of the pillars of AACSB International’s recently revised standards, and AACSB’s former board chair Robert Sullivan has discussed how the new standards might change the way research is assessed.
There is some indication that the metrics for evaluating professors are already beginning to broaden. Some individual schools do consider books, chapters, professional service, and other measures of impact when assessing faculty performance, though Shapiro notes that these rarely receive the same weight as publication in top journals. And some nations also are looking for ways to broaden their definition of impact—and using government funding as a way to ensure that schools follow through.
For instance, the Higher Education Funding Council for England (HEFCE) is implementing a new system for assessing the quality of research at universities. The Research Excellence Framework (REF) will be used by the four bodies that fund research in the U.K. to help them determine how to allocate funds, and among the criteria they’re considering is the impact of research. But the system is still being perfected: While submissions to REF include “impact case studies,” says Antonacopoulou, there has been little agreement about what impact is, how it’s demonstrated, and how it can be measured.
She says, “There is an urgent need to study these impact case studies to discover common perspectives and develop a coherent framework for capturing impact in tangible and intangible ways.”
And yet Antonacopoulou remains convinced that broadening the definition of scholarly output allows faculty to show impact through all of their activities. She finds it encouraging that other countries—including Finland, Canada, and some EU countries—are trying to establish more clear-cut definitions of what impact looks like in education.
Others are also optimistic. When government funding is tied to proof of impact, Aguinis says, “I think some of the metrics will change.”
That would be a win for schools, faculty, and general society, says Shapiro. “Positively evaluating faculty members for sharing knowledge in ways that supplement A journal publications seems likely to incentivize faculty to write articles for practitioner journals, to write book chapters, and even to write books that are likely to reach and benefit broader audiences.”
COUNTERWEIGHTS TO CHANGE
Yet even schools that want to broaden their metrics will find it difficult to abandon conventional measures, and Shapiro and Aguinis blame media rankings.
“One reason change is so slow is because the Financial Times rankings of business schools are so highly valued, and the Times bases rankings on the number of publications faculty have in particular journals,” says Shapiro. “As long as those rankings have the weight they have, it will be difficult for people to broaden the nature of impact indicators.”
She’d love to see business schools en masse refuse to participate in the rankings game. Failing that, she’d like to see the Financial Times metrics expanded to include not just specific journals in each discipline, but also handbooks and research-oriented volumes whose chapters have been rigorously critiqued and revised.
Aguinis proposes that business schools should devise their own set of performance standards. “We know that students look at the rankings when they’re choosing a school, so why don’t we produce a better source of data?” he asks. “We know our industry. We know what good research is and what good teaching is. We should find a way to create our own performance standards rather than chasing someone else’s.”
If the situation doesn’t change, Antonacopoulou paints a dark picture of the future. As government funding becomes scarcer, schools will raise tuition fees to make up for the funding shortfall. Like other industries that have responded to crises, they’ll also begin merging and consolidating to cut costs. In addition, they’ll aggressively recruit international students, who generally pay higher tuition—but if they recruit too heavily from one region of the world, they risk losing the classroom diversity that is necessary to foster a rich learning experience. There already appears to be a heavy reliance on dominant student markets, she says.
“We are supposed to be teaching students to think critically, emancipate their imaginations, and expand their capacity to learn differently,” says Antonacopoulou. “But that’s difficult to do when there is limited diversity in the classroom. When students invest the money to study in another country, it is imperative that they be exposed to the variety of perspectives that will enhance their capacity for critique.”
She continues, “This is the emerging picture of a future we are creating for ourselves if we don’t account for our impact in a richer way. We need to actively demonstrate our worth, so we can safeguard our role in society. We want to leave no room for critics to question the investment in education that cultivates conscience—not just at the business school level, but in the university at large. We begin by demonstrating how our scholarship has impact.”
SEARCHING FOR SOLUTIONS
But change is difficult, and these professors believe it won’t come unless faculty push for it—at their own schools and beyond. “During my performance reviews, I follow the guidelines. But I also tell my dean of the things I do that demonstrate my contributions to the scholarly community and the wider communities of business and policy,” says Antonacopoulou.
Faculty also must raise the issue among themselves. Says Antonacopoulou, “I’m inviting us to start discussing how what we do as scholars has an impact. If I recognize that what I do has impact, then I know that when I write my next article, I won’t just be mindful of where I can publish it so I can get the most brownie points. I’ll be thinking about ideas that I can bring into my classrooms and into the worlds of policy and practice so I can make an impact with my teaching, my research, and my engagement work.
I’ll seek feedback from students, executives, and policymakers so my ideas can be tested, enriched, and improved. I want to capture the variety of ways in which my work makes a difference.”
Such conversations are “absolutely happening” among members of the Academy of Management, says Shapiro, not only in the U.S. but at affiliate organizations in other parts of the world. For instance, at annual meetings of the organizations, participants in both scholarly symposia and professional development workshops debate how to broaden the impact of scholarly research, how to measure it, and how to operationalize it.
“The presidents’ speeches at Academy meetings also have increasingly touched on the need for research that advances scientific knowledge and solves practical problems,” Shapiro adds. “The solutions at this point are not clear. But we’re all speaking about the need for a solution.”
Today’s faculty labor under a growing number of pressures that are familiar to anyone in the higher education field. Debra Shapiro of the University of Maryland points to shrinking budgets, which have led to larger class sizes, and the rise of online education, which has erased geographic boundaries in the competition for students.
To attract students from around the world, Shapiro notes, many schools are offering a greater variety of classes, which they manage by shortening some classes to half a semester. While faculty might still teach for the same number of credit-hours per year, Shapiro says, “they’re teaching twice the number of students they would typically teach.”
In addition, teaching half-semester-long courses increases the courseload professors carry every year. Shapiro notes that, at research-intensive institutions, typical teaching loads might be three or four courses a year. But when those courses are cut in half, those loads might become six to eight courses per year. At teaching-oriented universities, that number easily could be doubled.
Not only that, to accommodate more students, classroom sizes have expanded—which has changed the pedagogy in certain kinds of courses, particularly those focused on skill-building. For instance, if there are 60 students in a course about learning how to manage teams or negotiate effectively, the professor can’t realistically videotape students and play back their performances for small-group critique. “It’s quite likely that the opportunity disappears for students to see themselves practicing negotiating or behaving as team members and receive individualized feedback from the instructor on how to improve,” says Shapiro.
To ease the teaching burden on tenure-track faculty, many schools are hiring more adjuncts, which Shapiro considers a mixed blessing. “Adjunct professors are highly valued by faculty, because they help cover classes that faculty would need to teach at the expense of doing research,” says Shapiro. “However, if schools begin hiring adjuncts in larger numbers than tenure-track faculty, it becomes difficult to sustain and energize the research culture. And because adjunct professors can’t train doctoral students, having too many of them on one campus hinders the school’s ability to train the next generation of management scholars.”
Thus, the problem remains: Tenure-track faculty continue to absorb responsibilities that take time away from conducting research, but the number of A-journal publications is still the primary metric that is considered in faculty evaluations. And that’s a problem still looking for an answer.