Solving for X

Solving for X: A New Approach to Faculty Assessment

The Vienna University of Economics and Business now takes a holistic approach to evaluating faculty performance, considering their teaching, service, and biographical factors, not just their research output.

By Edeltraud Hanappi-Egger • March 02, 2019
Illustration By Greg Mably

SOLVING FOR X: A New Approach to Faculty Assessment

THE CHALLENGE:

When hiring and assessing faculty, research-oriented academic institutions heavily emphasize publication output. But is this the best way to hire and promote the best candidates? In many cases, the answer is no. Rather, this approach rewards researchers in well-established fields who give their undivided attention to research, and it favors academics who follow linear, uninterrupted full-time academic careers.

That puts at distinct disadvantage those professors whose academic achievements are in newer, more innovative fields; those whose main achievements are in teaching; or those who have taken sabbaticals or pursued part-time employment so that they would have time to care for loved ones. In short, schools that evaluate faculty based primarily on publication output can develop a skewed view of what the ideal faculty candidate looks like.

Even more troubling, this measure ignores "third-mission activities," in which faculty engage with the local community, address social challenges, contribute to the transfer of university knowledge to market, and disseminate knowledge to nonacademic audiences. As third-mission activities become increasingly important to universities, more schools need to incorporate them into their evaluation procedures.

THE APPROACH:

At the Vienna University of Economics and Business (WU) in Austria, we asked ourselves whether, in to day's complex and increasingly interconnected world, we could justify reducing an applicant's performance to a publication list. Ultimately, our answer was no.

That's why we now assess faculty performance across many dimensions. We refer to our model as "uLike," which stands for universitiire Leistungsbeurteilung im Kontext entwickeln. In English, this translates to "Academic Performance Assessment in Context."

BROADER CONSIDERATIONS

More and more, we're seeing three elements on the resumes oftoday's academic scholars that make it unfair--and unwise--for us to ignore all factors except research output:

Nontraditional career paths. Researchers might have earned PhDs later in life, perhaps because of previous careers in business, or they might have taken breaks in their careers to act as caregivers. Their careers could have been interrupted by relocations or unforeseen events in their personal lives. Traditional systems of faculty evaluation often do not take such common career detours into account, at a time when the number of faculty who are affected by such factors is likely to increase. Today, both women and men are increasingly taking time off to raise children, and baby boomers are caring for their children and elderly parents simultaneously. Even millennials and members of Gen Z are following informal paths in their early careers, often by necessity: According to a 2012 study from the University of Western Australia, the number of secure early-career jobs is decreasing.

"Academic age." Each professor has not only a biological age, but also what we call an academic age, which refers to the number of years a professor has spent actively pursuing academic work. For example, consider two professors who both earned their PhDs the same year. One has produced eight publications while working full-time, while the other has produced five publications while working part-time. In absolute terms, the full-time professor has produced more work. However, the part-time professor has had less time to conduct research; she or he is both academically younger and potentially more productive. In this scenario, measuring each researcher's performance by publication output alone would not be neutral or objective. We also must account for the period of time each candidate had to accomplish the work.

Teaching and third-mission activities. As mentioned above, the performance portfolio of today's university must include not only research production, but also broader contributions to society. Therefore, faculty recruitment policies also should consider applicants' teaching and third-mission activities, including achievements related to university management and development, committee membership, management of academic units, conference organization, journal paper review, knowledge transfer, media appearances, participation in public events, and commitments to community service and social impact.

CHANGES TO OUR CRITERIA

In 2012-2013, a working group of full professors at WU began developing an alternative performance assessment model for the school. As a first step, they created a new job profile for all WU faculty, which included criteria in research, teaching, and third-mission activities. This profile also was applied to all new hires going forward.

In 2015, with the help of the Rector's Council, we re-evaluated our search criteria once again, this time to include each candidate's biographical history and career paths. We were inspired by the method of faculty evaluation used by many universities in Australia and New Zealand, where faculty's performance is evaluated relative to their opportunities. This method does not assume that all faculty follow traditional linear, uninterrupted academic career paths, largely because that career pattern no longer matches that of many of our professors.

The new job profile outlines the expectations WU has for faculty in the areas of teaching, research, and third-mission activities, and the school conducts regular evaluations. Above-average salary increases are granted only to those professors who excel in two of these three areas and who meet average standards in the remaining dimension. If faculty members are seeking promotion, we consider their academic age and biographical factors that could impact their performance, and we compare their performance to that of other full professors who work in similar fields and are at similar stages of their careers.

By clearly defining our criteria and the broad spectrum of responsibilities that we associate with a professorship, we positively manage faculty expectations. "Over the last decade, research output has become the dominant goal for academics on all career levels," says Michael Miiller-Camen, professor of human resource management at WU. "WU's job profile for full professors tries to reverse this development by proposing that professors have to demonstrate excellence in all three areas. As a result, the performance perspective of WU professors has broadened."

MORE NUANCED RECRUITMENT

We have made three significant changes to the language we use in our public announcements of open positions. One of the most telling changes is the requirement that states that applicants must have "an outstanding publication record in the relevant fields." Here, we have added the phrase "commensurate with academic age" to clarify that we consider how much time each candidate has had to publish.

In addition to "a record in attracting and conducting third-party funded research" and "experience in empirical research," we now ask applicants to demonstrate "teaching qualifications at the undergraduate and graduate level." Finally, we ask applicants to highlight their "gender mainstreaming skills," because our full professors must accept leadership roles with respect to the university's equal opportunities policy.

Once we begin reviewing applications, according to WU's bylaws, we are required to appoint at least three reviewers-including at least two external reviewers-to evaluate faculty candidates. The reviewers' highest-rated candidates are invited to a public hearing, after which the search committee proposes a short list. The rector then decides which candidate to approach with an offer.

We have drafted documents to provide both our search committee members and our reviewers with reference points throughout the process and remind them of their responsibility in evaluating faculty candidates. These documents include an evaluation sheet that breaks down each requirement in the language of the job posting. In addition, guideline and assessment sheets direct the committee to consider candidates' academic age, interruptions in their professional careers, reduction of working hours, and delays in completing individual educational stages due to caring for children or family members.

The documents also remind them to consider candidates' involvement in civil society activities and time spent in leadership and service positions. At no time can the committee use candidates' future intentions against them, whether they might plan to go on parental leave or opt for parental part-time hours.

Finally, the committee members are reminded that they must evaluate a candidate's "bibliometric data related to publication based on a comprehensive selection of parameters such as teaching, thesis supervision, knowledge transfer, and research and innovation management."

WU's Equal Opportunities Committee oversees the entire recruitment process. Guaranteeing transparency at every step is crucial, because Austrian law gives candidates the right to file complaints against alleged discrimination with a government anti-discrimination board.

See "Three Candidates, Three Assessments" below for an example of how our committee and reviewers might evaluate three representative candidates using the uLike model.

Let’s take a look at three potential candidates for a full-time faculty position at WU, as well as the criteria our search committee would consider for each one before selecting which individual to hire. All three candidates have excellent qualifications (see chart below):

APPLICANT A meets publication output requirements, has acquired two research grants, and has held positions at two lesser-known institutions.

APPLICANT B is the youngest academically—having earned a PhD in 2009, the applicant has an academic age of just ten years, but in that time has achieved an impressive publication output, including single-author papers in leading journals and research activities located at the intersection of two desired disciplinary areas. Applicant B also has worked internationally at two lesser-known academic institutions, but has not attracted any research funding.

APPLICANT C meets the requirement of work at the intersection of these fields and has a strong record of attracting grants. The applicant also has held three international positions at renowned universities.

All three candidates have international experience teaching at the undergraduate and graduate levels.

If the committee ranked these candidates based only on traditional criteria, without taking biographical factors into account, Applicant B would win out over Applicants C and A as our new hire due to a higher absolute publication output. However, once biographical factors are considered, the picture changes. As it turns out, Applicant C has two children and took time off for parental leave, dropping the applicant’s academic age from 12 years to eight years. This broader perspective makes the academic output all the more impressive, given that the applicant had fewer years to complete the work. In this scenario, Applicant C becomes the preferred candidate.

This example shows just how an alternative approach to performance assessment can change the outcome of faculty hiring and promotion decisions. It’s also a glimpse into how business schools can rethink their own assessment criteria in ways that suit their academic missions.

>Click to view larger image.

* Social Sciences Citation Index

**Each “+” symbol refers to relevant experience in that category. Each “3” symbol indicates that the candidate’s experience meets the search committee’s criteria in that category

ADDRESSING CHALLENGES

Through our new model, we have been able to help our university fulfill its third-mission activities and recruit more diverse top talent. But going against academic tradition can cause debate. For instance, a 2012 paper in the journal Environment and Planning A by Australian researchers Natascha Klocher and Danielle Drozdzweski asks, "How many papers is a baby 'worth'?" Klocher and Drozdzweski discuss the relative weight given to different criteria and life contexts and argue that a more comprehensive in-house approach to faculty evaluation must be based on a collective decision.

Of course, it's also essential to ensure that the new approach does not erode quality standards or excuse subpar performance. In addition, there is the issue of privacy. A faculty evaluation process can consider biographical factors only if they are disclosed. But how can we encourage faculty to share private information for performance assessment? Disclosing such information blurs the borders between private and work life, which some might view as problematic.

At WU, we still are exploring these questions. As one step to address the privacy issue, the school will establish an online application tool that informs applicants about WU's faculty recruiting approach. We will offer all applicants the opportunity to volunteer relevant information on private matters that they would like the search committee to know. Of course, we view it as critical that such private information is kept strictly confidential.

PERFORMANCE IN CONTEXT

In December 2016, Austria's Federal Ministry of Education, Science, and Research honored WU with one of eight Diversitas awards for the impact of uLike on promoting diversity. According to the Diversitas jury, our faculty assessment model was selected because it recognized candidates' wider range of contributions to society, viewed candidates as whole individuals with both personal and professional lives, and had the potential to eliminate hidden discrimination in the faculty recruitment process. The approach, the jury added, could be an incentive for young scholars to devote more effort to activities that go beyond research and teaching.

In the area of faculty evaluation, it all boils down to this: How can we attract and retain the best talent? If higher education institutions continue to cling to rigid notions, rewarding only those academics with standard biographies and A-journal publication lists, they may eventually find themselves lagging behind in the race for the best faculty.

Edeltraud Hanappi-Egger is professor of gender and diversity in organizations and president of the Vienna University of Economics and Business (WU) in Austria.

This article originally appeared in BizEd's March/April 2019 issue. Please send questions, comments, or letters to the editor to bized.editors@aacsb.edu.