Preparing for Impact

How the Franke College at Northern Arizona University defined its mission and created a system of measuring the impact of its faculty’s scholarship.

THE CHALLENGE: If you’re like many business school administrators, you have a keen interest in the concept of impact—particularly in light of AACSB International’s 2013 accreditation standards, which now require schools to show the educational, commercial, and social impact of their activities. Business schools can examine their impact in many areas, but one area looms especially large: intellectual contributions. Because other programs are undoubtedly struggling to address how to evaluate faculty scholarship in their own accreditation reviews, we wanted to share our experience.

THE APPROACH: As the Northern Arizona University’s W.A. Franke College of Business in Flagstaff recently completed its Continuous Improvement Review (CIR) under the 2013 standards, we faced challenges in defining four key areas: what drove the Franke College’s mission, what types of intellectual contributions we most valued and the impact we wanted those contributions to make, how to incentivize those types of contributions, and how to demonstrate impact at an individual and school level.

We first paid close attention to the wording of AACSB’s Standard 2, which requires a business school to produce “high-quality intellectual contributions that are consistent with its mission, expected outcomes, and strategies and that impact the theory, practice, and teaching of business and management.” In other words, what matters is not how many intellectual contributions our faculty generate, or in how many peer-reviewed journals (PRJs) their articles appear. What matters is that those contributions make an impact on teaching and practice—and that our pursuit of impact starts with our mission statement.

With these goals in mind, we didn’t want to view refining our mission statement as “checking a box,” but as an opportunity for our stakeholders to engage in discussion and debate. We wanted to ensure that our mission statement not only effectively communicates our school’s identity and purpose, but also serves as a touchstone that drives every strategy, decision, project, and activity. It also must clarify for our faculty what types of impact and research the school values most.

Using feedback from our prior CIR, we set up task forces to re-examine our mission statement. They determined that if our school was to fulfill its purpose to serve undergraduates, graduate students, and the regional Arizona business community, our research needed to maintain a broad focus. To that end, we revised our mission statement. It now states that we will “maintain and nurture an ongoing faculty commitment to a broad array of intellectual contributions that includes pedagogical research, contributions to practice, and discipline-based scholarship.”

A MANY-TIERED SYSTEM
In the past, we have measured the impact of our faculty’s intellectual contributions by using journal quality as a proxy—the more highly regarded the journal, the greater its impact. While we recognize that this isn’t a perfect proxy, it has worked for us. Under this system, we relied on tiers to indicate journal quality. Tier 1, for instance, included the most highly regarded three or four journals in each discipline, while Tier 2a included a broad range of journals in terms of quality. All other PRJs were categorized as Tier 2b and carried less weight during faculty’s annual reviews. Tiers 3 and 4 consisted of work such as conference papers, book chapters, and working papers. If faculty published work in untiered journals, our research group evaluated those journals and then recommended a tier to the dean, who made the final decision.

This system, however, had a problem. Because Tier 2a included journals ranging from high quality to mediocre, many faculty targeted less challenging Tier 2a journals over those in Tier 1. We unintentionally had created a disincentive to target high-quality journals.

Our research group led a semesterlong discussion with faculty to find ways to refine our system. The result was a new system in which we use two comprehensive journal lists: the Association of Business Schools (ABS) Academic Journal Quality Guide (www.charteredabs.org) and the Australian Business Deans Council (ABDC) Journal Quality List (www.abdc.edu.au/pages/abdc-journal-quality-list-2013.html). We now categorize journals as follows:

  • Tier 1 for articles in journals that ABS rates as Grade 4 or 3.
  • Tier 2a for articles in journals that ABS rates as Grade 2 or 1 or that appear on the ABDC list. This tier also includes widely adopted textbooks, feasibility studies with national scope, professional reports, consulting reports, and grants.
  • Tier 2b for articles in all other high- quality peer-reviewed and non-peer-reviewed journals. This tier also includes articles that appear in practitioner journals, chapters and cases in books that promote significantly adopted ideas, and innovative technological activities that have been significantly disseminated.
  • Tier 3 for conference papers, book chapters, and working papers.

Although faculty are encouraged to target outlets on these lists, they also are free to publish in other journals. As before, faculty can petition our research group to have a journal tiered by presenting evidence of impact based on factors such as acceptance rate, circulation, and editorial board quality. However, this new system now encourages faculty to focus on generating a wider range of impactful research because it expands the number of journals in Tier 1 (from two or three to more than 20 for most disciplines). It also recognizes journals in subdisciplines such as entrepreneurship and reference disciplines such as psychology.

Standard 2 recognizes intellectual contributions in three broad categories. Basic or discovery-based research develops new knowledge or methods—for this category, we recognize journals such as the Journal of Management Information Systems and Industrial Marketing Management. Applied or integrative/applied research combines new ideas or advances new methods based on existing knowledge—here, we recognize journals such as Academy of Management Executive and California Management Review. Finally, pedagogical research advances teaching and learning—in this category, we recognize journals such as Decision Sciences Journal of Innovative Education and Advances in Accounting Education.

WE WANTED TO ENSURE THAT OUR MISSION STATEMENT NOT ONLY EFFECTIVELY COMMUNICATES OUR IDENTITY AND PURPOSE, BUT ALSO SERVES AS A TOUCHSTONE THAT DRIVES EVERY STRATEGY, DECISION, PROJECT, AND ACTIVITY.

 

Each business school must decide what mix of research best supports its goals. Then, it must build evaluation and incentive systems, identify expected outcomes, and choose impact indicators that match that mix. Because the Franke College is an undergraduate teaching institution, it is especially important for us to recognize and value a broad range of intellectual work, including case studies, textbooks, and practitioner-oriented articles. We also have created a series of tables that provide us with a more global view of how faculty are achieving impact across the college, and how well their efforts support our mission (see “Collective Impact” sidebar at the end of this page).

ADDED INCENTIVES
Over and above our new rating system, we use three other mechanisms to encourage faculty to target intellectual work with higher impact. The first involves what we call publication “bounties,” which are small financial rewards we give faculty for articles published in Tier 1 and Tier 2a publications. The second is our annual review process, in which we ask faculty to show the impact of their body of work, both current and past. The third is the fact that their evaluations are based, in part, on how well they have met the requirements of their AACSB qualification status.

For example, for faculty to maintain their qualification as scholarly academics or scholarly practitioners, they must demonstrate both the quality and quantity of research outputs over the last four years, a review time frame that was encouraged by a previous dean and approved by our faculty. A scholarly academic, for example, must publish at least one item in Tier 1 or two items in Tier 2a; or one item in Tier 2a and two or more items in Tiers 2b or Tier 3; or two items in Tier 2b and one or more items in Tier 3. As you can see, faculty require fewer publications in highly ranked journals to maintain their qualification status.

At the same time, this system not only encourages higher-impact work, but also recognizes the additional effort required to publish in higher-quality outlets. Our tenure and promotion criteria encourage high-impact research—a promotion to tenured associate professor requires at least one publication in Tier 1 or Tier 2a journals. In addition, we take such publications into consideration for honors such as research awards, research chairs, and professorships.

NARRATIVES OF IMPACT
To communicate to faculty that we encourage a wider range of impactful activities, we provide them with the list of examples of such activities from Appendix A of AACSB’s 2013 accreditation standards. Then, we ask them to prepare narratives that describe the impact of their intellectual contributions. Below we have included slightly edited portions of these narratives, with names changed and some journal and book titles omitted.

The professors who wrote the following two narratives effectively demonstrated the academic impact of their work. The boldface text indicates different types of work that meets that objective:

  • Dr. A’s intellectual works have appeared in several high-impact journals. One article appeared in a 2012 issue of the Journal of Business and Industrial Marketing and received one of the journal’s highest citation counts. In the last four years, he also received a Best Paper Award at the National Conference in Sales Management.
  • Dr. B’s intellectual works have appeared in several high-impact journals, including Decision Sciences and Communications of the ACM. Three of his articles have received more than 100 citations, and five more have been cited more than 50 times. In addition, his 2011 textbook has been widely adopted by schools in the United States, Europe, and Africa. He noted his service on editorial boards as well as his multiple leadership positions at organizations within his discipline.

The two professors who submitted the narratives below effectively demonstrated the impact of their teaching, especially through the boldfaced items:

  • Professor E has authored two successful textbooks. She also wrote a book that has sold more than 5,000 copies. She has co-authored a digital textbook; the global adoption for this digital product exceeds 20,000 units per year. She has developed a series of “on location” business case studies for several editions of an operations management text and has been recognized at international Telly Award competitions. The global adoption of this text exceeds 25,000 units per year.
  • Dr. F has published in The Case Research Journal, as well as in the top business case journal. His cases are sold to faculty and students through a network of distributors. Dr. F currently serves as president of a discipline-based organization, and he has served as program chair for that organization’s 2012 annual meeting.

We use these narratives to demonstrate the impact of our faculty’s work to our CIR team—not as a basis for their annual reviews. But these annual narratives provide three other significant benefits. First, they give faculty frequent opportunities to examine how well their work supports our school’s mission and reflect on their contributions to the college’s impact. Second, they prepare faculty to submit more comprehensive portfolios for promotion or professorship consideration.

Finally, they make clear to faculty that we no longer are simply “counting PRJs.” They show we value contributions that go beyond the traditional academic indicators of journal quality and citation count, such as best paper awards, leadership of academic organizations, textbook authorship and sales, case production and adoption, and high-level service in discipline-based organizations. We recognize faculty for their impact in executive education, business practice, and the community.

CONTINUOUS IMPROVEMENT
After achieving our spring 2014 accreditation under the 2013 standards, we started to work on finding other ways to improve the impact of our research:

  • We now offer modest summer research grants for non-tenure-track professors to encourage them to publish practice-oriented articles. We awarded our first four grants this summer, paying the first half of the grants at the time of approval and the second upon submission of the research to a peer-reviewed journal.
  • Because most of our non-tenure-track faculty lack formal research training, we encourage them to partner with experienced researchers. One of our tenured accounting professors is working with a senior lecturer in computer information systems on research that already has produced a journal article.
  • In August, we hired a well-known researcher as a mentor for the college. She has held a professorship at a major research university and served a term as editor-in-chief of a top-ranked journal. With no teaching responsibilities, she works with non-tenure-track faculty on practice- or teaching-oriented research and with research faculty on projects that target Tier 1 journals. We believe her work will help us improve impact and increase the breadth of our faculty.

As we continue to define what constitutes impact for the Franke College, it will be crucial for us to maintain and clearly communicate to faculty what is required for promotion, tenure, and qualification. It will be equally important to ensure that our faculty’s disciplines are fairly represented in our promotion decisions. We do not want to overlook quality work, whether it’s a publication in the popular press, work in nontraditional disciplines such as legal studies or special areas of economics, or papers that are co-authored with professors from nonbusiness disciplines and published in nonbusiness journals.

As business schools acclimate to the requirements of AACSB’s 2013 accreditation standards, they will face the challenge of demonstrating the impact of their faculty’s intellectual contributions. But they’ll also reap long-term benefits from having a broader, more mission-driven view of how their intellectual work impacts students, schools, disciplines, and communities. Our approach is one of many that will emerge over the next few years. We hope that our case will provide a useful starting point for other schools as they develop their own ways to demonstrate the positive impacts they make on the world.

Craig Van Slyke is the dean of Northern Arizona University’s W.A. Franke College of Business in Flagstaff. Eric Yordy is an associate professor of business law and Stephen Wright is associate dean and professor of politics and international affairs at the Franke College.