Metrics for Mentors

Executive coaches help the University of Tennessee at Knoxville “close the loop” in its assessment of student learning.
Metrics for Mentors

Multiple-choice tests, written assignments, and oral presentations are traditional methods business faculty use to determine whether students have met a course’s learning objectives. But at the University of Tennessee, Knoxville, we wanted to complement those methods with a more hands-on approach.

For “Integrated Process Management,” a course required of all our undergraduate business students in their junior year, we trained executive coaches not only to mentor students, but also to evaluate their performance, share feedback based on those evaluations, and assign grades. By participating in our assessment process, our coaches are in a better position to help students improve their performance and strengthen their business models.

SIMULATION STRUCTURE

Informally called “Marketplace,” after the computer simulation that students use throughout the course, Integrated Process Management is delivered in sections of about 20 students each. In each section, students form teams of five to launch and build simulated computer companies, all while competing with other teams. In the simulation, they name their companies, formulate strategies, design logos, write mission statements, build and distribute computers, hire sales forces, determine employee compensation models, and choose when to expand or contract their reach. Their company must design computers with features that target any of five consumer segments—cost cutters, workhorses, travelers, innovators, and high performance users—in any of 20 geographical markets.

Ten executive coaches, mostly working practitioners or retired executives, mentor and evaluate the teams. They each handle two sections of the course and work with students approximately eight to 12 hours per week. We pay them US$1,800 per section.

The simulation takes place over eight financial quarters, with each quarter lasting approximately one week. After four quarters of play, teams pause for a couple of weeks to reflect on their business performance, tactics, and strategy, as well as prepare detailed business plans for future growth. At the end of this time, they participate in a venture capital fair where they present and defend their business plans to outside investors. And at the end of the course, teams must face these same investors to account for their performance by presenting stockholder reports.

At the end of every quarter, each team gives a 25-minute executive briefing to its coach, who assesses the students’ business acumen. The coach’s role is to challenge students’ analyses, plans, tactics, and projections. When students are having trouble with difficult concepts during the briefings, our coaches give “chalk talks”—brief lectures tailored to the problem the students are facing. Coaches lay out the components of the problem on a whiteboard—whether it’s creating a marketing plan or determining an investment strategy—and help students work through it.

When we created Marketplace nine years ago, our coaches made no formal assessments; they gave students only overall grades for the quality of their team briefings at the end of the course. However, as we refined our assessment process, we discovered new skills we wanted to help students improve. That’s why we decided to train our executive coaches to evaluate students with three different rubrics, one used for the executive briefings, one for the business plans, and one for the stockholder reports. As the coaches complete the rubrics, they evaluate the thought process, skill sets, and critical thinking that went into students’ decisions. The rubrics are shared with the students in advance to set expectations and provide guidance.

Before taking responsibility for mentoring student teams, our executive coaches attend a two-day workshop where they are introduced to the rubric framework. They also work through the simulation themselves and practice the “chalk talk” format. For the first semester of their service, they work side-by-side with a veteran coach to learn how to evaluate students effectively. Finally, they attend “norming” meetings, where coaches and faculty come together to ensure everyone is applying the rubrics consistently.

ASSESSMENT, BEFORE & AFTER

In the first years of the course, we based our objective assessment of students’ management skills on market share and profits. No matter how hard we pushed the students to use data to support their decisions and link their decisions across functions, we did not see the results we wanted. As a result, their executive briefings, business plans, and stockholder reports suffered. Our students could identify areas of necessary investment, but they struggled to apply principles of marginal cost and marginal revenue, decide which investments promised the best return, or determine where to focus resources when capital was low. They could identify problems but didn’t always know what solutions to apply.

We often were dismayed that students could conduct a SWOT (strengths, weaknesses, opportunities, and threats) analysis, but then would ignore it when they adjusted their strategic decisions. They also were challenged when asked to view their firms in an integrated way. They would give each other “high fives” when they increased market share but frown when we asked them about employee morale or the profitability of a segment. In chess, the best players think several steps ahead of the game, but too many of our students could not coordinate moves or anticipate how the board could or should look two or three financial quarters in the future.

Now, each quarter, the executive coaches score each student with a rubric that measures depth of understanding, breadth of understanding, and management by the numbers. We have learned that the very act of measurement changes what we measure. As a result, we have seen our students improve progressively throughout the semester.

Faculty and coaches began to apply even more comprehensive rubrics to the students’ business plans and stockholder reports, using 13 metrics for each. Almost immediately, our students’ business plans got better, which in turn improved their stockholder reports. Eighty percent of our students met our learning objectives in areas such as business knowledge and teamwork, as well as application, critical thinking, and interpersonal skills. However, many still lag in quantitative skills, cross-functional knowledge, and complex decision making.

To address this discrepancy, we recently added 15-minute computerbased micro-simulations to the curriculum to present important but difficult concepts to students. Today, students practice how to improve the value of their businesses via these simulations. They must properly choose the features of given products—like bicycles or computers—to provide the benefits that will most appeal to their consumer base. They complete assignments in which they experiment with ways to reduce inventory and allocate resources. We also are working on creating similar micro-simulations to guide their strategic planning and help them develop an integrated view of the firm.

Finally, we have devised a balanced scorecard to keep the students focused on many more measures than market share and profits, which span all functions of their businesses.

DATA-DRIVEN IMPROVEMENT

We have collected a treasure trove of data that has helped us make significant improvements in our curriculum, boosting student learning and confidence along the way. Our business coaches have played a critical role in our assessment process. They have a ringside seat on everything the students know, do, and think, and their observations provide a wonderful window into the workings of the students’ minds over an extended period of time.

In fact, the use of executive coaching has proven so successful for us that we have started conducting seminars on this approach with approximately 40 faculty each year from countries such as France, Poland, Slovakia, Brazil, Colombia, Egypt, South Africa, Japan, China, Korea, and India. We find that once faculty are well-versed in the pedagogy, they are more comfortable integrating executive coaches into their own courses.

At UT Knoxville, we’ve seen how executive coaching can transform a classroom. Marketplace has become a game-changer for the students. In the beginning, they are like deer in the headlights. Everything is new, and the idea of being accountable for their decisions is frightening. By the end, they exude confidence, because they’ve run simulated businesses and dealt with unrelenting challenges from every direction.

At the beginning and end of the course, we survey them on their confidence in making business decisions. On the last day, most students believe they have what it takes to run a successful business.

With our enhanced approach to assessment, we have the data to prove they really do.

Ernie Cadotte is a professor of marketing and the Fisher Professor of Innovative Learning at the University of Tennessee in Knoxville.