All-in-one or a la carte? In the cloud or locally hosted? Purchased off-the-shelf or designed in-house? So many questions are involved in choosing the right system to support two areas of utmost importance to the business school: faculty contributions and the assessment and assurance of learning. Putting the right system in place can be a high-stakes affair, especially when faculty development, promotion and tenure, student learning, and even accreditation hang in the balance.
The functions of different commercial platforms (often called “software as a service” systems, or SaaS) often overlap. That means that making the right choice can be as much about comfort level as it is about functionality, says Peter Ewell, vice president of the National Center for Higher Education Management Systems in Boulder, Colorado. “These are commercial products, so of course companies will claim a lot. You need to test drive a system with your own data to see if it can do what you need it to do,” says Ewell “What’s most important is how comfortable you feel with the interface and whether it’s compatible with how you work.”
However, choosing a software platform isn’t even a school’s most critical decision in this process. It’s more important for schools to first identify the problems they face and decide what their larger goals are, says Geoff Irvine, CEO of Chalk & Wire of Ridgeway, Ontario, in Canada. “Don’t rush to buy a tool based on a huge shopping list of items,” Irvine advises. “Instead, consider how you want people to feel and act differently a year, three years, or five years from now.”
But how can schools make sure that a system will add value to their programs over the long term? Voices from around the industry advise schools to think of their choice and implementation of a system of data management as a three-part process. First, establish the school’s objectives and the faculty and staff’s desired workflow. Second, promote a culture that sees the ultimate value of the system, through ongoing communication. Finally, design and implement the system, working out the kinks over time with the support and ongoing contributions of faculty.
“Take time to discuss valid measures of performance and develop a common vocabulary—it will make all the difference.”— Geoff Irvine, Chalk & Wire
For most administrators, choosing the right data management system starts with doing their research: speaking to company representatives at conferences, viewing demos, and hearing from current users about their experiences with certain products. But the process actually should start much sooner, with frank talks, goal setting, and consensus building, says Irvine.
It’s important to create a culture that’s ready to take the leap into a new way of doing things—and to contribute meaningfully to that process. “Take care of people first. Move slowly and bring in the early adopters first,” says Irvine. “Software cannot fix people issues—at least not the important ones.”
Once faculty and staff are on board with adopting a new reporting system, schools can take the following next steps:
Meet and discuss. Have early and frequent conversations about the new system, the value of reporting faculty data, and the difference between assessing student learning objectives and assigning grades. “Take time to discuss valid measures of performance and develop a common vocabulary—it will make all the difference,” says Irvine.
Know your goals. From these discussions should emerge a clear picture of the problems your school wants to solve and the goals its faculty want to achieve. With these in mind, you can present vendors with a user-specific scenario that outlines your present and future objectives. Then, ask providers for demonstrations that show how their products would achieve that scenario.
Likewise, says Ewell, avoid being enamored with a vendor’s generic pitch. Most software platforms offer a smorgasbord of interesting features, which may or may not offer value to an individual institution. For that reason, a general pitch or demo is rarely helpful to schools with specific requirements.
Consider compatibility. The business school may want to import registration data from the larger university or compare its students’ performance on a certain measure to that of students in other disciplines. “You want to be aware of what the rest of the institution is using and how compatible your choice will be with that system,” says Ewell.
Don’t wait for “perfect.” Establishing a data management plan is a work in progress—few schools get it 100 percent right on the first try. “Do not wait to have a perfectly developed plan before execution,” says Victoria Guzzo, director of communications for LiveText of LaGrange, Illinois. Once goals are set and the system is in place, continue to adjust and ref ne protocols and features as needed.
Home-Grown or OOTB?
Although an out-of-the-box software system can be the right option for many schools, it isn’t the only solution, particularly in the areas of assessment and assurance of learning. Sometimes a simpler, home-grown method can be just as effective to meet a school’s needs, whether it’s an adaptation of the school’s existing learning management system, the use of Microsoft Excel spreadsheets, or an in-house system designed by the school’s computer science and instructional design staff, says Kathryn Martell, dean of the College of Business at Central Washington University (CWU) in Ellensburg.
“In a survey I conducted a couple of years ago, I asked educators what they used for assessment. Excel was the program they used the most by far,” says Martell. “I think that’s because faculty already have it on their machines, and they’re familiar with it. Making the switch to another software platform involves a large learning curve.”
For instance, a member of Martell’s faculty created a rubric in Blackboard for his course. After students had completed the assessment, the results were downloaded into Excel. “The professor sent me the Excel spreadsheet with the data broken down by question,” says Martell. “I could see easily what areas proved most difficult for students.”
However, the more complicated a school’s data management needs are, the more a higher-order software platform might be a better option. That’s especially true for schools satisfying accreditation standards related to the documentation of faculty qualifications and contributions. “A number of schools use an OOTB solution to manage their faculty contributions, but use something like Excel for assessment,” she says.
Ewell agrees that Excel is a powerful tool, but he sees one downside of its being a school’s only data management tool—it offers no ability to create easy-to-read customized reports with appealing graphics. That could be important to schools that want to show their accomplishments to accreditors and stakeholders.
“To share reports with your dean or other stakeholders, you might want report templates with pre-programmed graphics capability so that you can show them well,” says Ewell. “The payoff of the home-grown system is that you get exactly what you want. But creating such a system is a lot of work—I would exhaust the commercial possibilities first.”
Keep Faculty Up-to-Date
A business school’s faculty do great things—teach, advise, research, write, manage, consult, present, volunteer—and keeping abreast of all of that activity is a challenge. “I spoke with administrators at one school who had to read in the newspaper that a member of their faculty won the Nobel Prize,” says Matt Bartel, CEO of Digital Measures in Milwaukee, Wisconsin. “It’s important for schools to know what their faculty are doing, so that they can better market their programs or demonstrate that their faculty are staying current.”
With the right system and procedures in place, administrators can have their faculty’s collective résumé at their fingertips, whether they want to use the information for accreditation, highlight their faculty’s best work in front of stakeholders, or have a starting point for their strategic planning.
However, for such a system to work well, it often requires faculty to input the data themselves. If a school is moving to a data management platform for the first time, faculty could be overwhelmed by the amount of data they’ll need to enter into the system. Therefore, schools should plan how to ease faculty into the transition.
For instance, schools could take an incremental approach, asking faculty to enter their historical information over a longer period of time. Or, it might make sense to hire staff or grad students to do the initial data entry, and then provide incentives to encourage faculty to keep the information up-to-date.
Says Jon Woodroof, co-founder of SEDONA Systems of Knoxville, Tennessee, “If schools base part of faculty’s annual performance evaluations on the information they have entered and maintained in the system, they will keep the database up-to-date.”
No matter what approach administrators choose, it’s important that they consider the full scope of their needs and adopt a system that can change with those needs, says Curt Naser, CEO of Axiom Education of Shelton, Connecticut, developer of the Mentor platform. “Cobbling different pieces of software together is bound to lead to frustration and limitations on use,” he says. “Look for a system that is pedagogically responsible and that can meet a variety of needs.”
“Schools generally purchase software to solve a specific problem they face,” says Bartel. “We often use the phrase, ‘Enter data one time. Use it many times.’”
With so many tools and approaches available, schools are sure to devise solutions that work well in their cultures—if they’ve done their homework. The ideas in this article and the table of companies available online (see box below, at left) are intended to be a starting point in that process.
To read about the experiences of users of some of the software platforms mentioned in this article, as well as read answers to other questions related to assessment, explore the discussion forum dedicated to “Assessment” at the AACSB Exchange, theexchange.aacsb.edu/Discussions. Access requires institutional log-in information.