Streamlined Decision-Making
in the University Program Assessment Process
Summary
The program assessment process involves countless meetings with faculty, administrators, industry advisors and others. ThinkTank group collaboration software dramatically streamlines the collection of input as well as consensus building and decision making on the committee. The result is a faster, more thorough, less painful process for agreeing on goals, outcomes, measures and program improvement opportunities. The entire process is automatically documented for accreditation officials.
Assurance of Learning is now a high priority
Recent years have seen dramatically increased attention to program assessment in higher education. Institutions of higher education are under increased scrutiny from their accrediting agencies to monitor achievement of program goals and missions, measure the degree to which courses accomplish intended outcomes, and gather evidence of actual student learning. As a result, the assessment (or assurance of learning) process has become critical.
The assurance of learning process also represents an opportunity for an institution to re-examine past assumptions and create a shared culture to truly enhancing higher learning. “Assessment requires making expectations and standards for quality explicit and public; systematically gathering evidence on how well performance matches those expectations and standards; analyzing and interpreting the evidence; and using the resulting information to document, explain, and improve performance.” (Thomas A. Angelo, AAHE Bulletin, April 1995, p.11). The assessment process typically contains these steps:
Collaboration is a necessity
Each of these steps requires a considerable amount of collaboration, consensus-building and decision making. The process typically involves a committee made up of faculty, staff and administrators from the department as well as key involvement from industry advisors, alumni, students and peers from other departments or universities. The outside feedback from these groups is critical to creating a well-grounded program assessment, and it’s often a key component that the accrediting agencies consider.
The challenges
Yet the process to gather this input remains difficult and time consuming. For example, asking industry leaders to visit campus for a focus group session imposes a significant time burden. Usually, the numbers of invited guests has to be restricted in order to respect their time commitment and insure that each individual has an adequate opportunity to be heard. Another option is to leverage a web survey. This offers the advantage of being easy to distribute to a large number of individuals, but surveys are static by nature. Once the questions are asked, there is very little opportunity for adjustment or follow up. A dynamic tool that enables easy capturing of feedback and that can easily be adjusted “on the fly” based on the flow of the discussion, would clearly be advantageous.
For the members of the program assessment team, there may be dozens of meetings required before the group has completed its work. Each of those meetings may bring another set of issues:
- Getting the team physically together to resolve issues is a challenge when schedules are tight.
- Dominant personalities on the team may monopolize the group’s time and attention.
- Inter-personal relations and politics can discourage the free flow of ideas
- People respond to the person, rather than to what is said
- Group’s can easily get sidetracked by issues not related to the task at hand.
- Consensus is hard to achieve as the group brings varying perspectives to the team’s goals.
- Minutes of the meeting, if any, are issued late or are inaccurate.
The solution
ThinkTankTM by GroupSystems can help a University with its assessment process in two key ways: it’s an unparalleled software tool for capturing feedback from stakeholders such as industry groups and alumni AND it’s un-matched in its ability to help a faculty team build consensus and make decisions. These capabilities make ThinkTankTM the ideal tool for accelerating the program assessment process. Academic research spanning over 150 studies and 4,000 projects has shown that software like ThinkTankTM helps cut meeting times by more than 50 percent, while improving the quality of the output and the buy-in of the participants.[1]
How ThinkTank works:
ThinkTankTM sessions are visibly different from conventional face to face meetings in that participants use their PCs to record questions, opinions, ideas, facts, suggestions or votes. There is no software installed on any of the PCs; users simply log into the ThinkTank web 2.0 software over the internet.
ThinkTankTM sessions differ from virtual meetings using web conferencing in that participants don’t merely passively view a PowerPoint broadcast. ThinkTankTM sessions are highly interactive and stimulating. The software is typically in use for 30-50% of the meeting and is supplemented with an audio conference. It assists in the communication process but does not dominate the meeting. In a real-time session, there is still plenty of scope for discussion, conversation, banter and humor.
Capturing input: fast and fun
A ThinkTankTM session is a series of problem solving activities corresponding to the Agenda that has been created by the Session Leader, or chairperson. During each session, statements, questions, or subject headings are sent to each participant’s PC asking for comments, answers or votes. For example:
- What are the goals of our program?
- What should our students know, do or value when they have completed our program?
- How can we measure student attainment of these outcomes?
- Given the results of assessment, what improvements should we make to our program?
The participants type their input into ThinkTankTM, check it and send it, like an instant message, to all the other participants. The input can be anonymous so no one knows who makes a particular comment. This helps focus on the quality of the ideas presented and encourages frank discussion of sensitive topics.
Typing skills are not important. In fact, the slow typist, by keeping input short in order to reduce the typing required, can communicate as effectively as the expert typist who enters long sentences at high speed.
Participants read all the input and add further comments as new thoughts are triggered. Thus, the subject is quickly explored and recorded. The enthusiasm of the participants to communicate what they know and to give ideas sometimes results in lot of input. The input can be reduced by categorizing or merging similar ideas, or by voting on the most important ideas.
At the end of the meeting, full documentation of the session can be generated in Microsoft Word, Excel or HTML formats. The entire content of the meeting, all ideas and comments, is captured verbatim from the participants and can be shared with anyone. This output becomes the organizational memory for the group allowing them to “remember” WHY decisions were made not just WHAT decisions were made.
ThinkTank removes the geographical and time constraints of traditional meetings. Since ThinkTankTM runs on the web, participants log in from anywhere and at any time. A session can be left open indefinitely so that participants can join in at their convenience or revisit when they have new ideas.
Assessments at California State University – Chico
“CSU Chico has been using GroupSystems products since 1992. We love the way meetings are more productive and we can get to the heart of issues without offending anyone. For our assurance of learning program we have used ThinkTankTM to brainstorm interventions given the results of our course embedded assessment activity and develop/validate the rubrics used in assessment activities. In both cases we use the web interface to get feedback from our stakeholder groups including our Industry Advisory Board. This has made our Business Information Systems (BIS) program current and relevant to our graduates. As a result, nearly 100% of our graduates get jobs related to their major and our assessment efforts are documented in a readily available format for the re-accreditation team. (We used prior products from Group Systems to develop, refine and validate our learning goals, student learning outcomes and comprehensive skill set needed by our BIS graduates.)” – Gail Corbitt, Department Chair, Accounting and Management Information Systems, California State University @ Chico
Why ThinkTankTM sessions are so effective
ThinkTankTM sessions have many features that make them so effective in driving toward solutions:
| Old School | ThinkTankTM |
| Hold a “whiteboard” session where one person speaks at a time, domineering personalities drone on and on… | Supplement discussion with simultaneous feedback by all stakeholders. Everyone gets a full say – ideas and comments are collected more quickly. |
| Conference call or web conference for virtual participants. Calls miss much of the context; web conferencing shares Powerpoints, but doesn’t collect feedback. | Use ThinkTankTM to get their active input. Participants see and add to all ideas/comments in REAL TIME. |
| People respond to personalities, rather than to what is said. | Enable anonymous input for FRANK discussions that focus on the quality of the idea. |
| Allow dominant personalities to dictate final outcomes of learning goals, outcomes, assessment measures, etc. | Instantly vote on topics. Measure team consensus automatically. |
| Rely on static surveys to get feedback from industry, alumni and other stakeholders | Supplement ThinkTank’s robust Survey functionality with ThinkTank’s real-time, interactive tools for gathering input |
| Transcribe all the meeting notes, hope you got the meaning correct, send out the document weeks later. | Automatically generate verbatim, detailed notes in MS Word, Excel or HTML formats. Send it to all participants in minutes. |
Structured agendas: keep the team focused
ThinkTankTM sessions have a structure that is much clearer than that of a conventional meeting. The structure is contained in the agenda of the meeting and the agenda is what the leader uses to guide the team through a series of collaboration activities.
The agenda is often prepared before the meeting, either specifically for that meeting or copied from a previous similar meeting. GroupSystems provides a series of free templates for common meetings including one for the Assessment Process. However, the agenda will often change during the meeting; to allow for this ThinkTankTM makes it easy for the leader to add agenda items so that the participants can:
- Exchange information and opinions on a new subject or a new angle that has spontaneously arisen in the meeting
- Vote to focus a discussion
- Liven up their creative juices with a suitable exercise.
Shared input: create a common focal point
Comments, facts, and opinions are shared instantly with all participants in the software. This avoids duplication of input and gives participants the opportunity to build on other’s input, for example “I agree/I don’t agree with…”
Parallel input: enable multi-tasking
Participants type in their contributions at the same time, a much quicker process than waiting for each person to speak in turn. Dominating personalities get their say, but so does everyone else in the session. Again ideas, regardless of the origin, are evaluated on their on merit.
Keyboard input: encourage deliberate thought
Recording one’s thoughts via the keyboard allows participants to compose their thoughts to be sure they have typed what they really mean before sending their input to others. Complicated thoughts and technical terms are more easily shared when typed and there is often less misunderstanding of a concept when it is written. Keyboard input is often an equalizer among people from different cultures and languages as well; they can respond in writing better than they express themselves orally.
Anonymous input: frank discussions
Distinctly different from a conventional meeting, ThinkTankTM sessions can be anonymous. As a result, each contribution is evaluated on its own merits, not who presented it. When input is anonymous, participants are free to enter ideas or proposals that they have not yet fully thought out or that are based on uncertain information. In a conventional meeting it can be embarrassing if something you propose is less than ecstatically received, and this risk of embarrassment often smothers potentially valuable ideas. In a ThinkTankTM session there is no loss of face if a half-baked idea is blown out of the water by other participants. Research shows that anonymity also diffuses confrontation. Participants find it easier to respond to anonymous criticism than to find the same criticism delivered eyeball to eyeball.
Of course, there are many times when anonymity is not appropriate, when it is important to know who said what. In this case, the meeting owner can turn on name tags and each person’s contributions are automatically identified when entered.
Voting: gauge consensus
ThinkTank’s voting tools can be used to record formal decisions after a discussion, but they also are used informally to help to focus discussion on the right topics. For example, if a quick vote shows 15-1 on a topic before a discussion starts, then perhaps the group doesn’t need to spend too much time on that issue.
Multi-criteria voting opens up new horizons in group decision making. Participants can assess each issue against each criterion and ThinkTankTM summarizes the votes. Participants can see exactly where there is agreement and disagreement for each issue for each criterion. Typical criterion might be “fit with University goals” “importance to industry” “effort to implement,” etc. Thus, levels of consensus can be measured in ways that is quite impossible in conventional meetings.
Large Meetings: maintain productivity
It is often considered in a conventional meeting that about eight people is the maximum effective number of participants. This is based on giving each person a “reasonable” amount of time to talk. In contrast, ThinkTankTM Sessions can be effective with 100 participants. This fundamentally changes how meetings are managed. You can now get all the people involved together, get their ideas, develop a deliverable and commit to it at one meeting.
Meeting minutes: automate documention
In a conventional meeting it is generally impossible to take accurate notes of everything that has been said. By contrast, in a ThinkTankTM session all the facts, opinions, ideas and votes are captured instantly. There is no delay and dependency on hastily scrawled notes. The record can be printed, emailed, or published to the web. The output is available in Word, Excel or HTML formats. Nothing is more impressive to an accrediting agency than detailed documentation on each of the industry validation sessions that were held during the assessment process.
Virtual and Different-Time Meetings
Today, everyone is so busy that even when faculty members are on the same campus, it’s frequently very difficult to pull everyone together face to face. The alternative is usually to have a conference call, perhaps supplemented with some form of web conferencing. The problem is that conference calls are a notoriously difficult forum for encouraging active participation of the group. It’s simply too easy for participants to check email, put the call on mute, or otherwise disengage. Web conferencing helps if one speaker is “broadcasting” a Powerpoint, but it doesn’t have much to offer when the goal is to capture feedback, build consensus and make decisions with the group. ThinkTankTM supplements the conference call with a real-time forum for soliciting ideas and organizing them. Shy people and non-native tongue speakers, who might otherwise be reluctant to speak up are given a easy opportunity to express themselves with ThinkTankTM.
For many aspects of a meeting, it may not be necessary that people participate at the same time. ThinkTankTM sessions can be left open for days or weeks while people join in from anywhere to make comments and submit proposals.
In short, ThinkTankTM can help streamline the program assessment process like no other tool or methodology. Its powerful features will help your team capture input, build consensus and make decisions in less than half the time of traditional approaches and with better results. Call today to get started with a free trial of ThinkTank!
[1] Jay Nunamaker and others. Lessons from a Dozen Years of Group Support Systems Research: A Discussion of Lab and Field Findings. (Journal of Management Information Systems, Winter 1996-97. Vol. 13, No. 3, pp 163-207)
