Accountability

[x-posted on The Long Eighteenth]

Accountability came up a lot in the course, as it does in thinking about assessment in general.  Even the quickest search on outcomes assessment in The Chronicle or Inside Higher Education will take you to debates over accountability more than any other topic.  I don’t generally write or speak about accountability, and most of the time I try not to think about it.  During the Bush years, one of my colleagues decided to pretend that Martin Sheen was the president.  That’s how I feel about accountability.

But I also recognize that you can’t get anywhere in conversations about student learning without facing this demon.  So I have been thinking about it more than usual lately, and this is what I have concluded, at least for now.

If you are an academic, it is not possible, without the gravest hypocrisy, to be opposed to accountability.  The academic project is built on accountability and has accountability at its core.

If you are a scientist, you are accountable for accurately reporting your results.  You can’t just say you performed experiments that you did not perform; you can’t just make up data.  When you do, it’s a scandal—or at least it should be, if you are caught.  Andrew Wakefield’s falsification of data that allowed him to claim a link between immunization and autism has become one such infamous case.  While the vetting process for research sometimes fails, it nevertheless serves as a (albeit imperfect) system of accountability.

If I am writing an essay on Clarissa for publication in a refereed journal, I will be held accountable for showing my familiarity with all other arguments on Clarissa that resemble mine.  Of course, if I forget to cite Terry Castle, no child will, as a result, contract a disease.  Nevertheless, most of us accept this system of accountability to limit the proliferation of essays on Clarissa that say more or less the same thing.  This accountability system has its flaws, but so far a good alternative to peer review (whether blind, open, or crowdsourced) has yet to emerge.  Digital work has brought more attention lately to this particular accountability problem, but the impulse, as far as I can tell, has been to try to figure out how to implement some kind of peer review process for digital work as well. (See, for example, 18^th Connect.)

The accountability in research, however, is only the tip of the iceberg.  We are accountable for office hours, turning in grades on time, generating credit hours, accommodating students with disabilities, affirmative action, withholding any curiosity about the personal lives of job candidates,  showing up for class, serving on committees, ordering books, grading without bias, submitting early warning grades for athletes, showing up for department meetings, holding classes that being at a given time that end at a given time, returning library books, using institutional equipment for institutional purposes.  We areevaluated every semester by the students we teach, every year by our departmental colleagues, and more or less constantly by presses, journals, and reviewers in the publication process. 

Institutions of higher education themselves are accountable for graduation rates, issuing credentials, vetting applicants, providing enough “seats” (as they say), counting the credit hours, ensuring a particular distribution of the credit hours, calculating GPAs, constructing a calendar, and reporting to their accrediting agencies.

But as Robert Barr and John Tagg pointed out more than twenty years ago, we have generally not been held accountable for student learning.  If we were, they propose, we could shed many other forms of accountability. And yet, this possibility (accountability for student learning) often comes across as outrageous, while many of those other forms do not.

Accountability, then, has a kind of ideological force.  The institutional context in which we operate renders most of our accountability invisible.  We don’t think about credit hours as a form of accountability; most of us don’t think much about them at all.  But as Barr and Tagg point out, the accumulation of credit hours tells you nothing about what student have learned. The language of “seats,” perhaps, reveals most clearly which end of the student occupies institutional attention.   The shock over accountability for student learning only suggests how far outside of the dominant ideology of accountability this function has remained.

Nevertheless, Barr and Tagg’s argument carries even more weight today.  How important are office hours, for example, when students can email with a question or to make an appointment?  I am far from the first to notice that new technologies offer the potential to change the way learning happens.  Unfortunately, though, too many of these debates focus on the wrong issues, such as whether we are “for” technology or “against” it, when we should be thinking instead is how best to use what we have to support learning.  Cathy Davidson has recently received a lot of attention, both positive and negative, for arguing that research on the brain suggests the advantages of integrating technology into learning.  Many have entered into this debate, which I won’t get into here.  I will only note, though, that a focus on cognitive science alone interestingly avoids the accountability issue by settling in advance what improves learning rather than focusing on empirical strategies that try to figure out whether or not what you’re doing is actually working.  Thus we can argue back and forth about whether student work improves more with blog posts or papers (I use both, so, as Woody Allen remarked about bisexuality, I double my chances).  But we could, alternatively, carry on this debate in the context of research (large-scale assessments), or of home-grown micro-assessments that aim to figure out which strategy works best in particular cases and for particular instructors.

So in conclusion, accountability is not the problem.  The problem is that we haven’t given enough though to which forms of accountability we would embrace (not fabricating results, accommodating students with disabilities), which are empty exercises in accountability for its own sake, which would inadvertently undermine research and learning, which are seemingly intended to undermine research and learning (hello Texas), which have been rendered vestigial by new technologies, and which should be getting more of our attention.

UMD’s Jewish Studies Program as a Case Study

In case anyone is curious, I posted the .ppt for my presentation from Friday (http://assessmentforlearning101.files.wordpress.com/2012/01/putting-loa-into-practice.pdf).

I especially look forward to seeing the .ppts of the presentations I missed because of the orientation I had to run.

Thanks for a great class!

The Grammar of Learning

Thinking about the problem of improving learning outcomes reminds me of one of the first things I learned as an undergraduate linguistics major: the difference between prescriptive and descriptive grammars.

A prescriptive grammar is a set of rules for language that suggest an ideal, in other words how the language should be. A descriptive grammar describes the way a language exists in nature, in other words the rules are discovered via observation. Between each different language, dialect, indeed between each speaker of one language, there are variations that can be observed; however, no language is any better or worse than another in its ability to express ideas to other speakers. Whereas a prescriptive grammar often contains rules that do not reflect the way that a language functions in nature, a descriptive grammar observes linguistic variations to discover new things about the phenomenon of language. This new knowledge can have practical applications. For example, phonologists and phoneticians can apply what they learn about language to create scientifically validated instruments for speech pathologists to treat speech disorders.

The current trend for institutions of higher learning seems to be one that mirrors a prescriptive grammar paradigm. To take what is occurring in the UK as an example, governmental authorities have decided that institutions of higher learning should focus on disciplines that are good for business (Holquist, 75). Humanities, on the other hand, are perceived to be so unimportant that the Shanghai Jiao Tong world ranking of Universities doesn’t even take humanities programs into consideration as criteria for evaluation (Holquist, 76).

However, those fighting for better learning outcomes seem to be working in the descriptive grammar paradigm. Barr and Tagg state that “the Learning Paradigm does not limit institutions to a single means for empowering students to learn; within its framework, effective learning technologies are continually identified, developed, tested, implemented, and assessed against one another” (4). In other words, the process of learning is observed in order to determine how the process of learning can be improved.

I think my metaphor might be a bit shaky, likely because I have only had a few weeks to learn about the field of learning outcomes. Nonetheless, it seems to me that learning processes can likely be described as a set of rules because by observing them in action we seem to be coming up with ways to improve them.

Emphasizing Value of Starting Small in Barr and Tagg

Although I am persuaded by Barr and Tagg’s arguments as to why we should shift from an Instruction Paradigm to a Learning paradigm, and think that their suggestions for making this shift are sensible, I find myself concerned by the way that Barr and Tagg have constructed their argument in From Teaching to Learning – A New Paradigm for Undergraduate Education.

The take home message contained in the closing anecdote of Buckminster Fuller’s observation that in order “to change the course of a great ship” one “should apply force to the trim-tab” because “a very small force will turn it left, thus moving the big rudder to the right, and the huge ship to the left” is what Barr and Tagg are arguing throughout their article (17). Indeed, they are trying to underscore what they said earlier on the page: “The change that is required to address today’s challenges is not vast or difficult or expensive. It is a small thing. But it is a small change that changes everything. Simply ask, how would we do things differently if we put learning first? Then do it” (17). However, when I read the article, and the responses of my fellow students to it, I can’t help but think that this message is lost in the overwhelming prospect that “Ultimately, changing paradigms means doing everything differently” even if they have previously stated that the shift in paradigms will be slow, and that “changes – even small ones – can create leverage for larger change in the future” (15).

Additionally, I see an issue with the adversarial tone in statements such as “The reason [few elements of the learning paradigm have been widely adopted] is that they have been applied piecemeal within the structures of a dominant paradigm that rejects or distorts them. Indeed, for two decades the response to calls for reform from national commissions and task forces generally has been an attempt to address the issues within the framework of the Instruction Paradigm” (2). The problem in my mind is that if the current Instructional Paradigm is the institution in which we as teachers function, then to demonize it creates an adversarial relationship with the institution that we are trying to change. To take a page from Gawande’s Better, positive deviance creates change by asking the culture in which dysfunction is occurring to look at itself. This culture acts as a change agent that can determine solutions to this dysfunction via the open exchange of ideas that positively deviate from the norm. These solutions can be more effectively implemented because they were identified within the culture and, as such, there are no cultural barriers to cross. I notice that they do try to alleviate this concern on page 15 by explaining how each side of the struggle will react, and providing an example of how paradigms do in fact shift; however, the adversarial tone persists in the images of tomatoes and jeers being thrown at paradigm shifters. I see value in warning that change will be hard, but doesn’t this undermine Barr and Tagg’s goals by scaring potential change agents?

Not everyone wants to be a revolutionary. Not even in academia. As such, I feel that Barr and Tagg would be better served by emphasizing in a more systematic manner what they are already saying: small acts of positive deviance can lead to great change. It seems to me that if the non-revolutionary academic were assured from the outset that the breaking down of institutional barriers necessary to shift from the Instructional Paradigm to the Learning Paradigm is something that they can contribute to with small actions, then they might be more willing to become engaged. Basically, I think statements like this should come in the beginning of the article: “First, you begin by speaking. You begin to speak within the new paradigm” (15).

Thoughts inspired by Holquist

Thus far, I’ve found Holquist’s article in the Rosenthal edition to be by far the most similar to my (still daily shifting) to my own thoughts and questions about learning outcomes assessment. Even while affirming the likely improvements to student learning, Holquist urges caution in their implementation, even at the disciplinary level. I’ll be talking about Holquist in more detail tomorrow, so I’ll spare a review of the reading here, and get straight to my thoughts.

I think we in the humanities have a tendency to relativize: to say, in the interest of politeness or fairness, that sure, math and chemistry departments are right that their disciplines aren’t necessarily quantifiable too. Perhaps they also resist assessment, so, to make the argument that we in the humanities are somehow any more resistant to assessment is a fallacy—one that only further reinforces the notion that we are useless precisely BECAUSE we can’t be measured. And yet, I can’t help thinking of last Friday’s class, when Romina (sp?) brought up the point that the only reason she even knew about our class was through a colleague, because in her experience (and please do correct me if I’m mischaracterizing what you said), the idea of teaching chem students how to teach, much less how to assess teaching, is not one that is privileged in her discipline. Perhaps STEM fields don’t agonize over this process as much as we do because it IS easier to measure what their students learn. What Holquist asks is, what if measuring the thing robs the thing of what makes it worthwhile)? In other words, to put it in terms of quantifiable science, is there a way in which LOA can mirror the process of the observer effect? http://en.wikipedia.org/wiki/Observer_effect_(physics)

I ask this question because, maybe there isn’t a way to measure how well a given graduating class of English majors understands Hamlet, but that perhaps there’s a way to turn that “ineffability” into a strength (I relish the idea of using a corporate interview strategy of casting weaknesses as strengths in order to resist the greater infiltration of corporate models in the university). This goes back to the Menand quote used by Professor Rosenthal in her essay, ““How did humanists get painted into a cultural corner such that everything that a social or  natural scientists sas that is counterintuitive receives public genuflection, but literature professors are expected to do nothing but reaffirm common sense?”

I think Professor Rosenthal’s advocacy for discipline based assessment is reasonable and vital, especially for how it could affect student learning. But when those measurements come in, I don’t see a scenario in which institutions or accrediting agencies or state regents or BOG’s or legislatures will ignore them. And that’s when I wonder how tenable assessment at only the disciplinary level is, long-term.

I’ll end with an example that’s close to home. I admit that I’m not familiar with all of the details, but as far as I know, the facts are these: someone, somewhere, decided that students who were placing out of English 101, First Year Composition, due to SAT scores did not, in fact, have skills already equivalent to those students who are required to take 101. To make this determination, some form of assessment must have been undertaken. Someone, somewhere (I have a hunch that former Provost Farvardin was involved) decided that, given the results, more students would need to take 101. I also suspect that this assessment was carried out, at least in part, by representatives from the English department. When this assessment was carried out, I wonder if those in charge recognized that the result would be many, many more sections of 101. Obviously teaching 101, especially more 101, is not something faculty want to do, yet I’d bet they’d all agree with the idea that writing proficiency is central to an undergraduate education. But still, I wonder about their opinion of having more sections added to their department of classes they don’t want to teach—did those who conducted the assessment foresee this? Did they have say in it? Will they be comfortable with the unavoidable necessity of hiring more non-tenure track faculty to teach it? With the likely increased class sizes?  I only want to use this example (filled with assumptions as it is) as one possible unforeseen consequence, though not in this case necessarily negative (as it hasn’t yet been implemented) of an assessment.

 

Feel free to tell me I’m being paranoid. I see so much good, so much that IS truly subversive in LOA (and Michelle’s post about specific ways to measure (though I might be more inclined to think of it as tracking) student engagement with literature is fascinating), that I often feel that way myself.

Evolving into a more educated society.

Somehow, being the only science major, I had to somehow bring in the topic of evolution into the discussion.

A friend recently posted this article, “Why do so many people have trouble believing in Evolution?” by Marcelo Gleiser. It hadn’t caught my attention until reading Louis Menand’s article, that I previously blogged about, “Why we have college”.  In this article one of the theories, which he called the democratic one, stated that ideally a college education “gets everyone on the same page.” College therefore is a way of “producing a society of like-minded grown-ups.”  If we look at the data given by Menand’s article it would seem to indicate that this isn’t exactly what college is doing, at least on the topic of evolution or to the degree I would of expected. He reported the findings of a Gallup Poll taken on the eve of Charles Darwin’s 200^th birthday (which I celebrated by going to a CBMG ice cream social).  The poll showed that 39 percent of Americans believed in evolution, 25 percent didn’t believe and 36 percent had no opinion. These percentages were further analyzed as follows:

“The same poll correlated belief in evolution with educational level: 21 percent of people with a high school education or less believed in evolution. That number rose to 41 percent for people with some college attendance, 53 percent for college graduates, and 74 percent for people with a postgraduate education.”http://www.npr.org/blogs/13.7/2012/01/18/145338804/why-do-so-many-have-trouble-with-evolution

The more education an individual has, the more likely they are to believe in the theory of evolution. I want to be like Emeril and say “BAM!” evidence that education is leading to a more like minded society. Yet as a scientist, who does believe in evolution and also is religious, having only 39 percent of the population believe in evolution seems tiny. And knowing that out of college graduates only  53 percent believed it down right shocking. I am now a believer in theory everyone needs an college education! If it gives them a chance to understand better the workings of nature, regardless of individual religious beliefs, it is a worthwhile concept. But I worry that even if they get an education, they are not understanding these theories if only 53 percent can state they believe in a basic scientific principle.

That so few believe in evolution, in our society is a source of worry.  I always have more questions than answers, so I wonder how can colleges create individuals who are like minded if our society is so diverse in its beliefs? How should we as educators and future education decision makers deal with the controversial issues that society does not agree upon, if our aim is to produce educated citizens?

I would think that perhaps we don’t want like minded individuals but, informed individuals, who know the facts, can analyze them and make their own decisions. These decisions then are individual, whether the professor or educator agrees with them or not, their job is to allow students to know how to think for themselves. Yet my own personal knowledge, leads me to think that if we were doing a good job of educating students, then more than 53 percent of college graduates would believe in a basic tenant of science.  I am torn.

If I’m sorted, I want to be Gryffindor… I mean intelligent… there is a hat, right?

“A Critic at Large Live and Learn: Why we have college” by Louis Menand was a wonderful read. It summarized many of the philosophies on the purpose of college, and some of its current debated failings, that I had previously not considered.  Menand presented the main theories on why we have college as either 1) meritocratic 2) democratic or 3) vocational.

I propose that every stake holder in education has a different view on the purpose of a college education.  Perception is based viewpoint and the view from the high steeples of a professor, differ from the familial point of view a parent, and even more to the invested or non-invested student. Which point of view is the most important to answering why we have college? Debatable but I will focus on the student.

As a student I had wished for an easy answer. I had hoped that by reading this article I would feel a sense of closure, I would learn what I need to about the reasons for college, and there was a right answer to the question of why we have colleges. But much like my own college education, it opened up more questions by the sheer act of learning, it left me feeling like there was so much more to know and that perhaps there was no right answer. My liberal arts education in science, like this article, got me to think but didn’t end up getting me to a particular conclusion. Now being that I started out as a vocational student, trailing the path blazed by my mother to earn a  living saving lives, or at least prescribing medications, this is not what I had at first envisioned college as being or the final service provided.

I would argue, some from my own experience and exposure to students as a TA, that students want to be sorted and that is what college means to them! And I believe this may be what we truly want college to be as a society, the sorting hat of life, not just intelligence. We want the hat to give us the right answer, we belong where we want to belong and provide a readily available house table (job/profession) that will accept us once the sorting has finished. All medical students just want college to give them the knowledge and grades to get into their chosen house. They want the classes to reinforce this goal. Others just have no idea, and want the hat, I mean college to do all the work in leading them to this job. In the JK Rawlings stories where the sorting hat would sing a song and place students where they belong, so do we hope that our college would do the same except we would want a nice paying job to cover our student loans.

Yet how disappointed are students when they find out this is not the case. They will be sorted, in a way, but it will be by being given grades for their performance. College will not sort them into jobs, that is they will not be ready for any specific job but perhaps have the option of many job possibilities and titles. That is if they do the work to find the jobs and network. Is it therefore so strange that students are spending more time socializing as the article states, if really college isn’t going to do what they want it to?  Or is it they don’t see how perhaps the point is to help them learn so they can do the sorting themselves?

I don’t know, but I’m OK with that answer, and will be open to possible varying opinions which I will evaluate and analyze. Perhaps the sorting processes has already occurred with me, just not the way  I would of imagined.

“For I’m the Hogwarts Sorting Hat 
And I can cap them all. 
There’s nothing hidden in your head 
The Sorting Hat can’t see, 
So try me on and I will tell you 
Where you ought to be.”

 –Harry Potter and the Sorcerer’s Stone, JK Rawling

 

University Heaven.

For my last posting, I actually started thinking about if an institution or school existed that functioned on a learning outcomes assessment model.  I was sure of it, but until this point had not sought out examples.  I did a search and randomly selected Capella University – an online degree-granting institution.

Here’s a little overview from a report done by NILOA:

http://www.learningoutcomeassessment.org/documents/Capella%20University.pdf

“Capella University, founded in 1993 and headquartered in Minneapolis, is an accredited, fully online university that provides degree opportunities for working adults. Nearly 80 percent of Capella’s students, called learners at Capella, are enrolled in graduate degree programs. Capella serves over 38,000 learners and has almost 1,300 faculty members…Capella University has received numerous awards for their innovative teaching and learning environment…National Institute for Learning Outcomes Assessment (NILOA) selected Capella University for a case study due to its systematic, embedded student learning outcomes assessment process; its administrative support and vision of what assessment can do for individual learners; its transparency efforts such as Capella Results, which publicizes assessment results, and its help in developing Transparency By Design; and its use of assessment results to enhance learner success.”

From their website at Capella:

“As an accredited university, Capella University takes online education to a higher level. We believe that real, life-changing learning happens in an environment of academic excellence, challenge, and personal connection.”

I was actually quite shocked and surprised at some of their initiatives and features.  They boast an action project which allows them to integrate data (in real time) into their decision making processes.  They also evaluate students based on surveys and progress reports, which then translates into a capstone type of project.

On the Capella University website, there is a separate page dedicated to creating and measuring learning outcomes. See picture below.  I think that the model provides a great outline that parallels some of the theories we have been reading in class, especially Suskie.  Theoretically and visually, this model seems to make sense; it also keeps the student at the forefront of his or her education.  http://www.capellaresults.com/index.asp

 

 

However, then the harsh realities set in.  In the report by NILOA, they note:

“Capella’s noteworthy achievements in assessing student learning have not come without challenges, and the institution recognizes there is more to accomplish. Since the impetus to become outcomes- based came from the executive leadership, engaging faculty in this effort and fostering understanding among them of what the institution was attempting to implement proved difficult at first. Further, building the necessary technological infrastructure as well as the institutional processes and protocols to support the development and implementation of assessment took several false starts and brought occasional realizations ‘that we needed to start over and begin the process again from scratch.’”

What do you mean start from scratch?  At whose cost? I completely understand the need to massage and finesse structures and regulations, but how do we justify when it happens?  What if I happen to enter the University during this first attempt and I spend all four years under the blanket of false starts?  When do we actually test and implement these plans?

And how effective is this program?  I did some basic research and found on USNEWS Online that “the average freshman retention rate, an indicator of student satisfaction, is 24.5 percent.”  And a large number of the postings seem to be in this nature:

“Capella (school of Business and Technology) is more interested and very efficient in sending you a quarterly bill than they are in partnering with you in the learning process to actually help you complete an advanced degree program.”

“I fist enrolled at Capella in 2007. It is 2011. The school prides itself on being an online higher educational program that provides its adult learners with the opportunity to maintain daily responsibilites, work and family obligations. HOWEVER,if a student chooses to enroll in the program that requires both a residency and internship, the school ACTUALLY advises its students quit their full-time job in order to complete their program….This school is not adult friendly and infact they bambozzle students into coming to their school when they are not supportive at all. Yes, they are a cash cow. Graduate students receive no assistance on landing internships or understanding with the position they place students in. This is disheartening and I can see why this school has been under scruntiny and have students take legal action. It is my hope to graduate soon and leave this awful place. I would never recommend this school to anyone.”

It is definitely not an attempt to focus specifically on this University, but it begs the question: how do we gauge the success of these learning outcomes-based institutions?

Let the Sun Shine In

In “From Teaching to Learning – A New Paradigm for Undergraduate Education” Robert B. Barr and John Tagg discuss the need for institutions to embrace a Learning Paradigm over the traditional Instruction Paradigm that tends to dominate the higher education system across the country (and even the world).  The chart at the end of the article nicely summarizes the main differences between the two paradigms in an easy-to-visualize manner.  While the purpose of the Instruction Paradigm is to transfer knowledge, the Learning Paradigm seeks to create powerful learning environments.  While the Instruction Paradigm establishes structures that tightly control learning and assessment (an implication that learning is hindered), the Learning Paradigm values the holistic – the sum of all of its parts.

 Despite this somewhat clear comparison of the two paradigms, it is the actual practicality or implementation of a 100% Learning Paradigm that raises questions or concerns.  It is almost as if Barr and Tagg paint a rather utopian picture of a system based on very theoretical and hypothetical situations, sometimes belittling important factors such as bureaucratic impediments, budget concerns (that come from way above the educational institutions themselves), and most importantly, student and instructor realities.

“Under it [Instructional Paradigm], colleges have created complex structures to provide for the activity of teaching conceived primarily as delivering 50-minute lectures-the mission of a college is to deliver instruction.” (1)  It seems rather utopian to think or hope that focusing a course on sole discussions would allow students to incorporate the same amount of information that may be required to meet minimum standards for a particular field.  For example, I am teaching theatre history – from the greeks through Shakespeare.  Unfortunately, it is a 50 minute lecture a couple of days a week.  If I based the class on simple discussions, it seems that it becomes much more difficult to ensure that the material you need to convey will be covered in a classroom with high tangent potential. 

 “In the Learning Paradigm, on the other hand, a college’s purpose is not to transfer knowledge but to create environments and experiences that bring students to discover and construct knowledge for themselves, to make students members of communities of learners that make discoveries and solve problems. The college aims, in fact, to create a series of ever more powerful learning environments.” (4)

Can that not be accomplished through a combination of lecture/discussion questions?  I would think one would be able to combine lectures with other forms of learning approaches (group projects, discussion groups) in order to maximize both the “transfer” of information while ensuring it is a condusive environment.  As a student, discussion-based classes that are not structured well are sometimes even more detrimental to the learning outcomes.

“We now see that our mission is not instruction but rather that of producing learning with every student by whatever means work best.” (1)

I would like to know which University (especially public) would allow professors to undertake whatever efforts they need to undertake in order to produce this effective learning.  Budgets, payroll, benefits, professor time – some of these are things that need to be factored in order to make the system work as a whole. 

 

Do I agree we need an overhaul of the system: yes.  Do I think this is the most practical and realistic method?  Based on the little experience I have in outcome assessments, it’s hard to say.  However, as an instructor these ideas seem very counterintuitive to the way we are supposed to function in an already established system.  The Learning Paradigm conjures up images of hippies and free-flowing communes with no rules; fun and ideal, but chaotic and disorganized.  

 

 

 

 

UMD Learning Outcomes

Commenting on Sarah’s post made me realize perhaps everyone might like to have a look at what the outcomes at UMD happen to be. Check out the main page, and the page from which you can navigate to your own department’s outcomes.