This post was originally published in the New England Journal of Higher Education

In many ways, higher education has not changed in the nearly 1,000 years since the first university was founded in Bologna, Italy in 1088. Many courses still have professors or “masters” lecturing in front of students, with exams being reproduction of facts learned in lectures. But in other ways, higher education changes daily. A brief perusal of headlines from the Chronicle of Higher Education or Inside Higher Ed demonstrates the changing landscape of colleges and universities. Given this perpetual state of chaos, colleges and universities not only need to withstand and manage change, they need to leverage that environmental change, and sometimes even foster change to better meet the needs of diverse stakeholders. To achieve their goals, colleges and universities need to become “learning organizations.”

Continue reading “Organized Anarchies: 13 Steps to Building a Learning Organization” →

NOTE: This was originally posted on here.

About 16 months ago our institution started an institutional assessment steering committee (IASC) as a vehicle to improve assessment on campus. Composed of representatives from administrative and academic units across campus, we have been engaging in a number of activities to help increase confidence and capacity in assessment and also begin to build a culture of assessment.

Continue reading “Assessment as Learning” →

While there are variations, the basic assessment cycle comprises four steps: identify outcomes, develop strategies to achieve the outcomes, gather and interpret data related to outcome achievement, and close the loop. A great deal of emphasis is placed on the first three steps, but less on the final and perhaps most important step: closing the loop. Assessment is meaningless without putting the findings into action.

Continue reading “Managing Change to Close the Loop” →

(NOTE: This blog posted was also posted on http://www.cas.edu/blog_home.asp?display=73)

Recently, I attended the Assessment Institute in Indianapolis. During a discussion with a friend she introduced me to the concept of improvement science. I had no idea there was such a field of study, so I immediately search Google to learn more. After reading a few online articles, I was intrigued enough to purchase for my flight how a copy of New Directions for Evaluation (2017), number 153 as the issue was dedicated to improvement science.

To those implementing assessment, the concept is familiar. According to Moen, Nolan, and Provost (as cited in Christie, Lemire, & Inkelas, 2017, p. 11), “improvement science is an approach to increasing knowledge that leads to an improvement of a product, process, or system. That certainly sounds like the continuous improvement purpose of assessment. What is a little different is that between improvement science explicitly focuses on systems thinking. Because of this systems approach, a central principle of this approach is change management, as it is critical for improvement (Christie, Lemire, & Inkelas, 2017).

The journal issue goes into detail regarding the components of improvement science, an operational model, and then case studies to illustrate what the authors are describing. But, there was one discussion point that jumped out to me. A cycle of testing and learning is foundational to improvement science (Lemire, Christie, & Inkelas, 2017). This cycle includes four steps: Plan, Do, Study, Act (PDSA).

In this cycle, improvement occurs in small steps to see how effective implementation is and to better understand concomitant issues. The PDSA cycle is critical to change management as there are many issues that impact implementation of an improvement including user buy-in, resources, and sometimes even politics.

The PDSA cycle can easily be applied to assessment in higher education. Once the assessment is complete and recommendations for change are made, those changes should be implemented in small steps starting first with planning out the execution. After the planning takes place, a small-scale version of the improvement is implemented. Third, there would be assessment of that small-scale improvement. With this information, the improvement is scaled up.

Here is a basic example of how this might look in assessment practice. The orientation office at a small college completed a CAS self-study and learned that their program regarding sexual assault prevention did not achieve the learning outcomes intended. A recommendation in their self-study report is to contract with a professional organization that has developed a program called “Consent4U.” However, the program costs almost half of the entire orientation budget. While the benefits could be great, the orientation director wants to test the program first before making the significance financial investment. The director ran a pilot of the program with one of the residence hall learning communities. To understand change in learning, the director developed a pre- and post-test regarding sexual assault. In the first step of the cycle, “plan,” the director collaborated with the residence hall director to schedule a time and partnered with a faculty member in sociology to create the pre- and post-test. The program was implemented as part step 2, “do.” After the program was done, the director administered the post-test and “studied” the results. Based on the data, the director determined that given the amount of learning students obtained from this program, they were going to implement the Consent4U program and they requested additional funding from the provost for the program.

Some of the tenets of improvement science mirror those of assessment. However, the Plan, Do, Study, Act model may provide a way to manage the change that comes with making an improvement.

References
Christie, C., Lemire, S., & Inkelas, M. (2017). Understanding the similarities and distinctions between improvement science and evaluation. New Directions for Evaluation, 153, 11-22.

Lemire, S., Christie, C., & Inkelas, M. (2017). The methods and tools of improvement science. New Directions for Evaluation, 153, 23-34.

NOTE: This blog post is inspired by Margaret Leary. During the ACPA16 Convention she and I discussed ideas she had regarding how assessment can foster organizational change. That conversation me to learn more about organizational change and how assessment might be related to it. This blog post is a result of that research and contemplation.

Continue reading “Assessment as a Tool for Organizational Change” →

Originally published November 3, 2015 on The Student Affairs Collective

Assessment isn’t an activity. It’s a state of mind.

The statement above has been my mantra for the past five years. Too often, assessment is seen as an afterthought rather than an integral part of the program or service planning and implementation process.

Interestingly, this statement didn’t occur to me when I was in the midst of an assessment project, teaching class, facilitating a workshop, or consulting with a division of student affairs. It occurred to me in a grocery store. Yes, in the deli aisle waiting on maple and brown sugar ham, to be exact. I won’t bore you with the details here, but you can read more regarding this “evaluative epiphany” here: http://bit.ly/1MSqT2x.

In this post, I don’t want to talk about the statement itself, but what it represents. Recently, I echoed this refrain during a webinar sponsored by ACPA’s Commission for Assessment and Evaluation. The topic was “Assessing Cultures of Assessment.” (You can access the webinar here by entering your name and email) At the outset, I suggested that this statement, Assessment isn’t an activity. It’s a state of mind was a definitive sign that a culture of assessment existed in an organization. When assessment is a state of mind, it is infused into every aspect of individual or organizational practice including planning, implementation, and – of course – evaluation. At this stage, assessment becomes an unconscious, embedded element of everyday work.

So, how does one arrive at this destination where assessment is a state of mind?

I think the four stages of competence, originally described as the four stages of learning, and outlined by Linda Adams (2011), can be a helpful guide. These stages are

• Unconscious incompetence
• Conscious incompetence
• Conscious competence
• Unconscious competence

In the unconscious incompetence stage, people are not engaged in assessment and don’t know why it’s important. They are unable articulate its value or purpose. This stage described most of our field 10-15 years ago. The prevailing attitude was that assessment was simply a fad and we just needed to “wait it out.”

Once awakened to the both the necessity and benefit of engaging in assessment, people realize they need to it, but are not sure how to do it. This is conscious incompetence. The first step is to identify the actual skills and knowledge needed to perform assessment. The 2^nd Edition of the ACPA/NASPA Professional Competencies (released in August 2015) provides a framework for skill and knowledge development regarding assessment, evaluation, and research. A great way to develop these skills is to seek out books, workshops, and other resources to improve skill and knowledge. Fortunately, more and more resources are available. There are four new student affairs assessment books coming out this academic year. How exciting! One centers on leading assessment for student success and the other focuses on coordinating divisional assessment. Two more covering more general assessment practice will come out in early March. Workshops and conferences are another way to build assessment competence. Each summer, ACPA holds its Student Affairs Assessment Institute and NASPA sponsors its Assessment and Persistence Conference. Both associations have special interest groups regarding assessment for networking and professional development. (ACPA’s Commission for Assessment and Evaluation and NASPA’s Assessment, Evaluation, and Research Knowledge Community). In addition, many master’s level preparation programs have assessment courses to help students move from this stage of competence to the next.

As people gain skill and knowledge, they become conscious competent. They value the assessment process and are continually attentive when performing it. Assessment takes effort and concentration at this stage but continues to become more comfortable and more frequent.

At the apex of this competence hierarchy is unconsciously competence. At this stage, assessment has been performed so often that it becomes habit or second nature and is integrated into daily practice and processes. It is important to note that learning should still take place. Even in assessment, lifelong learning is important as new scholarship is constantly being created.

As I mentioned at the beginning of this post, the statement Assessment isn’t an activity. It’s a state of mind came to me in a grocery store. In the deli aisle, I recognized that I was shopping in a way that mirrored the assessment process. My goal was to be both effective and efficient and those goals shaped by actions from planning to execution. Assessment had become so unconscious that it was integrated into other parts of my life. While, you may not be as much of a geek as me and want assessment to infiltrate your personal life, I do hope you develop your assessment skill and knowledge to the point that you are unconsciously competent.

At what stage are you? What can you do to get to the next level?

References
Adams, L. (2011). Learning a new skill is easier said than done. [Blog]. Retrieved from http://www.gordontraining.com/free-workplace-articles/learning-a-new-skill-is-easier-said-than-done/

I recently wrote a post for Student Affairs Feature entitled “Considering Assessment as Investment.” What would you get on your investment in assessment?