Tips & Myths for Competency Based Assessment in a New Era of CBME

In Education & Quality Improvement, Featured by Rob WoodsLeave a Comment

Competency Based Medical Education (CBME) curriculums are being launched across Canada for all Emergency Medicine Residency Programs in July 2018, although some programs have begun to make change as early as this year! This movement has come with much trepidation from Program Directors and faculty due to fears of implementation within the existing structure of our training programs. This transformative change will require a renewed focus on trainee assessment in the Emergency Department (ED) and much work is underway.

A few years ago, a team of educators at McMaster Emergency Medicine developed and implemented an EM specific competency based assessment program called McMAP.  They were kind enough to share this program with the University of Saskatchewan, and we now have some actual experience to address the challenges of this type of assessment system.  Competency Based Medical Education (CBME) is a major step forward in both formative and summative assessment of trainees which will translate to better Emergency Physicians and better patient care for Canadians in our Emergency Departments.

The following post discusses some key lessons that we’ve learned about CBME through the implementation of McMAP.  We will try to dispel the myths and provide some key tips on how to make this important transition a success!

Myth #1 – There is not enough time to do this and my faculty won’t do it!

McMAP uses short assessment tools, and only requires one specific task to be assessed per shift.  This assessment tool is designed to take a maximum of 10 minutes, usually only 5 minutes.  The residents drive the process.  They start the shift by showing their preceptor the list of potential assessments, and they choose one together.  The preceptor can usually fill out the assessment in real time while watching the encounter with the resident.  It really takes no more time to do this type of assessment than our traditional daily encounter cards.

Tip #1: Empower the residents, but support them.

At both of our institutions, we have greatly empowered our residents to “take charge” of their own assessment portfolios. Of course, at times it feels like “more paperwork” or “more checklists”, but based on the assessments that we’ve read having many structured tasks within our portfolios has greatly enriched the types of feedback generated when a resident is assessed at the end of shift.

One of the big things, though, is that with larger achievement portfolios, it is important to find ways to systematically support residents’ achievement.  At McMaster we do this by providing a summary “dashboard” that allows residents to review their achievements at a glance. Periodically, they also meet with a faculty member to go over these if they are having trouble. We spend a good amount of time making sure the residents know that the McMAP system is only as useful as they’re willing to make it! Notably, we explain that this system has greatly improved the quality of the feedback in their monthly assessments (in-training evaluation records, ITERs), but more specifically that these assessments allow you to ask attendings for specific recommendations for improvement.

Here are some strategies for communicating the importance of CBME and workplace-based assessments to your local shop.

If you’re a resident:  Take advantage of the opportunity to open a conversation about your performance and to get really useful feedback from them.  Remember, your staff are actually fountains of knowledge and experience, but when they’re tired/frustrated/busy at the end of a shift it can be tough for them to “find the words” to give you some great insights.  One strategy for overcoming this is to open up the discussion by reflecting about something you feel. For instance, if they looked at your charting today, it might be worthwhile to start with a statement like: “So, you’ve looked at my charting today… One of the challenges I have is that sometimes I run out of space on the page… Do you have a suggestion on what I could do to streamline my charting?”  The culture of feedback can and will be cultivated by residents, as evidenced in the discussion from this program evaluation on McMAP.

If you’re a staff/attending:  Remember that charting on residents is just as important as charting on your patients.  “Good job, read more” doesn’t help anyone improve – and your residents work really hard to help you during a shift, so one of the big ways you can pay them back is to provide them with at least one truly concrete suggestion for improvement.  As Nadim Lalani likes to say to his faculty members, you can think of your learner as your ‘last patient’ of the day.  Your day is not over until they’ve been tended to as well.

Myth #2 – Most of preceptors are not MedEd experts, and won’t have the skills to fill them out.

McMAP structures assessments with descriptions and checklists around a specific task.  Our experience is that preceptors embrace structure to their assessment, and feel they are giving more useful formative and summative assessment for their trainees.  It is a welcome change to the daily encounter card where they could write about anything, and often write about nothing (‘good job, keep reading’).

Tip #2:  Support all faculty members in their development, not just junior faculty.

The McMAP team has a done a great job of creating three different e-books with all of the tools, so faculty can get a start of the shift ‘just-in-time’ refresher of the assessment they are going to do that day, and the trainees are driving the assessment process.  This does NOT excuse us from providing faculty development. When you introduce these new assessments, you need warn faculty that the change is coming, and give them opportunities to practice with the tools before they use them assessing trainees.  This will require some reverse planning.  July 2018 is not that far away, so 2017-18 faculty development sessions should be planned with the purpose of bringing your faculty up to speed with the new assessment tools.  There will be more direct observation mini-CEX type assessments, so your faculty need to be introduced to this concept.

Myth #3 – We are going to have to watch residents suture 50 times before we deem them competent…how will we handle all of that data!

Competency Based Medical Education is based on the concept of Entrustable Professional Activities or EPAs.  These are the key elements of what defines a specialist in an area of Medicine.  One EPA might be: Perform procedural sedation for a painful procedure.  This one task employs many milestones and incorporates many CanMEDS competencies.  If you are competent to perform procedural sedation and all its elements, this will translate to other areas of practice as well.  Because of this, the list of EPAs and milestones that need to be observed and their frequency becomes actually quite manageable. We are certainly making progress from the see one do one teach one philosophy, but you’d be surprised how quickly certain EPAs will be met by trainees.

Tip #3:  Plan your workflow & consider your resources

Even though it is manageable, the amount of data collected will be more than what we are used to.  That’s a good thing for trainee development, but with increased data, comes increased responsibility!  Depending on the size of your program, it may be useful to take stock of how much administrative support you have and then design a system that allows you to optimize the workflow for your resources.  At Calgary, they use a written portfolio system that attendings then read and aggregate on their own. At McMaster, we have a 0.3 FTE admin person who helps process our reports.  The Saskatchewan program is quite small, so the usual admin person has been able to handle the load.

Moving forward, all trainee portfolios will be electronic, whether your university/academic centre uses the Royal College new ePortfolio system or another platform.  This will transform everyone’s ability to visualize each resident’s assessment portfolio and allow meaningful summary and aggregate data to inform future decisions.  It is an exciting time and it will not be long before we can permanently move away from paper assessments and towards hand held technologies.

So there you have it.  CBME can be done and should be done.  We will train better doctors and ultimately improve patient care along the way.

Rob Woods

Program Director at University of Saskatchewan
Rob is an emergency physician and STARS Transport Doc located in Saskatoon, Saskatchewan. He founded the Royal College emergency medicine residency program at the University of Saskatchewan (USask) and currently serves as Program Director. He has also recently founded the Clinician Educator Diploma AFC program at USask.

Teresa Chan

Senior Editor at CanadiEM
Emergency Physician. Medical Educator. #FOAMed Supporter, Producer and Researcher. Chief Strategy Officer of CanadiEM. Associate Professor, Division of Emergency Medicine, Department of Medicine, McMaster University.