Mini CEX (clinical evaluation exercise) as formative assessment –better appreciated assessment by post graduate students in Pediatrics
Original Research
Original Research
Abstract:
Objective: To assess the usefulness of mini clinical evaluation exercise (mini-CEX) as tool for formative assessment of clinical practices in post graduate trainees of pediatric department.
Design: Descriptive study.
Setting: Pediatric Outpatient department, MGM Medical College, Aurangabad from March 2015 to April 2015.
Participants: Post graduate trainees in Pediatric Department (M.D. and Diploma Students)
Intervention: Each post graduate student underwent 1 mini CEX encounter in OPD setting followed by entering the experience in structured feedback form.
Results: Twelve students in the second and third year of Pediatric post graduate training course underwent one mini-CEX encounter each. After completion, their perception of this method of assessment was sought through a structured questionnaire (Annexure 2). Perceptions were categorized for two areas: mini-CEX implementation and feedback. Results reported here are the opinions of participants, and words in quotation marks are direct quotes from trainees (T). Students were quite expressive with their responses related to this category. However, ten students agreed that the cases were of equal complexity. Eleven students agreed that the skills chosen to be assessed during the mini-CEX were taught during previous postings and nine said that a wide range of skills was chosen to be assessed. The students particularly appreciated that their communication skills were assessed. “Mini-CEX was a new format of examination of our communication skills…” (T); “I liked the mini-CEX format as I learnt to improve my communication skills” (T) this assessment is less stressful than traditional exam case presentation. However one student said that this could be combined to traditional assessment. All the twelve students agreed that periodic assessment of this kind will definitely improve our skills in all aspects. “Mini-CEX was not stressful but an opportunity for us to correct our mistakes and improve weak areas”. Two out of twelve students said that a teacher’s presence during the mini-CEX made them nervous but eventually they felt confident. Ten students agreed that mini-CEX provided more opportunities than the conventional format to demonstrate skills.
Conclusion: Various skills of doctors as in mini CEX along with communication skills with parents are important areas of learning. Currently, there is no formal evaluation of these skills in the Pediatric curriculum. Periodically taken Mini-CEX can bridge the gap in formative assessment along with traditional assessment method.
Key words: mini-CEX, formative assessment in medical teaching
Abbreviations: Trainee (T)
Introduction:
Although assessment of clinical competence is receiving increasing attention in educational research, it has not yet yielded a single ‘gold standard’ performance assessment tool that can confidently be said to be both reliable and valid. [1] The mini-CEX (clinical evaluation exercise) designed to conduct
Work based assessment of clinical performance was originally developed in 1995[2, 3] in the USA for the evaluation of Internal Medicine residents’ clinical skills. The principal characteristics of mini-
CEX are direct observation of real patient encounters, applicability in a broad range of settings and immediate structured feedback to the learner after the encounter. [2, 3] The traditional method of assessment only considers the final outcome, not how the students have reached it. The teachers do not observe the actual performance of the procedures by the students. This impacts both the ‘validity’ and the ‘reliability’. Moreover, communication skills are rarely assessed, there is very little scope for direct feedback, and some important skills may not be tested at all. Students may also feel dissatisfied by not receiving personal attention from the faculty and be confused about which particular skill is being tested.[4] Norcini has summarized the primary weak areas of the traditional assessment method, namely, lack of assessment of a large variety of cases, examination by a very limited number of examiners and testing of only a few competencies: These all lead to a lack of reproducibility of scores.[5] Mini-CEX, in contrast, has the potential to be a more practically suited assessment tool in situations involving patient–doctor interactions and where communication skills and professionalism are important. Mini-CEX formalizes the supervisory interaction between teachers and students, and promotes teaching interactions. As teachers are asked to observe their students, students have to take responsibility for the case, and teachers learn more about their students’ skills and decision-making processes. The structured nature of the rating form means that teachers give feedback across a broader range of topics and are more inclined to address issues that otherwise may not be addressed in evaluation. However, the assessment can be perceived as “threatening” to students and may alter how they perform, and also change the nature of the collegial relationship between the teacher and the student by strongly emphasizing the ‘assessor’ role of the teacher. [6]
Methods:
The aim of the study was to make students and faculty familiar with mini-CEX assessment pattern so that if it is feasible it can be applied in Indian pediatric curriculum as method of formative assessment.
This study was carried out in the Department of Pediatrics, MGM Medical College and Hospital, Aurangabad between March 2015 and April 2015. It was approved by the Institutional Review Board and exempted from ethical review. To sensitize the faculty to mini-CEX, an orientation session was first conducted. A presentation was made to the entire faculty of the department and hand outs were distributed. The faculty was made familiar with the mini-CEX rating form (Annexure 1). A similar orientation session was conducted for 12 post graduate students in the Pediatric department. A presentation was made to the students and hand outs were distributed. They were made familiar with the mini-CEX rating form. All 12 students voluntarily agreed to be a part of this pilot study. They were informed that results were known only to the research team, and as such had no impact on their formal assessment. It may be noted that under ideal circumstances, a student should undergo at least one mini-CEX with every assessor. However, this being only a pilot study each student in a group underwent only one mini-CEX encounter with their group assessor. Therefore, there were a total of 12 mini-CEX encounters in OPD settings. Every attempt was made to select 12 cases of equal complexity. Each mini-CEX encounter lasted roughly 15-20 min. The assessor directly observed the student and with the help of the checklist, rated the student’s performance under the following six domains using the mini-CEX rating form. Each student was also rated for his/her ‘Overall Clinical Competence,’ which was a global score, for which there were no checklist-determined guidelines. After the student–patient interaction was complete, a systematic feedback session of about 10 min took place in a separate room. The assessor first explained to the student the things that were done well, followed by the things that could be done better. These suggestions were put in writing on the mini-CEX rating form. The assessor and the student then agreed on a specific educational plan for the student to improve in the weak areas. Both the assessor and the student then carefully went through the completed rating form and signed it. After such structured feedback was given to the students as part of the mini-CEX, feedback was voluntarily obtained from them with a structured questionnaire: All 12 students participated. Since the purpose of this pilot study was not to compare the results obtained by mini-CEX with that of the traditional examination, no such comparison was made.
Results:
Twelve students in the second and third year of Pediatric post graduate training course underwent one mini-CEX encounter each. After completion, their perception of this method of assessment was sought through a structured questionnaire (Annexure 2). Perceptions were categorized for two areas: mini-CEX implementation and feedback. Results reported here are the opinions of participants, and words in quotation marks are direct quotes from trainees (T), and a number for which trainee the quote is attributed to. Students were quite expressive with their responses related to this category. However, ten students agreed that the cases were of equal complexity. Eleven students agreed that the skills chosen to be assessed during the mini-CEX were taught during previous postings and nine said that a wide range of skills was chosen to be assessed. The students particularly appreciated that their communication skills were assessed. “Mini-CEX was a new format of examination of our communication skill” (T). This assessment is less stressful than traditional exam case presentation. However one student said that this could be combined to traditional assessment. All the twelve students agreed that periodic assessment of this kind will definitely improve our skills in all aspects.
Likewise, there was the same split of opinion on whether the mini-CEX was a more stressful experience than the traditional format. “Mini-CEX was not stressful but an opportunity for us to correct our mistakes and improve weak areas”. Two out of twelve students said that a teacher’s presence during the mini-CEX made them nervous but eventually they felt confident. Ten students agreed that mini-CEX provided more opportunities than the conventional format to demonstrate skills.
Feedback:
The feedback session of the mini-CEX was most appreciated by the students. All students felt that the constructive feedback helped reinforce the skills that they did well, and helped them identify weak areas. “The immediate feedback helps because we immediately correlate the case and the way we approached it and the way we could have” (T). “It was an interesting experience because of the immediate feedback” (T). All but one student agreed that the feedback motivated them to learn further. Twelve out of the twelve students were satisfied that the rating reflected their level of performance. Overall, the students were found to perceive mini-CEX positively – “Mini-CEX was a new format of examination of our communication skills as well as our handling efficiency. This made us aware about the points and skills in which we were lacking” (T). “Mini-CEX has helped me to improve my skills, motivated me to learn some basic areas which will help me in the future” (T); “It was a good experience. Helps us learn our strengths and weak points” (T).
Discussion:
The goal of this study was to introduce mini- CEX as a tool for formative assessment of post graduate students in the subject of Pediatrics. An important component of the mini-CEX is the structured one on one feedback that takes place immediately following the student–patient encounter. This was greatly appreciated by all students. Research on formative assessment and feedback suggests that these are powerful tools to change trainees’ behavior. [7, 8, 9] The feedback session in the mini-CEX involves first reinforcing those skills that were done well, and then discussing the areas where improvement is possible. The trainee and assessor then agree on an individually tailored educational plan to bring about this improvement and formalize it with their signatures. Positive effects of narrative feedback have been reported by various authors like Overeem [10] who found higher satisfaction with such feedback, and Govaerts[11] who suggested that narrative feedback can improve in training evaluation. Archer[12] additionally concluded that feedback should not be exclusively trainer driven but a two-way process in which trainers provide comments and at the same time encourage trainees to self reflect on their performance. Archer’s model for effective feedback includes: Self monitoring (reflection on action) supported by external feedback and linkage with personal goals (action plan) in a coherent process rather than a series of unrelated events. [13] The overall perception of trainees toward mini-CEX was positive and they felt that this assessment method was a good experience would motivate them to improve in specific areas. Selecting cases of equal complexity was a daunting task. The presence of a teacher definitely had an impact on the students’ performance. The feedback session was the most favored part of the mini-CEX, but was possible only because direct observation of the student’s performance took place during the student–patient encounter.
Limitations of this study:
This study did not examine whether mini- CEX actually improved learning, clinical skills and ultimately the quality of patient care. Given the formative nature of this instrument, effects on learning and performance should be the prime objective of this type of assessment evaluates perceptions of users, and although the outcomes are overwhelmingly positive, they do not provide compelling evidence for learning effects. This study also did not collect feedback from patients. As they were the real “subjects” of the mini–CEX, their perception about it could have thrown up some new perspectives. Studies have shown that evaluation of performance by patients draws attention to different aspects of performance than is elicited by evaluation of consultations by health professionals.[14] Moreover, if such patient feedback could be incorporated and discussed in the mini-CEX feedback session, rather than simply be passed on to the trainees in a written format, it would be more significant in stimulating trainees’ interest and in improving the quality of patient care.[15] Lastly, this study was done in a single institution and with only one discipline.
Conclusion:
Various skills of doctors as in mini CEX along with communication skills with parents are important areas of learning. Currently, there is no formal evaluation of these skills in the Pediatric curriculum in Indian curriculum. Mini-CEX can bridge this gap in formative assessment along with traditional assessment method.
The data arising from this pilot study supports the implementation of novel assessment methods such as mini-CEX to improve the learning experience for Pediatric Post graduate students. Our pilot study goes some way to understanding why students find mini-CEX useful, how it could be improved, and the barriers to using it more effectively. Furthermore, all students felt that along with traditional assessment this can be definitely considered for periodic evaluation during training. We have log books for periodic assessment of students, so mini –CEX encounters can also be part of such periodic assessments. More studies on its feasibility for other disciplines, also adding patients’ feedback need to be carried out.
Contribution of Each Author:
SM: was involved in the review of literature, data collection and wrote the first draft, analysis and interpretation of data and will act as guarantor
AVK: was involved in management of patients, designing of study, drafting the article, analysis and interpretation of data and will and will act as guarantor
SS: was involved in management of patients, designing of study and drafting the manuscript
AK: was involved in management of patients, designing of study and drafting the manuscript
Disclaimers, if any: None
Funding / Source(s) of support: Nil
Acknowledgements
Authors thank the post graduate students who were part of this study.
Declaration of competing interests: None
References:
1. Wass V, van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001; 357:945-9.
2. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini-CEX (clinical evaluation exercise): A preliminary investigation. Ann Intern Med 1995; 123:795-9.
3. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003; 138:476-81.
4. Lele SM. A Mini-OSCE for Formative Assessment of Diagnostic and Radiographic Skills at a Dental College in India. J Dent Educ 2011; 75:1583-9.
5. Norcini JJ. Death of the long case? BMJ 2002; 324:408-9.
6. Weller JM, Jones A, Merry AF, Jolly B, Saunders D. Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: Implications for implementation. Br J Anaesth 2009; 103:524-30.
7. Black P, William D. Inside the black box-raising standards through classroom assessment. Phi Delta Kappan 1998; 80:139-49.
8. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach 2007; 29:855-71.
9. Yorke M. Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. High Educ 2003; 45:477-501.
10. Overeem K, Lombarts MJ, Arah OA, Klazinga NS, Grol RP, Wollersheim HC. Three methods of multi-source feedback compared: A plea for narrative comments and coworkers’ perspectives. Med Teach 2010; 32:141-7. 18. Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. The use of observational diaries in in-training evaluation: Student perceptions. Adv Health Sci Educ Theory Pract 2005; 10:171-88.
11. Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. The use of observational diaries in in-training evaluation: Student perceptions. Adv Health Sci Educ Theory Pract 2005; 10:171-88.
12. Archer JC. State of the science in health professional education: Effective feedback. Med Educ 2010; 44:101-8.
13. Pelgrim EA, Kramer AW, Mokkink HG, van den Vleuten CP. Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: An observational study. BMC Med Educ 2012; 12:97.
14. McKinley RK, Fraser RC, Baker RH, Riley RD. The relationship between measures of patient satisfaction n measures of patient satisfaction and enablement and professional assessments of consultation competence. Med Teach 2004; 26:223-8.
15. Baker R, Smith A, Tarrant C, McKinley RK, Taub N. Patient feedback in revalidation: An exploratory study using the consultation satisfaction questionnaire. Br J Gen Pract 2011; 61:e638-44.
Appendices and proforma:
Proforma 1:
Students’ perception of ‘Mini-CEX (Mini- clinical evaluation Exercise) as a method of assessment.
1) Was basic working of this method adequately understood by residents?
2) Were the cases of equal complexity?
3) Were all the skills checked in mini CEX taught to the residents before?
4) What did you particularly like about assessment?
5) Was this more stressful experience than case presentation in examination?
6) What is your impression when teacher is present during your assessment?
7) Do you fill that you could be assessed better as compared to traditional case presentation in examination?
8) Would you be interested in further periodic assessment of such kind?
9) What is your experience about feedback given by teacher?
10) Are you satisfied with your assessment?
Proforma2:
Mini CEX (mini clinical evaluation Exercise) proforma
Date of assessment (DD/MM/YY)-
Student’s surname- Student’s forename-
Assessor’s Registration No.(Maharashtra Medical Council)-
Assessor’s Name – Assessor’s Email-
Assessor’s Position-
Professor Associate Professor Assistant Professor Senior Registrar
Brief Summary of case:
Setting for Assessment (e.g. OPD, Inpatient, Pediatric ICU)-
Please score the student on the scale shown. Please note that your scoring should reflect the performance of the student against that which you would reasonably expect at their stage of training and level of experience. Please mark ‘unable to comment’ If you feel you have not observed the behavior
Issue: October-December 2017 [Volume 6.4]