Results of Operative Competency Evaluation Committee’s first technical skills assessment exam “very exciting”
Preliminary results from the Society’s Operative Competency Evaluation Committee’s first technical skills assessment examination are “very exciting,” says Committee Chair Dr. Patricia L. Roberts Burlington, MA. The examination was conducted at the University of Toronto, Toronto, ON, Canada, under the direction of Drs. Helen M. MacRae and Sandra de Montbrun.
“The results are very exciting. They have answered the key question of whether we can discriminate between general surgery graduating chief residents, many of whom are going into colon and rectal surgery, and colorectal fellows,” Dr. Roberts says.
Of ten colorectal fellows tested, nine passed and one failed. Seven of ten general surgery residents failed and only three passed. “The overall checklist scores and overall global rating scores effectively discriminate between GS residents and CR fellows,” Dr. de Montbrun reported.
The Committee is discussing the results and determining next steps for using this examination for final summative assessment, according to Dr. Roberts.
The new technical skills assessment tool is known as Colorectal Objective Structured Assessment of Technical Skill (COSATS). It includes a series of stations on which skills can be tested, using either models constructed from synthetic materials, animal tissue, or cadaveric tissue, or simulators. The pilot study examined colorectal and general surgery trainees to assess the validity and reliability of the COSATS stations.
Participants in the pilot study were asked to perform a series of surgical tasks while being directly observed by an expert colorectal surgeon, who determined whether or not each procedure was performed correctly and rated them, using a global scale of operative performance on skills such as respect for tissue, instrument handling, knowledge of instruments, and flow of operation. They also received ratings on overall performance and quality of final product.
“With the increasing public pressure for high quality health care and the professional duty of self-regulation, it is imperative this type of objective assessment of surgical skills be incorporated into certification,” Dr. Roberts says.
Until now, written and oral examinations and other tools have been used to assess a trainee’s cognitive knowledge, judgment, and professionalism. The COSATS will be the first test of its kind assessing subspecialty surgical skill. If implemented as a component of the ABCRS certification examination, it would also be the first test to be incorporated into subspecialty certification.
“There’s been no objective assessment of technical competence by a validated model,” says Dr. Roberts. “In effect, we’ve been able to evaluate how well a candidate has learned how to perform colorectal surgery, but we’ve not had an objective test of whether or not the candidate can actually perform in an operating room.”
In addition to Drs. Roberts and MacRae, members of the Operative Competency Evaluation Committee are Vice Chair Dr. Ann Lowry, St. Paul, MN, Advisor Dr. Richard Reznick, Kingston, ON, Canada, Council Rep Dr. Steven Wexner, Weston, FL, and Drs. Glenn Ault, Los Angeles, CA, Richard Billingham, Seattle, WA, Marcus Burnstein, Toronto, ON, Canada, Peter Cataldo, Burlington, VT, Eric Dozois, Rochester, MN, Gary Dunn, Oklahoma City, OK, James Fleshman, St. Louis, MO, Gerald Isenberg, Philadelphia, PA, Najjia Mahmoud, Philadelphia, PA, Peter Marcello, Burlington, MA, David Schoetz, Jr. (ex-officio), Burlington, MA, Michael Stamos, Orange, CA, Scott Strong, Cleveland, OH, Judith Trudel, St. Paul, MN, and Eric Weiss, Weston, FL.