How A Graduate Nursing Program Implemented a Virtual Oral Examination during the COVID-19 Pandemic

Most academic institutions require students to demonstrate a level of knowledge about their specific major. A knowledge that should compound as they progress throughout the program. Therefore, it is paramount educators develop and implement evidenced-based assessment and evaluation methodologies that can adequately assess the students’ knowledge, skills, and attitudes (KSAs) specific to that profession and the students' expected level of expertise [1]. The assessment and evaluation of KSAs depend heavily on the evaluator's ability to utilize diverse assessment methodologies that allow examinees to demonstrate their understanding and knowledge retention. Examinations can be written, oral, or a combination of both forms. Compared to written examination, the benefit of oral examination is that the evaluator can witness the test taker's thought process in real-time [2, 3]. In the United States, oral examinations are utilized in pre-licensure and licensed professions [4, 5].

Oral exams typically involve one or more examiners assessing an examinee's KSAs in spoken form. Then, the examinee responds verbally, demonstrating their competency regarding the given subject matter [6]. These oral exams are generally conducted live and in-person for many disciplines, requiring face-to-face interaction between the examiner-examinee dyad. In 2020, as the entire world experienced the rapid restraints and uncertainties of the COVID-19 pandemic, innovative methods had to be utilized to achieve the same pre-pandemic goals and objectives [7]. Many population sectors used the virtual environment to meet and fulfill visions, missions, goals, and dreams. Academia was not an exception to this change; in fact, there became a massive demand for the virtual environment at every level of American education with the arrival of COVID-19 [8].

Prior to the pandemic, a family nurse /women’s health nurse practitioner (FNP/WHNP) program aimed to develop and implement an oral examination as an evaluation methodology. Initially, faculty planned to conduct the traditional live, in-person oral examinations based on best practices utilized for the past 10 years by the certified nurse anesthesia (CRNA) program at the same institution.  In order to attempt the execution of their pre-pandemic goals, they instead chose to utilize a virtual platform in response to the in-person limitations imposed by the COVID-19 pandemic.

Oral examinations in academia have been around for centuries, dating back to the early years of Cambridge and Oxford [9] and for decades in medical professions [7, 10, 11]. However, there is a gap in the literature exploring oral evaluations in nursing. In a review of the literature concerning graduate nursing programs, the authors of this article only came across one article about oral examinations in CRNA programs [11]. The search illuminated the reliability and feasibility of oral assessment in medical and CRNA programs [12]. The authors’ primary goal was to assess the applicability and feasibility of virtually conducting oral examinations. This article aims to reduce the gap in the literature as it pertains to virtual oral examination usage in nursing programs.

Methods

Execution of the FNP/WHNP virtual oral examination required consideration of several factors. Factors that included: translating the CRNA oral examination framework into a feasible framework conducive for the FNP/WHNP content selected for virtual oral examination, determining assessment methods that aligned with the framework, selecting a suitable interactive web-based software platform, training participating faculty, explaining to learners the expectations regarding virtual oral examination processes, and ensuring the assessment process aligned with the courses objectives and clinical domains [13], while also evaluating the effectiveness and usability of virtual oral examinations in the future, from the learners’ and faculty perspective.

The first step in developing the FNP/WHNP virtual oral examination was to consult with expert CRNA faculty experienced in executing in-person mock oral examinations. Inperson, virtual, and email communication with the CRNA program assisted in providing a framework adapted for our virtual oral examination. Meetings with the CRNA faculty and FNP/WHNP faculty provided the opportunity for discussions regarding the methods, challenges, strengths, weaknesses, and recommendations from the CRNA faculty to assist the FNP/WHNP faculty with developing their virtual oral examinations. This information was then translated by the FNP/WHNP faculty into the process and content for the virtual oral examinations. 

Three to four graduate nursing faculty (three-course faculty and one graduate nurse faculty) participated in the virtual oral examinations. Although two WHNP faculty were assigned to the course (one coordinator and one co-coordinator), the course coordinator took the lead in three areas: selecting the virtual platform, choosing the content, and standardizing the grading and the examination process. The course coordinator sought the expertise of the CRNA faculty and the literature. Although the literature validated virtual oral examination within the health sciences, there was limited literature on virtual oral examination in CRNA programs. Since the faculty found no literature regarding oral boards in FNP/WHNP programs, collaboration with the CRNA faculty at the university was paramount.

Although there are many virtual platforms—Zoom, GoToMeeting, Microsoft Teams, or

Google Meet—that could have been selected for the virtual oral examinations, the WHNP coordinator of the event chose Google Meet. The platform was selected for several reasons. The learners and faculty had prior experience and comfort with Google Meet. Thus, their comfort would possibly reduce some stress for both learners and faculty. Also, the Google Meet was supported by the university’s informational technology and security team.

After the faculty selected the platform, the next step was creating and assigning learners and participating faculty to their Google Meet sessions. Course faculty created sessions based on the number of participating faculty (four in 2020 and three in 2021) with 25-minute time slots (one slot for each student). Students and faculty were randomly assigned to one of the groups. Once assigned to a group, the students were provided the link to their group assignment and allowed to self-select their session time.  Students were unaware of what faculty would be their evaluator until they logged on for their oral examination. Each student then participated in a 20-minute information session with the two-course faculty. This session provided the students with verbal instructions regarding the virtual oral examinations and the opportunity to ask questions and education regarding the grading rubric that the faculty would utilize. The three-faculty participating in the FNP/WHNP oral examination attended a 30-minute training session, led by the course coordinator The session provided the faculty the opportunity to review the documents and resources required for the virtual oral examination, receive verbal instructions, and ask any questions before the event [14].

In 2020, 25 second-year graduate students, and in 2021, 19 students participated in the virtual oral examinations. The participating faculty consisted of the three WHNP faculty and one FNP faculty solicited to assist with the oral examination. The learners were assigned a time to log into the Google Meet meeting to participate in their 25 minutes oral examination, facilitated by one of the three faculty members.

Results/Findings

Analysis of the virtual oral examination included data from the years 2020 and 2021. Twenty-five students and four faculty participated in 2020 and 19 students and three faculty in 2021. Students and participating faculty all reported being able to access the Google sign-up document and sessions links without any technical issues. All students were able to complete their virtual oral examinations within the allotted 25-minutes sessions. Students’ examination scores ranged from 84% to 100%, with a possible total score of 44.5 points. An informal survey was conducted with faculty to assess feasibility and usability of the virtual oral boards. Faculty stated the chosen virtual platform was convenient and easy to navigate. The method was successfully implemented for two years using the Google Meet virtual platform.

Discussion

Virtual oral examinations have proven efficacy, applicability, and feasibility in diverse professional disciplines, including an FNP/WHNP advanced nursing program. The success of the FNP/WHNP virtual oral examinations provide evidence of the feasibility and sustainability of future implementations of virtual oral examinations within the FNP/WHNP program. The transition of the traditional in-person oral examination platform to the virtual one was prudent in response to the COVID-19 pandemic, which relegated the need to implement virtual oral examination within the FNP/WHNP program. While programs like The American Board of Emergency Medicine (ABEM) were adapting in-person oral examinations to a virtual platform [9], our FNP/WHNP program was prepared to execute the virtual oral examinations immediately without delay or stress with minimum adjustments from an in-person oral examination framework. Course faculty, therefore, developed and implemented the virtual oral examination after reviewing the oral examination methodology used for the past 10 years by the CRNA program at the same institution. Utilization of the virtual oral examination aligned with the current course assessment and evaluation strategies, quizzes, and exams. The virtual oral examinations allowed faculty to evaluate the students’ knowledge, skills, and attitudes toward assessing and managing common women’s health-related conditions.

The virtual oral examination platform allowed the examiner-examinee dyad to meet in real-time at specified times for 25-minute sessions. Conducting the oral examination in the virtual platform provided a safe environment during the COVID-19 pandemic that achieved the verbal objective of the study to assess learners’ knowledge, skills, and attitudes regarding women’s health topics. The collaboration with CRNA faculty and advanced planning proved invaluable to both the faculty and students’ preparation for the exam. The adopted exam outline, the exam script, student and faculty instructions, and the clear rubric increased the feasibility success of the oral examination in the FNP/WHNP program.

As in any inaugural event, several lessons were learned from implementing the oral exam on the virtual platform. One of the growth opportunities would be to utilize a virtual platform that offers a waiting room for learners to wait until their turn to be assessed. This opportunity allowed examiners to properly dismiss examinees from their exam and enable the next examinee access into the virtual examination room at the appropriate time without interruptions to the previous learner’s session. Secondly and related to the first opportunity for growth, two examiners, at minimum, should give the oral examination to one student at a time to help with the interrater reliability and decrease the risk of learners accusing a sole examiner of being “against” them for any reason and therefore, scoring them lower on their rubrics. Another opportunity may be

to assess acceptability for the utilization of an oral examination in the virtual environment. Perhaps taking a brief survey of faculty and students can highlight stakeholders’ feelings towards the evaluation tool [8]. Lastly, more attention should be given to the comparison to meet objectives between the live, in-person, and virtual oral examination.

Conclusion

The oral examination is an evidence-based assessment and evaluation tool with historical data in various disciplines and professions. Traditionally the examination is given live and in-person with the examiner-examinee using oral communication. There is very little literature regarding oral examinations in advanced nursing education, particularly in FNP/WHNP programs. COVID-19 turned the world upside down and presented a real challenge in achieving goals and objectives in both academic and medical arenas. During a global pandemic, virtual and online platforms offered easy adaptability and safety opportunities. Feasible, reliable, and validated assessment/evaluation methods are needed to measure advanced practice nursing students’ knowledge, communication, skills, and attitudes. Innovative approaches will evaluate students' critical thinking on essential aspects of care that they will eventually be delivered to the public. The authors and faculty achieved these goals and objectives for two years by implementing the oral examination in their FNP/WHNP program via a virtual platform.

References

About the Authors

Janice K. Williams is an assistant professor at Uniformed Services University of the Health Professions Graduate School of Nursing. She graduated from John Hopkins School of Nursing with a DNP, as MSN in nursing from Bowie State University, and a BSN from Christopher Newport. She was previously an adjunct clinical instructor for several colleges and universities in the D.C., Maryland, and Virginia region. Her clinical practice has focused on providing primary care to homeless and underserved populations in Washington, D.C. with a primary focus on the treatment of clients diagnosed with HIV/HCV. She has also assisted with developing health care programs aimed at providing primary and preventative care for clients residing in half-way houses. She has participated in programs aimed at coordinating patient care for clients coinfected with HIV and Hepatitis C services. She has also collaborated with the National Institutes of Health (NIH) and other healthcare providers in research studies for Hepatitis C. Her scholarly studies have focused on addressing burnout and developing wellness programs for providers employed in community settings. She has also been a speaker for continuing education organizations aimed at educating health care providers about HIV /HCV.

Lt. Col. Natasha I. Best is an assistant professor in the Family & Women's Health Nurse Practitioner Program at the Uniformed Services University of the Health Sciences (USU), Daniel K. Inouye Graduate School of Nursing. A women’s health nurse practitioner by trade, she instructs graduate nursing and medical school students at USU in a variety of women’s health needs, advanced practice nursing, and military readiness skills. In May 2021, she assumed the role of Director of Clinical Education, Phase II, responsible for collaborating and coordinating with clinical site directors for the placement of more than 50 graduate family nurse practitioner/ women’s health nurse practitioner students in 11 separate world-wide tri-service clinical sites. Lt. Col. Best is also an active member in the Military Women’s Health Research Interest Group (MWHRIG) and performs patient care in the Women’s Health Clinic in Malcolm Grow Medical Clinics and Surgery Center on Joint Base Andrews. She received a BSN from Syracuse University, an MSN from the University of Cincinnati, and a DNP from the University of North Carolina, Chapel Hill.

Lt. Col. Stacy Carr is a certified registered nurse anesthetist (CRNA) at Christus Alamo Heights Surgery Center in San Antonio, TX. She previously served as assistant professor for the Daniel K. Inouye Graduate School of Nursing at the Uniformed Services University in Bethesda, MD from 2019–2021.

Candy Wilson is a retired U.S. Air Force colonel and an associate professor at the Christine E. Lynn College of Nursing at Florida Atlantic University, Boca Raton, FL. Her research interests include clinical care for military members from a sex-difference perspective and integrative therapy interventions.

Dr. Diane Seibert is professor and associate dean for academic affairs at the Uniformed Services University in Bethesda, MD, and is a certified nurse practitioner, maintaining an active clinical practice at the Walter Reed National Military Medical Center. She is a fellow in the American Association of Nurse Practitioners and in the American Academy of Nursing.  Dr. Seibert has led two university-level interprofessional education and distributed learning taskforces, published and presented widely in the areas of women’s health and genetics, is the co-editor of a monthly genomics column in the Journal of the American Association of Nurse Practitioners, and is involved in national task forces and committees working toward improving the genetics competency of the nursing workforce across all practice settings.