Background: The Pediatric Advanced Life Support (PALS) course is designed to teach the pediatric provider initial stabilization of critically ill children and residency programs often employ it as the main method to achieve this. We hypothesized that although PALS may initially teach residents the knowledge and skills needed to resuscitate children, this information is not retained.
Methods: Fourteen first and 14 third year pediatric residents were evaluated before PALS (PRE), within 1 month following the course (POST) and 6-8 months later (F/U). Evaluation included assessments of knowledge (multiple choice and casebased questions), skills (bag-mask ventilation, intubation, intraosseous catheter insertion and defibrillation using checklists) and a confidence questionnaire, although the participants did not complete all evaluations.
Results: For first and third year residents, scores on multiple choice testing improved at POST, but in neither group was this information retained at F/U. Performance of bag-mask ventilation, defibrillation, and intubation did not significantly improve in either group at POST or F/U. However, intraosseous catheter insertion was better at POST and F/U in first years, but only at F/U in third years. Third year resident confidence was greater at both POST and F/U even though knowledge and skills were largely unchanged.
Conclusions: Given the lack of retention of knowledge and minimal improvement in skill performance, residency programs should be cautious about basing resident resuscitation education on PALS. Other instructional techniques need to be utilized but learning should be formally assessed to ensure that residents are prepared for the emergencies that they may encounter in practice.
Keywords: PALS; Pediatric Resuscitation; Skills; Competency; Education; Training; Resident
ACLS: Advanced Cardiac Life Support;
BMV: Bag-Mask Ventilation;
ED: Emergency Department;
F/U: Follow-Up at 6-8 Months After PALS;
IO: Intraosseous Catheter;
NICU: Neonatal Intensive Care Unit;
PALS: Pediatric Advanced Life Support;
PICU: Pediatric Intensive Care Unit;
PRE: before PALS;
POST: 1 Month after PALS
Because pediatric resuscitations are rare and residents are spending less time in intensive care units due to changes in requirements and work hour regulations, there is concern that graduates of pediatric residency programs are inadequately prepared to perform resuscitation. Studies have shown that trainees are receiving fewer opportunities to perform skills such as intubation  and are completing residency prior to achieving competency in basic resuscitation skills .
One method used by many U.S. pediatric residency programs to teach trainees resuscitation is the American Heart Association’s Pediatric Advanced Life Support (PALS) course. The stated purpose of PALS is “…is to improve the quality of care provided to seriously ill or injured children, resulting in improved outcomes” and the targeted audience is healthcare providers who respond to emergencies in infants and children . During PALS, participants receive didactic lectures as well as training in key resuscitation skills. Certification is for two years. The course is not specifically targeted to trainees and it is unclear whether it is effective in teaching residents resuscitation and can be used to increase resident competency. Nonetheless, there is evidence that it does improve knowledge when assessed at the end of the course .
Since PALS is one of the primary means of teaching resuscitation to residents, it is important that it be evaluated to determine its efficacy when used in this setting. Although there have been a few studies which suggest that PALS is effective in teaching resuscitation, less information exists regarding long-term retention. A meta-analysis of all studies evaluating life support courses found that in those studies that examined knowledge acquisition as assessed by multiple choice tests, there was either an increase or no significant difference in scores after the course . Half of these investigations showed a significant decline in scores between the immediate posttest and follow-up testing periods. In addition, this analysis also evaluated studies in which the acquisition of resuscitation skills was examined and found there was a significant improvement in scores among participants who scored poorly prior to taking the course. However, test scores declined in follow-up testing as early as three months after the course was completed. The goal of this study was to evaluate pediatric residents’ acquisition and retention of both the knowledge and skills taught in PALS and to assess resident confidence in performing resuscitations after the course. The hypothesis was that although PALS may initially teach pediatric residents the knowledge and skills needed to resuscitate children, this information is not retained.
Materials and Methods
The study population consisted of 1st year residents taking the initial PALS certification course at the start of their pediatric residency and pediatric residents taking the PALS recertification course at the beginning of their 3rd year of training. Participants were recruited from two different pediatric residency programs in two separate hospitals. The study was deemed IRB exempt at both institutions and participation was strictly voluntary. Subjects were provided the PALS course by their respective hospitals and testing was performed independent of the course. In order to be PALS certified, individuals are required to pass a written test and demonstrate skill competency at the end of the course. In addition to PALS, mock codes are provided at both institutions with most trainees directing a simulated resuscitation once annually. Additional instruction about resuscitation is also provided during the Emergency Department (ED) and intensive care unit rotations. At one of the programs, 1st year residents participate in an airway skills course designed to improve their knowledge and skills of bag-mask ventilation (BMV) and intubation. This is given late in the first year of training.
Participants were evaluated at three time periods: prior to PALS (PRE), within 1 month following the course (POST), and 6 to 8 months later (F/U). Residents completed several testing instruments at each time point. Knowledge was evaluated using multiple choice and case-based questions. Each test included 20 multiple choice questions selected from a question bank available in the PALS Student CD Self- Assessment Test . In addition, 14 questions centered on 2 different clinical scenarios were included in the testing to assess participants’ knowledge of the PALS algorithms (case-based). These questions utilized cases presented in the PALS Student CD Practice Cases . Three expert pediatric intensivists with extensive experience in resident education reviewed the proposed tests to assure content validity. To evaluate the influence of question recall on test performance, half of the multiple choice questions were repeated at each time point and half of the questions were new.
Residents were asked to demonstrate their competence in performing four key resuscitation skills: BMV, endotracheal intubation, intraosseous catheter (IO) placement, and defibrillation. One of the authors (N.B.) directly observed the participants demonstrate each of the four skills on a mannequin. Subjects were evaluated using task-based checklists and skill performance was judged based on the percentage of subcomponents performed correctly. The checklists were similar to those that have been used in prior studies involving skill assessment in PALS [7, 8, 9]. Residents were also administered a confidence questionnaire upon which they were asked to rate their confidence in various resuscitation scenarios using a 5-point Likert scale, with 1 representing “very unconfident” and 5 representing “very confident.” This questionnaire was derived from the study by Grant et al .
Since previous experiences may affect participants’ knowledge and acquisition of resuscitation concepts, additional information was obtained via questionnaire. Data collected included prior residency training, history of prior resuscitation courses (PALS or Advanced Cardiac Life Support, ACLS), participation in mock and actual resuscitations, participation in sessions geared at teaching resuscitation skills, past performance of skills taught in the PALS course, and number of months spent working in the Pediatric Intensive Care Unit (PICU), Neonatal Intensive Care Unit (NICU), and ED.
Subject performance was evaluated based on the percentage of questions answered or skill subcomponents performed correctly. Mean differences (95% confidence interval) between POST or F/U with PRE are reported. To allow for longitudinal assessment, data were included only if the trainee participated in at least the PRE and POST or PRE and F/U periods. Statistical analysis was performed using the paired or unpaired Student t-test, as appropriate. When scores at the PRE period were compared with those at the POST and at the F/U period, the P value was adjusted for multiple comparisons using the Bonferroni correction. As only two comparisons were made (PRE vs. POST and PRE vs. F/U), the adjusted P value was 0.025. Cohen’s d was utilized for effect size. Generally, 0.2 represents a small effect size,0.5 medium and 0.8 or greater, large.
Fourteen 1st year (6 from program A and 8 from program B) and fourteen 3rd third year (6 from program A and 8 from program B) residents participated in the evaluation at the PRE time period and at least one other time point (either POST or F/U). Of these, nine 1st year and eleven 3rd year residents returned for testing at the POST period and ten 1st year and twelve 3rd year residents were assessed at F/U, although trainees did not complete all tasks at each period (Table 1).
Background data describing resident prior experience are presented in Table 2. As would be expected, 1st year residents had little prior experience in resuscitation. None had previously taken PALS, although half had received ACLS training, and the vast majority had no prior experience in an ED, PICU, or NICU. These residents had minimal background experience with BMV, intubation, IO placement, or defibrillation. By contrast, all 3rd year residents had previously taken PALS and the majority had completed ACLS. They had a median of 1-2 months of experience in each of the ED, PICU, and NICU and a median of 4 prior BMV and 10 prior intubation attempts. Most, but not all, 3rd year trainees participated in mock code simulations. However, similar to the 1st year residents, they had minimal experience with IO placement and defibrillation.
Figure 1. Comparison of baseline scores of the assessments between 1st (filled bar; n=14) and 3rd (hatched bar; n=14) year
residents. Compared with 1st year residents, those in their 3rd year of training scored higher (p<0.05) on all items except BMV. Confidence is plotted using the right y-axis; all other assessments use the left y-axis. Data are mean (95% CI). MCQ=multiple choice questions, BMV=bag-mask ventilation, IO=intraossesous insertion, defib=defibrillation.
Figure 2. Percent change in scores from PRE at F/U for 1st and 3rd year residents. Third year confidence was increased (p<0.025) but only IO catheter insertion was improved (p<0.025). For 1st year residents, only case-based questions and IO catheter placement were improved (p<0.025) at F/U. Mean (95% CI). MCQ=multiple choice questions, BMV=bag-mark ventilation, IO=intraossesous insertion.
As would be expected, 3rd year residents performed better (p<0.05) than 1st year residents at PRE in all evaluations except for BMV (p=0.15; Figure). For all assessments at baseline except IO, there was no difference (P>0.05) in those 1st or 3rd year trainees who completed the evaluations at all three time points compared with those who participated only twice. For IO catheter insertion, 3rd year residents who were tested three times scored better (p<0.05) at baseline (66.7% [51.2-82.2]) than those who were assessed twice (48.6% [39.1-58.1]). This was not observed with 1st year residents.
Table 1. Mean difference (95% confidence interval) and effect size between scores at POST or F/U compared with PRE.*
*Due to multiple comparisons, the adjusted p value is 0.025.
For the knowledge-based tests, both 1st and 3rd year residents had improved scores on the multiple choice questions test after PALS at POST with a large effect size, but in neither group was this information retained at F/U (Table 1). For the case-based questions, 1st year residents had a large improvement but only at F/U. Third-year trainees had no improvement at either time point.
Performance of BMV, defibrillation, and intubation did not significantly improve in 1st or 3rd year residents at either time point (Table 1). However, IO catheter insertion was markedly better at POST and F/U in 1st year residents, but only at F/U in 3rd years with a moderate effect size. In spite of the lack of sustained improvement in knowledge and skills, 3rd year resident confidence was significantly greater (p<0.025) after PALS at F/U (Figure 2). First year residents had increased (p<0.025) confidence at POST but not at F/U.
In order to evaluate the influence of question recall on test performance, half of the multiple choice questions were repeated at each time point and half of the questions were new. When resident performance with these two groups of questions (new and repeat) were compared, there was no significant difference in test results (p = 0.9), indicating that resident recall of the questions did not influence the analysis.
Table 2. Experiences prior to participation in the study (median[range] except as noted).
Most residency programs use PALS as the main method to teach resuscitation to residents. Some may assume that successful completion of the course verifies resident competency in resuscitation but this investigation brings this practice into question. As demonstrated by their improved scores on the multiple choice tests, resident knowledge improved after PALS. However, this learning was lost by 6 to 8 months after the course was completed. Scores on the casebased questions did not have as clear of a trend. First year residents had significantly better scores at follow-up while 3rd year resident scores did not change over time. Given that the immediate post-PALS scores in 1st year residents were not significantly different from those pre-PALS, it is unclear whether the improvement observed at F/U is the result of increasing experience level or is a lasting effect of PALS. For skills, only IO placement was better after PALS. Even shortly after the course, there was no improvement in BMV, intubation or defibrillation.
It is concerning that knowledge about resuscitation decreased over a short period of time. Presumably, residents receive instruction about stabilizing sick infants and children in multiple venues, including simulation exercises and patient encounters, and it is easier to teach knowledge compared with skills. Surprisingly, Quan et al  assessed the acquisition of resuscitation knowledge and skills by 1st year pediatric residents with PALS and documented significant improvement in skills, but not knowledge, immediately following the course. Conversely, Grant et al  evaluated first through forth year pediatric residents and assessed resuscitation knowledge before and after PALS by testing the trainees every three months for one year. They reported significant improvement in resident knowledge both after PALS and at 12 months. However, it is likely that this repeated testing served as a method of instruction about resuscitation. Nonetheless, although there was retention of some knowledge, there was a significant decrease in recall of PALS algorithms.
Other studies have indicated that resuscitation knowledge deteriorates over time. Wolfram et al  studied retention of knowledge in paramedics following PALS and found that only 25% of subjects attained a passing score on the written test two years after the course. In an investigation of pediatric residents following the Neonatal Resuscitation Course, Duran et al. reported that trainee knowledge deteriorated 6 months after the program . A recent report  describes utilization of a “reconstructed” PALS course with several training sessions held over a 6 month period of time. At the end of the 6 months, Pediatric Intensive Care Unit nurses and respiratory therapists had improved performance compared with those that received standard PALS. However, whether there is long-term retention of these knowledge and skills is unknown and it is also unclear whether administration of the program to pediatric residents would have shown the same beneficial effect.
PALS seems to have minimal effect on residents’ ability to perform key resuscitation skills. Neither 1st nor 3rd year residents had any significant improvement in BMV, endotracheal intubation, or defibrillation even one month after the program. The one skill which did improve was IO placement, particularly among 1st year residents. These results contrast with the findings of Quan et al , who found that resident skill performance was significantly improved following PALS. However, this difference in outcome could be due to the method in which skill performance was evaluated. Whereas participants in our study were judged based on the percentage of skill subcomponents successfully performed, Quan et al defined an endpoint for each skill (e.g., tube in the trachea for intubation) and declared skill performance successful if that endpoint was reached. We chose to evaluate the percentage of the subcomponents completed because if one subcomponent of a skill is not completed, it may make achieving that endpoint impossible (e.g. may be unable to intubate without suction because the airway cannot be visualized). These investigators did not assess long-term retention of skills. BMV and intubation were also evaluated in the study by Grant et al . Performance was subjectively assessed by the attending anesthesiologist during a mandatory anesthesia rotation at various times after PALS and a non-validated three point Likert scale was used to rate the pediatric residents. For both skills, most trainees were in-between “able but requires multiple attempts or assistance” and “consistently able to perform skill independently.”
It is not surprising that PALS has little effect on skill performance as it is challenging to teach a complex psychomotor skill such as intubation in a two-day course. Mulcaster et al  studied students’ learning of intubation in an anesthesia rotation and found that 47 intubation attempts on patients were required before the students had a 90% probability of performing a successful intubation. Of concern is the lack of improvement in BMV, a skill that is likely of greater importance for a resident to master than intubation.
Resident confidence increased after PALS with 3rd year residents reporting a 42% improvement in confidence at the F/U period. However, given the lack of objective improvement in resuscitation ability in terms of both knowledge and skills, the relevance of this increased confidence is unclear. Previous studies [2, 9] have shown that self-confidence in resuscitation does not correlate with amount of training, experience, or cognitive score. Nadel et al evaluated pediatric residents’ training in resuscitation and found a “marked discordance between resident confidence and their performance in the technical skills… .” For example, although 100% of residents rated themselves as confident in performing bag-mask ventilation, only 18% performed the skill successfully.
Our study has several limitations. First, in order to secure IRB approval, participation was required to be completely voluntary, potentially introducing sample bias. Second, although the overall number of participants was small, the effect size for most items was low and a larger sample would not likely have altered this. Considering the amount of time and effort involved, educational programs generally should have a medium or large effect size. Third, as in the study by Grant et al., there was significant subject drop out at the POST and F/U periods in which there was a defined time window for testing. Many residents were unable to be tested at all-time points, likely due to scheduling constraints and duty hour requirements, as residents have busy work schedules and participation in a voluntary research study was a low priority. It is also possible that those residents who felt unprepared were not motivated to return for further testing. However, if this occurred, the scores at the POST and F/U time points would likely be elevated, and the true scores would be even lower than those reported. We found little difference in the scores of residents who participated at all-time points compared with those who were evaluated only twice. Fourth, one individual completed all the skill assessments, potentially introducing bias. However, having more than one rater could introduce error associated with inter-rater reliability, especially when using checklists to evaluate skills.
Another limitation is that although PALS has a standardized curriculum, the effectiveness of the course is influenced by the specific instructors and thus prone to variation. Our study surveyed two different residency programs in which the PALS courses are provided by two separate sets of teachers. While it is possible that PALS administered in some centers may provide more significant, lasting education, our design represents the “real world” and allows for some generalizability. Likewise, although there may be differences in the structure of mock codes at the two institutions, this too reflects actual practice. It is also possible that residents who volunteered to participate in the investigation represent a selected population, limiting the generalizability of the results. In addition, although it would have been very interesting to compare performance improvement between the residents of the two programs, our study lacked sufficient power to do so.
Our study shows that the PALS course is inadequate to teach pediatric residents resuscitation. Knowledge declines over time and important skills are not learned. This is particularly true for BMV, perhaps the most important skill for a trainee to master. Because pediatric resuscitations are relatively rare, PALS, occasional mock codes and clinical rotations cannot serve as the only means of education in resuscitation. A structured resuscitation curriculum needs to be utilized but, more importantly, trainee learning formally assessed to ensure that residents are prepared for the emergencies that they may encounter in practice.
We would like to thank the pediatric residency classes at Harbor-UCLA Medical Center and CHOC Children’s Hospital of Orange County for their participation and the Simulation and Skills Center at Harbor-UCLA for providing resources to conduct this investigation.