Posts Tagged ‘assessment’
Authors’ opinions on publication in relation to annual performance assessment
Background:
In the past 50 years there has been a substantial increase in the volume of published research and in the number of authors per scientific publication. There is also significant pressure exerted on researchers to produce publications. Thus, the purpose of this study was to survey corresponding authors in published medical journals to determine their opinion on publication impact in relation to performance review and promotion.
Methods:
Cross-sectional survey of corresponding authors of original research articles published in June 2007 among 72 medical journals. Measurement outcomes included the number of publications, number of authors, authorship order and journal impact factor in relation to performance review and promotion.
Results:
Of 687 surveys, 478 were analyzed (response rate 69.6%). Corresponding authors self-reported that number of publications (78.7%), journal impact factor (67.8%) and being the first author (75.9%) were most influential for their annual performance review and assessment. Only 17.6% of authors reported that the number of authors on a manuscript was important criteria for performance review and assessment. A higher percentage of Asian authors reported that the number of authors was key to performance review and promotion (41.4% versus 7.8 to 22.2%). compared to authors from other countries.
Conclusions:
The number of publications, authorship order and journal impact factor were important factors for performance reviews and promotion at academic and non-academic institutes. The number of authors was not identified as important criteria. These factors may be contributing to the increase in the number of authors per publication.
The resident-as-teacher educational challenge: a needs assessment survey at the National Autonomous University of Mexico Faculty of Medicine
Background:
The role of residents as educators is increasingly recognized, since it impacts residents, interns, medical students and other healthcare professionals. A widespread implementation of resident-as-teacher courses in developed countries’ medical schools has occurred, with variable results. There is a dearth of information about this theme in developing countries. The National Autonomous University of Mexico (UNAM) Faculty of Medicine has more than 50% of the residency programs’ physician population in Mexico. This report describes a needs assessment survey for a resident as teacher program at our institution.
Methods:
A cross-sectional descriptive survey was developed based on a review of the available literature and discussion by an expert multidisciplinary committee. The goal was to identify the residents’ attitudes, academic needs and preferred educational strategies regarding resident-as-teacher activities throughout the residency. The survey was piloted and modified accordingly. The paper anonymous survey was sent to 7,685 residents, the total population of medical residents in UNAM programs in the country.
Results:
There was a 65.7 % return rate (5,186 questionnaires), a broad and representative sample of the student population. The residents felt they had knowledge and were competent in medical education, but the majority felt a need to improve their knowledge and skills in this discipline. The majority (92.5 %) felt that their role as educators of medical students, interns and other residents was important/very important. They estimated that 45.5 % of their learning came from other residents. Ninety percent stated that it was necessary to be trained in teaching skills. The themes identified to include in the educational intervention were mostly clinically oriented. The educational strategies in order of preference were interactive lectures with a professor, small groups with a moderator, printed material for self-study and homework, material available in a website for self-learning, and small group web-based learning.
Conclusions:
There is a large unmet need to implement educational interventions to improve residents’ educational skills in postgraduate educational programs in developing countries. Most perceived needs of residents are practical and clinically oriented, and they prefer traditional educational strategies. Resident as teachers educational interventions need to be designed taking into account local needs and resources.
An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis
Background:Four- or five-option multiple choice questions (MCQs) are the standard in health-science disciplines, both on certification-level examinations and on in-house developed tests. Previous research has shown, however, that few MCQs have three or four functioning distractors. The purpose of this study was to investigate non-functioning distractors in teacher-developed tests in one nursing program in an English-language university in Hong Kong.MethodSUsing item-analysis data, we assessed the proportion of non-functioning distractors on a sample of seven test papers administered to undergraduate nursing students. A total of 514 items were reviewed, including 2056 options (1542 distractors and 514 correct responses). Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.Results:The proportion of items containing 0, 1, 2, and 3 functioning distractors was 12.3%, 34.8%, 39.1%, and 13.8% respectively. Overall, items contained an average of 1.54 (SD=0.88) functioning distractors. Only 52.2% (n=805) of all distractors were functioning effectively and 10.2% (n=158) had a choice frequency of 0. Items with more functioning distractors were more difficult and more discriminating.Conclusions:The low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.
Needs assessment of Wisconsin primary care residents and faculty regarding interest in global health training
Background:The primary objectives of this study were to assess Wisconsin’s primary care residents’ attitudes toward international health training, the interest among faculty to provide IH training, and the preferred modality of IH training.Methods:Surveys were administered using 505 residents and 413 medical faculty in primary care residencies in Wisconsin. Results from 128 residents and 118 medical school faculty members were collected during the spring of 2007 and analyzed.Results:In total, 25% of residents (128/505) and 28% of faculty (118/413) responded to the survey. A majority of residents (58%) and faculty (63%) were interested in global health issues. Among residents, 63% planned on spending professional time working abroad. Few residents (9%) and faculty (11%) assess their residencies as preparing residents well to address topics relating to international health. The survey indicates that adequate faculty in Wisconsin could provide mentorship in international health as 47% (55) of faculty had experience working as a physician internationally, 49% (58) of faculty spend more than 25% clinical time caring for patient from underserved communities and 39% (46) would be willing to be involved with developing curriculum, lecturing and/or mentoring residents in international health.Conclusions:Overall, the majority of the respondents expressed high interest in IH and few felt prepared to address IH issues indicating a need for increased training in this area. The findings of this survey are likely relevant as a prototype for other primary care residencies.
Training simulated patients: evaluation of a training approach using self-assessment and peer/tutor feedback to improve performance
Background:Most medical schools use simulated patients (SPs) for teaching. In this context the authenticity of role play and quality of feedback provided by SPs is of paramount importance. The available literature on SP training mostly addresses instructor led training where the SPs are given direction on their roles. This study focuses on the use of peer and self evaluation as a tool to train SPs.Methods:SPs at the medical school participated in a staff development and training programme which included a) self-assessment of their performance while observing video-tapes of their role play using a structured guide and b) peer group assessment of their performance under tutor guidance. The pre and post training performance in relation to authenticity of role play and quality of feedback was blindly assessed by students and tutors using a validated instrument and the scores were compared. A focus group discussion and a questionnaire assessed acceptability of the training programme by the SPs.Results:The post-training performance assessment scores were significantly higher (p<0.05) than the pre-training scores. The degree of improvement in the quality of feedback provided to students was more when compared to the improvement of role play. The acceptability of the training by the SPs was very satisfactory scoring an average of 7.6 out of 10. The majority of the SPs requested the new method of training to be included in their current training programme as a regular feature.Conclusions:Use of structured self-reflective and peer-interactive, practice based methods of SP training is recommended to improve SP performance. More studies on these methods of training may further refine SP training and lead to improvement of SP performance which in turn may positively impact medical education.
Students and tutors’ social representations of assessment in problem-based learning tutorials supporting change.
Background:Medical programmes that implement problem based learning (PBL) face several challenges when introducing this innovative learning method. PBL relies on small group as the foundation of study, and tutors facilitate learning by guiding the process rather than teaching the group. One of the major challenges is the use of strategies to assess students working in small groups. Self-, peer- and tutor-assessment are integral part of PBL tutorials and they’re not easy to perform, especially for non experienced students and tutors. The undergraduate PBL medical programme was introduced in 2003, and after two years the curriculum committee decided to evaluate the tutorial assessment in the new program.Methods:A random group of ten students, out of a cohort of sixty, and ten tutors (out of eighteen) were selected for semi-structured interviews. The social representations’ theory was used to explore how the students and tutors made sense of “assessment in tutorials”. The data were content analyzed using software for qualitative and quantitative processing of text according to lexicological distribution patterns.Results:Even though students and tutors are aware of the broader purpose of assessment, they felt that they were not enough trained and confident to the tutorial assessment. Assigning numbers to complex behaviors on a regular basis, as in tutorials, is counter productive to cooperative group learning and self assessment. Tutors believe that students are immature and not able to assess themselves and tutors. Students believe that good grades are closely related to good oral presentation skills and also showed a corporative attitude among themselves (protecting each other from poor grades).Conclusions:Faculty training on PBL tutorials’ assessment process and a systematic strategy to evaluate new programs is absolutely necessary to review and correct directions. It is envisaged that planners can make better-informed decisions about curricular implementation, review and reform when information of this nature is made available to them.
Assessment of a national network: the case of the French teacher training colleges’ health education network
The French teacher training colleges’ health education (HE) network was set up in 2005 to encourage the inclusion of HE in courses for primary and secondary school teachers. A systematic process of monitoring the activity and the impact of this initiative was implemented. This analysis was systematically compared with the perceptions of teaching staff involved in the network. This paper assesses the network after 2 years using documents produced and interviews with 24 coordinators. Twenty-nine teacher training colleges out of a total of 31 are involved in the network. The network has helped to create links between teacher training colleges, extend HE training and encourage partnerships with other public health organizations. By 2007, HE was included in courses offered by 19 teacher training colleges as opposed to only 3 in 2005. This study not only showed the positive impact of the network but also revealed issues in its management and presented new challenges to ensure the effectiveness of the network. The network has succeeded in attracting and training trainers who were already providing or were interested in HE. Reaching other trainers who are not familiar with HE remains a challenge for the future.
Rasch scaling procedures for informing development of a valid Fetal Surveillance Education Program multiple-choice assessment
Background:It is widely recognised that deficiencies in fetal surveillance practice continue to contribute significantly to the burden of adverse outcomes. This has prompted the development of evidence-based clinical practice guidelines by the Royal Australian and New Zealand College of Obstetricians and Gynaecologists and an associated Fetal Surveillance Education Program to deliver the associated learning. This article describes initial steps in the validation of a corresponding multiple-choice assessment of the relevant educational outcomes through a combination of item response modelling and expert judgement.Methods:The Rasch item response model was employed for item and test analysis and to empirically derive the substantive interpretation of the assessment variable. This interpretation was then compared to the hierarchy of competencies specified a priori by a team of eight subject-matter experts. Classical Test Theory analyses were also conducted.Results:A high level of agreement between the hypothesised and derived variable provided evidence of construct validity. Item and test indices from Rasch analysis and Classical Test Theory analysis suggested that the current test form was of moderate quality. However, the analyses made clear the required steps for establishing a valid assessment of sufficient psychometric quality. These steps included: increasing the number of items from 40 to 50 in the first instance, reviewing ineffective items, targeting new items to specific content and difficulty gaps, and formalising the assessment blueprint in light of empirical information relating item structure to item difficulty.Conclusions:The application of the Rasch model for criterion-referenced assessment validation with an expert stakeholder group is herein described. Recommendations for subsequent item and test construction are also outlined in this article.
Interrater reliability of the mind map assessment rubric in a cohort of medical students
Background:Learning strategies are thinking tools that students can use to actively acquire information. Examples of learning strategies include mnemonics, charts, and maps. One strategy that may help students master the tsunami of information presented in medical school is the mind map learning strategy. Currently, there is no valid and reliable rubric to grade mind maps and this may contribute to their underutilization in medicine. Because concept maps and mind maps engage learners similarly at a metacognitive level, a valid and reliable concept map assessment scoring system was adapted to form the mind map assessment rubric (MMAR). The MMAR can assess mind map depth based upon concept-links, cross-links, hierarchies, examples, pictures, and colors. The purpose of this study was to examine interrater reliability of the MMAR.Methods:This exploratory study was conducted at a US medical school as part of a larger investigation on learning strategies. Sixty-six (N = 66) first-year medical students were given a 394-word text passage followed by a 30-minute presentation on mind mapping. After the presentation, subjects were again given the text passage and instructed to create mind maps based upon the passage. The mind maps were collected and independently scored using the MMAR by 3 examiners. Interrater reliability was measured using the intraclass correlation coefficient (ICC) statistic. Statistics were calculated using SPSS version 12.0 (Chicago, IL).Results:Analysis of the mind maps revealed the following: concept-links ICC = .05 (95% CI, -.42 to .38), cross-links ICC = .58 (95% CI, .37 to .73), hierarchies ICC = .23 (95% CI, -.15 to .50), examples ICC = .53 (95% CI, .29 to .69), pictures ICC = .86 (95% CI, .79 to .91), colors ICC = .73 (95% CI, .59 to .82), and total score ICC = .86 (95% CI, .79 to .91).Conclusions:The high ICC value for total mind map score indicates strong MMAR interrater reliability. Pictures and colors demonstrated moderate to strong interrater reliability. We conclude that the MMAR may be a valid and reliable tool to assess mind maps in medicine. However, further research on the validity and reliability of the MMAR is necessary.
Pediatric resident and faculty attitudes toward self-assessment and self-directed learning: a cross-sectional study
Background:The development of self-assessment and self-directed learning skills is essential to lifelong learning and becoming an effective physician. Pediatric residents in the United States are now required to use Individualized Learning Plans (ILPs) to document self-assessment and self-directed learning. A better understanding of resident and faculty attitudes and skills about self-assessment and self-directed learning will allow more successful integration of lifelong learning into residency education. The objective of this study was to compare faculty and resident attitudes, knowledge and skills about self-assessment, self-directed learning and ILPs. Methods:Survey of pediatric residents and faculty at a single institution. Respondents rated their attitudes, knowledge, and self-perceived skills surrounding self-assessment, self-directed learning and ILPs.Results:Overall survey response rate was 81% (79/97); 100% (36/36) residents and 70% (43/61) faculty. Residents and faculty agreed that lifelong learning is a necessary part of being a physician. Both groups were comfortable with assessing their own strengths and weaknesses and developing specific goals to improve their own performance. However, residents were less likely than faculty to continuously assess their own performance (44% vs. 81%; p<0.001) or continuously direct their own learning (53% vs. 86%; p<0.001). Residents were more likely than faculty to believe that residents should be primarily responsible for directing their own learning (64% vs. 19%; p<0.0001), but at the same time, more residents believed that assigned clinical (31% vs. 0%; p<0.0001) or curricular (31% vs. 0%; p<0.0001) experiences were sufficient to make them competent physicians. Interns were less likely than senior residents to have a good understanding of how to assess their own skills (8% vs. 58%; p=0.004) or what it means to be a self-directed learner (50% vs. 83%; p=0.04).Qualitative comments indicated that while ILPs have the potential to help learners develop individualized, goal-directed learning plans based on strengths and weaknesses, successful implementation will require dedicated time and resident and faculty development.Conclusions:These findings suggest that training and experience are necessary for physicians to understand the role of self-directed learning in education. Deliberate practice, for example by requiring residents to use ILPs, may facilitate self-directed, lifelong learning.
Rx Drug Compare
Translator
By N2H