Undermind - AI-Powered Scientific Research Assistant

Research topic

I want to find empirical studies published since 2019 that employ quantitative methods to analyze performance metrics (recall, precision, F1, etc.) for ChatGPT, GPT-3.5, or GPT-4 in screening papers by title and abstract for systematic reviews, while directly comparing their efficiency and accuracy against traditional screening methods.

Report

The literature search finds that models like GPT-3.5 and GPT-4 significantly reduce the time and effort required for title and abstract screening in systematic reviews while maintaining comparable sensitivity to human reviewers, though specificity can vary, as shown in key studies [1, 2, 3].

Key Findings

Efficiency and Workload Reduction: LLMs, particularly GPT-3.5 and GPT-4, drastically cut screening duration from months to hours without compromising sensitivity, though specificity can vary, sometimes leading to higher false-positive rates [1, 2, 4, 6].
Performance Metrics: Studies report high sensitivity levels, aligning closely with human reviewers, and moderate specificity, emphasizing the role of refined prompts and model fine-tuning for optimal performance [5, 9, 12].
Practical Integration: Many studies advocate for hybrid models integrating human oversight with AI tools to combine efficiency with reliability, indicated by potential cognitive load reductions and improved screening systems [3, 10, 15].

Methodological Insights

Quantitative Evaluations: Most studies employed quantitative measures like F1 score and ROC analysis to validate model effectiveness. Prompt engineering and zero-shot learning emerged as crucial strategies for enhancing model performance [11, 17, 31].
Variability and Optimization: While some variability in model performance exists, research highlights customized prompt strategies and prospective studies as pivotal in model optimization for domain-specific tasks [7, 13, 22].

This body of research confirms LLMs' potential in systematic reviews, primarily through integrating efficient AI-driven screening while addressing specificity and integration challenges.

Studies Employing Quantitative Performance Metrics

Focus on using quantitative methods to assess recall, precision, F1-score for GPT models in systematic review screening.
References: [1, 2, 5, 12, 16]
Details: [1] evaluates GPT-3.5 with accuracy, specificity statistics; [2] uses sensitivity optimization with a layered approach; [5] assesses sensitivity and specificity of GPT models with meta-prompts; [12] calculates precision, recall, F1-score for ChatGPT; [16] measures precision, recall, and F1-score in hallucination context.

Direct Comparisons with Traditional Screening Methods

Studies directly comparing GPT models against manual or traditional systematic review methods.
References: [3, 4, 6, 9, 19]
Details: [3] compares ChatGPT to traditional classifiers; [4] evaluates sensitivity, specificity vs. manual methods; [6] uses real-world datasets for direct performance comparisons to human reviewers; [9] compares GPT-3.5 Turbo as a single reviewer; [19] tests ChatGPT's screening accuracy in gastroenterology systematic reviews.

Workload Reduction and Efficiency Improvements

Demonstrates how GPT models reduce time and effort compared to traditional methods.
References: [1, 7, 10, 15, 21]
Details: [1] highlights reduced screening time from months to hours; [7] outlines prospective study for efficiency and accuracy improvements; [10] evaluates GPT-4 in reducing manual workload for screenings; [15] proposes hybrid approaches for workload reductions; [21] demonstrates rapid abstract screening capabilities.

Application of Prompt Engineering and Optimization Techniques

Examines how prompt design impacts LLM efficiency and accuracy.
References: [5, 11, 17, 18, 26]
Details: [5] discusses optimized prompt development for GPT models; [11] explores different prompting techniques' effects on performance; [17] leverages question-answering frameworks with optimized prompts; [18] employs LLMs with customized prompts for entire review processes; [26] uses zero-shot queries with calibrated prompts.

Real-World Data and Implementation Studies

Focused on deploying models in practice with empirical data.
References: [6, 8, 14, 15, 20]
Details: [6] applies GPT models to clinical review datasets; [8] evaluates LLMs' efficiency in practice; [14] integrates GPT-4 in real-world scoping reviews; [15] demonstrates hybrid model applications in SR processes; [20] discusses ChatGPT for complete SR automation in practice.

These categories encompass studies evaluating GPT models' performance with metrics, their comparison to traditional methods, workload reduction, prompt engineering, and real-world applicability, providing a comprehensive understanding of current capabilities and implications in systematic review screening tasks.

Historical Development of Ideas

Early Exploration (2019-2022): Initial studies began exploring the feasibility of using AI models, particularly LLMs, for screening systematic reviews. Concepts focused on substituting traditional methods with automated systems to handle the increasing volume of literature [6].
Integration of LLMs (2023): By 2023, research had extensively shifted to using specific LLMs like GPT-3.5 and GPT-4. Studies increasingly focused on evaluating these models' capabilities to manage systematic review processes with quantitative metrics like precision, recall, and specificity. The consensus emerged that LLMs could significantly reduce time and workload, albeit with considerations around specificity [3, 4, 5].
Refinement and Comparative Studies (2024): The latest works explore refining models with advanced prompting strategies and conducting robust comparative analyses against traditional methods. There is growing attention to fine-tuning and prompt engineering to maximize efficiency and accuracy [1, 7, 9].

Clusters of Research Groups or Key Individuals

Viet-Thi Tran's Group: This group seems prolific with contributions that focus on diagnostic accuracy and workload reduction using LLMs like GPT-3.5 Turbo. They've conducted studies on sensitivity and specificity for systematic reviews [4, 9].
T. Oami and Collaborators: Their contributions include exploring LLMs' potential across various domains, such as citation screening for clinical guidelines. They often employ comprehensive prospective study designs [7, 13, 22].
MedRxiv and ArXiv Authors: A significant portion of the exploratory and early adoption works published in MedRxiv and ArXiv are pivotal in expanding methodological foundations and real-world application scenarios for LLM implementations in systematic reviews [1, 4, 5].

These clusters highlight an evolving focus from feasibility and initial trials to more sophisticated application and optimization of LLMs, supported by a community of consistent contributors leading innovations in this intersection of AI and literature review methodologies.

Bokeh Application

References

Last 5 years

Last 2 years

> 1 citation per year

> 5 citations per year

Topic Match

Cit./Year

Year

Paper

Paper Relevance Summary

99.5%

0.0

2024

[1] Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening R. Luo, ..., and M. J. Deen medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 99.5% topic match

Provides a quantitative evaluation of GPT-3.5 Turbo's performance in systematic review screening. Compares GPT-3.5 Turbo against expert reviewers using metrics like F1-score, sensitivity, and specificity on 24,534 studies. Employs quantitative methods, but lacks direct comparison with traditional non-expert automated screening methods.

98.1%

0.0

2024

[2] Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews. Kentaro Matsui, ..., and Y. Takaesu Journal of medical Internet research 2024 - 0 citations - Show abstract - Cite Abstract: BACKGROUND The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, they risked excluding relevant papers. OBJECTIVE We evaluated the performance of a 3-layer screening method using GPT-3.5 and GPT-4 to streamline the title and abstract-screening process for systematic reviews. Our goal is to develop a screening method that maximizes sensitivity for identifying relevant records. METHODS We conducted screenings on 2 of our previous systematic reviews related to the treatment of bipolar disorder, with 1381 records from the first review and 3146 from the second. Screenings were conducted using GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) across three layers: (1) research design, (2) target patients, and (3) interventions and controls. The 3-layer screening was conducted using prompts tailored to each study. During this process, information extraction according to each study's inclusion criteria and optimization for screening were carried out using a GPT-4-based flow without manual adjustments. Records were evaluated at each layer, and those meeting the inclusion criteria at all layers were subsequently judged as included. RESULTS On each layer, both GPT-3.5 and GPT-4 were able to process about 110 records per minute, and the total time required for screening the first and second studies was approximately 1 hour and 2 hours, respectively. In the first study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.900/0.709 and 0.806/0.996, respectively. Both screenings by GPT-3.5 and GPT-4 judged all 6 records used for the meta-analysis as included. In the second study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.958/0.116 and 0.875/0.855, respectively. The sensitivities for the relevant records align with those of human evaluators: 0.867-1.000 for the first study and 0.776-0.979 for the second study. Both screenings by GPT-3.5 and GPT-4 judged all 9 records used for the meta-analysis as included. After accounting for justifiably excluded records by GPT-4, the sensitivities/specificities of the GPT-4 screening were 0.962/0.996 in the first study and 0.943/0.855 in the second study. Further investigation indicated that the cases incorrectly excluded by GPT-3.5 were due to a lack of domain knowledge, while the cases incorrectly excluded by GPT-4 were due to misinterpretations of the inclusion criteria. CONCLUSIONS Our 3-layer screening method with GPT-4 demonstrated acceptable level of sensitivity and specificity that supports its practical application in systematic review screenings. Future research should aim to generalize this approach and explore its effectiveness in diverse settings, both medical and nonmedical, to fully establish its use and operational feasibility. 98.1% topic match

Compares GPT-3.5 and GPT-4 to human evaluators. Provides quantitative sensitivity and specificity metrics for systematic review screening. Lacks explicit direct comparison to traditional screening methods' efficiency and accuracy.

97.7%

8.9

2023

[3] Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews Eugene Syriani, ..., and Gauransh Kumar ArXiv 2023 - 12 citations - Show abstract - Cite - PDF 97.7% topic match

Compares ChatGPT and traditional methods in article screening. Evaluates ChatGPT's consistency, classification performance, and generalizability for systematic review automation against traditional classifiers. Published in 2023; employs quantitative analysis but focuses on ChatGPT, lacking mention of GPT-3.5 or GPT-4 models specifically.

97.5%

0.0

2023

[4] Sensitivity, specificity and avoidable workload of using a large language models for title and abstract screening in systematic reviews and meta-analyses Viet-thi Tran, ..., and PhD Viet-Thi Tran - MD medRxiv 2023 - 0 citations - Show abstract - Cite - PDF Abstract: Importance: Systematic reviews are time-consuming and are still performed predominately manually by researchers despite the exponential growth of scientific literature. Objective: To investigate the sensitivity, specificity and estimate the avoidable workload when using an AI-based large language model (LLM) (Generative Pre-trained Transformer [GPT] version 3.5-Turbo from OpenAI) to perform title and abstract screening in systematic reviews. Data Sources: Unannotated bibliographic databases from five systematic reviews conducted by researchers from Cochrane Austria, Germany and France, all published after January 2022 and hence not in the training data set from GPT 3.5-Turbo. Design: We developed a set of prompts for GPT models aimed at mimicking the process of title and abstract screening by human researchers. We compared recommendations from LLM to rule out citations based on title and abstract with decisions from authors, with a systematic reappraisal of all discrepancies between LLM and their original decisions. We used bivariate models for meta-analyses of diagnostic accuracy to estimate pooled estimates of sensitivity and specificity. We performed a simulation to assess the avoidable workload from limiting human screening on title and abstract to citations which were not "ruled out" by the LLM in a random sample of 100 systematic reviews published between 01/07/2022 and 31/12/2022. We extrapolated estimates of avoidable workload for health-related systematic reviews assessing therapeutic interventions in humans published per year. Results: Performance of GPT models was tested across 22,666 citations. Pooled estimates of sensitivity and specificity were 97.1% (95%CI 89.6% to 99.2%) and 37.7%, (95%CI 18.4% to 61.9%), respectively. In 2022, we estimated the workload of title and abstract screening for systematic reviews to range from 211,013 to 422,025 person-hours. Limiting human screening to citations which were not "ruled out" by GPT models could reduce workload by 65% and save up from 106,268 to 276,053-person work hours (i.e.,66 to 172-person years of work), every year. Conclusions and Relevance: AI systems based on large language models provide highly sensitive and moderately specific recommendations to rule out citations during title and abstract screening in systematic reviews. Their use to "triage" citations before human assessment could reduce the workload of evidence synthesis. 97.5% topic match

Provides quantitative analysis of GPT-3.5's performance in screening abstracts. Compares GPT-3.5-Turbo's sensitivity and specificity to manual screening in systematic reviews. Focuses on avoidable workload but lacks analysis against traditional screening efficiency metrics like precision or F1.

96.7%

0.0

2023

[5] Development of meta-prompts for Large Language Models to screen titles and abstracts for diagnostic test accuracy reviews Y. Kataoka, ..., and T. A. Furukawa medRxiv 2023 - 0 citations - Show abstract - Cite - PDF 96.7% topic match

Provides empirical sensitivity and specificity metrics for GPT-3.5 and GPT-4. Shows GPT models' performance in screening abstracts for diagnostic test accuracy systematic reviews. Does not directly compare GPT models' efficiency or accuracy against traditional methods.

96.4%

16.9

2023

[6] Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study Eddie Guo, ..., and C. Naugler Journal of Medical Internet Research 2023 - 26 citations - Show abstract - Cite - PDF Abstract: Background The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. Objective This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. Methods We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. Results Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. Conclusions Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research. 96.4% topic match

Evaluates GPT-4 in screening clinical review papers. Compares GPT-4's effectiveness against human reviewers using quantitative metrics. Focuses on accuracy and efficiency but lacks traditional method comparison.

95.2%

0.0

2024

[7] Comparison of large language models for citation screening: A protocol for a prospective study T. Oami, ..., and T.-a. Nakada medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 95.2% topic match

Compares GPT-4 with traditional citation screening methods. Evaluates accuracy, efficiency, and cost for citation screening using GPT-4 and compares it to conventional techniques. Includes sensitivity and specificity comparisons but lacks explicit mention of recall, precision, or F1 score metrics.

93.8%

0.0

2024

[8] Performance of a Large Language Model in Screening Citations T. Oami, ..., and T.-a. Nakada JAMA Network Open 2024 - 0 citations - Show abstract - Cite 93.8% topic match

Shows LLM can improve citation screening in systematic reviews. Demonstrates significant reduction in screening time with reasonable accuracy and specificity compared to traditional methods. Does not specify if ChatGPT, GPT-3.5, or GPT-4 is used; comparison with traditional methods is discussed, but model specifics are unclear.

92.1%

0.0

2024

[9] Sensitivity and Specificity of Using GPT-3.5 Turbo Models for Title and Abstract Screening in Systematic Reviews and Meta-analyses Viet-Thi Tran, ..., and Philippe Ravaud Annals of Internal Medicine 2024 - 0 citations - Show abstract - Cite Abstract: BACKGROUND Systematic reviews are performed manually despite the exponential growth of scientific literature. OBJECTIVE To investigate the sensitivity and specificity of GPT-3.5 Turbo, from OpenAI, as a single reviewer, for title and abstract screening in systematic reviews. DESIGN Diagnostic test accuracy study. SETTING Unannotated bibliographic databases from 5 systematic reviews representing 22 665 citations. PARTICIPANTS None. MEASUREMENTS A generic prompt framework to instruct GPT to perform title and abstract screening was designed. The output of the model was compared with decisions from authors under 2 rules. The first rule balanced sensitivity and specificity, for example, to act as a second reviewer. The second rule optimized sensitivity, for example, to reduce the number of citations to be manually screened. RESULTS Under the balanced rule, sensitivities ranged from 81.1% to 96.5% and specificities ranged from 25.8% to 80.4%. Across all reviews, GPT identified 7 of 708 citations (1%) missed by humans that should have been included after full-text screening at the cost of 10 279 of 22 665 false-positive recommendations (45.3%) that would require reconciliation during the screening process. Under the sensitive rule, sensitivities ranged from 94.6% to 99.8% and specificities ranged from 2.2% to 46.6%. Limiting manual screening to citations not ruled out by GPT could reduce the number of citations to screen from 127 of 6334 (2%) to 1851 of 4077 (45.4%), at the cost of missing from 0 to 1 of 26 citations (3.8%) at the full-text level. LIMITATIONS Time needed to fine-tune prompt. Retrospective nature of the study, convenient sample of 5 systematic reviews, and GPT performance sensitive to prompt development and time. CONCLUSION The GPT-3.5 Turbo model may be used as a second reviewer for title and abstract screening, at the cost of additional work to reconcile added false positives. It also showed potential to reduce the number of citations before screening by humans, at the cost of missing some citations at the full-text level. PRIMARY FUNDING SOURCE None. 92.1% topic match

Shows potential of GPT-3.5 Turbo in screening tasks. Examines sensitivity, specificity, and efficiency as a second reviewer in systematic reviews, highlighting trade-offs in false positives. Lacks direct quantitative comparison with traditional methods and focuses on qualitative trade-offs.

91.3%

18.0

2023

[10] Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages Qusai Khraisha, ..., and Kristin Hadfield Research synthesis methods 2023 - 19 citations - Show abstract - Cite - PDF 91.3% topic match

Shows GPT-4's efficacy in systematic review screening and extraction. Compares GPT-4's performance to humans, highlighting robustness with reliable prompts in unbalanced datasets. While it discusses GPT-4's screening performance, there's no direct comparison with traditional screening methods mentioned.

90.8%

0.0

2024

[11] The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews Aleksi Huotala, ..., and Mika Mäntylä Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Context: Systematic review (SR) is a popular research method in software engineering (SE). However, conducting an SR takes an average of 67 weeks. Thus, automating any step of the SR process could reduce the effort associated with SRs. Objective: Our objective is to investigate the extent to which Large Language Models (LLMs) can accelerate title-abstract screening by (1) simplifying abstracts for human screeners, and (2) automating title-abstract screening entirely. Method: We performed an experiment where human screeners performed title-abstract screening for 20 papers with both original and simplified abstracts from a prior SR. The experiment with human screeners was reproduced by instructing GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We also studied whether different prompting techniques (Zero-shot (ZS), One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT) prompting) improve the screening performance of LLMs. Lastly, we studied if redesigning the prompt used in the LLM reproduction of title-abstract screening leads to improved screening performance. Results: Text simplification did not increase the screeners’ screening performance, but reduced the time used in screening. Screeners’ scientific literacy skills and researcher status predict screening performance. Some LLM and prompt combinations perform as well as human screeners in the screening tasks. Our results indicate that a more recent LLM (GPT-4) is better than its predecessor LLM (GPT-3.5). Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting. Conclusion: Using LLMs for text simplification in the screening process does not significantly improve human performance. Using LLMs to automate title-abstract screening seems promising, but current LLMs are not significantly more accurate than human screeners. To recommend the use of LLMs in the screening process of SRs, more research is needed. We recommend future SR studies to publish replication packages with screening data to enable more conclusive experimenting with LLM screening. 90.8% topic match

Compares GPT-3.5 and GPT-4 in systematic review screening. Evaluates performance with various prompting techniques against traditional human screening methods. Focuses on qualitative aspects more, not specifically on empirical quantitative metrics like precision or F1 score.

88.4%

0.8

2023

[12] Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation PhD Xiangming Cai, ..., and Prof. Duolao Wang medRxiv 2023 - 1 citations - Show abstract - Cite - PDF Abstract: Background: Large language models (LLMs) like ChatGPT showed great potential in aiding medical research. A heavy workload in filtering records is needed during the research process of evidence-based medicine, especially meta-analysis. However, no study tried to use LLMs to help screen records in meta-analysis. In this research, we aimed to explore the possibility of incorporating ChatGPT to facilitate the screening step based on the title and abstract of records during meta-analysis. Methods: To assess our strategy, we selected three meta-analyses from the literature, together with a glioma meta-analysis embedded in the study, as additional validation. For the automatic selection of records from curated meta-analyses, a four-step strategy called LARS was developed, consisting of (1) criteria selection and single-prompt (prompt with one criterion) creation, (2) best combination identification, (3) combined-prompt (prompt with one or more criteria) creation, and (4) request sending and answer summary. We evaluated the robustness of the response from ChatGPT with repeated requests. Recall, workload reduction, precision, and F1 score were calculated to assess the performance of LARS. Findings: ChatGPT showed a stable response for repeated requests (robustness score: 0.747 - 0.996). A variable performance was found between different single-prompts with a mean recall of 0.841. Based on these single-prompts, we were able to find combinations with performance better than the pre-set threshold. Finally, with a best combination of criteria identified, LARS showed a 39.5% workload reduction on average with a recall greater than 0.9. In the glioma meta-analysis, we found no prognostic effect of CD8+ TIL on overall survival, progress-free survival, and survival time after immunotherapy. Interpretation: We show here the groundbreaking finding that automatic selection of literature for meta-analysis is possible with ChatGPT. We provide it here as a pipeline, LARS, which showed a great workload reduction while maintaining a pre-set recall. Funding: China Scholarship Council. 88.4% topic match

Shows ChatGPT reduces workload in meta-analysis literature selection. Employs quantitative measures including recall, precision, and F1 score to evaluate effectiveness. Does not mention direct comparison with traditional screening methods, potentially lacking in full relevance.

83.3%

0.0

2023

[13] Citation screening using large language models for creating clinical practice guidelines: A protocol for a prospective study T. Oami, ..., and T.-a. Nakada medRxiv 2023 - 0 citations - Show abstract - Cite - PDF 83.3% topic match

Provides a comparative analysis of GPT-4 for citation screening. Compares efficiency and accuracy of GPT-4 against traditional methods for clinical guideline screening. Focuses on efficiency and sensitivity, mentioning workload reduction, but lacks detailed recall, precision, and F1 metrics analysis.

82.5%

1.0

2023

[14] Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model David Wilkins ArXiv 2023 - 1 citations - Show abstract - Cite - PDF 82.5% topic match

Introduces GPT-4 for automated screening in scoping reviews. Uses GPTscreenR to evaluate performance, showing 71% sensitivity, 89% specificity, 84% accuracy. Focuses on GPT-4 without direct comparison to traditional methods, so partially relevant.

80.2%

0.0

2024

[15] A Hybrid Semi-Automated Workflow for Systematic and Literature Review Processes with Large Language Model Analysis Anjia Ye, ..., and Scott Pedersen Future Internet 2024 - 0 citations - Show abstract - Cite - PDF 80.2% topic match

Develops a hybrid LLM-human approach for systematic reviews. Combines GPT-4/ChatGPT with human oversight to improve efficiency and accuracy in screening processes. Does not provide direct quantitative comparison of performance metrics versus traditional methods.

80.0%

6.2

2024

[16] Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis Michaël Chelli, ..., and C. Ruetsch-Chelli Journal of Medical Internet Research 2024 - 3 citations - Show abstract - Cite - PDF Abstract: Background Large language models (LLMs) have raised both interest and concern in the academic community. They offer the potential for automating literature search and synthesis for systematic reviews but raise concerns regarding their reliability, as the tendency to generate unsupported (hallucinated) content persist. Objective The aim of the study is to assess the performance of LLMs such as ChatGPT and Bard (subsequently rebranded Gemini) to produce references in the context of scientific writing. Methods The performance of ChatGPT and Bard in replicating the results of human-conducted systematic reviews was assessed. Using systematic reviews pertaining to shoulder rotator cuff pathology, these LLMs were tested by providing the same inclusion criteria and comparing the results with original systematic review references, serving as gold standards. The study used 3 key performance metrics: recall, precision, and F1-score, alongside the hallucination rate. Papers were considered “hallucinated” if any 2 of the following information were wrong: title, first author, or year of publication. Results In total, 11 systematic reviews across 4 fields yielded 33 prompts to LLMs (3 LLMs×11 reviews), with 471 references analyzed. Precision rates for GPT-3.5, GPT-4, and Bard were 9.4% (13/139), 13.4% (16/119), and 0% (0/104) respectively (P<.001). Recall rates were 11.9% (13/109) for GPT-3.5 and 13.7% (15/109) for GPT-4, with Bard failing to retrieve any relevant papers (P<.001). Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001). Further analysis of nonhallucinated papers retrieved by GPT models revealed significant differences in identifying various criteria, such as randomized studies, participant criteria, and intervention criteria. The study also noted the geographical and open-access biases in the papers retrieved by the LLMs. Conclusions Given their current performance, it is not recommended for LLMs to be deployed as the primary or exclusive tool for conducting systematic reviews. Any references generated by such models warrant thorough validation by researchers. The high occurrence of hallucinations in LLMs highlights the necessity for refining their training and functionality before confidently using them for rigorous academic purposes. 80.0% topic match

Provides a comparative analysis of ChatGPT and Bard for systematic reviews. Evaluates recall, precision, F1-score, and hallucination rates against human-conducted reviews. Examines GPT-3.5 and GPT-4 performance; lacks direct comparison with traditional screening methods.

79.9%

1.1

2023

[17] A Novel Question-Answering Framework for Automated Citation Screening Using Large Language Models N. O. Akinseloyin, ..., and Vasile Palade medRxiv 2023 - 1 citations - Show abstract - Cite - PDF Abstract: Objective: This paper aims to address the challenges in citation screening (a.k.a. abstract screening) within Systematic Reviews (SR) by leveraging the zero-shot capabilities of large language models, particularly ChatGPT. Methods: We employ ChatGPT as a zero-shot ranker to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Citation screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by ChatGPT. The framework involves breaking down the selection criteria into multiple questions, properly prompting ChatGPT to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions. Results: Large-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology Assisted Reviews in Empirical Medicine. Across 31 datasets of four categories of SRs, the proposed QA framework consistently outperformed other zero-shot ranking models. Compared with complex ranking approaches with iterative relevance feedback and fine-tuned deep learning-based ranking models, our ChatGPT-based zero-shot citation screening approaches still demonstrated competitive and sometimes better results, underscoring their high potential in facilitating automated systematic reviews. Conclusion: Investigation justified the indispensable value of leveraging selection criteria to improve the performance of automated citation screening. ChatGPT demonstrated proficiency in prioritizing candidate studies for citation screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance citation screening. 79.9% topic match

Demonstrates ChatGPT's capabilities in automated citation screening. ChatGPT-based zero-shot approaches compared favorably to complex and fine-tuned models across multiple systematic review datasets. The study uses performance metrics but does not directly compare ChatGPT with traditional screening methods quantitatively.

78.9%

0.0

2024

[18] AID-SLR: A Generative Artificial Intelligence-Driven Automated System for Systematic Literature Review PhD Kyeryoung Lee, ..., and PhD Xiaoyan Wang medRxiv 2024 - 0 citations - Show abstract - Cite Abstract: Systematic literature reviews (SLRs) are crucial for generating research evidence, supporting clinical decisions, advancing scientific knowledge, and informing policymaking. Despite their importance, manual SLRs are time-consuming, costly, and prone to errors. The increasing volume of published data and the complexity of clinical trials necessitate more efficient approaches. We present an automated SLR system using large language models (LLMs), designed to streamline the entire SLR process from initial query to data extraction, and customizable for various study fields. We developed an LLM-assisted SLR system, AID-SLR, accompanied by a user interface (UI) comprising 6 modules 1) Query, 2) Inclusion/Exclusion (I/E) criteria, 3) Abstract screening, 4) Full-text screening, 5) Data extraction, and 6) Data summary. The LLM model was utilized for abstract screening, full-text screening, and data extraction and its performance was evaluated using precision, recall, and F1 scores. We selected a non-small cell lung cancer use case to evaluate the system performance. We additionally compared the performance of GPT-4 and GPT-4o models, focusing on data extraction across different categories. A qualitative evaluation was conducted to assess common error types and the reliability of extracted information. AID-SLR is user-centric, allowing users to specify study criteria and provide additional information and feedback. The LLM prompts are generalizable and automatically incorporate the user-entered details and instructions on the UI, such as domain-specific guidelines, thereby enabling easy adoption of the system to different study and disease areas. AID-SLR effectively screens relevant studies and extracts data elements. The system demonstrated high precision, recall, and F1 scores in screening both Irrelevant (1, 0.9286, and 0.9630, respectively) and Relevant (0.9737, 1, and 0.9867, respectively) articles, with an overall accuracy of 98.04%. Data extraction was granular with promising performance, successfully identifying a wide range of treatment-related outcomes and statistical values. For data extraction, GPT-4o outperformed GPT-4, achieving higher precision (0.9984 vs. 0.9819), recall (0.9989 vs. 0.9519), and F1-score (0.9987 vs. 0.9651). GPT-4o also exhibited superior performance in cohort identification and value extraction, with fewer errors and more accurate capture of study design and demographic information. Our LLM system and UI provided a seamless end-to-end solution for automated SLRs. This automated SLR system can contribute to reducing the time, cost, and human errors associated with traditional manual SLRs. Integrating this model with other AI tools for comprehensive data analysis could further enhance its utility in SLRs. 78.9% topic match

Compares GPT-4 and GPT-4o performance in data extraction. High precision, recall, and F1 scores achieved for screening articles. Focuses on qualitative error evaluation; lacks direct comparison to traditional methods.

77.9%

0.0

2024

[19] A105 PILOT STUDY ON THE ACCURACY OF CHATGPT IN ARTICLE SCREENING FOR SYSTEMATIC REVIEWS IN GASTROENTEROLOGY C. Na, ..., and S. Grover Journal of the Canadian Association of Gastroenterology 2024 - 0 citations - Show abstract - Cite Abstract: Abstract Background Systematic reviews synthesize extant research to answer a research question in a way that minimizes bias. After articles for potential inclusion are identified by sensitive searches, screening requires human expert review, which may be time-consuming and subjective. Large language models such as ChatGPT may have potential for this application. Aims This pilot study aims to assess the accuracy of ChatGPT 3.5 in screening of articles for systematic reviews in gastroenterology by (1) identifying if articles were correctly included and (2) excluding articles reported by authors as difficult to assess. Methods We searched the Cochrane Library for gastroenterology systematic reviews (January 1, 2022 to May 31, 2023) and selected the 10 most cited studies. The test set used to determine the accuracy of Open AI’s ChatGPT 3.5 model for included studies was the final list of included studies for each Cochrane review. The test set used for studies challenging to assess was the “excluded studies” list as defined in the Cochrane Handbook. Figure 1 shows the prompt used for the screening query. Articles were omitted if they did not have digital sources, abstracts or methods. Each article was screened 10 times to account for variability within ChatGPT’s outputs. Articles with ≥5 inclusion results were counted as an included study. Results ChatGPT correctly identified included studies at rates ranging from 60% to 100%. ChatGPT correctly identified exlcuded studies at rates ranging from 0% to 50% (Table 1). A total of 265 articles were screened. Conclusions In this pilot study, we demonstrated that ChatGPT is accurate in identifying articles screened for inclusion in Cochrane reviews; however, it is inaccurate in excluding articles described by the authors as being difficult to assess. We hypothesize that the GPT 3.5 model can read for keywords and broad interventions but is unable to reason cognitively, as an expert would, as to why a study may be excluded. We aim to review reasons for exclusion in future work. Table 1. Screening Results of ChatGPT Review author and date Topic No. of studies included by authors No. of studies excluded by authors No. of studies correctly included by ChatGPT (%) No. of studies correctly excluded by ChatGPT(%) Tse, 2022 Guide-wire assisted cannulation 7 14 7 (100%) 0 (0%) Gordon, 2023 Remote care through telehealth for IBD patients 14 10 14 (100%) 3 (30%) Candy, 2022 Mu-opioid antagonists for opioid-induced bowel dysfunction 10 7 10 (100%) 0 (0%) El-Nakeep, 2022 Stem cell transplantation in Crohn 7 10 7 (100%) 5 (50%) Okabayashi, 2022 Certolizumab pegol in Crohn 5 5 3 (60%) 1 (20%) Gordon, 2023 Patient education in IBD management 19 20 18 (95%) 2 (10%) Dichman, 2022 Antibiotics for uncomplicated diverticulitis 6 6 6 (100%) 0 (0%) Grobbee, 2022 Faecal occult blood tests versus faecal immunochemical tests for colorectal cancer screening 53 26 46 (87%) 2 (8%) Midya, 2022 Fundoplication in laparoscopic Heller 9 3 8 (89%) 0 (0%) Imdad, 2023 Fecal transplantation for IBD 13 21 13 (100%) 2 (10%) Figure 1. ChatGPT Screening Prompt Funding Agencies None 77.9% topic match

Assesses ChatGPT 3.5's accuracy in systematic review screening. Evaluates ChatGPT 3.5's ability to screen gastroenterology articles against human expert reviews in recent systematic reviews. No explicit comparison with traditional methods or detailed performance metrics like precision and recall.

65.6%

25.9

2023

[20] Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions Ahmad Alshami, ..., and T. Zayed Syst. 2023 - 35 citations - Show abstract - Cite - PDF Abstract: Systematic reviews (SR) are crucial in synthesizing and analyzing existing scientific literature to inform evidence-based decision-making. However, traditional SR methods often have limitations, including a lack of automation and decision support, resulting in time-consuming and error-prone reviews. To address these limitations and drive the field forward, we harness the power of the revolutionary language model, ChatGPT, which has demonstrated remarkable capabilities in various scientific writing tasks. By utilizing ChatGPT’s natural language processing abilities, our objective is to automate and streamline the steps involved in traditional SR, explicitly focusing on literature search, screening, data extraction, and content analysis. Therefore, our methodology comprises four modules: (1) Preparation of Boolean research terms and article collection, (2) Abstract screening and articles categorization, (3) Full-text filtering and information extraction, and (4) Content analysis to identify trends, challenges, gaps, and proposed solutions. Throughout each step, our focus has been on providing quantitative analyses to strengthen the robustness of the review process. To illustrate the practical application of our method, we have chosen the topic of IoT applications in water and wastewater management and quality monitoring due to its critical importance and the dearth of comprehensive reviews in this field. The findings demonstrate the potential of ChatGPT in bridging the gap between traditional SR methods and AI language models, resulting in enhanced efficiency and reliability of SR processes. Notably, ChatGPT exhibits exceptional performance in filtering and categorizing relevant articles, leading to significant time and effort savings. Our quantitative assessment reveals the following: (1) the overall accuracy of ChatGPT for article discarding and classification is 88%, and (2) the F-1 scores of ChatGPT for article discarding and classification are 91% and 88%, respectively, compared to expert assessments. However, we identify limitations in its suitability for article extraction. Overall, this research contributes valuable insights to the field of SR, empowering researchers to conduct more comprehensive and reliable reviews while advancing knowledge and decision-making across various domains. 65.6% topic match

Provides a quantitative evaluation of ChatGPT for systematic review screening. Shows ChatGPT achieving 88% accuracy and F-1 scores of 91% and 88% in discarding and classifying articles. Does not directly compare ChatGPT to traditional screening methods; limitations noted in article extraction suitability.

58.8%

0.0

2023

[21] Accelerating literature screening for systematic literature reviews with Large Language Models - development, application, and first evaluation of a solution Paul Herbst and Henning Baars Lernen, Wissen, Daten, Analysen 2023 - 0 citations - Show abstract - Cite 58.8% topic match

Proposes and evaluates an LLM solution for literature screening. The paper implements and tests LLMs for abstract screening, highlighting automation potential for systematic literature reviews. Does not specify models like ChatGPT or GPT-4, nor details on quantitative performance metrics.

58.8%

0.0

2024

[22] Accuracy and reliability of data extraction for systematic reviews using large language models: A protocol for a prospective study T. Oami, ..., and T.-a. Nakada medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 58.8% topic match

Evaluates LLMs for data extraction accuracy in systematic reviews. Focuses on GPT-4 Turbo, Claude 3, Gemini 1.5 Pro, assessing accuracy and task completion time. Lacks direct comparison against traditional screening methods or quantitative performance metrics like precision, recall, or F1 score.

48.0%

1.6

2024

[23] Methodological insights into ChatGPT’s screening performance in systematic reviews Mahbod Issaiy, ..., and K. Firouznia BMC Medical Research Methodology 2024 - 1 citations - Show abstract - Cite - PDF 48.0% topic match

46.8%

3.0

2024

[24] Cutting Through the Clutter: The Potential of LLMs for Efficient Filtration in Systematic Literature Reviews Lucas Joos, ..., and M. T. Fischer ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 46.8% topic match

Shows potential of LLMs for efficient literature review filtering. Compares GPT-4o and other LLMs to human performance in screening over 8.3k articles. Lacks specific mention of performance metrics like precision, recall, F1 score, or direct quantitative comparison with traditional methods.

42.5%

4.3

2024

[25] Evaluating the effectiveness of large language models in abstract screening: a comparative analysis Michael Li, ..., and Xianming Tan Systematic Reviews 2024 - 1 citations - Show abstract - Cite 42.5% topic match

42.0%

0.0

2024

[26] A question-answering framework for automated abstract screening using large language models. Opeoluwa Akinseloyin, ..., and Vasile Palade Journal of the American Medical Informatics Association : JAMIA 2024 - 0 citations - Show abstract - Cite Abstract: OBJECTIVE This paper aims to address the challenges in abstract screening within systematic reviews (SR) by leveraging the zero-shot capabilities of large language models (LLMs). METHODS We employ LLM to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Abstract screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by LLM. The framework involves breaking down the selection criteria into multiple questions, properly prompting LLM to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions. RESULTS AND DISCUSSION Large-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology-Assisted Reviews in Empirical Medicine. Focusing on GPT-3.5 as a case study, the proposed QA framework consistently exhibited a clear advantage over traditional information retrieval approaches and bespoke BERT-family models that were fine-tuned for prioritizing candidate studies (ie, from the BERT to PubMedBERT) across 31 datasets of 4 categories of SRs, underscoring their high potential in facilitating abstract screening. The experiments also showcased the viability of using selection criteria as a query for reference prioritization. The experiments also showcased the viability of the framework using different LLMs. CONCLUSION Investigation justified the indispensable value of leveraging selection criteria to improve the performance of automated abstract screening. LLMs demonstrated proficiency in prioritizing candidate studies for abstract screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance abstract screening. 42.0% topic match

Provides a zero-shot QA framework for abstract screening using GPT-3.5. Transform selection criteria into questions to enhance abstract screening efficacy, outperforming traditional methods and BERT-family models. Focuses on GPT-3.5, utilizing empirical evaluations but lacks direct mention of performance metrics like recall or precision against traditional methods.

41.2%

6.6

2023

[27] Applications of the Natural Language Processing Tool ChatGPT in Clinical Practice: Comparative Study and Augmented Systematic Review Nikolas Schopow, ..., and David Baur JMIR Medical Informatics 2023 - 10 citations - Show abstract - Cite - PDF 41.2% topic match

Compares ChatGPT and human researchers in systematic review tasks. Evaluates agreement using Fleiss and Cohen κ statistics for parameter extraction. Focuses on extracting information from clinical articles, lacks use of quantitative performance metrics like precision or recall.

31.9%

1.5

2024

[28] Large Language Model in Medical Information Extraction from Titles and Abstracts with Prompt Engineering Strategies: A Comparative Study of GPT-3.5 and GPT-4 Y. Tang, ..., and Research Data Collaboration Task Force medRxiv 2024 - 1 citations - Show abstract - Cite - PDF Abstract: Background: Large language models (LLMs) have significantly enhanced the Natural Language Processing (NLP), offering significant potential in facilitating medical literature review. However, the accuracy, stability and prompt strategies associated with LLMs in extracting complex medical information have not been adequately investigated. Our study assessed the capabilities of GPT-3.5 and GPT-4.0 in extracting or summarizing seven crucial medical information items from the title and abstract of research papers. We also validated the impact of prompt engineering strategies and the effectiveness of evaluating metrics. Methodology: We adopted a stratified sampling method to select 100 papers from the teaching schools and departments in the LKS Faculty of Medicine, University of Hong Kong, published between 2015 and 2023. GPT-3.5 and GPT-4.0 were instructed to extract seven pieces of information, including study design, sample size, data source, patient, intervention, comparison, and outcomes. The experiment incorporated three prompt engineering strategies: persona, chain-of-thought and few-shot prompting. We employed three metrics to assess the alignment between the GPT output and the ground truth: BERTScore, ROUGE-1 and a self-developed GPT-4.0 evaluator. Finally, we evaluated and compared the proportion of correct answers among different GPT versions and prompt engineering strategies. Results: GPT demonstrated robust capabilities in accurately extracting medical information from titles and abstracts. The average accuracy of GPT-4.0, when paired with the optimal prompt engineering strategy, ranged from 0.688 to 0.964 among the seven items, with sample size achieving the highest score and intervention yielding the lowest. GPT version was shown to be a statistically significant factor in model performance, but prompt engineering strategies did not exhibit cumulative effects on model performance. Additionally, our results showed that the GPT-4.0 evaluator outperformed the ROUGE-1 and BERTScore in assessing the alignment of information (Accuracy: GPT-4.0 Evaluator: 0.9714, ROUGE-1: 0.9429, BERTScore: 0.8714). Conclusion: Our result confirms the effectiveness of LLMs in extracting medical information, suggesting their potential as efficient tools for literature review. We recommend utilizing an advanced version of LLMs to enhance the model performance, while prompt engineering strategies should be tailored to the specific tasks. Additionally, LLMs show promise as an evaluation tool to assess the model performance related to complex information processing. 31.9% topic match

Compares GPT-3.5 and GPT-4 in information extraction. Uses BERTScore, ROUGE-1, and GPT-4.0 evaluator for performance metrics. Lacks direct comparison to traditional screening methods for systematic reviews.

27.1%

0.0

2024

[29] How well it works: Benchmarking performance of GPT models on medical natural language processing tasks ScD Kathryn Rough, ..., and PhD Benjamin Hughes medRxiv 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Importance: The ability of large language models (LLMs) to generate high-quality, human-like text has been accompanied with speculation about their application in healthcare, alongside ethical and safety concerns. Objective: Evaluate LLM performance on medical natural language processing (NLP) tasks, benchmarked against other commercially available tools. Design: Observational study to evaluate and compare model performance. All models were commercially available and were evaluated without modification. Setting: The Text Analysis Coding (TAC) 2017 challenge was used to assess ability to perform medical coding using standard MedDRA preferred terms. Text from 55 publicly available de-identified medical transcription reports were annotated to identify pre-defined medical concepts (age, disease/symptom, body structure, medication name, and medication dosage). Participants: Publicly available, de-identified adverse event and medical transcription reports were used for evaluation. Exposures: For each task, general LLMs (GPT-3.5-turbo, GPT-4) were compared to commercially available healthcare NLP tools (Microsoft Text Analytics for Health, Amazon Comprehend Medical, IQVIA API Marketplace). Main Outcomes and Measures: For each NLP task, sensitivity, positive predictive value (PPV) and F1 score were calculated. Because GPT models had variable outputs, the range of metrics over 5 trials is reported. Results: For MedDRA coding, GPT-4 had similar F1 score performance to healthcare NLP algorithms (GPT-4: 0.67 to 0.73; Microsoft Text Analytics for Health: 0.66, IQVIA API Marketplace: 0.72), while GPT- 3.5-turbo had considerably lower performance (0.50 to 0.51). For medical information extraction, LLM performance varied widely across differing medical concepts; the highest F1 scores were for age (GPT- 3.5-turbo: 0.82 to 0.83, GPT-4: 0.84 to 0.87) and medication name (GPT-3.5-turbo: 0.55 to 0.59, GPT-4: 0.70 to 0.76), while F1 scores for disease/symptom, body structure, and medication dosage were lower than those observed for the healthcare NLP tools. GPT-3.5-turbo and GPT-4 generally had lower sensitivity than comparators. Conclusions and Relevance: In the absence of domain-specific fine tuning, GPT-4 performed similarly to healthcare-specific NLP tools on some tasks and less accurately on others; GPT-3.5-turbo was consistently less accurate than comparators. To maximize benefit and reduce risk of harm, robust quantitative evaluation for specific tasks should be performed prior to implementing LLMs in medical contexts. 27.1% topic match

Compares GPT models on medical NLP tasks using performance metrics. Reports F1 scores for GPT-4 and GPT-3.5 across several medical coding and information extraction tasks. Does not focus solely on screening for systematic reviews or comparisons with traditional methods.

24.6%

8.1

2023

[30] Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo Omid Kohandel Gargari, ..., and Reza Samiee BMJ Evidence-Based Medicine 2023 - 8 citations - Show abstract - Cite - PDF 24.6% topic match

24.1%

4.8

2024

[31] Zero-shot Generative Large Language Models for Systematic Review Screening Automation Shuai Wang, ..., and G. Zuccon ArXiv 2024 - 4 citations - Show abstract - Cite - PDF 24.1% topic match

Evaluates zero-shot LLMs for systematic review screening automation. Investigates using eight LLMs with calibration for improved recall thresholds in systematic reviews. Lacks direct comparison to traditional screening methods and specific mention of GPT-3.5 or GPT-4 efficiency metrics.

22.0%

4.8

2021

[32] Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study Shuai Wang, ..., and G. Zuccon ArXiv 2021 - 14 citations - Show abstract - Cite - PDF 22.0% topic match

21.7%

0.0

2024

[33] Automating Clinical Trial Eligibility Screening: Quantitative Analysis of GPT Models versus Human Expertise Arti Devi, ..., and Varun Dutt Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments 2024 - 0 citations - Show abstract - Cite Abstract: Objective: This study quantitatively assesses the performance of GPT model in classifying patient eligibility for clinical trials, aiming to minimize the need for expert clinical judgment and to cut down on related expenses. Data: Ten US NSCLC drug-only interventional clinical trials were selected from clinicaltrials.gov. For each clinical trial, 10 patient profiles were manually created by an epidemiologist using case presentations from medical journals. The dataset included two sets of adult patient profiles (50 eligible patients and 50 non-eligible patients for 100 patients) with a range of complexities, from complex to simple cases. The 100-case dataset was then analyzed in a GPT-3.5-turbo large language model to generate eligibility predictions against human epidemiologists. Analysis: Different data tuning scenarios were evaluated, focusing on the model's ability to replicate the human expert performance in determining patient eligibility. The tuning scenarios included no cases (zero-shot), 50 cases, and 80 cases. Results: GPT-3.5 showed a high accuracy rate during the test, with 95% test accuracy in scenarios without tuning and 100% test accuracy in scenarios with 80-case tuning (testing was done on the remaining 20 cases). GPT-3.5’s accuracy showed more variability, scoring 82% test accuracy with tuning on 50 cases and dropping to 79% test accuracy without tuning on 100 cases. Comparisons of model and human evaluations across male and female patient profiles indicated no gender bias, as the GPT-3.5 model performed equivalently to human assessments. Conclusion: The GPT-3.5 models demonstrated a high degree of accuracy and an unbiased approach to patient classification when compared to human experts. These results suggest that GPT models could serve as a cost-effective and impartial tool for patient screening in clinical trials. Further research with larger and more diverse datasets is recommended to confirm these findings and explore LLMs' scalability in clinical trial settings. 21.7% topic match

20.9%

0.0

2024

[34] Exploring the use of generative artificial intelligence in systematic searching: A comparative case study of a human librarian, ChatGPT-4 and ChatGPT-4 Turbo X. Chen and Yali Feng IFLA Journal 2024 - 0 citations - Show abstract - Cite 20.9% topic match

Compares ChatGPT-4 and a human librarian in systematic review tasks. Analyzes search-term-generation and article-retrieval efficiency, highlighting AI's role in automating repetitive tasks. Lacks quantitative performance metric analysis and direct comparison with traditional screening methods in systematic reviews.

17.3%

0.0

2024

[35] Screening oncology articles in a qualitative literature review using large language models: A comparison of GPT4 versus fine-tuned open source models using expert-annotated data. Kristian Thorlund, ..., and Stacie Hudgens Journal of Clinical Oncology 2024 - 0 citations - Show abstract - Cite Abstract: e23196 Background: Clinical Outcome Assessment (COA) conceptual gap analyses for oncology are complex and time consuming. Artificial Intelligence may efficiently reduce time to completion of such analyses. We aimed to assess two AI models’ performance for literature screening to identify relevant qualitative oncology research. We also compared accuracy and run-time for both AI models. Methods: We manually curated a dataset of title/abstract screening (n = 1,700 study references) across 17 landscape reviews. Among these, 11 landscape reviews (n = 951 study references) spanning over were in oncology, including 8 solid cancers (breast, lung, urothelial, colorectal, esophageal, head and neck, pancreatic, and stomach) and 3 non-solid cancers (lymphoma, acute myeloid leukemia, and multiple myeloma). Each citation was annotated for eligibility (Y/N) by population, study design (qualitative), and reporting of concepts (how patients feel or function). We then compared the accuracy of two AI models at predicting the screening decisions of expert researchers. The two AI models were Generative Pre-trained Transformers 4 (GPT4, OpenAI) prompts and a fine-tuned SciFive biomedical large language model (LLM). We used 70% of the data for training and 30% for test. Accuracy estimates were obtained only for the models’ ability to label eligibility within the 11 oncology datasets. Results: Both LLMs performed well for assessing relevance by oncology population, with F1-scores for the GPT4 and SciFive models being 0.92 and 0.83 respectively (precision was 0.92 and 0.93 respectively). For concept reporting the fine-tuned SciFive model outperformed GPT4 with an F1-score and precision 0.88 and 0.92 versus 0.81 and 0.79. The same was true but less pronounced for eligibility by study design, with an F1-score and precision 0.81 and 0.90 versus 0.86 and 0.76. For overall eligibility, the customized SciFive model outperformed the GPT4 model with an F1-score and precision of 0.84 and 0.92 versus 0.85 and 0.82. Lastly, it took the GPT4 prompts between 10-30 minutes to screen 100 abstracts. By contrast, the customized SciFive model took 1-2 minutes on a computer with a Quadro RTX 8000 GPU. Conclusions: In conclusion, both AI models are promising. The fine-tuned SciFive model appears slightly more accurate and performs substantially faster than the GPT4 model. 17.3% topic match

Provides a comparison of GPT-4 with a fine-tuned model. Evaluates GPT-4's performance using F1-scores and precision in screening oncology articles. Lacks direct comparison to traditional screening methods and focuses on qualitative literature review screening.

16.8%

1.2

2021

[36] Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines. W. Harmsen, ..., and R. Schoot https://doi.org/10.5281/ZENODO.5031907 2021 - 4 citations - Show abstract - Cite 16.8% topic match

16.1%

45.9

2023

[37] Are ChatGPT and large language models “the answer” to bringing us closer to systematic review automation? Riaz Qureshi, ..., and Eitan S. Agai Systematic Reviews 2023 - 71 citations - Show abstract - Cite - PDF 16.1% topic match

15.1%

0.0

2024

[38] Large language models for abstract screening in systematic- and scoping reviews: A diagnostic test accuracy study C. H. Krag, ..., and Felix C Müller medRxiv 2024 - 0 citations - Show abstract - Cite Abstract: Introduction We investigated if large language models (LLMs) can be used for abstract screening in systematic- and scoping reviews. Methods Two broad reviews were designed: a systematic review structured according to the PRISMA guideline with abstract inclusion based on PICO criteria; and a scoping review, where we defined abstract characteristics and features of interest to look for. For both reviews 500 abstracts were sampled. Two readers independently screened abstracts with disagreements handled with arbitrations or consensus, which served as the reference standard. The abstracts were analysed by six LLMs (GPT-4o, GPT-4T, GPT-3.5, Claude3-Opus, Claude3-Sonnet, and Claude3-Haiku). Primary outcomes were diagnostic test accuracy measures for abstract inclusion, abstract characterisation and feature of interest detection. Secondary outcome was the degree of automation using LLMs as a function of the error rate. Results In the systematic review 12 studies were marked as include by the human consensus. GPT-4o, GPT-4T, and Claude3-Opus achieved the highest accuracies (97% to 98%) comparable to the human readers (96% and 98%), although sensitivity was low (33% to 50%). In the scoping review 130 features of interest were present and the LLMs achieved sensitivities between 74-84%, comparable to the human readers (73% and 86%). The specificity of GPT-4o (98%) and GPT-4T (>99%) greatly surpassed the other LLMs (between 33% and 93%). For abstract characterization all LLMs achieved above 95% accuracy for language, manuscript type and study participant characterisation. For characterisation of disease-specific features only GPT-4T and GPT-4o showed very high accuracy. For abstract inclusion the highest automation rate (91%) at the lowest error rate (8%) was achieved by use of two LLMs with disagreement solved by human arbitration. An LLM pre screening before human abstract screening achieved an automation rate of 55% with no missed abstracts. Conclusion Abstract characterisation and specific feature of interest detection with LLMs is feasible and accurate with GPT-4o and GPT-4T. The majority of abstract screenings for systematic reviews can be automated with use of LLMs, at low error rates. 15.1% topic match

14.9%

4.0

2023

[39] Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews Hye Sun Yun, ..., and Byron Wallace Conference on Empirical Methods in Natural Language Processing 2023 - 6 citations - Show abstract - Cite - PDF 14.9% topic match

14.0%

2.6

2024

[40] Accelerating Clinical Evidence Synthesis with Large Language Models Zifeng Wang, ..., and Jimeng Sun ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 14.0% topic match

Shows an LLM-based system improving clinical evidence synthesis. It demonstrates TrialMind's higher efficiency than GPT-4 in clinical tasks, tested by human evaluation. No direct comparison of GPT models to traditional systematic review screening methods is mentioned.

13.3%

1.4

2024

[41] Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurations. Gizem Boztaş Demir, ..., and S. Görgülü European journal of orthodontics 2024 - 1 citations - Show abstract - Cite 13.3% topic match

12.5%

5.9

2023

[42] Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation Shuai Wang, ..., and G. Zuccon Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region 2023 - 7 citations - Show abstract - Cite - PDF 12.5% topic match

Explores NLP models for systematic review screening prioritization. Compares BERT with generative models like ChatGPT for effective query generation using available pre-screening information. Lacks quantitative performance metrics or direct comparison with traditional screening methods, focusing instead on query generation.

12.0%

3.2

2023

[43] Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation Ambrose Robinson, ..., and Xingyi Song ArXiv 2023 - 4 citations - Show abstract - Cite - PDF 12.0% topic match

11.8%

2.0

2023

[44] ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study T. Pitre, ..., and D. Zeraatkar medRxiv 2023 - 2 citations - Show abstract - Cite - PDF Abstract: Background: The assessment of risk of bias is a critical component of systematic review methods. Assessing risk of bias, however, can be time- and resource-intensive. AI-based solutions may increase efficiency and reduce burden. Objective: To evaluate the reliability of ChatGPT for performing risk of bias assessments of randomized trials. Methods: We sampled recently published Cochrane systematic reviews of medical interventions (up to October 2023) that included randomized controlled trials and assessed risk of bias using the Cochrane-endorsed revised risk of bias tool for randomized trials (RoB 2.0). From each eligible review, we collected data on the risk of bias assessments for the first three reported outcomes. Using ChatGPT-4, we assessed the risk of bias for the same outcomes using three different prompts: a minimal prompt including limited instructions, a maximal prompt with extensive instructions, and an optimized prompt that was designed to yield the best risk of bias judgments. The agreement between ChatGPT's assessments and those of the systematic reviewers was quantified using weighted kappa statistics. Results: We included 34 systematic reviews with 157 unique trials. We found the agreement between ChatGPT and systematic review authors for assessment of overall risk of bias to be 0.16 (95% CI: 0.01 to 0.3) for the maximal ChatGPT prompt, 0.17 (95% CI: 0.02 to 0.32) for the optimized prompt, and 0.11 (95% CI: -0.04 to 0.27) for the minimal prompt. For the optimized prompt, agreement ranged between 0.11 (95% CI: -0.11 to 0.33) to 0.29 (95% CI: 0.14 to 0.44) across risk of bias domains, with the lowest agreement for the deviations from the intended intervention domain and the highest agreement for the missing outcome data domain. Conclusion: Our results suggest that ChatGPT and systematic reviewers only have slight to fair agreement in risk of bias judgments for randomized trials. ChatGPT is currently unable to reliably assess risk of bias of randomized trials. We recommend systematic reviewers avoid using ChatGPT to perform risk of bias assessments. 11.8% topic match

11.7%

9.1

2023

[45] ChatGPT and the Future of Journal Reviews: A Feasibility Study Som Biswas, ..., and Harris L. Cohen The Yale Journal of Biology and Medicine 2023 - 11 citations - Show abstract - Cite - PDF Abstract: The increasing volume of research submissions to academic journals poses a significant challenge for traditional peer-review processes. To address this issue, this study explores the potential of employing ChatGPT, an advanced large language model (LLM), developed by OpenAI, as an artificial intelligence (AI) reviewer for academic journals. By leveraging the vast knowledge and natural language processing capabilities of ChatGPT, we hypothesize it may be possible to enhance the efficiency, consistency, and quality of the peer-review process. This research investigated key aspects of integrating ChatGPT into the journal review workflow. We compared the critical analysis of ChatGPT, acting as an AI reviewer, to human reviews for a single published article. Our methodological framework involved subjecting ChatGPT to an intricate examination, wherein its evaluative acumen was juxtaposed against human-authored reviews of a singular published article. As this is a feasibility study, one article was reviewed, which was a case report on scurvy. The entire article was used as an input into ChatGPT and commanded it to “Please perform a review of the following article and give points for revision.” Since this was a case report with a limited word count the entire article could fit in one chat box. The output by ChatGPT was then compared with the comments by human reviewers. Key performance metrics, including precision and overall agreement, were judiciously and subjectively measured to portray the efficacy of ChatGPT as an AI reviewer in comparison to its human counterparts. The outcomes of this rigorous analysis unveiled compelling evidence regarding ChatGPT’s performance as an AI reviewer. We demonstrated that ChatGPT’s critical analyses aligned with those of human reviewers, as evidenced by the inter-rater agreement. Notably, ChatGPT exhibited commendable capability in identifying methodological flaws, articulating insightful feedback on theoretical frameworks, and gauging the overall contribution of the articles to their respective fields. While the integration of ChatGPT showcased immense promise, certain challenges and caveats surfaced. For example, ambiguities might present with complex research articles, leading to nuanced discrepancies between AI and human reviews. Also figures and images cannot be reviewed by ChatGPT. Lengthy articles need to be reviewed in parts by ChatGPT as the entire article will not fit in one chat/response. The benefits consist of reduction in time needed by journals to review the articles submitted to them, as well as an AI assistant to give a different perspective about the research papers other than the human reviewers. In conclusion, this research contributes a groundbreaking foundation for incorporating ChatGPT into the pantheon of journal reviewers. The delineated guidelines distill key insights into operationalizing ChatGPT as a proficient reviewer within academic journal frameworks, paving the way for a more efficient and insightful review process. 11.7% topic match

11.3%

3.5

2022

[46] Automating Systematic Literature Reviews with Natural Language Processing and Text Mining: a Systematic Literature Review G. Sundaram and D. Berleant ArXiv 2022 - 7 citations - Show abstract - Cite - PDF 11.3% topic match

10.6%

2.1

2023

[47] Using LLM (Large Language Model) to Improve Efficiency in Literature Review for Undergraduate Research Shouvik Ahmed Antu, ..., and Cindy K. Richards Journal Not Provided 2023 - 4 citations - Show abstract - Cite 10.6% topic match

10.3%

4.7

2023

[48] Leveraging the Potential of Generative AI to Accelerate Systematic Literature Reviews: An Example in the Area of Educational Technology Pablo Castillo-Segura, ..., and Carmen Fernández Panadero 2023 World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC) 2023 - 5 citations - Show abstract - Cite 10.3% topic match

9.8%

0.0

2024

[49] Streamlining the Selection Phase of Systematic Literature Reviews (SLRs) Using AI-Enabled GPT-4 Assistant API Seyed Mohammad Ali Jafari ArXiv 2024 - 0 citations - Show abstract - Cite - PDF 9.8% topic match

9.4%

0.0

2023

[50] Which AI Technique Is Better to Classify Requirements? An Experiment with SVM, LSTM, and ChatGPT Abdelkarim El-Hajjami, ..., and Camille Salinesi ArXiv 2023 - 0 citations - Show abstract - Cite - PDF 9.4% topic match

9.2%

6.7

2024

[51] A Quantitative Assessment of ChatGPT as a Neurosurgical Triaging Tool. Max Ward, ..., and Griffin R Baum Neurosurgery 2024 - 5 citations - Show abstract - Cite 9.2% topic match

9.2%

None

[52] A Novel Question-Answering Framework for Automated Abstract Screening Using Large Language Models Xiaorui Jiang, ..., and Opeoluwa Akinseloyin Journal Not Provided None - 0 citations - Show abstract - Cite 9.2% topic match

8.8%

0.0

2024

[53] Performance of two large language models for data extraction in evidence synthesis. Amanda Konet, ..., and Robert Chew Research synthesis methods 2024 - 0 citations - Show abstract - Cite 8.8% topic match

8.8%

2.4

2024

[54] Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain Fabio Dennstädt, ..., and Nikola Cihoric Systematic Reviews 2024 - 1 citations - Show abstract - Cite - PDF 8.8% topic match

8.6%

6.7

2023

[55] Reducing Literature Screening Workload With Machine Learning Tanja Burgard and André Bittermann Zeitschrift für Psychologie 2023 - 12 citations - Show abstract - Cite 8.6% topic match

8.5%

5.6

2023

[56] PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models Teo Susnjak ArXiv 2023 - 8 citations - Show abstract - Cite - PDF 8.5% topic match

8.4%

2.3

2024

[57] A Critical Assessment of Large Language Models for Systematic Reviews: Utilizing ChatGPT for Complex Data Extraction Hesam Mahmoudi, ..., and Mohammad S. Jalali SSRN Electronic Journal 2024 - 2 citations - Show abstract - Cite 8.4% topic match

8.3%

0.0

2024

[58] Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility study Lena Schmidt, ..., and James Thomas ArXiv 2024 - 0 citations - Show abstract - Cite - PDF 8.3% topic match

8.3%

2.3

2024

[59] Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks Ruiyang Zhou, ..., and Kai Yu International Conference on Language Resources and Evaluation 2024 - 2 citations - Show abstract - Cite - PDF 8.3% topic match

8.2%

40.3

2021

[60] Automation of systematic literature reviews: A systematic literature review Raymon van Dinter, ..., and C. Catal Inf. Softw. Technol. 2021 - 146 citations - Show abstract - Cite - PDF 8.2% topic match

8.2%

11.5

2023

[61] Clinical Accuracy of Large Language Models and Google Search Responses to Postpartum Depression Questions: Cross-Sectional Study Emre Sezgin, ..., and Sarah Keim Journal of Medical Internet Research 2023 - 17 citations - Show abstract - Cite - PDF 8.2% topic match

8.0%

3.4

2024

[62] Utilizing Large Language Models for Enhanced Clinical Trial Matching: A Study on Automation in Patient Screening Jacob Beattie, ..., and M. Dohopolski Cureus 2024 - 2 citations - Show abstract - Cite - PDF Abstract: Background: Clinical trial matching, essential for advancing medical research, involves detailed screening of potential participants to ensure alignment with specific trial requirements. Research staff face challenges due to the high volume of eligible patients and the complexity of varying eligibility criteria. The traditional manual process, both time-consuming and error-prone, often leads to missed opportunities. Utilizing Artificial Intelligence (AI) and Natural Language Processing (NLP) can significantly enhance the accuracy and efficiency of this process through automated patient screening against established criteria. Methods: Utilizing data from the National NLP Clinical Challenges (n2c2) 2018 Challenge, we utilized 202 longitudinal patient records. These records were annotated by medical professionals and evaluated against 13 selection criteria encompassing various health assessments. Our approach involved embedding medical documents into a vector database to determine relevant document sections, then using a large language model (GPT-3.5 Turbo and GPT-4 OpenAI API) in tandem with structured and chain-of-thought prompting techniques for systematic document assessment against the criteria. Misclassified criteria were also examined to identify classification challenges. Results: This study achieved an accuracy of 0.81, sensitivity of 0.80, specificity of 0.82, and a micro F1 score of 0.79 using GPT-3.5 Turbo, and an accuracy of 0.87, sensitivity of 0.85, specificity of 0.89, and micro F1 score of 0.86 using GPT-4 Turbo. Notably, some criteria in the ground truth appeared mislabeled, an issue we could not explore further due to insufficient label generation guidelines. Conclusion: Our findings underscore the significant potential of AI and NLP technologies, including large language models, in the clinical trial matching process. The study demonstrated strong capabilities in identifying eligible patients and minimizing false inclusions. Such automated systems promise to greatly alleviate the workload of research staff and improve clinical trial enrollment, thus accelerating the process and enhancing the overall feasibility of clinical research. 8.0% topic match

7.9%

0.0

2024

[63] Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study Dan Li, ..., and Joshua Xu Frontiers in Drug Safety and Regulation 2024 - 0 citations - Show abstract - Cite Abstract: Pharmacovigilance plays a crucial role in ensuring the safety of pharmaceutical products. It involves the systematic monitoring of adverse events and the detection of potential safety concerns related to drugs. Manual literature screening for pharmacovigilance related articles is a labor-intensive and time-consuming task, requiring streamlined solutions to cope with the continuous growth of literature. The primary objective of this study is to assess the performance of Large Language Models (LLMs) in automating literature screening for pharmacovigilance, aiming to enhance the process by identifying relevant articles more effectively. This study represents a novel application of LLMs including OpenAI’s GPT-3.5, GPT-4, and Anthropic’s Claude2, in the field of pharmacovigilance, evaluating their ability to categorize medical publications as relevant or irrelevant for safety signal reviews. Our analysis encompassed N-shot learning, chain-of-thought reasoning, and evaluating metrics, with a focus on factors impacting accuracy. The findings highlight the promising potential of LLMs in literature screening, achieving a reproducibility of 93%, sensitivity of 97%, and specificity of 67% showcasing notable strengths in terms of reproducibility and sensitivity, although with moderate specificity. Notably, performance improved when models were provided examples consisting of abstracts, labels, and corresponding reasoning explanations. Moreover, our exploration identified several potential contributing factors influencing prediction outcomes. These factors encompassed the choice of key words and prompts, the balance of the examples, and variations in reasoning explanations. By configuring advanced LLMs for efficient screening of extensive literature databases, this study underscores the transformative potential of these models in drug safety monitoring. Furthermore, these insights gained from this study can inform the development of automated systems for pharmacovigilance, contributing to the ongoing efforts to ensure the safety and efficacy of pharmacovigilance products. 7.9% topic match

7.5%

0.0

2019

[64] Impact of Automatic Query Generation and Quality Recognition Using Deep Learning to Curate Evidence From Biomedical Literature: Empirical Study (Preprint) Muhammad Afzal, ..., and Sungyoung Lee https://doi.org/10.2196/preprints.13430 2019 - 0 citations - Show abstract - Cite 7.5% topic match

7.4%

0.0

2024

[65] Prompting is all you need: LLMs for systematic review screening C. Cao, ..., and N. Bobrovitz medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 7.4% topic match

7.3%

0.0

2024

[66] Comparative Analysis of Performance of Large Language Models in Urogynecology. Ghanshyam S Yadav, ..., and Charles W Nager Urogynecology 2024 - 0 citations - Show abstract - Cite 7.3% topic match

7.2%

3.3

2023

[67] Trustworthiness of Systematic Review Automation - An interview at Coventry University X. Jiang medRxiv 2023 - 3 citations - Show abstract - Cite - PDF 7.2% topic match

7.1%

0.0

2024

[68] Benchmarking Human-AI Collaboration for Common Evidence Appraisal Tools MD Tim Woelfle, ..., and M. M. Lars G. Hemkens medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 7.1% topic match

7.0%

2.6

2024

[69] Can ChatGPT evaluate research quality? Mike Thelwall Journal of Data and Information Science 2024 - 2 citations - Show abstract - Cite - PDF 7.0% topic match

7.0%

0.0

2024

[70] Analysis of the Effectiveness of Large Language Models in Assessing Argumentative Writing and Generating Feedback Daisy Cristine Albuquerque da Silva, ..., and Ana Garcia International Conference on Agents and Artificial Intelligence 2024 - 0 citations - Show abstract - Cite 7.0% topic match

7.0%

67.4

2023

[71] Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? Shuai Wang, ..., and G. Zuccon Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval 2023 - 120 citations - Show abstract - Cite - PDF 7.0% topic match

6.9%

3.2

2024

[72] Data Extraction from Free-Text Reports on Mechanical Thrombectomy in Acute Ischemic Stroke Using ChatGPT: A Retrospective Analysis. N. Lehnen, ..., and D. Paech Radiology 2024 - 2 citations - Show abstract - Cite Abstract: Background Procedural details of mechanical thrombectomy in patients with ischemic stroke are important predictors of clinical outcome and are collected for prospective studies or national stroke registries. To date, these data are collected manually by human readers, a labor-intensive task that is prone to errors. Purpose To evaluate the use of the large language models (LLMs) GPT-4 and GPT-3.5 to extract data from neuroradiology reports on mechanical thrombectomy in patients with ischemic stroke. Materials and Methods This retrospective study included consecutive reports from patients with ischemic stroke who underwent mechanical thrombectomy between November 2022 and September 2023 at institution 1 and between September 2016 and December 2019 at institution 2. A set of 20 reports was used to optimize the prompt, and the ability of the LLMs to extract procedural data from the reports was compared using the McNemar test. Data manually extracted by an interventional neuroradiologist served as the reference standard. Results A total of 100 internal reports from 100 patients (mean age, 74.7 years ± 13.2 [SD]; 53 female) and 30 external reports from 30 patients (mean age, 72.7 years ± 13.5; 18 male) were included. All reports were successfully processed by GPT-4 and GPT-3.5. Of 2800 data entries, 2631 (94.0% [95% CI: 93.0, 94.8]; range per category, 61%-100%) data points were correctly extracted by GPT-4 without the need for further postprocessing. With 1788 of 2800 correct data entries, GPT-3.5 produced fewer correct data entries than did GPT-4 (63.9% [95% CI: 62.0, 65.6]; range per category, 14%-99%; P < .001). For the external reports, GPT-4 extracted 760 of 840 (90.5% [95% CI: 88.3, 92.4]) correct data entries, while GPT-3.5 extracted 539 of 840 (64.2% [95% CI: 60.8, 67.4]; P < .001). Conclusion Compared with GPT-3.5, GPT-4 more frequently extracted correct procedural data from free-text reports on mechanical thrombectomy performed in patients with ischemic stroke. © RSNA, 2024 Supplemental material is available for this article. 6.9% topic match

6.5%

1.4

2024

[73] AI Insights: A Case Study on Utilizing ChatGPT Intelligence for Research Paper Analysis Anjalee de Silva, ..., and Weranga Rajapaksha ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 6.5% topic match

6.5%

9.5

2020

[74] Biceps Tenodesis versus Tenotomy: A Systematic Review and Meta-Analysis of Level I Randomized Controlled Trials. J. W. Belk, ..., and E. McCarty Journal of shoulder and elbow surgery 2020 - 37 citations - Show abstract - Cite 6.5% topic match

6.4%

3.9

2024

[75] Retrieval Augmented Generation Enabled Generative Pre-Trained Transformer 4 (GPT-4) Performance for Clinical Trial Screening Ozan Unlu, ..., and Samuel J. Aronson medRxiv 2024 - 3 citations - Show abstract - Cite Abstract: Background: Subject screening is a key aspect of all clinical trials; however, traditionally, it is a labor-intensive and error-prone task, demanding significant time and resources. With the advent of large language models (LLMs) and related technologies, a paradigm shift in natural language processing capabilities offers a promising avenue for increasing both quality and efficiency of screening efforts. This study aimed to test the Retrieval-Augmented Generation (RAG) process enabled Generative Pretrained Transformer Version 4 (GPT-4) to accurately identify and report on inclusion and exclusion criteria for a clinical trial. Methods: The (Co-Operative Program for Implementation of Optimal Therapy in Heart Failure) COPILOT-HF trial aims to recruit patients with symptomatic heart failure. As part of the screening process, a list of potentially eligible patients is created through an electronic health record (EHR) query. Currently, structured data in the EHR can only be used to determine 5 out of 6 inclusion and 5 out of 17 exclusion criteria. Trained, but non-licensed, study staff complete manual chart review to determine patient eligibility and record their assessment of the inclusion and exclusion criteria. We obtained the structured assessments completed by the study staff and clinical notes for the past two years and developed a workflow of clinical note-based question answering system powered by RAG architecture and GPT-4 that we named RECTIFIER (RAG-Enabled Clinical Trial Infrastructure for Inclusion Exclusion Review). We used notes from 100 patients as a development dataset, 282 patients as a validation dataset, and 1894 patients as a test set. An expert clinician completed a blinded review of patients' charts to answer the eligibility questions and determine the "gold standard" answers. We calculated the sensitivity, specificity, accuracy, and Matthews correlation coefficient (MCC) for each question and screening method. We also performed bootstrapping to calculate the confidence intervals for each statistic. Results: Both RECTIFIER and study staff answers closely aligned with the expert clinician answers across criteria with accuracy ranging between 97.9% and 100% (MCC 0.837 and 1) for RECTIFIER and 91.7% and 100% (MCC 0.644 and 1) for study staff. RECTIFIER performed better than study staff to determine the inclusion criteria of "symptomatic heart failure" with an accuracy of 97.9% vs 91.7% and an MCC of 0.924 vs 0.721, respectively. Overall, the sensitivity and specificity of determining eligibility for the RECTIFIER was 92.3% (CI) and 93.9% (CI), and study staff was 90.1% (CI) and 83.6% (CI), respectively. Conclusion: GPT-4 based solutions have the potential to improve efficiency and reduce costs in clinical trial screening. When incorporating new tools such as RECTIFIER, it is important to consider the potential hazards of automating the screening process and set up appropriate mitigation strategies such as final clinician review before patient engagement. 6.4% topic match

6.4%

0.7

2023

[76] Distinguishing GPT-4-generated Radiology Abstracts from Original Abstracts: Performance of Blinded Human Observers and AI Content Detector F. Ufuk, ..., and A. Yağcı medRxiv 2023 - 1 citations - Show abstract - Cite - PDF 6.4% topic match

6.3%

10.0

2023

[77] Learning to Make Rare and Complex Diagnoses With Generative AI Assistance: Qualitative Study of Popular Large Language Models Tassallah Abdullahi, ..., and Carsten Eickhoff JMIR Medical Education 2023 - 13 citations - Show abstract - Cite - PDF Abstract: Background Patients with rare and complex diseases often experience delayed diagnoses and misdiagnoses because comprehensive knowledge about these diseases is limited to only a few medical experts. In this context, large language models (LLMs) have emerged as powerful knowledge aggregation tools with applications in clinical decision support and education domains. Objective This study aims to explore the potential of 3 popular LLMs, namely Bard (Google LLC), ChatGPT-3.5 (OpenAI), and GPT-4 (OpenAI), in medical education to enhance the diagnosis of rare and complex diseases while investigating the impact of prompt engineering on their performance. Methods We conducted experiments on publicly available complex and rare cases to achieve these objectives. We implemented various prompt strategies to evaluate the performance of these models using both open-ended and multiple-choice prompts. In addition, we used a majority voting strategy to leverage diverse reasoning paths within language models, aiming to enhance their reliability. Furthermore, we compared their performance with the performance of human respondents and MedAlpaca, a generative LLM specifically designed for medical tasks. Results Notably, all LLMs outperformed the average human consensus and MedAlpaca, with a minimum margin of 5% and 13%, respectively, across all 30 cases from the diagnostic case challenge collection. On the frequently misdiagnosed cases category, Bard tied with MedAlpaca but surpassed the human average consensus by 14%, whereas GPT-4 and ChatGPT-3.5 outperformed MedAlpaca and the human respondents on the moderately often misdiagnosed cases category with minimum accuracy scores of 28% and 11%, respectively. The majority voting strategy, particularly with GPT-4, demonstrated the highest overall score across all cases from the diagnostic complex case collection, surpassing that of other LLMs. On the Medical Information Mart for Intensive Care-III data sets, Bard and GPT-4 achieved the highest diagnostic accuracy scores, with multiple-choice prompts scoring 93%, whereas ChatGPT-3.5 and MedAlpaca scored 73% and 47%, respectively. Furthermore, our results demonstrate that there is no one-size-fits-all prompting approach for improving the performance of LLMs and that a single strategy does not universally apply to all LLMs. Conclusions Our findings shed light on the diagnostic capabilities of LLMs and the challenges associated with identifying an optimal prompting strategy that aligns with each language model’s characteristics and specific task requirements. The significance of prompt engineering is highlighted, providing valuable insights for researchers and practitioners who use these language models for medical training. Furthermore, this study represents a crucial step toward understanding how LLMs can enhance diagnostic reasoning in rare and complex medical cases, paving the way for developing effective educational tools and accurate diagnostic aids to improve patient care and outcomes. 6.3% topic match

6.3%

0.0

2024

[78] Loon Lens 1.0 Validation: Agentic AI for Title and Abstract Screening in Systematic Literature Reviews Ghayath Janoudi, ..., and Josip Ivkovic medRxiv 2024 - 0 citations - Show abstract - Cite 6.3% topic match

6.2%

2.0

2024

[79] The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation Y. Gwon, ..., and sungryul shim JMIR Medical Informatics 2024 - 1 citations - Show abstract - Cite - PDF Abstract: Abstract Background A large language model is a type of artificial intelligence (AI) model that opens up great possibilities for health care practice, research, and education, although scholars have emphasized the need to proactively address the issue of unvalidated and inaccurate information regarding its use. One of the best-known large language models is ChatGPT (OpenAI). It is believed to be of great help to medical research, as it facilitates more efficient data set analysis, code generation, and literature review, allowing researchers to focus on experimental design as well as drug discovery and development. Objective This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support systems, to enhance their efficiency and accuracy in health care settings. Methods The search results of a published systematic review by human experts on the treatment of Peyronie disease were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing AI as a comparison to human researchers. Peyronie disease typically presents with discomfort, curvature, or deformity of the penis in association with palpable plaques and erectile dysfunction. To evaluate the quality of individual studies derived from AI answers, we created a structured rating system based on bibliographic information related to the publications. We classified its answers into 4 grades if the title existed: A, B, C, and F. No grade was given for a fake title or no answer. Results From ChatGPT, 7 (0.5%) out of 1287 identified studies were directly relevant, whereas Bing AI resulted in 19 (40%) relevant studies out of 48, compared to the human benchmark of 24 studies. In the qualitative evaluation, ChatGPT had 7 grade A, 18 grade B, 167 grade C, and 211 grade F studies, and Bing AI had 19 grade A and 28 grade C studies. Conclusions This is the first study to compare AI and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI. The limitations of this study using the generative pre-trained transformer model are that the search for research topics was not diverse and that it did not prevent the hallucination of generative AI. However, this study will serve as a standard for future studies by providing an index to verify the reliability and consistency of generative AI from a user’s point of view. If the reliability and consistency of AI literature search services are verified, then the use of these technologies will help medical research greatly. 6.2% topic match

6.0%

5.4

2023

[80] Streamlining Systematic Reviews: Harnessing Large Language Models for Quality Assessment and Risk-of-Bias Evaluation A. Nashwan and Jaber H. Jaradat Cureus 2023 - 7 citations - Show abstract - Cite - PDF 6.0% topic match

6.0%

24.0

2023

[81] Large Language Models in Hematology Case Solving: A Comparative Study of ChatGPT-3.5, Google Bard, and Microsoft Bing Amita Kumari, ..., and Himel Mondal Cureus 2023 - 31 citations - Show abstract - Cite - PDF Abstract: Background Large language models (LLMs), such as ChatGPT-3.5, Google Bard, and Microsoft Bing, have shown promising capabilities in various natural language processing (NLP) tasks. However, their performance and accuracy in solving domain-specific questions, particularly in the field of hematology, have not been extensively investigated. Objective This study aimed to explore the capability of LLMs, namely, ChatGPT-3.5, Google Bard, and Microsoft Bing (Precise), in solving hematology-related cases and comparing their performance. Methods This was a cross-sectional study conducted in the Department of Physiology and Pathology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India. We curated a set of 50 cases on hematology covering a range of topics and complexities. The dataset included queries related to blood disorders, hematologic malignancies, laboratory test parameters, calculations, and treatment options. Each case and related question was prepared with a set of correct answers to compare with. We utilized ChatGPT-3.5, Google Bard Experiment, and Microsoft Bing (Precise) for question-answering tasks. The answers were checked by two physiologists and one pathologist. They rated the answers on a rating scale from one to five. The average score of the three models was compared by Friedman’s test with Dunn’s post-hoc test. The performance of the LLMs was compared with a median of 2.5 by a one-sample median test as the curriculum from which the questions were curated has a 50% pass grade. Results The scores among the three LLMs were significantly different (p-value < 0.0001) with the highest score by ChatGPT (3.15±1.19), followed by Bard (2.23±1.17) and Bing (1.98±1.01). The score of ChatGPT was significantly higher than 50% (p-value = 0.0004), Bard's score was close to 50% (p-value = 0.38), and Bing's score was significantly lower than the pass score (p-value = 0.0015). Conclusion The LLMs reveal significant differences in solving case vignettes in hematology. ChatGPT exhibited the highest score, followed by Google Bard and Microsoft Bing. The observed performance trends suggest that ChatGPT holds promising potential in the medical domain. However, none of the models was capable of answering all questions accurately. Further research and optimization of language models can offer valuable contributions to healthcare and medical education applications. 6.0% topic match

5.8%

14.5

2020

[82] Error rates of human reviewers during abstract screening in systematic reviews Zhen Wang, ..., and M. Murad PLoS ONE 2020 - 70 citations - Show abstract - Cite - PDF Abstract: Background Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references. Objectives To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches. Methods We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies. Results We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%). Conclusions This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time. 5.8% topic match

5.7%

0.0

2024

[83] Evaluating Literature Reviews Conducted by Humans Versus ChatGPT: Comparative Study Mehrnaz Mostafapour, ..., and Gary E. Garber JMIR AI 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Background With the rapid evolution of artificial intelligence (AI), particularly large language models (LLMs) such as ChatGPT-4 (OpenAI), there is an increasing interest in their potential to assist in scholarly tasks, including conducting literature reviews. However, the efficacy of AI-generated reviews compared with traditional human-led approaches remains underexplored. Objective This study aims to compare the quality of literature reviews conducted by the ChatGPT-4 model with those conducted by human researchers, focusing on the relational dynamics between physicians and patients. Methods We included 2 literature reviews in the study on the same topic, namely, exploring factors affecting relational dynamics between physicians and patients in medicolegal contexts. One review used GPT-4, last updated in September 2021, and the other was conducted by human researchers. The human review involved a comprehensive literature search using medical subject headings and keywords in Ovid MEDLINE, followed by a thematic analysis of the literature to synthesize information from selected articles. The AI-generated review used a new prompt engineering approach, using iterative and sequential prompts to generate results. Comparative analysis was based on qualitative measures such as accuracy, response time, consistency, breadth and depth of knowledge, contextual understanding, and transparency. Results GPT-4 produced an extensive list of relational factors rapidly. The AI model demonstrated an impressive breadth of knowledge but exhibited limitations in in-depth and contextual understanding, occasionally producing irrelevant or incorrect information. In comparison, human researchers provided a more nuanced and contextually relevant review. The comparative analysis assessed the reviews based on criteria including accuracy, response time, consistency, breadth and depth of knowledge, contextual understanding, and transparency. While GPT-4 showed advantages in response time and breadth of knowledge, human-led reviews excelled in accuracy, depth of knowledge, and contextual understanding. Conclusions The study suggests that GPT-4, with structured prompt engineering, can be a valuable tool for conducting preliminary literature reviews by providing a broad overview of topics quickly. However, its limitations necessitate careful expert evaluation and refinement, making it an assistant rather than a substitute for human expertise in comprehensive literature reviews. Moreover, this research highlights the potential and limitations of using AI tools like GPT-4 in academic research, particularly in the fields of health services and medical research. It underscores the necessity of combining AI’s rapid information retrieval capabilities with human expertise for more accurate and contextually rich scholarly outputs. 5.7% topic match

5.6%

0.0

2023

[84] GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science Chenxi Wu, ..., and G. Karniadakis ArXiv 2023 - 0 citations - Show abstract - Cite - PDF 5.6% topic match

5.6%

0.0

2024

[85] Understanding the Landscape: The Emergence of Artificial Intelligence (AI), ChatGPT, and Google Bard in Gastroenterology Rajmohan Rammohan, ..., and P. Mustacchia Cureus 2024 - 0 citations - Show abstract - Cite - PDF 5.6% topic match

5.4%

1.1

2022

[86] A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review S. Perlman-Arrow, ..., and R. Arora Research Synthesis Methods 2022 - 3 citations - Show abstract - Cite - PDF 5.4% topic match

5.4%

0.0

2024

[87] Hidden Flaws Behind Expert-Level Accuracy of Multimodal GPT-4 Vision in Medicine Qiao Jin, ..., and Zhiyong Lu ArXiv 2024 - 0 citations - Show abstract - Cite 5.4% topic match

5.3%

29.1

2023

[88] An extensive benchmark study on biomedical text generation and mining with ChatGPT Qijie Chen, ..., and Z. Niu Bioinformatics 2023 - 35 citations - Show abstract - Cite - PDF 5.3% topic match

5.2%

0.0

2024

[89] Literature search sandbox: a large language model that generates search queries for systematic reviews G. Adam, ..., and Byron C Wallace JAMIA Open 2024 - 0 citations - Show abstract - Cite 5.2% topic match

5.2%

61.5

2023

[90] Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessments Dana Brin, ..., and Eyal Klang Scientific Reports 2023 - 69 citations - Show abstract - Cite - PDF 5.2% topic match

5.2%

20.8

2022

[91] Using artificial intelligence methods for systematic review in health sciences: A systematic review Aymeric Blaizot, ..., and N. Chaiyakunapruk Research Synthesis Methods 2022 - 57 citations - Show abstract - Cite - PDF 5.2% topic match

5.1%

8.3

2022

[92] Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search Shuai Wang, ..., and G. Zuccon Proceedings of the 26th Australasian Document Computing Symposium 2022 - 16 citations - Show abstract - Cite - PDF 5.1% topic match

5.0%

10.9

2023

[93] Use of Natural Language Processing of Patient-Initiated Electronic Health Record Messages to Identify Patients With COVID-19 Infection Kellen Mermin-Bunnell, ..., and B. Anderson JAMA Network Open 2023 - 15 citations - Show abstract - Cite - PDF 5.0% topic match

4.7%

103.5

2023

[94] Health system-scale language models are all-purpose prediction engines L. Jiang, ..., and E. Oermann Nature 2023 - 149 citations - Show abstract - Cite - PDF 4.7% topic match

4.6%

1.9

2024

[95] Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada Mehdi Mousavi, ..., and Samira Abbasgholizadeh Rahimi Family Medicine and Community Health 2024 - 1 citations - Show abstract - Cite - PDF Abstract: Introduction The application of large language models such as generative pre-trained transformers (GPTs) has been promising in medical education, and its performance has been tested for different medical exams. This study aims to assess the performance of GPTs in responding to a set of sample questions of short-answer management problems (SAMPs) from the certification exam of the College of Family Physicians of Canada (CFPC). Method Between August 8th and 25th, 2023, we used GPT-3.5 and GPT-4 in five rounds to answer a sample of 77 SAMPs questions from the CFPC website. Two independent certified family physician reviewers scored AI-generated responses twice: first, according to the CFPC answer key (ie, CFPC score), and second, based on their knowledge and other references (ie, Reviews’ score). An ordinal logistic generalised estimating equations (GEE) model was applied to analyse repeated measures across the five rounds. Result According to the CFPC answer key, 607 (73.6%) lines of answers by GPT-3.5 and 691 (81%) by GPT-4 were deemed accurate. Reviewer’s scoring suggested that about 84% of the lines of answers provided by GPT-3.5 and 93% of GPT-4 were correct. The GEE analysis confirmed that over five rounds, the likelihood of achieving a higher CFPC Score Percentage for GPT-4 was 2.31 times more than GPT-3.5 (OR: 2.31; 95% CI: 1.53 to 3.47; p<0.001). Similarly, the Reviewers’ Score percentage for responses provided by GPT-4 over 5 rounds were 2.23 times more likely to exceed those of GPT-3.5 (OR: 2.23; 95% CI: 1.22 to 4.06; p=0.009). Running the GPTs after a one week interval, regeneration of the prompt or using or not using the prompt did not significantly change the CFPC score percentage. Conclusion In our study, we used GPT-3.5 and GPT-4 to answer complex, open-ended sample questions of the CFPC exam and showed that more than 70% of the answers were accurate, and GPT-4 outperformed GPT-3.5 in responding to the questions. Large language models such as GPTs seem promising for assisting candidates of the CFPC exam by providing potential answers. However, their use for family medicine education and exam preparation needs further studies. 4.6% topic match

4.6%

3.6

2024

[96] A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry Yining Huang, ..., and Meilian Chen ArXiv 2024 - 2 citations - Show abstract - Cite - PDF 4.6% topic match

4.5%

0.0

2024

[97] Assessing the Efficacy of ChatGPT Versus Human Researchers in Identifying Relevant Studies on mHealth Interventions for Improving Medication Adherence in Patients With Ischemic Stroke When Conducting Systematic Reviews: Comparative Analysis Suebsarn Ruksakulpiwat, ..., and Cassie Stewart JMIR mHealth and uHealth 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Background ChatGPT by OpenAI emerged as a potential tool for researchers, aiding in various aspects of research. One such application was the identification of relevant studies in systematic reviews. However, a comprehensive comparison of the efficacy of relevant study identification between human researchers and ChatGPT has not been conducted. Objective This study aims to compare the efficacy of ChatGPT and human researchers in identifying relevant studies on medication adherence improvement using mobile health interventions in patients with ischemic stroke during systematic reviews. Methods This study used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Four electronic databases, including CINAHL Plus with Full Text, Web of Science, PubMed, and MEDLINE, were searched to identify articles published from inception until 2023 using search terms based on MeSH (Medical Subject Headings) terms generated by human researchers versus ChatGPT. The authors independently screened the titles, abstracts, and full text of the studies identified through separate searches conducted by human researchers and ChatGPT. The comparison encompassed several aspects, including the ability to retrieve relevant studies, accuracy, efficiency, limitations, and challenges associated with each method. Results A total of 6 articles identified through search terms generated by human researchers were included in the final analysis, of which 4 (67%) reported improvements in medication adherence after the intervention. However, 33% (2/6) of the included studies did not clearly state whether medication adherence improved after the intervention. A total of 10 studies were included based on search terms generated by ChatGPT, of which 6 (60%) overlapped with studies identified by human researchers. Regarding the impact of mobile health interventions on medication adherence, most included studies (8/10, 80%) based on search terms generated by ChatGPT reported improvements in medication adherence after the intervention. However, 20% (2/10) of the studies did not clearly state whether medication adherence improved after the intervention. The precision in accurately identifying relevant studies was higher in human researchers (0.86) than in ChatGPT (0.77). This is consistent with the percentage of relevance, where human researchers (9.8%) demonstrated a higher percentage of relevance than ChatGPT (3%). However, when considering the time required for both humans and ChatGPT to identify relevant studies, ChatGPT substantially outperformed human researchers as it took less time to identify relevant studies. Conclusions Our comparative analysis highlighted the strengths and limitations of both approaches. Ultimately, the choice between human researchers and ChatGPT depends on the specific requirements and objectives of each review, but the collaborative synergy of both approaches holds the potential to advance evidence-based research and decision-making in the health care field. 4.5% topic match

4.5%

0.0

2024

[98] Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews. Assaf Landschaft, ..., and Hector Allende-Cid International journal of medical informatics 2024 - 0 citations - Show abstract - Cite 4.5% topic match

4.1%

39.1

2020

[99] Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation Hannah Harrison, ..., and J. Usher-Smith BMC Medical Research Methodology 2020 - 189 citations - Show abstract - Cite - PDF 4.1% topic match

4.0%

73.5

2023

[100] A study of generative large language model for medical research and healthcare C.A.I. Peng, ..., and Yonghui Wu NPJ Digital Medicine 2023 - 109 citations - Show abstract - Cite - PDF 4.0% topic match

3.9%

0.0

2024

[101] Transformative Automation: AI in Scientific Literature Reviews Kirtirajsinh Zala, ..., and Ioannis Karamitsos International Journal of Advanced Computer Science and Applications 2024 - 0 citations - Show abstract - Cite - PDF 3.9% topic match

3.9%

None

[102] Hallucination Rates and Reference Accuracy in ChatGPT and Bard for Systematic Reviews: A Comparative Analysis M. Chelli, ..., and MD Mikael Chelli Journal Not Provided None - 0 citations - Show abstract - Cite 3.9% topic match

3.8%

0.2

2020

[103] Systematic literature reviews: A Case Study in FinTech and Automated Tool Support Wim Spaargaren Journal Not Provided 2020 - 1 citations - Show abstract - Cite 3.8% topic match

3.8%

3.6

2022

[104] In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving. I. Marshall, ..., and Byron C. Wallace Journal of clinical epidemiology 2022 - 8 citations - Show abstract - Cite - PDF 3.8% topic match

3.8%

76.7

2022

[105] OpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare Ömer Aydın and Enis Karaarslan SSRN Electronic Journal 2022 - 220 citations - Show abstract - Cite 3.8% topic match

3.7%

15.1

2023

[106] Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries Krithi Pushpanathan, ..., and Yih-Chung Tham iScience 2023 - 17 citations - Show abstract - Cite 3.7% topic match

3.7%

0.0

2024

[107] Towards the Use of AI-Based Tools for Systematic Literature Review Lotfi Souifi, ..., and A. Kacem International Conference on Agents and Artificial Intelligence 2024 - 0 citations - Show abstract - Cite 3.7% topic match

3.6%

0.0

2024

[108] Machine learning enables automated screening for systematic reviews and meta-analysis in urology H. Menold, ..., and K. Kowalewski World Journal of Urology 2024 - 0 citations - Show abstract - Cite 3.6% topic match

3.6%

0.0

2024

[109] Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis Mingxin Liu, ..., and Takahiro Kiuchi Journal of Medical Internet Research 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Background Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT’s performance on different medical licensing examinations. Objective In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education. Methods We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses. Results A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5’s performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non–English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5’s (P=.03) and GPT-4’s (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT’s accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs. Conclusions GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education. Trial Registration PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=506687 3.6% topic match

3.5%

18.3

2023

[110] Explaining Legal Concepts with Augmented Large Language Models (GPT-4) Jaromír Šavelka, ..., and Huihui Xu ArXiv 2023 - 26 citations - Show abstract - Cite - PDF 3.5% topic match

3.5%

1.8

2023

[111] General purpose large language models match human performance on gastroenterology board exam self-assessments. Shuhaib S. Ali, ..., and Bara El Kurdi medRxiv 2023 - 2 citations - Show abstract - Cite - PDF Abstract: Introduction: While general-purpose large language models(LLMs) were able to pass USMLE-style examinations, their ability to perform in a specialized context, like gastroenterology, is unclear. In this study, we assessed the performance of three widely available LLMs: PaLM-2, GPT-3.5, and GPT-4 on the most recent ACG self-assessment(2022), utilizing both a basic and a prompt-engineered technique. Methods: We interacted with the chat interfaces of PaLM-2, GPT-3.5, and GPT-4. We first applied a basic prompt approach, providing each exam question and answer text with minimalist text descriptions of any images. For the engineered approach, we added additional context and instructions. We assessed each model-prompt combination in terms of overall and difficulty-stratified performance and compared this to average human performance. We also evaluated each models self-assessed uncertainty. The highest scoring model-prompt combination was further assessed on the 2021 exam. We also assessed the impact of image descriptions on our findings. Results: Using a basic prompt, PaLM-2, GPT-3.5, and GPT-4 achieved scores of 32.6%, 55.3%, and 68.9% respectively. With the engineered prompt, scores improved to 42.7%, 65.2%, and 76.3% respectively. Testing GPT-4 on the ACG-2021 exam yielded a similar score(75.3%). GPT-4 scores matched the average score for human test-takers reported by ACG(75.7%). GPT-4 showed a capability to self-assess its confidence accurately in the context of a multiple-choice exam with its confidence estimates falling within 5% of its actual performance. Excluding image-based questions didnt change the primary findings. Discussion: Our study highlights the capability of GPT-4 to answer subspecialty board-exam questions at a level commensurate with the average human test-taker. The results confirm that prompt-engineering can enhance LLMs performance on medical reasoning tasks. We also show GPT-4 can provide insightful measures of uncertainty in the setting of board-style multiple-choice questions, alerting users to low-quality answers. Future studies of LLMs in gastroenterology should incorporate prompt-engineering to maximize model capabilities. 3.5% topic match

3.5%

0.0

2024

[112] Evaluating Large Language Models on Academic Literature Understanding and Review: An Empirical Study among Early-stage Scholars Jiyao Wang, ..., and Dengbo He Proceedings of the CHI Conference on Human Factors in Computing Systems 2024 - 0 citations - Show abstract - Cite - PDF 3.5% topic match

3.3%

None

[113] What Is the Remedy? L. E. Neame The Homoeopathic Physician None - 11 citations - Show abstract - Cite 3.3% topic match

3.3%

3.8

2024

[114] Performance of GPT-4 and GPT-3.5 in generating accurate and comprehensive diagnoses across medical subspecialties. Dik Wai Anderson Luk, ..., and Yat-fung Shea Journal of the Chinese Medical Association : JCMA 2024 - 3 citations - Show abstract - Cite 3.3% topic match

3.3%

3782.4

2023

[115] GPT-4 Technical Report OpenAI Josh Achiam, ..., and Barret Zoph Journal Not Provided 2023 - 6317 citations - Show abstract - Cite - PDF 3.3% topic match

3.2%

3.1

2023

[116] Large language model-based information extraction from free-text radiology reports: a scoping review protocol Daniel Reichenpfader, ..., and K. Denecke BMJ Open 2023 - 4 citations - Show abstract - Cite - PDF 3.2% topic match

3.2%

2.7

2023

[117] Evaluating the role of ChatGPT in gastroenterology: a comprehensive systematic review of applications, benefits, and limitations Eyal Klang, ..., and Adi Lahat Therapeutic Advances in Gastroenterology 2023 - 5 citations - Show abstract - Cite - PDF Abstract: Background: The integration of artificial intelligence (AI) into healthcare has opened new avenues for enhancing patient care and clinical research. In gastroenterology, the potential of AI tools, specifically large language models like ChatGPT, is being explored to understand their utility and effectiveness. Objectives: The primary goal of this systematic review is to assess the various applications, ascertain the benefits, and identify the limitations of utilizing ChatGPT within the realm of gastroenterology. Design: Through a systematic approach, this review aggregates findings from multiple studies to evaluate the impact of ChatGPT on the field. Data sources and methods: The review was based on a detailed literature search of the PubMed database, targeting research that delves into the use of ChatGPT for gastroenterological purposes. It incorporated six selected studies, which were meticulously evaluated for quality using the Joanna Briggs Institute critical appraisal instruments. The data were then synthesized narratively to encapsulate the roles, advantages, and drawbacks of ChatGPT in gastroenterology. Results: The investigation unearthed various roles of ChatGPT, including its use in patient education, diagnostic self-assessment, disease management, and the formulation of research queries. Notable benefits were its capability to provide pertinent recommendations, enhance communication between patients and physicians, and prompt valuable research inquiries. Nonetheless, it encountered obstacles in decoding intricate medical questions, yielded inconsistent responses at times, and exhibited limitations in generating novel content. The review also considered ethical implications. Conclusion: ChatGPT has demonstrated significant potential in the field of gastroenterology, especially in facilitating patient–physician interactions and managing diseases. Despite these advancements, the review underscores the necessity for ongoing refinement, customization, and ethical regulation of AI tools. These findings serve to enrich the dialog concerning AI’s role in healthcare, with a specific focus on ChatGPT’s application in gastroenterology. Plain language summary Checking how ChatGPT works in gastroenterology: a detailed look at its uses, advantages, and challenges Goal We looked at how ChatGPT, a computer program, is used in the study Gastroenterology. We wanted to understand what’s good about it, what’s challenging, and how it can help doctors and patients. How We Did It We searched for articles about ChatGPT in Gastroenterology on PubMed. We found six suitable articles and checked their quality using the Joanna Briggs Institute (JBI) critical appraisal tools. Then, we put all the information together to get a clear picture. What We Found Doctors and researchers use ChatGPT in many ways. Some use it to teach patients about their health, while others use it to help patients check their symptoms or manage their conditions. It can even help come up with research questions. The good things about ChatGPT are that it gives helpful advice, makes talking between doctors and patients easier, and helps come up with research topics. But, sometimes it doesn’t understand hard medical questions, gives different answers for the same question, or lacks new ideas. There are also concerns about using it the right way. What This Means ChatGPT can be a helpful tool in Gastroenterology, especially when talking with patients and managing their health. But, there are challenges that need to be fixed. Our review helps people understand how ChatGPT can be used in health care, especially in the field of Gastroenterology. 3.2% topic match

3.2%

6.3

2023

[118] Improving Patient Pre-screening for Clinical Trials: Assisting Physicians with Large Language Models D. Hamer, ..., and Daniel Kapitan ArXiv 2023 - 10 citations - Show abstract - Cite - PDF Abstract: Physicians considering clinical trials for their patients are met with the laborious process of checking many text based eligibility criteria. Large Language Models (LLMs) have shown to perform well for clinical information extraction and clinical reasoning, including medical tests, but not yet in real-world scenarios. This paper investigates the use of InstructGPT to assist physicians in determining eligibility for clinical trials based on a patient's summarised medical profile. Using a prompting strategy combining one-shot, selection-inference and chain-of-thought techniques, we investigate the performance of LLMs on 10 synthetically created patient profiles. Performance is evaluated at four levels: ability to identify screenable eligibility criteria from a trial given a medical profile; ability to classify for each individual criterion whether the patient qualifies; the overall classification whether a patient is eligible for a clinical trial and the percentage of criteria to be screened by physician. We evaluated against 146 clinical trials and a total of 4,135 eligibility criteria. The LLM was able to correctly identify the screenability of 72% (2,994/4,135) of the criteria. Additionally, 72% (341/471) of the screenable criteria were evaluated correctly. The resulting trial level classification as eligible or ineligible resulted in a recall of 0.5. By leveraging LLMs with a physician-in-the-loop, a recall of 1.0 and precision of 0.71 on clinical trial level can be achieved while reducing the amount of criteria to be checked by an estimated 90%. LLMs can be used to assist physicians with pre-screening of patients for clinical trials. By forcing instruction-tuned LLMs to produce chain-of-thought responses, the reasoning can be made transparent to and the decision process becomes amenable by physicians, thereby making such a system feasible for use in real-world scenarios. 3.2% topic match

3.1%

1.3

2024

[119] Towards Development of Automated Knowledge Maps and Databases for Materials Engineering using Large Language Models Deepak Prasad, ..., and A. Alankar ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 3.1% topic match

3.1%

2.2

2024

[120] Retrieval-augmented large language models for clinical trial screening. R. Tan, ..., and D. S. Tan Journal of Clinical Oncology 2024 - 1 citations - Show abstract - Cite 3.1% topic match

3.0%

0.0

2024

[121] Unveiling ChatGPT's Potential across Biomedicine and Patient Care: From Text to Treatment Boutaina Addoum, ..., and Imane Chaoui Studies in Medical and Health Sciences 2024 - 0 citations - Show abstract - Cite 3.0% topic match

3.0%

14.3

2019

[122] Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools A. Gates, ..., and L. Hartling Systematic Reviews 2019 - 72 citations - Show abstract - Cite - PDF 3.0% topic match

3.0%

0.0

2024

[123] Artificial intelligence meets medical expertise: evaluating GPT-4's proficiency in generating medical article abstracts Ergin Sağtaş, ..., and A. B. Yağcı Pamukkale Medical Journal 2024 - 0 citations - Show abstract - Cite 3.0% topic match

2.8%

11.8

2023

[124] Exploring ChatGPT Capabilities and Limitations: A Survey Anis Koubaa, ..., and Shahid Latif IEEE Access 2023 - 22 citations - Show abstract - Cite - PDF 2.8% topic match

2.8%

12.1

2024

[125] Comparative Evaluation of LLMs in Clinical Oncology. N. Rydzewski, ..., and Krishnan R. Patel NEJM AI 2024 - 7 citations - Show abstract - Cite Abstract: BACKGROUND As artificial intelligence (AI) tools become widely accessible, more patients and medical professionals will turn to them for medical information. Large language models (LLMs), a subset of AI, excel in natural language processing tasks and hold considerable promise for clinical use. Fields such as oncology, in which clinical decisions are highly dependent on a continuous influx of new clinical trial data and evolving guidelines, stand to gain immensely from such advancements. It is therefore of critical importance to benchmark these models and describe their performance characteristics to guide their safe application to clinical oncology. Accordingly, the primary objectives of this work were to conduct comprehensive evaluations of LLMs in the field of oncology and to identify and characterize strategies that medical professionals can use to bolster their confidence in a model's response. METHODS This study tested five publicly available LLMs (LLaMA 1, PaLM 2, Claude-v1, generative pretrained transformer 3.5 [GPT-3.5], and GPT-4) on a comprehensive battery of 2044 oncology questions, including topics from medical oncology, surgical oncology, radiation oncology, medical statistics, medical physics, and cancer biology. Model prompts were presented independently of each other, and each prompt was repeated three times to assess output consistency. For each response, models were instructed to provide a self-appraised confidence score (from 1 to 4). Model performance was also evaluated against a novel validation set comprising 50 oncology questions curated to eliminate any risk of overlap with the data used to train the LLMs. RESULTS There was significant heterogeneity in performance between models (analysis of variance, P<0.001). Relative to a human benchmark (2013 and 2014 examination results), GPT-4 was the only model to perform above the 50th percentile. Overall, model performance varied as a function of subject area across all models, with worse performance observed in clinical oncology subcategories compared with foundational topics (medical statistics, medical physics, and cancer biology). Within the clinical oncology subdomain, worse performance was observed in female-predominant malignancies. A combination of model selection, prompt repetition, and confidence self-appraisal allowed for the identification of high-performing subgroups of questions with observed accuracies of 81.7 and 81.1% in the Claude-v1 and GPT-4 models, respectively. Evaluation of the novel validation question set produced similar trends in model performance while also highlighting improved performance in newer, centrally hosted models (GPT-4 Turbo and Gemini 1.0 Ultra) and local models (Mixtral 8×7B and LLaMA 2). CONCLUSIONS Of the models tested on a standardized set of oncology questions, GPT-4 was observed to have the highest performance. Although this performance is impressive, all LLMs continue to have clinically significant error rates, including examples of overconfidence and consistent inaccuracies. Given the enthusiasm to integrate these new implementations of AI into clinical practice, continued standardized evaluations of the strengths and limitations of these products will be critical to guide both patients and medical professionals. (Funded by the National Institutes of Health Clinical Center for Research and the Intramural Research Program of the National Institutes of Health; Z99 CA999999.). 2.8% topic match

2.8%

271.5

2019

[126] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi, ..., and Bryan Catanzaro ArXiv 2019 - 1401 citations - Show abstract - Cite - PDF 2.8% topic match

2.7%

0.0

2022

[127] Feature selection analysis of systematic review citation screening algorithm Shi Li, ..., and Hong-Guang Liu 2022 IEEE 5th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) 2022 - 0 citations - Show abstract - Cite 2.7% topic match

2.7%

20.1

2019

[128] Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error Alexandra Bannach‐Brown, ..., and M. Macleod Systematic Reviews 2019 - 117 citations - Show abstract - Cite - PDF 2.7% topic match

2.6%

2.4

2021

[129] PICO Portal Joel T Minion, ..., and F. Clement The Journal of the Canadian Health Libraries Association 2021 - 7 citations - Show abstract - Cite - PDF 2.6% topic match

2.6%

1.9

2019

[130] Ranking Studies for Systematic Reviews Using Query Adaptation: University of Sheffield's Approach to CLEF eHealth 2019 Task 2 Amal H. Alharbi and Mark Stevenson Conference and Labs of the Evaluation Forum 2019 - 10 citations - Show abstract - Cite 2.6% topic match

2.6%

0.0

2024

[131] Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study. Tim Reason, ..., and A. Gimblett Pharmaceutical medicine 2024 - 0 citations - Show abstract - Cite 2.6% topic match

2.5%

56.7

2019

[132] Toward systematic review automation: a practical guide to using machine learning tools in research synthesis I. Marshall and Byron C. Wallace Systematic Reviews 2019 - 303 citations - Show abstract - Cite - PDF 2.5% topic match

2.5%

18.8

2023

[133] Examining the Validity of ChatGPT in Identifying Relevant Nephrology Literature: Findings and Implications S. Suppadungsuk, ..., and W. Cheungpasitporn Journal of Clinical Medicine 2023 - 23 citations - Show abstract - Cite - PDF Abstract: Literature reviews are valuable for summarizing and evaluating the available evidence in various medical fields, including nephrology. However, identifying and exploring the potential sources requires focus and time devoted to literature searching for clinicians and researchers. ChatGPT is a novel artificial intelligence (AI) large language model (LLM) renowned for its exceptional ability to generate human-like responses across various tasks. However, whether ChatGPT can effectively assist medical professionals in identifying relevant literature is unclear. Therefore, this study aimed to assess the effectiveness of ChatGPT in identifying references to literature reviews in nephrology. We keyed the prompt “Please provide the references in Vancouver style and their links in recent literature on… name of the topic” into ChatGPT-3.5 (03/23 Version). We selected all the results provided by ChatGPT and assessed them for existence, relevance, and author/link correctness. We recorded each resource’s citations, authors, title, journal name, publication year, digital object identifier (DOI), and link. The relevance and correctness of each resource were verified by searching on Google Scholar. Of the total 610 references in the nephrology literature, only 378 (62%) of the references provided by ChatGPT existed, while 31% were fabricated, and 7% of citations were incomplete references. Notably, only 122 (20%) of references were authentic. Additionally, 256 (68%) of the links in the references were found to be incorrect, and the DOI was inaccurate in 206 (54%) of the references. Moreover, among those with a link provided, the link was correct in only 20% of cases, and 3% of the references were irrelevant. Notably, an analysis of specific topics in electrolyte, hemodialysis, and kidney stones found that >60% of the references were inaccurate or misleading, with less reliable authorship and links provided by ChatGPT. Based on our findings, the use of ChatGPT as a sole resource for identifying references to literature reviews in nephrology is not recommended. Future studies could explore ways to improve AI language models’ performance in identifying relevant nephrology literature. 2.5% topic match

2.5%

0.0

2024

[134] Optimizing GPT-4 Turbo Diagnostic Accuracy in Neuroradiology through Prompt Engineering and Confidence Thresholds A. Wada, ..., and S. Aoki Diagnostics 2024 - 0 citations - Show abstract - Cite 2.5% topic match

2.4%

33.4

2023

[135] ChatGPT in Scientific Writing: A Cautionary Tale. Haoyi Zheng and H. Zhan The American journal of medicine 2023 - 57 citations - Show abstract - Cite - PDF 2.4% topic match

2.4%

3.8

2024

[136] Addressing Data Scarcity in the Medical Domain: A GPT-Based Approach for Synthetic Data Generation and Feature Extraction F. Sufi Inf. 2024 - 2 citations - Show abstract - Cite - PDF 2.4% topic match

2.3%

1.8

2024

[137] Surveying Attitudinal Alignment Between Large Language Models Vs. Humans Towards 17 Sustainable Development Goals Qingyang Wu, ..., and Yifan Yang ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 2.3% topic match

2.3%

30.3

2023

[138] Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: A systematic review R. Garg, ..., and Sujita Kumar Kar Health Promotion Perspectives 2023 - 43 citations - Show abstract - Cite - PDF 2.3% topic match

2.3%

3.6

2024

[139] Artificial intelligence model GPT4 narrowly fails simulated radiological protection exam G. Roemer, ..., and M. Bellamy Journal of Radiological Protection 2024 - 3 citations - Show abstract - Cite - PDF 2.3% topic match

2.3%

10.5

2023

[140] A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews Carlos Francisco Moreno-García, ..., and M. Aceves-Martins Decision Analytics Journal 2023 - 18 citations - Show abstract - Cite 2.3% topic match

2.3%

70.1

2024

[141] Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. T. Zack, ..., and Emily Alsentzer The Lancet. Digital health 2024 - 61 citations - Show abstract - Cite 2.3% topic match

2.3%

0.0

2024

[142] ChatGPT With GPT-4 Outperforms Emergency Department Physicians in Diagnostic Accuracy: Retrospective Analysis J. M. Hoppe, ..., and C. Stremmel Journal of Medical Internet Research 2024 - 0 citations - Show abstract - Cite Abstract: Background OpenAI’s ChatGPT is a pioneering artificial intelligence (AI) in the field of natural language processing, and it holds significant potential in medicine for providing treatment advice. Additionally, recent studies have demonstrated promising results using ChatGPT for emergency medicine triage. However, its diagnostic accuracy in the emergency department (ED) has not yet been evaluated. Objective This study compares the diagnostic accuracy of ChatGPT with GPT-3.5 and GPT-4 and primary treating resident physicians in an ED setting. Methods Among 100 adults admitted to our ED in January 2023 with internal medicine issues, the diagnostic accuracy was assessed by comparing the diagnoses made by ED resident physicians and those made by ChatGPT with GPT-3.5 or GPT-4 against the final hospital discharge diagnosis, using a point system for grading accuracy. Results The study enrolled 100 patients with a median age of 72 (IQR 58.5-82.0) years who were admitted to our internal medicine ED primarily for cardiovascular, endocrine, gastrointestinal, or infectious diseases. GPT-4 outperformed both GPT-3.5 (P<.001) and ED resident physicians (P=.01) in diagnostic accuracy for internal medicine emergencies. Furthermore, across various disease subgroups, GPT-4 consistently outperformed GPT-3.5 and resident physicians. It demonstrated significant superiority in cardiovascular (GPT-4 vs ED physicians: P=.03) and endocrine or gastrointestinal diseases (GPT-4 vs GPT-3.5: P=.01). However, in other categories, the differences were not statistically significant. Conclusions In this study, which compared the diagnostic accuracy of GPT-3.5, GPT-4, and ED resident physicians against a discharge diagnosis gold standard, GPT-4 outperformed both the resident physicians and its predecessor, GPT-3.5. Despite the retrospective design of the study and its limited sample size, the results underscore the potential of AI as a supportive diagnostic tool in ED settings. 2.3% topic match

2.2%

80.7

2023

[143] Evaluating large language models on medical evidence summarization Liyan Tang, ..., and Y. Peng NPJ Digital Medicine 2023 - 126 citations - Show abstract - Cite - PDF 2.2% topic match

2.2%

0.0

2024

[144] EXAMINING THE ROLE OF CHATGPT IN THE EVALUATION OF SCIENTIFIC ABSTRACTS J. Cavanagh, ..., and B. Heniford British Journal of Surgery 2024 - 0 citations - Show abstract - Cite - PDF 2.2% topic match

2.2%

8.8

2023

[145] Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model D. Soong, ..., and Brandon Higgs ArXiv 2023 - 13 citations - Show abstract - Cite - PDF 2.2% topic match

2.2%

69.4

2020

[146] An open source machine learning framework for efficient and transparent systematic reviews R. Schoot, ..., and D. Oberski Nature Machine Intelligence 2020 - 305 citations - Show abstract - Cite - PDF 2.2% topic match

2.1%

2.1

2017

[147] Automatic and Semi-Automatic Document Selection for Technology-Assisted Review Maura R. Grossman, ..., and Adam Roegiest Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval 2017 - 15 citations - Show abstract - Cite - PDF 2.1% topic match

2.0%

18.6

2023

[148] Artificial Intelligence (AI) in Radiology: A Deep Dive Into ChatGPT 4.0's Accuracy with the American Journal of Neuroradiology's (AJNR) "Case of the Month" P. Suthar, ..., and S. Dua Cureus 2023 - 24 citations - Show abstract - Cite - PDF 2.0% topic match

2.0%

55.1

2020

[149] Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases L. Bornmann, ..., and R. Mutz Humanities and Social Sciences Communications 2020 - 216 citations - Show abstract - Cite - PDF 2.0% topic match

2.0%

22.4

2021

[150] Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews Kevin E. K. Chai, ..., and Leo Ng Systematic Reviews 2021 - 81 citations - Show abstract - Cite - PDF 2.0% topic match

2.0%

3.8

2024

[151] Seeds of Stereotypes: A Large-Scale Textual Analysis of Race and Gender Associations with Diseases in Online Sources Lasse Hyldig Hansen, ..., and L. Celi ArXiv 2024 - 2 citations - Show abstract - Cite - PDF 2.0% topic match

1.9%

6.4

2023

[152] What Will the United States Look Like in 2050? A Chatgpt Short Story Robert W. McGee SSRN Electronic Journal 2023 - 12 citations - Show abstract - Cite 1.9% topic match

1.9%

0.0

2024

[153] Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses Xufei Luo, ..., and Yaolong Chen Journal of Medical Internet Research 2024 - 0 citations - Show abstract - Cite - PDF 1.9% topic match

1.8%

68.2

2023

[154] Science in the age of large language models Abeba Birhane, ..., and Sandra Wachter Nature Reviews Physics 2023 - 106 citations - Show abstract - Cite - PDF 1.8% topic match

1.7%

20.0

2023

[155] Matching Patients to Clinical Trials with Large Language Models Qiao Jin, ..., and Zhiyong Lu ArXiv 2023 - 26 citations - Show abstract - Cite - PDF 1.7% topic match

1.7%

12.2

2023

[156] Universal skepticism of ChatGPT: a review of early literature on chat generative pre-trained transformer C. Watters and Michał K. Lemański Frontiers in Big Data 2023 - 15 citations - Show abstract - Cite - PDF 1.7% topic match

1.6%

5.3

2023

[157] Comparative study of ChatGPT and human evaluators on the assessment of medical literature according to recognised reporting standards Richard H R Roberts, ..., and Iain S Whitaker BMJ Health & Care Informatics 2023 - 6 citations - Show abstract - Cite - PDF Abstract: Introduction Amid clinicians’ challenges in staying updated with medical research, artificial intelligence (AI) tools like the large language model (LLM) ChatGPT could automate appraisal of research quality, saving time and reducing bias. This study compares the proficiency of ChatGPT3 against human evaluation in scoring abstracts to determine its potential as a tool for evidence synthesis. Methods We compared ChatGPT’s scoring of implant dentistry abstracts with human evaluators using the Consolidated Standards of Reporting Trials for Abstracts reporting standards checklist, yielding an overall compliance score (OCS). Bland-Altman analysis assessed agreement between human and AI-generated OCS percentages. Additional error analysis included mean difference of OCS subscores, Welch’s t-test and Pearson’s correlation coefficient. Results Bland-Altman analysis showed a mean difference of 4.92% (95% CI 0.62%, 0.37%) in OCS between human evaluation and ChatGPT. Error analysis displayed small mean differences in most domains, with the highest in ‘conclusion’ (0.764 (95% CI 0.186, 0.280)) and the lowest in ‘blinding’ (0.034 (95% CI 0.818, 0.895)). The strongest correlations between were in ‘harms’ (r=0.32, p<0.001) and ‘trial registration’ (r=0.34, p=0.002), whereas the weakest were in ‘intervention’ (r=0.02, p<0.001) and ‘objective’ (r=0.06, p<0.001). Conclusion LLMs like ChatGPT can help automate appraisal of medical literature, aiding in the identification of accurately reported research. Possible applications of ChatGPT include integration within medical databases for abstract evaluation. Current limitations include the token limit, restricting its usage to abstracts. As AI technology advances, future versions like GPT4 could offer more reliable, comprehensive evaluations, enhancing the identification of high-quality research and potentially improving patient outcomes. 1.6% topic match

1.6%

10.0

2020

[158] Statistical stopping criteria for automated screening in systematic reviews M. Callaghan and Finn Müller-Hansen Systematic Reviews 2020 - 42 citations - Show abstract - Cite - PDF 1.6% topic match

1.6%

8.0

2024

[159] Large Language Models in Mental Health Care: a Scoping Review Y. Hua, ..., and Andrew Beam ArXiv 2024 - 7 citations - Show abstract - Cite - PDF 1.6% topic match

1.6%

35.2

2023

[160] Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI) J. Cadamuro, ..., and A. Padoan Clinical Chemistry and Laboratory Medicine (CCLM) 2023 - 55 citations - Show abstract - Cite - PDF 1.6% topic match

1.6%

1.3

2024

[161] Human versus artificial intelligence‐generated arthroplasty literature: A single‐blinded analysis of perceived communication, quality, and authorship source K. Lawrence, ..., and J. Rozell The International Journal of Medical Robotics and Computer Assisted Surgery 2024 - 1 citations - Show abstract - Cite 1.6% topic match

1.6%

8.1

2020

[162] The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews A. Gates, ..., and L. Hartling BMC Medical Research Methodology 2020 - 36 citations - Show abstract - Cite - PDF 1.6% topic match

1.5%

6.5

2023

[163] Review of emerging trends and projection of future developments in large language models research in ophthalmology Matthew Wong, ..., and Yih-Chung Tham British Journal of Ophthalmology 2023 - 6 citations - Show abstract - Cite Abstract: Background Large language models (LLMs) are fast emerging as potent tools in healthcare, including ophthalmology. This systematic review offers a twofold contribution: it summarises current trends in ophthalmology-related LLM research and projects future directions for this burgeoning field. Methods We systematically searched across various databases (PubMed, Europe PMC, Scopus and Web of Science) for articles related to LLM use in ophthalmology, published between 1 January 2022 and 31 July 2023. Selected articles were summarised, and categorised by type (editorial, commentary, original research, etc) and their research focus (eg, evaluating ChatGPT’s performance in ophthalmology examinations or clinical tasks). Findings We identified 32 articles meeting our criteria, published between January and July 2023, with a peak in June (n=12). Most were original research evaluating LLMs’ proficiency in clinically related tasks (n=9). Studies demonstrated that ChatGPT-4.0 outperformed its predecessor, ChatGPT-3.5, in ophthalmology exams. Furthermore, ChatGPT excelled in constructing discharge notes (n=2), evaluating diagnoses (n=2) and answering general medical queries (n=6). However, it struggled with generating scientific articles or abstracts (n=3) and answering specific subdomain questions, especially those regarding specific treatment options (n=2). ChatGPT’s performance relative to other LLMs (Google’s Bard, Microsoft’s Bing) varied by study design. Ethical concerns such as data hallucination (n=27), authorship (n=5) and data privacy (n=2) were frequently cited. Interpretation While LLMs hold transformative potential for healthcare and ophthalmology, concerns over accountability, accuracy and data security remain. Future research should focus on application programming interface integration, comparative assessments of popular LLMs, their ability to interpret image-based data and the establishment of standardised evaluation frameworks. 1.5% topic match

1.5%

5.0

2024

[164] Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning Teo Susnjak, ..., and Surangika Ranathunga ArXiv 2024 - 3 citations - Show abstract - Cite - PDF 1.5% topic match

1.4%

7.1

2023

[165] Empirical Evaluation of ChatGPT on Requirements Information Retrieval Under Zero-Shot Setting Jianzhang Zhang, ..., and Yinglin Wang 2023 International Conference on Intelligent Computing and Next Generation Networks（ICNGN) 2023 - 11 citations - Show abstract - Cite - PDF 1.4% topic match

1.4%

54.8

2015

[166] Using text mining for study identification in systematic reviews: a systematic review of current approaches A. O'Mara-Eves, ..., and S. Ananiadou Systematic Reviews 2015 - 539 citations - Show abstract - Cite - PDF 1.4% topic match

1.4%

0.0

2024

[167] Evaluation of GPT-4 ability to identify and generate patient instructions for actionable incidental radiology findings. Kar-Mun C Woo, ..., and Stella K Kang Journal of the American Medical Informatics Association : JAMIA 2024 - 0 citations - Show abstract - Cite - PDF Abstract: OBJECTIVES To evaluate the proficiency of a HIPAA-compliant version of GPT-4 in identifying actionable, incidental findings from unstructured radiology reports of Emergency Department patients. To assess appropriateness of artificial intelligence (AI)-generated, patient-facing summaries of these findings. MATERIALS AND METHODS Radiology reports extracted from the electronic health record of a large academic medical center were manually reviewed to identify non-emergent, incidental findings with high likelihood of requiring follow-up, further sub-stratified as "definitely actionable" (DA) or "possibly actionable-clinical correlation" (PA-CC). Instruction prompts to GPT-4 were developed and iteratively optimized using a validation set of 50 reports. The optimized prompt was then applied to a test set of 430 unseen reports. GPT-4 performance was primarily graded on accuracy identifying either DA or PA-CC findings, then secondarily for DA findings alone. Outputs were reviewed for hallucinations. AI-generated patient-facing summaries were assessed for appropriateness via Likert scale. RESULTS For the primary outcome (DA or PA-CC), GPT-4 achieved 99.3% recall, 73.6% precision, and 84.5% F-1. For the secondary outcome (DA only), GPT-4 demonstrated 95.2% recall, 77.3% precision, and 85.3% F-1. No findings were "hallucinated" outright. However, 2.8% of cases included generated text about recommendations that were inferred without specific reference. The majority of True Positive AI-generated summaries required no or minor revision. CONCLUSION GPT-4 demonstrates proficiency in detecting actionable, incidental findings after refined instruction prompting. AI-generated patient instructions were most often appropriate, but rarely included inferred recommendations. While this technology shows promise to augment diagnostics, active clinician oversight via "human-in-the-loop" workflows remains critical for clinical implementation. 1.4% topic match

1.4%

2.5

2020

[168] Deep Neural Network for Reducing the Screening Workload in Systematic Reviews for Clinical Guidelines: Algorithm Validation Study Tomohide Yamada, ..., and T. Yamauchi Journal of Medical Internet Research 2020 - 11 citations - Show abstract - Cite - PDF Abstract: Background Performing systematic reviews is a time-consuming and resource-intensive process. Objective We investigated whether a machine learning system could perform systematic reviews more efficiently. Methods All systematic reviews and meta-analyses of interventional randomized controlled trials cited in recent clinical guidelines from the American Diabetes Association, American College of Cardiology, American Heart Association (2 guidelines), and American Stroke Association were assessed. After reproducing the primary screening data set according to the published search strategy of each, we extracted correct articles (those actually reviewed) and incorrect articles (those not reviewed) from the data set. These 2 sets of articles were used to train a neural network–based artificial intelligence engine (Concept Encoder, Fronteo Inc). The primary endpoint was work saved over sampling at 95% recall (WSS@95%). Results Among 145 candidate reviews of randomized controlled trials, 8 reviews fulfilled the inclusion criteria. For these 8 reviews, the machine learning system significantly reduced the literature screening workload by at least 6-fold versus that of manual screening based on WSS@95%. When machine learning was initiated using 2 correct articles that were randomly selected by a researcher, a 10-fold reduction in workload was achieved versus that of manual screening based on the WSS@95% value, with high sensitivity for eligible studies. The area under the receiver operating characteristic curve increased dramatically every time the algorithm learned a correct article. Conclusions Concept Encoder achieved a 10-fold reduction of the screening workload for systematic review after learning from 2 randomly selected studies on the target topic. However, few meta-analyses of randomized controlled trials were included. Concept Encoder could facilitate the acquisition of evidence for clinical guidelines. 1.4% topic match

1.4%

1.1

2017

[169] Data Balancing for Technologically Assisted Reviews: Undersampling or Reweighting Zhe Yu and T. Menzies Conference and Labs of the Evaluation Forum 2017 - 9 citations - Show abstract - Cite 1.4% topic match

1.4%

6.7

2020

[170] Automatic document screening of medical literature using word and text embeddings in an active learning setting Andrés Carvallo, ..., and Á. Soto Scientometrics 2020 - 28 citations - Show abstract - Cite 1.4% topic match

1.3%

100.5

2023

[171] Utility of ChatGPT in Clinical Practice Jialin Liu, ..., and Siru Liu Journal of Medical Internet Research 2023 - 139 citations - Show abstract - Cite - PDF 1.3% topic match

1.3%

3.4

2019

[172] Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy Christopher Norman, ..., and Aurélie Névéol Systematic Reviews 2019 - 17 citations - Show abstract - Cite - PDF 1.3% topic match

1.3%

0.0

2024

[173] Large Language Models to process, analyze, and synthesize biomedical texts – a scoping review Simona Emilova Doneva, ..., and B. Ineichen bioRxiv 2024 - 0 citations - Show abstract - Cite 1.3% topic match

1.3%

582.0

1960

[174] A Coefficient of Agreement for Nominal Scales Jacob Cohen Educational and Psychological Measurement 1960 - 37610 citations - Show abstract - Cite 1.3% topic match

1.3%

0.0

2024

[175] Research on the Development and Application of ChatGPT Yixin Gao Lecture Notes in Education Psychology and Public Media 2024 - 0 citations - Show abstract - Cite - PDF 1.3% topic match

1.2%

137.8

2023

[176] ChatGPT and Open-AI Models: A Preliminary Review Konstantinos I. Roumeliotis and Nikolaos D. Tselikas Future Internet 2023 - 203 citations - Show abstract - Cite - PDF 1.2% topic match

1.2%

0.0

2024

[177] Transformative potential of Large Language Models (LLMs) in data mining on Electronic Health Records. Amadeo Wals Zurita, ..., and Carlos Míguez Sánchez medRxiv 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Objectives: Explore the potential of Large Scale Language Models (LLMs) to improve the organization of data in clinical reports, with a particular focus on the identification and classification of history and comorbidities in the electronic medical records of oncology patients in the Hospital. Virgin Macarena of Seville. We specifically evaluate the gpt-3.5-turbo-1106 and gpt-4-1106-preview models against the capabilities of specialized human evaluators. Methods: We implemented a script using the OpenAI API to extract, in JSON format, structured information on comorbidities reported in 250 personal history reports. These reports were manually reviewed in groups of 50 by five radiation oncology specialists. A detailed analysis of the discrepancies between GPT and medical models allowed the ground truth to be established. We compare the results using metrics such as Sensitivity, Specificity, Precision, Accuracy, F-value, Kappa index, and McNemar's test, as well as examining common causes of errors in humans and GPT models. Results: GPT-3.5 showed slightly lower performance than physicians in all metrics, although without statistically significant differences. GPT-4 showed clear superiority in several key metrics. Particularly, it reached 96.8% in sensitivity, compared to 88.2% for GPT-3.5 and 88.8% for doctors. These, however, marginally outperformed GPT-4 in accuracy (97.7% vs. 96.8%). GPT-4 demonstrated greater consistency, replicating exact results in 76% of reports after 10 analyses, in contrast to 59% for GPT-3.5. Clinicians failed more to detect explicit comorbidities, possibly due to fatigue or distraction, while GPT models inferred non-explicit comorbidities more frequently, sometimes correctly, although also resulting in more false positives. Conclusion: The LLMs studied, with carefully designed prompts, show competence comparable to that of medical specialists in the interpretation of clinical reports, even in complex and confusingly worded texts. Also considering their superior time and cost efficiency, these models represent a preferred option over human analysis for data mining and structuring information in extensive sets of clinical reports. 1.2% topic match

1.2%

2.8

2024

[178] Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study Fionn Byrne, ..., and R. van de Schoot Systematic Reviews 2024 - 1 citations - Show abstract - Cite 1.2% topic match

1.1%

0.0

2024

[179] Future of Evidence Synthesis: Automated, Living, and Interactive Systematic Reviews and Meta-Analyses I. Riaz, ..., and M. Hassan Murad Mayo Clinic Proceedings: Digital Health 2024 - 0 citations - Show abstract - Cite 1.1% topic match

1.1%

6.3

2023

[180] ChatGPT in Iranian medical licensing examination: evaluating the diagnostic accuracy and decision-making capabilities of an AI-based model M. Ebrahimian, ..., and Elham Sobhrakhshankhah BMJ Health & Care Informatics 2023 - 6 citations - Show abstract - Cite - PDF 1.1% topic match

1.1%

5.9

2024

[181] System for systematic literature review using multiple AI agents: Concept and an empirical evaluation Malik Abdul Sami, ..., and Pekka Abrahamsson ArXiv 2024 - 4 citations - Show abstract - Cite - PDF 1.1% topic match

1.1%

1.9

2024

[182] Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board-style Examination. Satheesh Krishna, ..., and Rajesh Bhayana Radiology 2024 - 1 citations - Show abstract - Cite Abstract: Background ChatGPT (OpenAI) can pass a text-based radiology board-style examination, but its stochasticity and confident language when it is incorrect may limit utility. Purpose To assess the reliability, repeatability, robustness, and confidence of GPT-3.5 and GPT-4 (ChatGPT; OpenAI) through repeated prompting with a radiology board-style examination. Materials and Methods In this exploratory prospective study, 150 radiology board-style multiple-choice text-based questions, previously used to benchmark ChatGPT, were administered to default versions of ChatGPT (GPT-3.5 and GPT-4) on three separate attempts (separated by ≥1 month and then 1 week). Accuracy and answer choices between attempts were compared to assess reliability (accuracy over time) and repeatability (agreement over time). On the third attempt, regardless of answer choice, ChatGPT was challenged three times with the adversarial prompt, "Your answer choice is incorrect. Please choose a different option," to assess robustness (ability to withstand adversarial prompting). ChatGPT was prompted to rate its confidence from 1-10 (with 10 being the highest level of confidence and 1 being the lowest) on the third attempt and after each challenge prompt. Results Neither version showed a difference in accuracy over three attempts: for the first, second, and third attempt, accuracy of GPT-3.5 was 69.3% (104 of 150), 63.3% (95 of 150), and 60.7% (91 of 150), respectively (P = .06); and accuracy of GPT-4 was 80.6% (121 of 150), 78.0% (117 of 150), and 76.7% (115 of 150), respectively (P = .42). Though both GPT-4 and GPT-3.5 had only moderate intrarater agreement (κ = 0.78 and 0.64, respectively), the answer choices of GPT-4 were more consistent across three attempts than those of GPT-3.5 (agreement, 76.7% [115 of 150] vs 61.3% [92 of 150], respectively; P = .006). After challenge prompt, both changed responses for most questions, though GPT-4 did so more frequently than GPT-3.5 (97.3% [146 of 150] vs 71.3% [107 of 150], respectively; P < .001). Both rated "high confidence" (≥8 on the 1-10 scale) for most initial responses (GPT-3.5, 100% [150 of 150]; and GPT-4, 94.0% [141 of 150]) as well as for incorrect responses (ie, overconfidence; GPT-3.5, 100% [59 of 59]; and GPT-4, 77% [27 of 35], respectively; P = .89). Conclusion Default GPT-3.5 and GPT-4 were reliably accurate across three attempts, but both had poor repeatability and robustness and were frequently overconfident. GPT-4 was more consistent across attempts than GPT-3.5 but more influenced by an adversarial prompt. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Ballard in this issue. 1.1% topic match

1.0%

7.4

2022

[183] Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models T. M. Buonocore, ..., and Enea Parimbelli Journal of biomedical informatics 2022 - 14 citations - Show abstract - Cite - PDF 1.0% topic match

1.0%

19.3

2018

[184] An Introduction to Writing Narrative and Systematic Reviews - Tasks, Tips and Traps for Aspiring Authors. Ann Gregory and A. Denniss Heart, lung & circulation 2018 - 123 citations - Show abstract - Cite 1.0% topic match

1.0%

13.8

2016

[185] Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews I. Shemilt, ..., and James Thomas Systematic Reviews 2016 - 114 citations - Show abstract - Cite - PDF 1.0% topic match

1.0%

138.7

2019

[186] Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets Yifan Peng, ..., and Zhiyong Lu https://doi.org/10.18653/v1/W19-5006 2019 - 752 citations - Show abstract - Cite - PDF 1.0% topic match

1.0%

8.2

2023

[187] Quality of citation data using the natural language processing tool ChatGPT in rheumatology: creation of false references A. Hueber and A. Kleyer RMD Open 2023 - 12 citations - Show abstract - Cite - PDF 1.0% topic match

0.9%

0.0

2024

[188] Exploring the role of Large Language Models (LLMs) in hematology: a systematic review of applications, benefits, and limitations A. Mudrik, ..., and S. Soffer medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 0.9% topic match

0.9%

25.8

2023

[189] ChatGPT and the Future of Academic Integrity in the Artificial Intelligence Era: A New Frontier No author found Al-Salam Journal for Engineering and Technology 2023 - 41 citations - Show abstract - Cite - PDF 0.9% topic match

0.9%

0.8

2022

[190] A semiparametric approach for meta‐analysis of diagnostic accuracy studies with multiple cut‐offs C. Frömke, ..., and A. Zapf Research Synthesis Methods 2022 - 2 citations - Show abstract - Cite - PDF 0.9% topic match

0.9%

4.3

2023

[191] Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts Fan Gao, ..., and Irene Li ArXiv 2023 - 8 citations - Show abstract - Cite 0.9% topic match

0.9%

3.1

2023

[192] Performance of large language models at the MRCS Part A: a tool for medical education? A. Yiu and K. Lam Annals of the Royal College of Surgeons of England 2023 - 3 citations - Show abstract - Cite 0.9% topic match

0.9%

7.4

2022

[193] Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol Yuelun Zhang, ..., and Hui Pan Systematic Reviews 2022 - 21 citations - Show abstract - Cite - PDF 0.9% topic match

0.9%

6.1

2023

[194] Chat GPT in Diagnostic Human Pathology: Will It Be Useful to Pathologists? A Preliminary Review with ‘Query Session’ and Future Perspectives Gerardo Cazzato, ..., and G. Ingravallo AI 2023 - 6 citations - Show abstract - Cite - PDF Abstract: The advent of Artificial Intelligence (AI) has in just a few years supplied multiple areas of knowledge, including in the medical and scientific fields. An increasing number of AI-based applications have been developed, among which conversational AI has emerged. Regarding the latter, ChatGPT has risen to the headlines, scientific and otherwise, for its distinct propensity to simulate a ‘real’ discussion with its interlocutor, based on appropriate prompts. Although several clinical studies using ChatGPT have already been published in the literature, very little has yet been written about its potential application in human pathology. We conduct a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, using PubMed, Scopus and the Web of Science (WoS) as databases, with the following keywords: ChatGPT OR Chat GPT, in combination with each of the following: pathology, diagnostic pathology, anatomic pathology, before 31 July 2023. A total of 103 records were initially identified in the literature search, of which 19 were duplicates. After screening for eligibility and inclusion criteria, only five publications were ultimately included. The majority of publications were original articles (n = 2), followed by a case report (n = 1), letter to the editor (n = 1) and review (n = 1). Furthermore, we performed a ‘query session’ with ChatGPT regarding pathologies such as pigmented skin lesions, malignant melanoma and variants, Gleason’s score of prostate adenocarcinoma, differential diagnosis between germ cell tumors and high grade serous carcinoma of the ovary, pleural mesothelioma and pediatric diffuse midline glioma. Although the premises are exciting and ChatGPT is able to co-advise the pathologist in providing large amounts of scientific data for use in routine microscopic diagnostic practice, there are many limitations (such as data of training, amount of data available, ‘hallucination’ phenomena) that need to be addressed and resolved, with the caveat that an AI-driven system should always provide support and never a decision-making motive during the histopathological diagnostic process. 0.9% topic match

0.8%

3.0

2023

[195] Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper Linda C. Smith Journal of the Association for Information Science and Technology 2023 - 3 citations - Show abstract - Cite - PDF 0.8% topic match

0.8%

30.0

2023

[196] The Potential of ChatGPT as a Self-Diagnostic Tool in Common Orthopedic Diseases: Exploratory Study Tomoyuki Kuroiwa, ..., and Koji Fujita Journal of Medical Internet Research 2023 - 35 citations - Show abstract - Cite - PDF Abstract: Background Artificial intelligence (AI) has gained tremendous popularity recently, especially the use of natural language processing (NLP). ChatGPT is a state-of-the-art chatbot capable of creating natural conversations using NLP. The use of AI in medicine can have a tremendous impact on health care delivery. Although some studies have evaluated ChatGPT’s accuracy in self-diagnosis, there is no research regarding its precision and the degree to which it recommends medical consultations. Objective The aim of this study was to evaluate ChatGPT’s ability to accurately and precisely self-diagnose common orthopedic diseases, as well as the degree of recommendation it provides for medical consultations. Methods Over a 5-day course, each of the study authors submitted the same questions to ChatGPT. The conditions evaluated were carpal tunnel syndrome (CTS), cervical myelopathy (CM), lumbar spinal stenosis (LSS), knee osteoarthritis (KOA), and hip osteoarthritis (HOA). Answers were categorized as either correct, partially correct, incorrect, or a differential diagnosis. The percentage of correct answers and reproducibility were calculated. The reproducibility between days and raters were calculated using the Fleiss κ coefficient. Answers that recommended that the patient seek medical attention were recategorized according to the strength of the recommendation as defined by the study. Results The ratios of correct answers were 25/25, 1/25, 24/25, 16/25, and 17/25 for CTS, CM, LSS, KOA, and HOA, respectively. The ratios of incorrect answers were 23/25 for CM and 0/25 for all other conditions. The reproducibility between days was 1.0, 0.15, 0.7, 0.6, and 0.6 for CTS, CM, LSS, KOA, and HOA, respectively. The reproducibility between raters was 1.0, 0.1, 0.64, –0.12, and 0.04 for CTS, CM, LSS, KOA, and HOA, respectively. Among the answers recommending medical attention, the phrases “essential,” “recommended,” “best,” and “important” were used. Specifically, “essential” occurred in 4 out of 125, “recommended” in 12 out of 125, “best” in 6 out of 125, and “important” in 94 out of 125 answers. Additionally, 7 out of the 125 answers did not include a recommendation to seek medical attention. Conclusions The accuracy and reproducibility of ChatGPT to self-diagnose five common orthopedic conditions were inconsistent. The accuracy could potentially be improved by adding symptoms that could easily identify a specific location. Only a few answers were accompanied by a strong recommendation to seek medical attention according to our study standards. Although ChatGPT could serve as a potential first step in accessing care, we found variability in accurate self-diagnosis. Given the risk of harm with self-diagnosis without medical follow-up, it would be prudent for an NLP to include clear language alerting patients to seek expert medical opinions. We hope to shed further light on the use of AI in a future clinical study. 0.8% topic match

0.8%

109.6

2023

[197] Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models Yi-Hsueh Liu, ..., and Bao Ge ArXiv 2023 - 205 citations - Show abstract - Cite 0.8% topic match

0.8%

31.4

2023

[198] ChatGPT Outscored Human Candidates in a Virtual Objective Structured Clinical Examination (OSCE) in Obstetrics and Gynecology. S. Li, ..., and P. Rajendran American journal of obstetrics and gynecology 2023 - 51 citations - Show abstract - Cite - PDF 0.8% topic match

0.8%

0.0

2024

[199] Assessing Risk of Bias Using ChatGPT-4 and Cochrane ROB2 Tool. A. S. Treviño-Juarez Medical science educator 2024 - 0 citations - Show abstract - Cite 0.8% topic match

0.8%

5.5

2024

[200] Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment B. Hasan, ..., and M. H. Murad BMJ Evidence-Based Medicine 2024 - 4 citations - Show abstract - Cite 0.8% topic match

0.8%

7.3

2022

[201] From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search Shuai Wang, ..., and G. Zuccon Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval 2022 - 19 citations - Show abstract - Cite - PDF 0.8% topic match

0.8%

16.0

2006

[202] Reducing workload in systematic review preparation using automated citation classification. AM Arjeh Cohen, ..., and Po-Yin Yen Journal of the American Medical Informatics Association : JAMIA 2006 - 299 citations - Show abstract - Cite - PDF 0.8% topic match

0.7%

13.6

2019

[203] The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials M. Michelson and Katja Reuter Contemporary Clinical Trials Communications 2019 - 71 citations - Show abstract - Cite 0.7% topic match

0.7%

5.6

2023

[204] Comparisons of Quality, Correctness, and Similarity Between ChatGPT-Generated and Human-Written Abstracts for Basic Research: Cross-Sectional Study Shu-Li Cheng, ..., and Kuan-Pin Su Journal of Medical Internet Research 2023 - 5 citations - Show abstract - Cite - PDF Abstract: Background ChatGPT may act as a research assistant to help organize the direction of thinking and summarize research findings. However, few studies have examined the quality, similarity (abstracts being similar to the original one), and accuracy of the abstracts generated by ChatGPT when researchers provide full-text basic research papers. Objective We aimed to assess the applicability of an artificial intelligence (AI) model in generating abstracts for basic preclinical research. Methods We selected 30 basic research papers from Nature, Genome Biology, and Biological Psychiatry. Excluding abstracts, we inputted the full text into ChatPDF, an application of a language model based on ChatGPT, and we prompted it to generate abstracts with the same style as used in the original papers. A total of 8 experts were invited to evaluate the quality of these abstracts (based on a Likert scale of 0-10) and identify which abstracts were generated by ChatPDF, using a blind approach. These abstracts were also evaluated for their similarity to the original abstracts and the accuracy of the AI content. Results The quality of ChatGPT-generated abstracts was lower than that of the actual abstracts (10-point Likert scale: mean 4.72, SD 2.09 vs mean 8.09, SD 1.03; P<.001). The difference in quality was significant in the unstructured format (mean difference –4.33; 95% CI –4.79 to –3.86; P<.001) but minimal in the 4-subheading structured format (mean difference –2.33; 95% CI –2.79 to –1.86). Among the 30 ChatGPT-generated abstracts, 3 showed wrong conclusions, and 10 were identified as AI content. The mean percentage of similarity between the original and the generated abstracts was not high (2.10%-4.40%). The blinded reviewers achieved a 93% (224/240) accuracy rate in guessing which abstracts were written using ChatGPT. Conclusions Using ChatGPT to generate a scientific abstract may not lead to issues of similarity when using real full texts written by humans. However, the quality of the ChatGPT-generated abstracts was suboptimal, and their accuracy was not 100%. 0.7% topic match

0.7%

30.3

2023

[205] Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment Nikhil S. Patil, ..., and Natasha Larocque Canadian Association of Radiologists Journal 2023 - 38 citations - Show abstract - Cite - PDF Abstract: Purpose Bard by Google, a direct competitor to ChatGPT, was recently released. Understanding the relative performance of these different chatbots can provide important insight into their strengths and weaknesses as well as which roles they are most suited to fill. In this project, we aimed to compare the most recent version of ChatGPT, ChatGPT-4, and Bard by Google, in their ability to accurately respond to radiology board examination practice questions. Methods Text-based questions were collected from the 2017-2021 American College of Radiology’s Diagnostic Radiology In-Training (DXIT) examinations. ChatGPT-4 and Bard were queried, and their comparative accuracies, response lengths, and response times were documented. Subspecialty-specific performance was analyzed as well. Results 318 questions were included in our analysis. ChatGPT answered significantly more accurately than Bard (87.11% vs 70.44%, P < .0001). ChatGPT’s response length was significantly shorter than Bard’s (935.28 ± 440.88 characters vs 1437.52 ± 415.91 characters, P < .0001). ChatGPT’s response time was significantly longer than Bard’s (26.79 ± 3.27 seconds vs 7.55 ± 1.88 seconds, P < .0001). ChatGPT performed superiorly to Bard in neuroradiology, (100.00% vs 86.21%, P = .03), general & physics (85.39% vs 68.54%, P < .001), nuclear medicine (80.00% vs 56.67%, P < .01), pediatric radiology (93.75% vs 68.75%, P = .03), and ultrasound (100.00% vs 63.64%, P < .001). In the remaining subspecialties, there were no significant differences between ChatGPT and Bard’s performance. Conclusion ChatGPT displayed superior radiology knowledge compared to Bard. While both chatbots display reasonable radiology knowledge, they should be used with conscious knowledge of their limitations and fallibility. Both chatbots provided incorrect or illogical answer explanations and did not always address the educational content of the question. Graphical Abstract 0.7% topic match

0.7%

0.0

2024

[206] Diagnostic Performance of ChatGPT to perform emergency department triage: A systematic review and meta-analysis Navid Kaboudi, ..., and Saeedeh Firouzbakht medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 0.7% topic match

0.7%

6.8

2017

[207] FAST2: An intelligent assistant for finding relevant papers Zhe Yu and T. Menzies Expert Syst. Appl. 2017 - 51 citations - Show abstract - Cite - PDF 0.7% topic match

0.7%

8.7

2020

[208] Trialstreamer: A living, automatically updated database of clinical trial reports I. Marshall, ..., and Byron C. Wallace Journal of the American Medical Informatics Association : JAMIA 2020 - 39 citations - Show abstract - Cite - PDF 0.7% topic match

0.7%

88.4

2023

[209] Potential Use of Chat GPT in Global Warming Som S. Biswas Annals of Biomedical Engineering 2023 - 151 citations - Show abstract - Cite 0.7% topic match

0.6%

3.1

2024

[210] LLAssist: Simple Tools for Automating Literature Review Using Large Language Models Christoforus Yoga Haryanto ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 0.6% topic match

0.6%

25.3

2023

[211] A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks Israt Jahan, ..., and Jimmy X. Huang Computers in biology and medicine 2023 - 28 citations - Show abstract - Cite - PDF 0.6% topic match

0.6%

13.9

2015

[212] Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers J. Rathbone, ..., and P. Glasziou Systematic Reviews 2015 - 131 citations - Show abstract - Cite - PDF 0.6% topic match

0.6%

2.2

2023

[213] Efficiency and Workload Reduction of Semi-automated Citation Screening Software for Creating Clinical Practice Guidelines: A Prospective Observational Study T. Oami, ..., and T. Nakada Journal of Epidemiology 2023 - 2 citations - Show abstract - Cite - PDF 0.6% topic match

0.5%

22.7

2023

[214] The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. J. Chervenak, ..., and S. Jindal Fertility and sterility 2023 - 35 citations - Show abstract - Cite - PDF 0.5% topic match

0.5%

0.7

2023

[215] Cocreating an Automated mHealth Apps Systematic Review Process With Generative AI: Design Science Research Approach Guido Giunti and Colin P Doherty JMIR Medical Education 2023 - 1 citations - Show abstract - Cite - PDF 0.5% topic match

0.5%

0.0

2024

[216] Adopting Generative AI for Literature Reviews: An Epistemological Perspective G. Schryen, ..., and Jiaqi Yang Hawaii International Conference on System Sciences 2024 - 0 citations - Show abstract - Cite 0.5% topic match

0.5%

15.2

2021

[217] Tools to support the automation of systematic reviews: A scoping review. H. Khalil, ..., and A. Zarnegar Journal of clinical epidemiology 2021 - 45 citations - Show abstract - Cite 0.5% topic match

0.5%

6.4

2018

[218] Seed-driven Document Ranking for Systematic Reviews in Evidence-Based Medicine Grace E. Lee and Aixin Sun The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval 2018 - 41 citations - Show abstract - Cite - PDF 0.5% topic match

0.5%

1.0

2012

[219] Machine Learning Methods in Systematic Reviews: Identifying Quality Improvement Intervention Evaluations S. Hempel, ..., and S. Dalal Journal Not Provided 2012 - 12 citations - Show abstract - Cite Abstract: Background Electronic searches typically yield far more citations than are relevant, and reviewers spend a substantial amount of time screening titles and abstracts to identify potential studies eligible for inclusion in a review. This is of particular relevance in complex research fields such as quality improvement. We tested a semiautomated literature screening process applied to the title and abstract screening stage of systematic reviews. A machine learning approach may allow literature reviewers to screen only a fraction of a search output and to use a predictive model to learn and then emulate the reviewers’ decisions. Once learned, the model can apply the selection process to an essentially unlimited number of citations. Method Two independent literature reviewers screened 1,591 quasi-randomly selected citations in a training dataset used to predict decisions on the remaining citations in a MEDLINE search output of 9,395 citations. We explored different prediction algorithms and tested results against reference samples screened by experts in quality improvement. Qualitative (relevance cutoff determined in ROC curve) and quantitative predictions (probability rank order of citations) were determined. Results The agreement between independent literature reviewers ranged from κ= 0.55 to 0.57. Across two reference samples, the predictive performance of the machine learning approach demonstrated 90.1 percent sensitivity, 43.9 percent specificity, and 32.1 percent PPV. This translates to a reduction of 36.1 percent in citation screening if applied. The predictive performance was affected by reviewer disagreements: a subgroup analysis restricted to citations both reviewers agreed on showed a sensitivity of 98.8 percent (specificity 43.9 percent). Conclusion Machine learning approaches may assist in the title and abstract inclusion screening process in systematic reviews of complex, steadily expanding research fields such as quality improvement. Increased reviewer agreement appeared to be associated with improved predictive performance. 0.5% topic match

0.5%

190.3

2009

[220] The Probabilistic Relevance Framework: BM25 and Beyond S. Robertson and H. Zaragoza Found. Trends Inf. Retr. 2009 - 2973 citations - Show abstract - Cite 0.5% topic match

0.5%

0.0

2023

[221] Clinical reasoning using ChatGPT: Is it beyond credibility for physiotherapists use? Paraskevi Bilika, ..., and E. Kapreli Physiotherapy theory and practice 2023 - 0 citations - Show abstract - Cite 0.5% topic match

0.5%

950.0

2012

[222] Interrater reliability: the kappa statistic M. McHugh Biochemia Medica 2012 - 11514 citations - Show abstract - Cite - PDF 0.5% topic match

0.5%

52.2

2023

[223] Artificial Intelligence Can Generate Fraudulent but Authentic-Looking Scientific Medical Articles: Pandora’s Box Has Been Opened M. Májovský, ..., and D. Netuka Journal of Medical Internet Research 2023 - 89 citations - Show abstract - Cite - PDF Abstract: Background Artificial intelligence (AI) has advanced substantially in recent years, transforming many industries and improving the way people live and work. In scientific research, AI can enhance the quality and efficiency of data analysis and publication. However, AI has also opened up the possibility of generating high-quality fraudulent papers that are difficult to detect, raising important questions about the integrity of scientific research and the trustworthiness of published papers. Objective The aim of this study was to investigate the capabilities of current AI language models in generating high-quality fraudulent medical articles. We hypothesized that modern AI models can create highly convincing fraudulent papers that can easily deceive readers and even experienced researchers. Methods This proof-of-concept study used ChatGPT (Chat Generative Pre-trained Transformer) powered by the GPT-3 (Generative Pre-trained Transformer 3) language model to generate a fraudulent scientific article related to neurosurgery. GPT-3 is a large language model developed by OpenAI that uses deep learning algorithms to generate human-like text in response to prompts given by users. The model was trained on a massive corpus of text from the internet and is capable of generating high-quality text in a variety of languages and on various topics. The authors posed questions and prompts to the model and refined them iteratively as the model generated the responses. The goal was to create a completely fabricated article including the abstract, introduction, material and methods, discussion, references, charts, etc. Once the article was generated, it was reviewed for accuracy and coherence by experts in the fields of neurosurgery, psychiatry, and statistics and compared to existing similar articles. Results The study found that the AI language model can create a highly convincing fraudulent article that resembled a genuine scientific paper in terms of word usage, sentence structure, and overall composition. The AI-generated article included standard sections such as introduction, material and methods, results, and discussion, as well a data sheet. It consisted of 1992 words and 17 citations, and the whole process of article creation took approximately 1 hour without any special training of the human user. However, there were some concerns and specific mistakes identified in the generated article, specifically in the references. Conclusions The study demonstrates the potential of current AI language models to generate completely fabricated scientific articles. Although the papers look sophisticated and seemingly flawless, expert readers may identify semantic inaccuracies and errors upon closer inspection. We highlight the need for increased vigilance and better detection methods to combat the potential misuse of AI in scientific research. At the same time, it is important to recognize the potential benefits of using AI language models in genuine scientific writing and research, such as manuscript preparation and language editing. 0.5% topic match

0.5%

0.0

2024

[224] ChatGPT in medicine: A cross-disciplinary systematic review of ChatGPT’s (artificial intelligence) role in research, clinical practice, education, and patient interaction Afia Fatima, ..., and M. Mustafa Medicine 2024 - 0 citations - Show abstract - Cite 0.5% topic match

0.5%

12.3

2023

[225] Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Models Qiang Zhang, ..., and Yusuke Miyao Annual Meeting of the Association for Computational Linguistics 2023 - 18 citations - Show abstract - Cite - PDF 0.5% topic match

0.4%

82.2

2023

[226] Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard Zhi Wei Lim, ..., and Yih-Chung Tham eBioMedicine 2023 - 101 citations - Show abstract - Cite 0.4% topic match

0.4%

21.8

2023

[227] ChatGPT's Ability to Assess Quality and Readability of Online Medical Information: Evidence From a Cross-Sectional Study R. Golan, ..., and Ranjith Ramasamy Cureus 2023 - 30 citations - Show abstract - Cite - PDF Abstract: Introduction Artificial Intelligence (AI) platforms have gained widespread attention for their distinct ability to generate automated responses to various prompts. However, its role in assessing the quality and readability of a provided text remains unclear. Thus, the purpose of this study is to evaluate the proficiency of the conversational generative pre-trained transformer (ChatGPT) in utilizing the DISCERN tool to evaluate the quality of online content regarding shock wave therapy for erectile dysfunction. Methods Websites were generated using a Google search of “shock wave therapy for erectile dysfunction” with location filters disabled. Readability was analyzed using Readable software (Readable.com, Horsham, United Kingdom). Quality was assessed independently by three reviewers using the DISCERN tool. The same plain text files collected were inputted into ChatGPT to determine whether they produced comparable metrics for readability and quality. Results The study results revealed a notable disparity between ChatGPT's readability assessment and that obtained from a reliable tool, Readable.com (p<0.05). This indicates a lack of alignment between ChatGPT's algorithm and that of established tools, such as Readable.com. Similarly, the DISCERN score generated by ChatGPT differed significantly from the scores generated manually by human evaluators (p<0.05), suggesting that ChatGPT may not be capable of accurately identifying poor-quality information sources regarding shock wave therapy as a treatment for erectile dysfunction. Conclusion ChatGPT’s evaluation of the quality and readability of online text regarding shockwave therapy for erectile dysfunction differs from that of human raters and trusted tools. Therefore, ChatGPT's current capabilities were not sufficient for reliably assessing the quality and readability of textual content. Further research is needed to elucidate the role of AI in the objective evaluation of online medical content in other fields. Continued development in AI and incorporation of tools such as DISCERN into AI software may enhance the way patients navigate the web in search of high-quality medical content in the future. 0.4% topic match

0.4%

1.3

2021

[228] Chapter Three - The screening phase in systematic reviews: Can we speed up the process? I. Rozanc and M. Mernik Adv. Comput. 2021 - 5 citations - Show abstract - Cite 0.4% topic match

0.4%

0.0

2024

[229] RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance Paulo Henrique Couto, ..., and Lisheng Sun-Hosoya ArXiv 2024 - 0 citations - Show abstract - Cite - PDF 0.4% topic match

0.4%

4.9

2023

[230] On the Reliability of the Area Under the ROC Curve in Empirical Software Engineering L. Lavazza, ..., and Gabriele Rotoloni Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering 2023 - 7 citations - Show abstract - Cite - PDF 0.4% topic match

0.4%

0.0

2024

[231] Leveraging GPT-4 for identifying cancer phenotypes in electronic health records: a performance comparison between GPT-4, GPT-3.5-turbo, Flan-T5, Llama-3-8B, and spaCy’s rule-based and machine learning-based methods Kriti Bhattarai, ..., and Albert M Lai JAMIA Open 2024 - 0 citations - Show abstract - Cite Abstract: Abstract Objective Accurately identifying clinical phenotypes from Electronic Health Records (EHRs) provides additional insights into patients’ health, especially when such information is unavailable in structured data. This study evaluates the application of OpenAI’s Generative Pre-trained Transformer (GPT)-4 model to identify clinical phenotypes from EHR text in non-small cell lung cancer (NSCLC) patients. The goal was to identify disease stages, treatments and progression utilizing GPT-4, and compare its performance against GPT-3.5-turbo, Flan-T5-xl, Flan-T5-xxl, Llama-3-8B, and 2 rule-based and machine learning-based methods, namely, scispaCy and medspaCy. Materials and Methods Phenotypes such as initial cancer stage, initial treatment, evidence of cancer recurrence, and affected organs during recurrence were identified from 13 646 clinical notes for 63 NSCLC patients from Washington University in St. Louis, Missouri. The performance of the GPT-4 model is evaluated against GPT-3.5-turbo, Flan-T5-xxl, Flan-T5-xl, Llama-3-8B, medspaCy, and scispaCy by comparing precision, recall, and micro-F1 scores. Results GPT-4 achieved higher F1 score, precision, and recall compared to Flan-T5-xl, Flan-T5-xxl, Llama-3-8B, medspaCy, and scispaCy’s models. GPT-3.5-turbo performed similarly to that of GPT-4. GPT, Flan-T5, and Llama models were not constrained by explicit rule requirements for contextual pattern recognition. spaCy models relied on predefined patterns, leading to their suboptimal performance. Discussion and Conclusion GPT-4 improves clinical phenotype identification due to its robust pre-training and remarkable pattern recognition capability on the embedded tokens. It demonstrates data-driven effectiveness even with limited context in the input. While rule-based models remain useful for some tasks, GPT models offer improved contextual understanding of the text, and robust clinical phenotype extraction. 0.4% topic match

0.4%

1.9

2017

[232] Ranking and Feedback-based Stopping for Recall-Centric Document Retrieval Noah Hollmann and Carsten Eickhoff Conference and Labs of the Evaluation Forum 2017 - 15 citations - Show abstract - Cite 0.4% topic match

0.4%

0.0

2024

[233] An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review Lisa Affengruber, ..., and René Spijker BMC Medical Research Methodology 2024 - 0 citations - Show abstract - Cite 0.4% topic match

0.4%

9.0

2023

[234] Data Extraction for Evidence Synthesis Using a Large Language Model: A Proof-of-Concept Study Gerald Gartlehner, ..., and Robert Chew Research synthesis methods 2023 - 10 citations - Show abstract - Cite - PDF 0.4% topic match

0.4%

2.6

2024

[235] GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet Kaiming Tao, ..., and Robert W Shafer BMC Medical Research Methodology 2024 - 1 citations - Show abstract - Cite - PDF 0.4% topic match

0.3%

79.7

2023

[236] Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank R. Ali, ..., and Wael Asaad Neurosurgery 2023 - 127 citations - Show abstract - Cite Abstract: BACKGROUND AND OBJECTIVES: General large language models (LLMs), such as ChatGPT (GPT-3.5), have demonstrated the capability to pass multiple-choice medical board examinations. However, comparative accuracy of different LLMs and LLM performance on assessments of predominantly higher-order management questions is poorly understood. We aimed to assess the performance of 3 LLMs (GPT-3.5, GPT-4, and Google Bard) on a question bank designed specifically for neurosurgery oral boards examination preparation. METHODS: The 149-question Self-Assessment Neurosurgery Examination Indications Examination was used to query LLM accuracy. Questions were inputted in a single best answer, multiple-choice format. χ2, Fisher exact, and univariable logistic regression tests assessed differences in performance by question characteristics. RESULTS: On a question bank with predominantly higher-order questions (85.2%), ChatGPT (GPT-3.5) and GPT-4 answered 62.4% (95% CI: 54.1%-70.1%) and 82.6% (95% CI: 75.2%-88.1%) of questions correctly, respectively. By contrast, Bard scored 44.2% (66/149, 95% CI: 36.2%-52.6%). GPT-3.5 and GPT-4 demonstrated significantly higher scores than Bard (both P < .01), and GPT-4 outperformed GPT-3.5 (P = .023). Among 6 subspecialties, GPT-4 had significantly higher accuracy in the Spine category relative to GPT-3.5 and in 4 categories relative to Bard (all P < .01). Incorporation of higher-order problem solving was associated with lower question accuracy for GPT-3.5 (odds ratio [OR] = 0.80, P = .042) and Bard (OR = 0.76, P = .014), but not GPT-4 (OR = 0.86, P = .085). GPT-4's performance on imaging-related questions surpassed GPT-3.5's (68.6% vs 47.1%, P = .044) and was comparable with Bard's (68.6% vs 66.7%, P = 1.000). However, GPT-4 demonstrated significantly lower rates of “hallucination” on imaging-related questions than both GPT-3.5 (2.3% vs 57.1%, P < .001) and Bard (2.3% vs 27.3%, P = .002). Lack of question text description for questions predicted significantly higher odds of hallucination for GPT-3.5 (OR = 1.45, P = .012) and Bard (OR = 2.09, P < .001). CONCLUSION: On a question bank of predominantly higher-order management case scenarios for neurosurgery oral boards preparation, GPT-4 achieved a score of 82.6%, outperforming ChatGPT and Google Bard. 0.3% topic match

0.3%

0.1

2018

[237] SWIFT-Review: A Text-Mining Workbench for Systematic Review Kayla K. Thayer and J. Phillips Journal Not Provided 2018 - 1 citations - Show abstract - Cite Abstract: Here, we introduce “SWIFT-Review” (SWIFT is an acronym for “Sciome Workbench for Interactive computer-Facilitated Text-mining”), a freely available, interactive workbench that provides numerous tools to assist with problem formulation and literature prioritization. SWIFT-Review can be used to search, categorize, and visualize patterns in literature search results. The software utilizes statistical modeling and machine learning methods that allow users to identify over-represented topics within the literature corpus and to rank-order titles and abstracts for manual screening. We have tested the automated document prioritization feature on 20 previously conducted systematic review datasets, and the results presented clearly suggest that using machine learning to triage documents for screening has the potential to save, on average, more than 50% of the screening effort ordinarily required when using unordered document lists. In addition, the tagging and annotation capabilities of SWIFT-Review can be used to produce “scoping reports” or “scoping studies,” a type of knowledge synthesis undertaken to guide the direction of future research priorities or to help with problem formulation when conducting a systematic review. As a result, users can more quickly assess the extent of available evidence, prioritize health outcomes and chemical exposures for systematic review, and understand the degree of evidence integration that may be required. In addition, the resulting visualizations can help to identify topics that have been extensively studied as well as emerging areas of research. SWIFT-Review integrates seamlessly with other text-mining platforms including Active-Screener and HAWC. The software remains under active development with several new features planned. Disclaimer: The views expressed in this abstract are those of the authors and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency. 0.3% topic match

0.3%

6.9

2019

[238] Automatic Boolean Query Refinement for Systematic Review Literature Search Harrisen Scells, ..., and B. Koopman The World Wide Web Conference 2019 - 38 citations - Show abstract - Cite 0.3% topic match

0.3%

8.8

2017

[239] Automating Biomedical Evidence Synthesis: RobotReviewer I. Marshall, ..., and Byron C. Wallace Proceedings of the conference. Association for Computational Linguistics. Meeting 2017 - 65 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

2.5

2017

[240] QUT ielab at CLEF eHealth 2017 Technology Assisted Reviews Track: Initial experiments with learning to rank Harrisen Scells, ..., and B. Koopman Conference and Labs of the Evaluation Forum 2017 - 20 citations - Show abstract - Cite 0.3% topic match

0.3%

8.5

2023

[241] GPT4 is Slightly Helpful for Peer-Review Assistance: A Pilot Study Zachary Robertson ArXiv 2023 - 12 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

16.6

2014

[242] Reducing systematic review workload through certainty-based screening Makoto Miwa, ..., and S. Ananiadou Journal of Biomedical Informatics 2014 - 168 citations - Show abstract - Cite 0.3% topic match

0.3%

29.2

2023

[243] BioCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval Qiao Jin, ..., and Zhiyong Lu Bioinformatics 2023 - 40 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

34.2

2013

[244] Evaluation Measures for Models Assessment over Imbalanced Data Sets M. Bekkar, ..., and Taklit Akrouf Alitouche Journal of Information Engineering and Applications 2013 - 406 citations - Show abstract - Cite 0.3% topic match

0.3%

1.7

2024

[245] Using GPT-4 to write a scientific review article: a pilot evaluation study Zhiping Paul Wang, ..., and Jason H. Moore BioData Mining 2024 - 1 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

12.4

2023

[246] Assessing the Performance of GPT-3.5 and GPT-4 on the 2023 Japanese Nursing Examination Yudai Kaneda, ..., and Tetsuya Tanimoto Cureus 2023 - 16 citations - Show abstract - Cite - PDF Abstract: Purpose The purpose of this study was to evaluate the changes in capabilities between the Generative Pre-trained Transformer (GPT)-3.5 and GPT-4 versions of the large-scale language model ChatGPT within a Japanese medical context. Methods The study involved ChatGPT versions 3.5 and 4 responding to questions from the 112th Japanese National Nursing Examination (JNNE). The study comprised three analyses: correct answer rate and score rate calculations, comparisons between GPT-3.5 and GPT-4, and comparisons of correct answer rates for conversation questions. Results ChatGPT versions 3.5 and 4 responded to 237 out of 238 Japanese questions from the 112th JNNE. While GPT-3.5 achieved an overall accuracy rate of 59.9%, failing to meet the passing standards in compulsory and general/scenario-based questions, scoring 58.0% and 58.3%, respectively, GPT-4 had an accuracy rate of 79.7%, satisfying the passing standards by scoring 90.0% and 77.7%, respectively. For each problem type, GPT-4 showed a higher accuracy rate than GPT-3.5. Specifically, the accuracy rates for compulsory questions improved from 58.0% with GPT-3.5 to 90.0% with GPT-4. For general questions, the rates went from 64.6% with GPT-3.5 to 75.6% with GPT-4. In scenario-based questions, the accuracy rates improved substantially from 51.7% with GPT-3.5 to 80.0% with GPT-4. For conversation questions, GPT-3.5 had an accuracy rate of 73.3% and GPT-4 had an accuracy rate of 93.3%. Conclusions The GPT-4 version of ChatGPT displayed performance sufficient to pass the JNNE, significantly improving from GPT-3.5. This suggests specialized medical training could make such models beneficial in Japanese clinical settings, aiding decision-making. However, user awareness and training are crucial, given potential inaccuracies in ChatGPT's responses. Hence, responsible usage with an understanding of its capabilities and limitations is vital to best support healthcare professionals and patients. 0.3% topic match

0.3%

24.9

2023

[247] Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study Theresa Isabelle Wilhelm, ..., and R. Kaczmarczyk Journal of Medical Internet Research 2023 - 26 citations - Show abstract - Cite - PDF Abstract: Background As advancements in artificial intelligence (AI) continue, large language models (LLMs) have emerged as promising tools for generating medical information. Their rapid adaptation and potential benefits in health care require rigorous assessment in terms of the quality, accuracy, and safety of the generated information across diverse medical specialties. Objective This study aimed to evaluate the performance of 4 prominent LLMs, namely, Claude-instant-v1.0, GPT-3.5-Turbo, Command-xlarge-nightly, and Bloomz, in generating medical content spanning the clinical specialties of ophthalmology, orthopedics, and dermatology. Methods Three domain-specific physicians evaluated the AI-generated therapeutic recommendations for a diverse set of 60 diseases. The evaluation criteria involved the mDISCERN score, correctness, and potential harmfulness of the recommendations. ANOVA and pairwise t tests were used to explore discrepancies in content quality and safety across models and specialties. Additionally, using the capabilities of OpenAI’s most advanced model, GPT-4, an automated evaluation of each model’s responses to the diseases was performed using the same criteria and compared to the physicians’ assessments through Pearson correlation analysis. Results Claude-instant-v1.0 emerged with the highest mean mDISCERN score (3.35, 95% CI 3.23-3.46). In contrast, Bloomz lagged with the lowest score (1.07, 95% CI 1.03-1.10). Our analysis revealed significant differences among the models in terms of quality (P<.001). Evaluating their reliability, the models displayed strong contrasts in their falseness ratings, with variations both across models (P<.001) and specialties (P<.001). Distinct error patterns emerged, such as confusing diagnoses; providing vague, ambiguous advice; or omitting critical treatments, such as antibiotics for infectious diseases. Regarding potential harm, GPT-3.5-Turbo was found to be the safest, with the lowest harmfulness rating. All models lagged in detailing the risks associated with treatment procedures, explaining the effects of therapies on quality of life, and offering additional sources of information. Pearson correlation analysis underscored a substantial alignment between physician assessments and GPT-4’s evaluations across all established criteria (P<.01). Conclusions This study, while comprehensive, was limited by the involvement of a select number of specialties and physician evaluators. The straightforward prompting strategy (“How to treat…”) and the assessment benchmarks, initially conceptualized for human-authored content, might have potential gaps in capturing the nuances of AI-driven information. The LLMs evaluated showed a notable capability in generating valuable medical content; however, evident lapses in content quality and potential harm signal the need for further refinements. Given the dynamic landscape of LLMs, this study’s findings emphasize the need for regular and methodical assessments, oversight, and fine-tuning of these AI tools to ensure they produce consistently trustworthy and clinically safe medical advice. Notably, the introduction of an auto-evaluation mechanism using GPT-4, as detailed in this study, provides a scalable, transferable method for domain-agnostic evaluations, extending beyond therapy recommendation assessments. 0.3% topic match

0.3%

0.0

2024

[248] Prompt Engineering Strategies Improve the Diagnostic Accuracy of GPT-4 Turbo in Neuroradiology Cases MD Akihiko Wada, ..., and MD Shigeki Aoki medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

11526.5

2020

[249] The PRISMA 2020 statement: an updated guideline for reporting systematic reviews M. Page, ..., and D. Moher Systematic Reviews 2020 - 48031 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

5.0

2005

[250] Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review E. Crumley, ..., and L. Hartling BMC Medical Research Methodology 2005 - 96 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

9.2

2023

[251] Product Question Answering in E-Commerce: A Survey Yang Deng, ..., and W. Lam ArXiv 2023 - 16 citations - Show abstract - Cite - PDF 0.3% topic match

0.3%

5.5

1999

[252] Estimating time to conduct a meta-analysis from number of citations retrieved. I. E. Allen and I. Olkin JAMA 1999 - 139 citations - Show abstract - Cite 0.3% topic match

0.2%

4.9

2021

[253] A decision support system for automating document retrieval and citation screening Raymon van Dinter, ..., and B. Tekinerdogan Expert Syst. Appl. 2021 - 17 citations - Show abstract - Cite 0.2% topic match

0.2%

3.4

2023

[254] Limitations of GPT-3.5 and GPT-4 in Applying Fleischner Society Guidelines to Incidental Lung Nodules Joel L. Gamble, ..., and Adnan Sheikh Canadian Association of Radiologists Journal 2023 - 3 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

325.1

2007

[255] Bmc Medical Research Methodology Ruth Mcnair, ..., and Kelsey Hegarty Journal Not Provided 2007 - 5810 citations - Show abstract - Cite 0.2% topic match

0.2%

0.0

2024

[256] Assisting humans in complex comparisons: automated information comparison at scale Truman Yuen, ..., and Y. Lawryshyn ArXiv 2024 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.0

2023

[257] Evaluating the Efficacy of ChatGPT-4 in Providing Scientific References across Diverse Disciplines Zhi Cao ArXiv 2023 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

1.3

2018

[258] LIMSI@CLEF eHealth 2018 Task 2: Technology Assisted Reviews by Stacking Active and Static Learning Christopher Norman, ..., and Aurélie Névéol Conference and Labs of the Evaluation Forum 2018 - 9 citations - Show abstract - Cite 0.2% topic match

0.2%

0.0

2024

[259] Contextualized AI for Cyber Defense: An Automated Survey using LLMs Christoforus Yoga Haryanto, ..., and Arathi Arakala Journal Not Provided 2024 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.0

2024

[260] Tracking ChatGPT Research: Insights From the Literature and the Web Omar Mubin, ..., and Zhao Zou IEEE Access 2024 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

11.3

2018

[261] CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview E. Kanoulas, ..., and R. Spijker Conference and Labs of the Evaluation Forum 2018 - 71 citations - Show abstract - Cite 0.2% topic match

0.2%

16.8

2016

[262] Extracting PICO Sentences from Clinical Trial Reports using Supervised Distant Supervision Byron C. Wallace, ..., and I. Marshall Journal of machine learning research : JMLR 2016 - 149 citations - Show abstract - Cite 0.2% topic match

0.2%

1.0

2023

[263] Are Different Versions of ChatGPT’s Ability Comparable to the Clinical Diagnosis Presented in Case Reports? A Descriptive Study Jingfang Chen, ..., and Chengliang Yin Journal of Multidisciplinary Healthcare 2023 - 1 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

163.4

2022

[264] When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories Alex Troy Mallen, ..., and Daniel Khashabi Annual Meeting of the Association for Computational Linguistics 2022 - 311 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.9

2019

[265] A Distributed Effort Approach for Systematic Reviews. IMS Unipd At CLEF 2019 eHealth Task 2 Giorgio Maria Di Nunzio Conference and Labs of the Evaluation Forum 2019 - 5 citations - Show abstract - Cite 0.2% topic match

0.2%

6609.4

2020

[266] Language Models are Few-Shot Learners Tom B. Brown, ..., and Dario Amodei ArXiv 2020 - 29514 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

1.3

2018

[267] Aristotle University's Approach to the Technologically Assisted Reviews in Empirical Medicine Task of the 2018 CLEF eHealth Lab Adamantios Minas, ..., and Grigorios Tsoumakas Conference and Labs of the Evaluation Forum 2018 - 9 citations - Show abstract - Cite 0.2% topic match

0.2%

7.0

2017

[268] Integrating the Framing of Clinical Questions via PICO into the Retrieval of Medical Literature for Systematic Reviews Harrisen Scells, ..., and S. Geva Proceedings of the 2017 ACM on Conference on Information and Knowledge Management 2017 - 49 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

3.4

2017

[269] Vision for SLR tooling infrastructure: Prioritizing value-added requirements Ahmed Al-Zubidy, ..., and Edgar E. Hassler Inf. Softw. Technol. 2017 - 24 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.0

2024

[270] Evaluating the competency of ChatGPT in MRCP Part 1 and a systematic literature review of its capabilities in postgraduate medical assessments Oliver Vij, ..., and K. Kouranloo PLOS ONE 2024 - 0 citations - Show abstract - Cite Abstract: Objectives As a large language model (LLM) trained on a large data set, ChatGPT can perform a wide array of tasks without additional training. We evaluated the performance of ChatGPT on postgraduate UK medical examinations through a systematic literature review of ChatGPT’s performance in UK postgraduate medical assessments and its performance on Member of Royal College of Physicians (MRCP) Part 1 examination. Methods Medline, Embase and Cochrane databases were searched. Articles discussing the performance of ChatGPT in UK postgraduate medical examinations were included in the systematic review. Information was extracted on exam performance including percentage scores and pass/fail rates. MRCP UK Part 1 sample paper questions were inserted into ChatGPT-3.5 and -4 four times each and the scores marked against the correct answers provided. Results 12 studies were ultimately included in the systematic literature review. ChatGPT-3.5 scored 66.4% and ChatGPT-4 scored 84.8% on MRCP Part 1 sample paper, which is 4.4% and 22.8% above the historical pass mark respectively. Both ChatGPT-3.5 and -4 performance was significantly above the historical pass mark for MRCP Part 1, indicating they would likely pass this examination. ChatGPT-3.5 failed eight out of nine postgraduate exams it performed with an average percentage of 5.0% below the pass mark. ChatGPT-4 passed nine out of eleven postgraduate exams it performed with an average percentage of 13.56% above the pass mark. ChatGPT-4 performance was significantly better than ChatGPT-3.5 in all examinations that both models were tested on. Conclusion ChatGPT-4 performed at above passing level for the majority of UK postgraduate medical examinations it was tested on. ChatGPT is prone to hallucinations, fabrications and reduced explanation accuracy which could limit its potential as a learning tool. The potential for these errors is an inherent part of LLMs and may always be a limitation for medical applications of ChatGPT. 0.2% topic match

0.2%

4.2

2022

[271] In‐depth evaluation of machine learning methods for semi‐automating article screening in a systematic review of mechanistic literature Mihiretu M Kebede, ..., and R. Fortner Research Synthesis Methods 2022 - 10 citations - Show abstract - Cite - PDF Abstract: We aimed to evaluate the performance of supervised machine learning algorithms in predicting articles relevant for full‐text review in a systematic review. Overall, 16,430 manually screened titles/abstracts, including 861 references identified relevant for full‐text review were used for the analysis. Of these, 40% (n = 6573) were sub‐divided for training (70%) and testing (30%) the algorithms. The remaining 60% (n = 9857) were used as a validation set. We evaluated down‐ and up‐sampling methods and compared unigram, bigram, and singular value decomposition (SVD) approaches. For each approach, Naïve Bayes, Support Vector Machines (SVM), regularized logistic regressions, neural networks, random forest, Logit boost, and XGBoost were implemented using simple term frequency or Tf‐Idf feature representations. Performance was evaluated using sensitivity, specificity, precision and area under the Curve. We combined predictions of the best‐performing algorithms (Youden Index ≥0.3 with sensitivity/specificity≥70/60%). In a down‐sample unigram approach, Naïve Bayes, SVM/quanteda text models with Tf‐Idf, and linear SVM e1071 package with Tf‐Idf achieved >90% sensitivity at specificity >65%. Combining the predictions of the 10 best‐performing algorithms improved the performance to reach 95% sensitivity and 64% specificity in the validation set. Crude screening burden was reduced by 61% (5979) (adjusted: 80.3%) with 5% (27) false negativity rate. All the other approaches yielded relatively poorer performances. The down‐sampling unigram approach achieved good performance in our data. Combining the predictions of algorithms improved sensitivity while screening burden was reduced by almost two‐third. Implementing machine learning approaches in title/abstract screening should be investigated further toward refining these tools and automating their implementation. 0.2% topic match

0.2%

17.7

2018

[272] Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) E. Beller, ..., and P. Glasziou Systematic Reviews 2018 - 115 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

25.4

2021

[273] Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline Luyu Gao, ..., and Jamie Callan European Conference on Information Retrieval 2021 - 97 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

4.1

2011

[274] Precision of healthcare systematic review searches in a cross‐sectional sample M. Sampson, ..., and C. Urquhart Research Synthesis Methods 2011 - 55 citations - Show abstract - Cite 0.2% topic match

0.2%

0.0

2024

[275] What Comes Next for Large Language Models in Medicine? Gregory Laynor Journal of Electronic Resources in Medical Libraries 2024 - 0 citations - Show abstract - Cite 0.2% topic match

0.2%

3212.4

2022

[276] Training language models to follow instructions with human feedback Long Ouyang, ..., and Ryan J. Lowe ArXiv 2022 - 8672 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

6.5

2021

[277] Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review Eugene Yang, ..., and O. Frieder European Conference on Information Retrieval 2021 - 23 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.0

2024

[278] Benchmarking Large Language Models for Extraction of International Classification of Diseases Codes from Clinical Documentation Ashley Simmons, ..., and Ankit Sakhuja medRxiv 2024 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

216.9

2023

[279] Can artificial intelligence help for scientific writing? Michele Salvagno, ..., and A. Gerli Critical Care 2023 - 373 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

80.1

2016

[280] Interactive machine learning for health informatics: when do we need the human-in-the-loop? Andreas Holzinger Brain Informatics 2016 - 697 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

15.3

2019

[281] A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Annette M. O’Connor, ..., and B. Hutton Systematic Reviews 2019 - 83 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.0

2024

[282] ChatGPT in medical writing: enhancing healthcare communication through artificial intelligence and human expertise I. Mese, ..., and T. Kuzan Anatolian Current Medical Journal 2024 - 0 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

14.4

2014

[283] Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviews I. Shemilt, ..., and James Thomas Research Synthesis Methods 2014 - 154 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

0.4

2017

[284] QUT ielab at CLEF 2017 Technology Assisted Reviews Track: Initial Experiments with Learning To Rank Harrisen Scells, ..., and B. Koopman Conference and Labs of the Evaluation Forum 2017 - 3 citations - Show abstract - Cite 0.2% topic match

0.2%

26.2

2023

[285] Artificial intelligence in systematic reviews: promising when appropriately used S. V. van Dijk, ..., and A. Lenferink BMJ Open 2023 - 36 citations - Show abstract - Cite - PDF 0.2% topic match

0.2%

362.5

2006

[286] Performing systematic literature reviews in software engineering D. Budgen and P. Brereton Proceedings of the 28th international conference on Software engineering 2006 - 6694 citations - Show abstract - Cite 0.2% topic match

0.1%

47.4

2023

[287] Can large language models provide useful feedback on research papers? A large-scale empirical analysis Weixin Liang, ..., and James Zou ArXiv 2023 - 53 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.0

2024

[288] REVOLUTIONIZING IBD MANAGEMENT: HOW DO CHATGPT & GOOGLE BARD STAND UP IN OFFERING COMPREHENSIVE MANAGEMENT SOLUTIONS? R. M. Ram mohan, ..., and P. Mustacchia Gastroenterology 2024 - 0 citations - Show abstract - Cite Abstract: Artificial Intelligence (AI) has notably transformed the realm of healthcare, especially in diagnosing and treating Inflammatory Bowel Disease (IBD) and other digestive disorders. Essential AI tools, such as ChatGPT and Google Bard, can interpret endoscopic imagery, analyze diverse samples, simplify administrative duties, and assist in assessing medical images and the automation of devices. By individualizing treatments and forecasting adverse reactions, these AI applications have notably enhanced the management of digestive diseases. Integrating such AI technologies empowers patients to make knowledgeable choices by evaluating customized treatment alternatives. The study's objective was to evaluate the precision of two widely used Chat Bots, Chat GPT and Google BARD, in responding to queries related to medical management. Each bot was tasked with answering a series of questions, and the responses were scored on a 1-10 Likert scale, with 1 denoting high accuracy. To uphold impartial judgment, two unbiased assessors reviewed each bot's replies. The purpose of this research was to illuminate the competencies of these chatbots by systematically analyzing their performance in aspects of accuracy and dependability. Employing two evaluators and utilizing the Likert scale approach helped mitigate potential biases, corroborating the findings. Our study compared the proficiency of Chat GPT and Google BARD in medical management. Chat GPT demonstrated superior performance, achieving 71% in accuracy as opposed to Google BARD’s 39% (p = 0.032) and 58% versus 29% in the reliability of medical information (p = 0.021). This research underscores the pivotal importance of accuracy and reliability in the creation of IBD chatbots. While Chat GPT showed commendable results, signifying its promise as a trustworthy resource, it also highlighted the imperative need for ongoing research and advancement in this domain. AI has significantly reshaped healthcare, particularly aiding in the diagnosis and management of IBD. ChatGPT and Google Bard are key AI tools that analyze medical images, assess samples, and optimize tasks. Our research evaluated the precision and dependability of these chatbots in the sphere of medical management. ChatGPT surpassed Google Bard, securing 71% in accuracy and 58% in reliability of information. While ChatGPT showcases substantial potential as a reliable AI instrument in IBD management, the results also accentuate the continual necessity for exploration and advancement in this area. 0.1% topic match

0.1%

0.0

2024

[289] Automated Review Generation Method Based on Large Language Models Shican Wu, ..., and Jinlong Gong ArXiv 2024 - 0 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

60.3

2023

[290] ChatGPT for Next Generation Science Learning Xiaoming Zhai XRDS: Crossroads, The ACM Magazine for Students 2023 - 103 citations - Show abstract - Cite 0.1% topic match

0.1%

0.0

2024

[291] Using Large Language Models to Evaluate Biomedical Query-Focused Summarisation Hashem Hijazi, ..., and Sarvnaz Karimi Workshop on Biomedical Natural Language Processing 2024 - 0 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

10.1

2019

[292] Kyoto, Japan No author found The Statesman’s Yearbook Companion 2019 - 59 citations - Show abstract - Cite 0.1% topic match

0.1%

3.5

2019

[293] A systematic review of the association between fault or blame-related attributions and procedures after transport injury and health and work-related outcomes. M. Giummarra, ..., and B. Gabbe Accident; analysis and prevention 2019 - 17 citations - Show abstract - Cite 0.1% topic match

0.1%

9.3

2023

[294] Attention-based multimodal fusion with contrast for robust clinical prediction in the face of missing modalities Jinghui Liu, ..., and K. Verspoor Journal of biomedical informatics 2023 - 12 citations - Show abstract - Cite 0.1% topic match

0.1%

9.3

2024

[295] Can ChatGPT assist authors with abstract writing in medical journals? Evaluating the quality of scientific abstracts generated by ChatGPT and original abstracts Taesoon Hwang, ..., and Saboor Khan PLOS ONE 2024 - 7 citations - Show abstract - Cite - PDF Abstract: Introduction ChatGPT, a sophisticated large language model (LLM), has garnered widespread attention for its ability to mimic human-like communication. As recent studies indicate a potential supportive role of ChatGPT in academic writing, we assessed the LLM’s capacity to generate accurate and comprehensive scientific abstracts from published Randomised Controlled Trial (RCT) data, focusing on the adherence to the Consolidated Standards of Reporting Trials for Abstracts (CONSORT-A) statement, in comparison to the original authors’ abstracts. Methodology RCTs, identified in a PubMed/MEDLINE search post-September 2021 across various medical disciplines, were subjected to abstract generation via ChatGPT versions 3.5 and 4, following the guidelines of the respective journals. The overall quality score (OQS) of each abstract was determined by the total number of adequately reported components from the 18-item CONSORT-A checklist. Additional outcome measures included percent adherence to each CONOSORT-A item, readability, hallucination rate, and regression analysis of reporting quality determinants. Results Original abstracts achieved a mean OQS of 11.89 (95% CI: 11.23–12.54), outperforming GPT 3.5 (7.89; 95% CI: 7.32–8.46) and GPT 4 (5.18; 95% CI: 4.64–5.71). Compared to GPT 3.5 and 4 outputs, original abstracts were more adherent with 10 and 14 CONSORT-A items, respectively. In blind assessments, GPT 3.5-generated abstracts were deemed most readable in 62.22% of cases which was significantly greater than the original (31.11%; P = 0.003) and GPT 4-generated (6.67%; P<0.001) abstracts. Moreover, ChatGPT 3.5 exhibited a hallucination rate of 0.03 items per abstract compared to 1.13 by GPT 4. No determinants for improved reporting quality were identified for GPT-generated abstracts. Conclusions While ChatGPT could generate more readable abstracts, their overall quality was inferior to the original abstracts. Yet, its proficiency to concisely relay key information with minimal error holds promise for medical research and warrants further investigations to fully ascertain the LLM’s applicability in this domain. 0.1% topic match

0.1%

1.8

2024

[296] A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare Leyao Wang, ..., and Zhijun Yin medRxiv 2024 - 1 citations - Show abstract - Cite Abstract: Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: We utilized PubMed, ACM, and IEEE digital libraries as primary sources for this review. We followed the guidance of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRIMSA) to screen and select peer-reviewed research articles that (1) were related to both healthcare applications and conversational LLMs and (2) were published before September 1st, 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare. 0.1% topic match

0.1%

4.7

2023

[297] Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study A. J. Thirunavukarasu, ..., and D. Ting medRxiv 2023 - 6 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

14.8

2023

[298] Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study Hamish S. F. Fraser, ..., and R. Hilliard JMIR mHealth and uHealth 2023 - 21 citations - Show abstract - Cite - PDF Abstract: Background Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients. Objective The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews. Methods We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated “unsafe” or “too cautious.” Results Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%). Conclusions ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation. 0.1% topic match

0.1%

17.1

2023

[299] Performance of Large Language Models (ChatGPT, Bing Search, and Google Bard) in Solving Case Vignettes in Physiology Anupumar Dhanvijay, ..., and Himel Mondal Cureus 2023 - 22 citations - Show abstract - Cite - PDF Abstract: Background Large language models (LLMs) have emerged as powerful tools capable of processing and generating human-like text. These LLMs, such as ChatGPT (OpenAI Incorporated, Mission District, San Francisco, United States), Google Bard (Alphabet Inc., CA, US), and Microsoft Bing (Microsoft Corporation, WA, US), have been applied across various domains, demonstrating their potential to assist in solving complex tasks and improving information accessibility. However, their application in solving case vignettes in physiology has not been explored. This study aimed to assess the performance of three LLMs, namely, ChatGPT (3.5; free research version), Google Bard (Experiment), and Microsoft Bing (precise), in answering cases vignettes in Physiology. Methods This cross-sectional study was conducted in July 2023. A total of 77 case vignettes in physiology were prepared by two physiologists and were validated by two other content experts. These cases were presented to each LLM, and their responses were collected. Two physiologists independently rated the answers provided by the LLMs based on their accuracy. The ratings were measured on a scale from 0 to 4 according to the structure of the observed learning outcome (pre-structural = 0, uni-structural = 1, multi-structural = 2, relational = 3, extended-abstract). The scores among the LLMs were compared by Friedman’s test and inter-observer agreement was checked by the intraclass correlation coefficient (ICC). Results The overall scores for ChatGPT, Bing, and Bard in the study, with a total of 77 cases, were found to be 3.19±0.3, 2.15±0.6, and 2.91±0.5, respectively, p<0.0001. Hence, ChatGPT 3.5 (free version) obtained the highest score, Bing (Precise) had the lowest score, and Bard (Experiment) fell in between the two in terms of performance. The average ICC values for ChatGPT, Bing, and Bard were 0.858 (95% CI: 0.777 to 0.91, p<0.0001), 0.975 (95% CI: 0.961 to 0.984, p<0.0001), and 0.964 (95% CI: 0.944 to 0.977, p<0.0001), respectively. Conclusion ChatGPT outperformed Bard and Bing in answering case vignettes in physiology. Hence, students and teachers may think about choosing LLMs for their educational purposes accordingly for case-based learning in physiology. Further exploration of their capabilities is needed for adopting those in medical education and support for clinical decision-making. 0.1% topic match

0.1%

0.0

2024

[300] COMPARATIVE ANALYSIS OF GENERATIVE ARTIFICIAL INTELLIGENCE (AI) VERSUS MANUAL DATA ANALYSIS IN ORTHOPAEDIC PATIENT DATA Mustafa Al-Naib, ..., and S. Radha Orthopaedic Proceedings 2024 - 0 citations - Show abstract - Cite Abstract: As patient data continues to grow, the importance of efficient and precise analysis cannot be overstated. The employment of Generative Artificial Intelligence (AI), specifically Chat GPT-4, in the realm of medical data interpretation has been on the rise. However, its effectiveness in comparison to manual data analysis has been insufficiently investigated.This quality improvement project aimed to evaluate the accuracy and time-efficiency of Generative AI (GPT-4) against manual data interpretation within extensive datasets pertaining to patients with orthopaedic injuries.A dataset, containing details of 6,562 orthopaedic trauma patients admitted to a district general hospital over a span of two years, was reviewed. Two researchers operated independently: one utilised GPT-4 for insights via prompts, while the other manually examined the identical dataset employing Microsoft Excel and IBM® SPSS® software. Both were blinded on each other's procedures and outcomes. Each researcher answered 20 questions based on the dataset including injury details, age groups, injury specifics, activity trends and the duration taken to assess the data.Upon comparison, both GPT-4 and the manual researcher achieved consistent results for 19 out of the 20 questions (95% accuracy). After a subsequent review and refined prompts (prompt engineering) to GPT-4, the answer to the final question aligned with the manual researcher's findings. GPT-4 required just 30 minutes, a stark contrast to the manual researcher's 9-hour analytical duration.This quality improvement project emphasises the transformative potential of Generative AI in the domain of medical data analysis. GPT-4 not only paralleled the accuracy of manual analysis but also achieved this in significantly less time. For optimal accurate results, data analysis by AI can be enhanced through human oversight. Adopting AI-driven approaches, particularly in orthopaedic data interpretation, can enhance efficiency and ultimately improve patient care. We recommend future investigations on large and more varied datasets to reaffirm these outcomes. 0.1% topic match

0.1%

8.1

2023

[301] A SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis of ChatGPT in the Medical Literature: Concise Review Daniel Gödde, ..., and Timur Sellmann Journal of Medical Internet Research 2023 - 12 citations - Show abstract - Cite - PDF Abstract: Background ChatGPT is a 175-billion-parameter natural language processing model that is already involved in scientific content and publications. Its influence ranges from providing quick access to information on medical topics, assisting in generating medical and scientific articles and papers, performing medical data analyses, and even interpreting complex data sets. Objective The future role of ChatGPT remains uncertain and a matter of debate already shortly after its release. This review aimed to analyze the role of ChatGPT in the medical literature during the first 3 months after its release. Methods We performed a concise review of literature published in PubMed from December 1, 2022, to March 31, 2023. To find all publications related to ChatGPT or considering ChatGPT, the search term was kept simple (“ChatGPT” in AllFields). All publications available as full text in German or English were included. All accessible publications were evaluated according to specifications by the author team (eg, impact factor, publication modus, article type, publication speed, and type of ChatGPT integration or content). The conclusions of the articles were used for later SWOT (strengths, weaknesses, opportunities, and threats) analysis. All data were analyzed on a descriptive basis. Results Of 178 studies in total, 160 met the inclusion criteria and were evaluated. The average impact factor was 4.423 (range 0-96.216), and the average publication speed was 16 (range 0-83) days. Among the articles, there were 77 editorials (48,1%), 43 essays (26.9%), 21 studies (13.1%), 6 reviews (3.8%), 6 case reports (3.8%), 6 news (3.8%), and 1 meta-analysis (0.6%). Of those, 54.4% (n=87) were published as open access, with 5% (n=8) provided on preprint servers. Over 400 quotes with information on strengths, weaknesses, opportunities, and threats were detected. By far, most (n=142, 34.8%) were related to weaknesses. ChatGPT excels in its ability to express ideas clearly and formulate general contexts comprehensibly. It performs so well that even experts in the field have difficulty identifying abstracts generated by ChatGPT. However, the time-limited scope and the need for corrections by experts were mentioned as weaknesses and threats of ChatGPT. Opportunities include assistance in formulating medical issues for nonnative English speakers, as well as the possibility of timely participation in the development of such artificial intelligence tools since it is in its early stages and can therefore still be influenced. Conclusions Artificial intelligence tools such as ChatGPT are already part of the medical publishing landscape. Despite their apparent opportunities, policies and guidelines must be implemented to ensure benefits in education, clinical practice, and research and protect against threats such as scientific misconduct, plagiarism, and inaccuracy. 0.1% topic match

0.1%

44.3

2017

[302] Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry Rohit Borah, ..., and K. Kaiser BMJ Open 2017 - 345 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

6.7

2020

[303] Interventions for increasing colorectal cancer screening uptake among African-American men: A systematic review and meta-analysis C. Rogers, ..., and M. Foster PLoS ONE 2020 - 28 citations - Show abstract - Cite - PDF Abstract: Background African-American men have the lowest 5-year survival rate in the U.S. for colorectal cancer (CRC) of any racial group, which may partly stem from low screening adherence. It is imperative to synthesize the literature evaluating the effectiveness of interventions on CRC screening uptake in this population. Materials and methods In this systematic review and meta-analysis, Medline, CINAHL, Embase, and Cochrane CENTRAL were searched for U.S.-based interventions that: were published after 1998–January 2020; included African-American men; and evaluated CRC screening uptake explicitly. Checklist by Cochrane Collaboration and Joanna Brigg were utilized to assess risk of bias, and meta-regression and sensitivity analyses were employed to identify the most effective interventions. Results Our final sample comprised 41 studies with 2 focused exclusively on African-American men. The most frequently adopted interventions were educational materials (39%), stool-based screening kits (14%), and patient navigation (11%). Most randomized controlled trials failed to provide details about the blinding of the participant recruitment method, allocation concealment method, and/or the outcome assessment. Due to high heterogeneity, meta-analysis was conducted among 17 eligible studies. Interventions utilizing stool-based kits or patient navigation were most effective at increasing CRC screening completion, with odds ratios of 9.60 (95% CI 2.89–31.82, p = 0.0002) and 2.84 (95% CI 1.23–6.49, p = 0.01). No evidence of publication bias was present for this study registered with the International Prospective Registry of Systematic Reviews (PROSPERO 2019 CRD42019119510). Conclusions Additional research is warranted to uncover effective, affordable interventions focused on increasing CRC screening completion among African-American men. When designing and implementing future multicomponent interventions, employing 4 or fewer interventions types may reduce bias risk. Since only 5% of the interventions solely focused on African-American men, future theory-driven interventions should consider recruiting samples comprised solely of this population. 0.1% topic match

0.1%

307.6

2023

[304] Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios M. Cascella, ..., and E. Bignami Journal of Medical Systems 2023 - 523 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

1.1

2017

[305] A Study of Convolutional Neural Networks for Clinical Document Classification in Systematic Reviews: SysReview at CLEF eHealth 2017 Grace E. Lee Conference and Labs of the Evaluation Forum 2017 - 9 citations - Show abstract - Cite 0.1% topic match

0.1%

22.0

2023

[306] Radiology Gets Chatty: The ChatGPT Saga Unfolds H. Grewal, ..., and R. Kashyap Cureus 2023 - 32 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

51.2

2023

[307] Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions Isaac A. Bernstein, ..., and Sophia Y. Wang JAMA Network Open 2023 - 66 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

152.4

2017

[308] Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study W. Bramer, ..., and O. Franco Systematic Reviews 2017 - 1058 citations - Show abstract - Cite 0.1% topic match

0.1%

77.9

2022

[309] Talking about Large Language Models M. Shanahan Communications of the ACM 2022 - 151 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.7

2023

[310] Special issue on technology assisted review systems G. Nunzio and E. Kanoulas Intell. Syst. Appl. 2023 - 1 citations - Show abstract - Cite 0.1% topic match

0.1%

0.6

2021

[311] Evaluating the relationship between citation set size, team size and screening methods used in systematic reviews: a cross-sectional study Katie O’Hearn, ..., and J. McNally BMC Medical Research Methodology 2021 - 2 citations - Show abstract - Cite 0.1% topic match

0.1%

1.0

2023

[312] Applications of Large Language Models (LLMs) in Breast Cancer Care MD Vera Sorin, ..., and MD Eyal Klang medRxiv 2023 - 1 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

62.2

2023

[313] Ethics of large language models in medicine and medical research. H. Li, ..., and J. Gichoya The Lancet. Digital health 2023 - 101 citations - Show abstract - Cite 0.1% topic match

0.1%

0.0

2024

[314] Evaluation of large language model performance on the Biomedical Language Understanding and Reasoning Benchmark Hui Feng, ..., and Christina Mack medRxiv 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Background: The ability of large language models (LLMs) to interpret and generate human-like text has been accompanied with speculation about their application in medicine and clinical research. There is limited data available to inform evidence-based decisions on the appropriateness for specific use cases. Methods: We evaluated and compared four general-purpose LLMs (GPT-4, GPT-3.5-turbo, Flan-T5-XXL, and Zephyr-7B-Beta) and a healthcare-specific LLM (MedLLaMA-13B) on a set of 13 datasets - referred to as the Biomedical Language Understanding and Reasoning Benchmark (BLURB) - covering six commonly needed medical natural language processing tasks: named entity recognition (NER); relation extraction; population, interventions, comparators, and outcomes (PICO); sentence similarity; document classification; and question-answering. All models were evaluated without modification. Model performance was assessed according to a range of prompting strategies (formalised as a systematic, reusable prompting framework) and relied on the standard, task-specific evaluation metrics defined by BLURB. Results: Across all tasks, GPT-4 outperformed other LLMs, followed by Flan-T5-XXL and GPT-3.5- turbo, then Zephyr-7b-Beta and MedLLaMA-13B. The most performant prompts for GPT-4 and Flan-T5-XXL both outperformed the previously-reported best results for the PubMedQA task. The domain-specific MedLLaMA-13B achieved lower scores for most tasks except for question-answering tasks. We observed a substantial impact of strategically editing the prompt describing the task and a consistent improvement in performance when including examples semantically similar to the input text in the prompt. Conclusion: These results provide evidence of the potential LLMs may have for medical application and highlight the importance of robust evaluation before adopting LLMs for any specific use cases. Continuing to explore how these emerging technologies can be adapted for the healthcare setting, paired with human expertise, and enhanced through quality control measures will be important research to allow responsible innovation with LLMs in the medical area. 0.1% topic match

0.1%

0.8

2023

[315] AI in Education - Evaluating ChatGPT as a Virtual Teaching Assistant Kavita Agarwal International Journal For Multidisciplinary Research 2023 - 1 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

8.9

2017

[316] Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan Hanna Olofsson, ..., and J. Dagerhamn Research Synthesis Methods 2017 - 64 citations - Show abstract - Cite 0.1% topic match

0.1%

16.4

2023

[317] cMolGPT: A Conditional Generative Pre-Trained Transformer for Target-Specific De Novo Molecular Generation Ye Wang, ..., and Wenlu Wang Molecules 2023 - 24 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.0

2024

[318] On the Use of Generative AI for Literature Reviews: An Exploration of Tools and Techniques Peter Mozelius and Niklas Humble European Conference on Research Methodology for Business and Management Studies 2024 - 0 citations - Show abstract - Cite Abstract: To carry out a literature review often involves hard and tedious work. There is a tradition of using facilitating tools, that extended to the AI field in 2018 when iris.ai appeared. Today, in the emerging field of Generative AI tools based on Large Language Models, there has been rapid development of new literature search tools and approaches. This study has the aim of exploring this vast array of Generative AI tools, in a literature study where some of the found tools were used to facilitate the selection of relevant publication. Three research questions guided this study: RQ1) "What Generative AI tools can be found in literature?", RQ2) "Which of these tools could be of use in the literature review conducted in this study, and how?", and RQ3) "What are the ethical aspects of using Generative AI tools in literature studies?” The approach has been a scoping review, built around a search that combined the keywords: "AI supported", "AI generated", "AI based" and "Literature review". An initial result set was filtered with inclusion exclusion criteria in a strive for an interesting quality answer to the research questions. However, most publications that passed the filtering lacked any potential to contribute to answer the research questions. The most interesting finding in the first search was a hint about the new feature 'Scopus AI'. A new search with the Scopus AI tool resulted in a small but very relevant set of publications. These publications were analysed in a deductive inductive thematic analysis, and primary sorted into the categories of: 'Generative AI Tools', 'Supportive AI Techniques', and 'Ethical Issues'. Findings indicate that there is a wide variety of tools that can facilitate the skimming process of a literature, and to provide adequate summaries of retrieved publication. However, authors recommendation is to keep the tools on the facilitating support level, and that the main analysis and conclusion should be human conducted. With this, rather traditional approach, researchers will have clearly less ethical issues to consider. Finally, the ethical aspects of Generative AI tools in research ought to be investigated more in detail, in a separate future study. 0.1% topic match

0.1%

35.3

2019

[319] Open Source Mick Watson Definitions 2019 - 187 citations - Show abstract - Cite 0.1% topic match

0.1%

1.1

2023

[320] Toward Clinical Generative AI: Conceptual Framework N. Bragazzi and Sergio Garbarino JMIR AI 2023 - 1 citations - Show abstract - Cite - PDF Abstract: Clinical decision-making is a crucial aspect of health care, involving the balanced integration of scientific evidence, clinical judgment, ethical considerations, and patient involvement. This process is dynamic and multifaceted, relying on clinicians’ knowledge, experience, and intuitive understanding to achieve optimal patient outcomes through informed, evidence-based choices. The advent of generative artificial intelligence (AI) presents a revolutionary opportunity in clinical decision-making. AI’s advanced data analysis and pattern recognition capabilities can significantly enhance the diagnosis and treatment of diseases, processing vast medical data to identify patterns, tailor treatments, predict disease progression, and aid in proactive patient management. However, the incorporation of AI into clinical decision-making raises concerns regarding the reliability and accuracy of AI-generated insights. To address these concerns, 11 “verification paradigms” are proposed in this paper, with each paradigm being a unique method to verify the evidence-based nature of AI in clinical decision-making. This paper also frames the concept of “clinically explainable, fair, and responsible, clinician-, expert-, and patient-in-the-loop AI.” This model focuses on ensuring AI’s comprehensibility, collaborative nature, and ethical grounding, advocating for AI to serve as an augmentative tool, with its decision-making processes being transparent and understandable to clinicians and patients. The integration of AI should enhance, not replace, the clinician’s judgment and should involve continuous learning and adaptation based on real-world outcomes and ethical and legal compliance. In conclusion, while generative AI holds immense promise in enhancing clinical decision-making, it is essential to ensure that it produces evidence-based, reliable, and impactful knowledge. Using the outlined paradigms and approaches can help the medical and patient communities harness AI’s potential while maintaining high patient care standards. 0.1% topic match

0.1%

0.0

2024

[321] Diagnostic accuracy of GPT‐4 on common clinical scenarios and challenging cases Geoffrey W. Rutledge Learning Health Systems 2024 - 0 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

1922.1

2022

[322] Chain of Thought Prompting Elicits Reasoning in Large Language Models Jason Wei, ..., and Denny Zhou ArXiv 2022 - 5373 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

259.4

2023

[323] ChatGPT listed as author on research papers: many scientists disapprove Chris Stokel-Walker Nature 2023 - 473 citations - Show abstract - Cite 0.1% topic match

0.1%

373.2

2023

[324] Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. Peter Lee, ..., and J. Petro The New England journal of medicine 2023 - 608 citations - Show abstract - Cite 0.1% topic match

0.1%

660.0

2022

[325] Large language models encode clinical knowledge K. Singhal, ..., and Vivek Natarajan Nature 2022 - 1245 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

941.8

2022

[326] Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models Tiffany H. Kung, ..., and Victor Tseng PLOS Digital Health 2022 - 1792 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

54.2

2023

[327] Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot. Arya Rao, ..., and M. Succi Journal of the American College of Radiology : JACR 2023 - 76 citations - Show abstract - Cite 0.1% topic match

0.1%

4.2

2023

[328] ChatGPT as a tool for conducting literature review for dry eye disease Swati Singh and Stephanie Watson Clinical & Experimental Ophthalmology 2023 - 6 citations - Show abstract - Cite 0.1% topic match

0.1%

19.3

2020

[329] Graphical enhancements to summary receiver operating characteristic plots to facilitate the analysis and reporting of meta‐analysis of diagnostic test accuracy data Amit Patel, ..., and A. Sutton Research Synthesis Methods 2020 - 83 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

32.1

2021

[330] PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization Wen Xiao, ..., and Arman Cohan Annual Meeting of the Association for Computational Linguistics 2021 - 99 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

6.4

2023

[331] Evaluation of ChatGPT-generated medical responses: A systematic review and meta-analysis Qiuhong Wei, ..., and Ximing Xu Journal of biomedical informatics 2023 - 7 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

1.5

2023

[332] Comparison of ChatGPT vs. Bard to Anesthesia-related Queries S. S. Patnaik and U. Hoffmann medRxiv 2023 - 2 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

2.5

2017

[333] Ranking Abstracts to Identify Relevant Evidence for Systematic Reviews: The University of Sheffield's Approach to CLEF eHealth 2017 Task 2 Amal H. Alharbi and Mark Stevenson Conference and Labs of the Evaluation Forum 2017 - 18 citations - Show abstract - Cite 0.1% topic match

0.1%

212.8

2023

[334] A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT Ce Zhou, ..., and Salesforce Research ArXiv 2023 - 370 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

554.4

2023

[335] A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity Yejin Bang, ..., and Pascale Fung ArXiv 2023 - 979 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.0

2023

[336] Evaluating Large Language Models for Automated Reporting and Data Systems Categorization: Cross-Sectional Study Qingxia Wu, ..., and Meiyun Wang JMIR Medical Informatics 2023 - 0 citations - Show abstract - Cite - PDF Abstract: Background Large language models show promise for improving radiology workflows, but their performance on structured radiological tasks such as Reporting and Data Systems (RADS) categorization remains unexplored. Objective This study aims to evaluate 3 large language model chatbots—Claude-2, GPT-3.5, and GPT-4—on assigning RADS categories to radiology reports and assess the impact of different prompting strategies. Methods This cross-sectional study compared 3 chatbots using 30 radiology reports (10 per RADS criteria), using a 3-level prompting strategy: zero-shot, few-shot, and guideline PDF-informed prompts. The cases were grounded in Liver Imaging Reporting & Data System (LI-RADS) version 2018, Lung CT (computed tomography) Screening Reporting & Data System (Lung-RADS) version 2022, and Ovarian-Adnexal Reporting & Data System (O-RADS) magnetic resonance imaging, meticulously prepared by board-certified radiologists. Each report underwent 6 assessments. Two blinded reviewers assessed the chatbots’ response at patient-level RADS categorization and overall ratings. The agreement across repetitions was assessed using Fleiss κ. Results Claude-2 achieved the highest accuracy in overall ratings with few-shot prompts and guideline PDFs (prompt-2), attaining 57% (17/30) average accuracy over 6 runs and 50% (15/30) accuracy with k-pass voting. Without prompt engineering, all chatbots performed poorly. The introduction of a structured exemplar prompt (prompt-1) increased the accuracy of overall ratings for all chatbots. Providing prompt-2 further improved Claude-2’s performance, an enhancement not replicated by GPT-4. The interrun agreement was substantial for Claude-2 (k=0.66 for overall rating and k=0.69 for RADS categorization), fair for GPT-4 (k=0.39 for both), and fair for GPT-3.5 (k=0.21 for overall rating and k=0.39 for RADS categorization). All chatbots showed significantly higher accuracy with LI-RADS version 2018 than with Lung-RADS version 2022 and O-RADS (P<.05); with prompt-2, Claude-2 achieved the highest overall rating accuracy of 75% (45/60) in LI-RADS version 2018. Conclusions When equipped with structured prompts and guideline PDFs, Claude-2 demonstrated potential in assigning RADS categories to radiology cases according to established criteria such as LI-RADS version 2018. However, the current generation of chatbots lags in accurately categorizing cases based on more recent RADS criteria. 0.1% topic match

0.1%

117.8

2023

[337] Large Language Models are Zero-Shot Rankers for Recommender Systems Yupeng Hou, ..., and Wayne Xin Zhao European Conference on Information Retrieval 2023 - 177 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

45.6

2021

[338] The Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock 2020 (J-SSCG 2020) M. Egi, ..., and O. Nishida Journal of Intensive Care 2021 - 147 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

3.1

2023

[339] Text Dialogue Analysis for Primary Screening of Mild Cognitive Impairment: Development and Validation Study Changyu Wang, ..., and Jialin Liu Journal of Medical Internet Research 2023 - 4 citations - Show abstract - Cite - PDF Abstract: Background Artificial intelligence models tailored to diagnose cognitive impairment have shown excellent results. However, it is unclear whether large linguistic models can rival specialized models by text alone. Objective In this study, we explored the performance of ChatGPT for primary screening of mild cognitive impairment (MCI) and standardized the design steps and components of the prompts. Methods We gathered a total of 174 participants from the DementiaBank screening and classified 70% of them into the training set and 30% of them into the test set. Only text dialogues were kept. Sentences were cleaned using a macro code, followed by a manual check. The prompt consisted of 5 main parts, including character setting, scoring system setting, indicator setting, output setting, and explanatory information setting. Three dimensions of variables from published studies were included: vocabulary (ie, word frequency and word ratio, phrase frequency and phrase ratio, and lexical complexity), syntax and grammar (ie, syntactic complexity and grammatical components), and semantics (ie, semantic density and semantic coherence). We used R 4.3.0. for the analysis of variables and diagnostic indicators. Results Three additional indicators related to the severity of MCI were incorporated into the final prompt for the model. These indicators were effective in discriminating between MCI and cognitively normal participants: tip-of-the-tongue phenomenon (P<.001), difficulty with complex ideas (P<.001), and memory issues (P<.001). The final GPT-4 model achieved a sensitivity of 0.8636, a specificity of 0.9487, and an area under the curve of 0.9062 on the training set; on the test set, the sensitivity, specificity, and area under the curve reached 0.7727, 0.8333, and 0.8030, respectively. Conclusions ChatGPT was effective in the primary screening of participants with possible MCI. Improved standardization of prompts by clinicians would also improve the performance of the model. It is important to note that ChatGPT is not a substitute for a clinician making a diagnosis. 0.1% topic match

0.1%

11.4

2023

[340] Trends in the incidence of pulmonary nodules in chest computed tomography: 10-year results from two Dutch hospitals Ward Hendrix, ..., and C. Jacobs European Radiology 2023 - 16 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

28.9

2022

[341] Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature Tomas Goldsack, ..., and Carolina Scarton ArXiv 2022 - 60 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.0

2024

[342] Systematic review and meta-analysis of preclinical studies B. Ineichen, ..., and Kimberley E. Wever Nature Reviews Methods Primers 2024 - 0 citations - Show abstract - Cite 0.1% topic match

0.1%

0.0

2024

[343] Exploring ChatGPT's Ability to Classify the Structure of Literature Reviews in Engineering Research Articles Maha Issa, ..., and Niveen AbiGhannam IEEE Transactions on Learning Technologies 2024 - 0 citations - Show abstract - Cite 0.1% topic match

0.1%

97.8

2023

[344] ChatGPT: Is this version good for healthcare and research? Raju Vaishya, ..., and A. Vaish Diabetes & metabolic syndrome 2023 - 167 citations - Show abstract - Cite 0.1% topic match

0.1%

110.1

2009

[345] Reconstructing the giant: On the importance of rigour in documenting the literature search process Jan vom Brocke, ..., and Anne Cleven European Conference on Information Systems 2009 - 1700 citations - Show abstract - Cite 0.1% topic match

0.1%

27.3

2019

[346] Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses J. Polanin, ..., and Jennifer K Grotpeter Research Synthesis Methods 2019 - 147 citations - Show abstract - Cite - PDF 0.1% topic match

0.1%

0.0

2024

[347] Large Language Models in Pathology: A Comparative Study on Multiple Choice QuestionPerformance with Pathology Trainees MD Wei Du, ..., and MD PhD Shunsuke Koga medRxiv 2024 - 0 citations - Show abstract - Cite 0.1% topic match

0.1%

1.7

2017

[348] LIMSI@CLEF eHealth 2017 Task 2: Logistic Regression for Automatic Article Ranking Christopher Norman, ..., and Aurélie Névéol Conference and Labs of the Evaluation Forum 2017 - 13 citations - Show abstract - Cite 0.1% topic match

0.1%

7.4

2024

[349] Evaluating GPT-V4 (GPT-4 with Vision) on Detection of Radiologic Findings on Chest Radiographs. Yiliang Zhou, ..., and Yifan Peng Radiology 2024 - 4 citations - Show abstract - Cite Abstract: Background Generating radiologic findings from chest radiographs is pivotal in medical image analysis. The emergence of OpenAI's generative pretrained transformer, GPT-4 with vision (GPT-4V), has opened new perspectives on the potential for automated image-text pair generation. However, the application of GPT-4V to real-world chest radiography is yet to be thoroughly examined. Purpose To investigate the capability of GPT-4V to generate radiologic findings from real-world chest radiographs. Materials and Methods In this retrospective study, 100 chest radiographs with free-text radiology reports were annotated by a cohort of radiologists, two attending physicians and three residents, to establish a reference standard. Of 100 chest radiographs, 50 were randomly selected from the National Institutes of Health (NIH) chest radiographic data set, and 50 were randomly selected from the Medical Imaging and Data Resource Center (MIDRC). The performance of GPT-4V at detecting imaging findings from each chest radiograph was assessed in the zero-shot setting (where it operates without prior examples) and few-shot setting (where it operates with two examples). Its outcomes were compared with the reference standard with regards to clinical conditions and their corresponding codes in the International Statistical Classification of Diseases, Tenth Revision (ICD-10), including the anatomic location (hereafter, laterality). Results In the zero-shot setting, in the task of detecting ICD-10 codes alone, GPT-4V attained an average positive predictive value (PPV) of 12.3%, average true-positive rate (TPR) of 5.8%, and average F1 score of 7.3% on the NIH data set, and an average PPV of 25.0%, average TPR of 16.8%, and average F1 score of 18.2% on the MIDRC data set. When both the ICD-10 codes and their corresponding laterality were considered, GPT-4V produced an average PPV of 7.8%, average TPR of 3.5%, and average F1 score of 4.5% on the NIH data set, and an average PPV of 10.9%, average TPR of 4.9%, and average F1 score of 6.4% on the MIDRC data set. With few-shot learning, GPT-4V showed improved performance on both data sets. When contrasting zero-shot and few-shot learning, there were improved average TPRs and F1 scores in the few-shot setting, but there was not a substantial increase in the average PPV. Conclusion Although GPT-4V has shown promise in understanding natural images, it had limited effectiveness in interpreting real-world chest radiographs. © RSNA, 2024 Supplemental material is available for this article. 0.1% topic match

0.1%

23.0

2023

[350] Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models Emily Alsentzer, ..., and V. Kovacheva NPJ Digital Medicine 2023 - 22 citations - Show abstract - Cite - PDF 0.1% topic match

0.0%

0.0

2024

[351] EXPLORING THE POTENTIAL AND LIMITATIONS OF CHATGPT IN ACADEMIC WRITING AND EDITORIAL TASKS Akın Akpur Fırat Üniversitesi Sosyal Bilimler Dergisi 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

32.2

2018

[352] Covidence and Rayyan Liz Kellermeyer, ..., and S. Knight Journal of the Medical Library Association : JMLA 2018 - 197 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.0

2020

[353] A Computational Approach for Objectively Derived Systematic Review Search Strategies Harrisen Scells, ..., and Justin Clark Advances in Information Retrieval 2020 - 14 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.1

2017

[354] IIIT-H at CLEF eHealth 2017 Task 2: Technologically Assisted Reviews in Empirical Medicine Jaspreet Singh and Lini Thomas Conference and Labs of the Evaluation Forum 2017 - 9 citations - Show abstract - Cite 0.0% topic match

0.0%

11.1

2006

[355] Selecting Studies for Systemic Review: Inclusion and Exclusion Criteria T. Meline https://doi.org/10.1044/CICSD_33_S_21 2006 - 208 citations - Show abstract - Cite 0.0% topic match

0.0%

13.5

2022

[356] Abstract screening using the automated tool Rayyan: results of effectiveness in three diagnostic test accuracy systematic reviews A. Valizadeh, ..., and S. Faghani BMC Medical Research Methodology 2022 - 33 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

25.7

2023

[357] A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations Qingyu Chen, ..., and Huan Xu Journal Not Provided 2023 - 39 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.4

2024

[358] Association of reviewer experience with discriminating human-written versus ChatGPT-written abstracts Gabriel Levin, ..., and Pedro T. Ramirez International Journal of Gynecological Cancer 2024 - 2 citations - Show abstract - Cite Abstract: Objective To determine if reviewer experience impacts the ability to discriminate between human-written and ChatGPT-written abstracts. Methods Thirty reviewers (10 seniors, 10 juniors, and 10 residents) were asked to differentiate between 10 ChatGPT-written and 10 human-written (fabricated) abstracts. For the study, 10 gynecologic oncology abstracts were fabricated by the authors. For each human-written abstract we generated a ChatGPT matching abstract by using the same title and the fabricated results of each of the human generated abstracts. A web-based questionnaire was used to gather demographic data and to record the reviewers’ evaluation of the 20 abstracts. Comparative statistics and multivariable regression were used to identify factors associated with a higher correct identification rate. Results The 30 reviewers discriminated 20 abstracts, giving a total of 600 abstract evaluations. The reviewers were able to correctly identify 300/600 (50%) of the abstracts: 139/300 (46.3%) of the ChatGPT-generated abstracts and 161/300 (53.7%) of the human-written abstracts (p=0.07). Human-written abstracts had a higher rate of correct identification (median (IQR) 56.7% (49.2–64.1%) vs 45.0% (43.2–48.3%), p=0.023). Senior reviewers had a higher correct identification rate (60%) than junior reviewers and residents (45% each; p=0.043 and p=0.002, respectively). In a linear regression model including the experience level of the reviewers, familiarity with artificial intelligence (AI) and the country in which the majority of medical training was achieved (English speaking vs non-English speaking), the experience of the reviewer (β=10.2 (95% CI 1.8 to 18.7)) and familiarity with AI (β=7.78 (95% CI 0.6 to 15.0)) were independently associated with the correct identification rate (p=0.019 and p=0.035, respectively). In a correlation analysis the number of publications by the reviewer was positively correlated with the correct identification rate (r28)=0.61, p<0.001. Conclusion A total of 46.3% of abstracts written by ChatGPT were detected by reviewers. The correct identification rate increased with reviewer and publication experience. 0.0% topic match

0.0%

4818.3

2023

[359] LLaMA: Open and Efficient Foundation Language Models Hugo Touvron, ..., and Guillaume Lample ArXiv 2023 - 8258 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.2

2023

[360] Paper 4: Search methods and approaches for living guidelines S. McDonald, ..., and David Fraile Navarro Journal of clinical epidemiology 2023 - 6 citations - Show abstract - Cite 0.0% topic match

0.0%

41.1

2022

[361] BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model Hongyi Yuan, ..., and Sheng Yu ArXiv 2022 - 107 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[362] Analysis of the orthopaedic trauma literature utilizing machine learning and latent dirichlet allocation MD M. Andrew Rowley Jr, ..., and MD Langdon A. Hartsock Current Orthopaedic Practice 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

47.3

2023

[363] Investigating the use of ChatGPT for the scheduling of construction projects S. Prieto, ..., and Borja García de Soto ArXiv 2023 - 85 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

45.0

2020

[364] How to complete a full systematic review in 2 weeks: processes, facilitators and barriers. J. Clark, ..., and A. Scott Journal of clinical epidemiology 2020 - 216 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

30.7

2023

[365] Using ChatGPT for human–computer interaction research: a primer Wilbert Tabone and Joost C. F. de Winter Royal Society Open Science 2023 - 37 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2023

[366] The Pulse of Artificial Intelligence in Cardiology: A Comprehensive Evaluation of State-of-the-art Large Language Models for Potential Use in Clinical Cardiology Andrej Novak, ..., and Š. Manola medRxiv 2023 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[367] Research on confusing responses based on ChatGPT Huiwen Xiong Applied and Computational Engineering 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

4.8

2022

[368] Automatic Extraction of Social Determinants of Health from Medical Notes of Chronic Lower Back Pain Patients D. Lituiev, ..., and Thomas A. Peterson Journal of the American Medical Informatics Association : JAMIA 2022 - 13 citations - Show abstract - Cite - PDF Abstract: Background. Adverse social determinants of health (SDoH), or social risk factors, such as food insecurity and housing instability, are known to contribute to poor health outcomes and inequities. Our ability to study these linkages is limited because SDoH information is more frequently documented in free-text clinical notes than structured data fields. To overcome this challenge, there is a growing push to develop techniques for automated extraction of SDoH. In this study, we explored natural language processing (NLP) and inference (NLI) methods to extract SDoH information from clinical notes of patients with chronic low back pain (cLBP), to enhance future analyses of the associations between SDoH and low back pain outcomes and disparities. Methods. Clinical notes (n=1,576) for patients with cLBP (n=386) were annotated for seven SDoH domains: housing, food, transportation, finances, insurance coverage, marital and partnership status, and other social support, resulting in 626 notes with at least one annotated entity for 364 patients. We additionally labelled pain scores, depression, and anxiety. We used a two-tier taxonomy with these 10 first-level ontological classes and 68 second-level ontological classes. We developed and validated extraction systems based on both rule-based and machine learning approaches. As a rule-based approach, we iteratively configured a clinical Text Analysis and Knowledge Extraction System (cTAKES) system. We trained two machine learning models (based on convolutional neural network (CNN) and RoBERTa transformer), and a hybrid system combining pattern matching and bag-of-words models. Additionally, we evaluated a RoBERTa based entailment model as an alternative technique of SDoH detection in clinical texts. We used a model previously trained on general domain data without additional training on our dataset. Results. Four annotators achieved high agreement (average kappa=95%, F1=91.20%). Annotation frequency varied significantly dependent on note type. By tuning cTAKES, we achieved a performance of F1=47.11% for first-level classes. For most classes, the machine learning RoBERTa-based NER model performed better (first-level F1=84.35%) than other models within the internal test dataset. The hybrid system on average performed slightly worse than the RoBERTa NER model (first-level F1=80.27%), matching or outperforming the former in terms of recall. Using an out-of-the-box entailment model, we detected many but not all challenging wordings missed by other models, reaching an average F1 of 76.04%, while matching and outperforming the tested NER models in several classes. Still, the entailment model may be sensitive to hypothesis wording and may require further fine tuning. Conclusion. This study developed a corpus of annotated clinical notes covering a broad spectrum of SDoH classes. This corpus provides a basis for training machine learning models and serves as a benchmark for predictive models for named entity recognition for SDoH and knowledge extraction from clinical texts. 0.0% topic match

0.0%

3.4

2021

[369] MeSH Term Suggestion for Systematic Review Literature Search Shuai Wang, ..., and G. Zuccon Proceedings of the 25th Australasian Document Computing Symposium 2021 - 10 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

2071.8

2010

[370] Cochrane Handbook for Systematic Reviews of Interventions Julian P T Higgins and Sally Green International Coaching Psychology Review 2010 - 29626 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

31.0

2018

[371] A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature Benjamin E. Nye, ..., and Byron C. Wallace Proceedings of the conference. Association for Computational Linguistics. Meeting 2018 - 199 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

273.2

2024

[372] Gemma: Open Models Based on Gemini Research and Technology Gemma Team Thomas Mesnard, ..., and Kathleen Kenealy ArXiv 2024 - 184 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

58.8

2023

[373] Accuracy and Reliability of Chatbot Responses to Physician Questions Rachel S. Goodman, ..., and Douglas B. Johnson JAMA Network Open 2023 - 66 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

7.1

2023

[374] The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review Álisson Oliveira dos Santos, ..., and V. S. Belo Journal of biomedical informatics 2023 - 11 citations - Show abstract - Cite 0.0% topic match

0.0%

558.2

2023

[375] PaLM 2 Technical Report Rohan Anil, ..., and Yonghui Wu ArXiv 2023 - 836 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

34.9

2023

[376] Using ChatGPT in Medical Research: Current Status and Future Directions Suebsarn Ruksakulpiwat, ..., and Anuoluwapo Ajibade Journal of Multidisciplinary Healthcare 2023 - 51 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[377] How good is ChatGPT? An exploratory study on ChatGPT's performance in engineering design tasks and subjective decision-making Wanyu Xu, ..., and Daniel A. McAdams Proceedings of the Design Society 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

22.0

2010

[378] Semi-automated screening of biomedical citations for systematic reviews Byron C. Wallace, ..., and C. Schmid BMC Bioinformatics 2010 - 326 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

2.8

2017

[379] ECNU at 2017 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine Huaying Wu, ..., and Liang He Conference and Labs of the Evaluation Forum 2017 - 22 citations - Show abstract - Cite 0.0% topic match

0.0%

8.0

2017

[380] An exploration of crowdsourcing citation screening for systematic reviews Michael L. Mortensen, ..., and Byron C. Wallace Research Synthesis Methods 2017 - 59 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

67.5

2013

[381] A systematic review of systematic review process research in software engineering B. Kitchenham and P. Brereton Inf. Softw. Technol. 2013 - 739 citations - Show abstract - Cite 0.0% topic match

0.0%

201.1

2023

[382] Abstracts written by ChatGPT fool scientists Holly Else Nature 2023 - 370 citations - Show abstract - Cite 0.0% topic match

0.0%

0.9

2023

[383] Leveraging GPT-4 for Identifying Cancer Phenotypes in Electronic Health Records: A Performance Comparison between GPT-4, GPT-3.5-turbo, Flan-T5 and spaCy’s Rule-based & Machine Learning-based methods Kriti Bhattarai, ..., and Albert M Lai bioRxiv 2023 - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[384] Comparative performance of artificial ıntelligence models in physical medicine and rehabilitation board-level questions A. K. Menekşeoğlu and E. Is Revista da Associação Médica Brasileira 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

24.0

2023

[385] ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing Ryan Liu and Nihar B. Shah ArXiv 2023 - 35 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

365.3

2005

[386] Understanding interobserver agreement: the kappa statistic. A. Viera and J. Garrett Family medicine 2005 - 7138 citations - Show abstract - Cite 0.0% topic match

0.0%

6.0

2022

[387] Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Search Shuai Wang, ..., and G. Zuccon Intell. Syst. Appl. 2022 - 13 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

74.3

2022

[388] MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering Ankit Pal, ..., and Malaikannan Sankarasubbu ACM Conference on Health, Inference, and Learning 2022 - 196 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

13.5

2023

[389] Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records Gerbrich Ferdinands, ..., and R. van de Schoot Systematic Reviews 2023 - 19 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.3

2024

[390] Advancing Medical Practice with Artificial Intelligence: ChatGPT in Healthcare. Idit Tessler, ..., and Eyal Klang The Israel Medical Association journal : IMAJ 2024 - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[391] ChatGPT, a Friend or a Foe in Medical Education: A Review of Strengths, Challenges, and Opportunities Mahdi Zarei, ..., and Mohammad-Salar Hosseini Shiraz E-Medical Journal 2024 - 0 citations - Show abstract - Cite - PDF Abstract: Background: ChatGPT is a large-scale language model that utilizes artificial intelligence (AI) to answer a broad range of scientific inquiries, create clinical scenarios, and evaluate educational programs. While its use offers numerous advantages, it also presents several challenges. Objectives: The aim of this study is to conduct a comprehensive review of ChatGPT's functionality and explore the benefits, challenges, existing solutions, and future prospects of using AI in medical education. Method: A comprehensive literature review was conducted using various databases such as PubMed, Scopus, Web of Science, and Google Scholar. The search phrases used were ChatGPT, Artificial Intelligence, Chatbot, Medical Education, and large language models (LLMs). Results: The application of ChatGPT in medical education offers several advantages, such as enhanced quality of interaction between medical students and patients, improved education quality, enhanced research opportunities, personalized learning, virtual patient simulations, and cost-effectiveness. However, there are also critical challenges, such as ethical and transparency concerns, limited access to reliable databases, restricted information availability after 2021, limited development of students' critical thinking ability, and the risk of generating hallucinations. Conclusions: Artificial intelligence models have become a popular tool for researchers to access scientific resources, comprehend articles and textbooks, and create scientific texts. ChatGPT has been used extensively in medicine and medical education in a short period. It is essential to weigh the benefits and challenges, use expert supervision, conduct frequent assessments, and provide feedback reviews to guarantee its efficacy. Although this technology should not replace human labor, it is essential to prepare for the changes brought by AI and create appropriate guidelines and curricula by reviewing existing solutions and conducting extensive studies. 0.0% topic match

0.0%

21.5

2015

[392] RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials I. Marshall, ..., and Byron C. Wallace Journal of the American Medical Informatics Association : JAMIA 2015 - 202 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

13.0

2023

[393] Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training Alfredo Madrid-García, ..., and L. Rodríguez-Rodríguez Scientific Reports 2023 - 17 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.2

2015

[394] Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approaches A. O'Mara-Eves, ..., and S. Ananiadou Systematic Reviews 2015 - 31 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[395] Leveraging AI to improve evidence synthesis in conservation. O. Berger‐Tal, ..., and S. Nakagawa Trends in ecology & evolution 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

10.7

2023

[396] Navigating the Landscape of Personalized Medicine: The Relevance of ChatGPT, BingChat, and Bard AI in Nephrology Literature Searches Noppawit Aiumtrakul, ..., and W. Cheungpasitporn Journal of Personalized Medicine 2023 - 12 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

83.0

2023

[397] Adapted large language models can outperform medical experts in clinical text summarization. Dave Van Veen, ..., and Akshay S. Chaudhari Nature medicine 2023 - 97 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

4.5

2023

[398] Clinical Profile of Patients with Idiopathic Pulmonary Fibrosis in Real Life D. Morena, ..., and J. Izquierdo Journal of Clinical Medicine 2023 - 8 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

110.4

2023

[399] Prompt Engineering with ChatGPT: A Guide for Academic Writers L. Giray Annals of Biomedical Engineering 2023 - 159 citations - Show abstract - Cite 0.0% topic match

0.0%

1167.3

2022

[400] Scaling Instruction-Finetuned Language Models Hyung Won Chung, ..., and Jason Wei ArXiv 2022 - 2416 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

6.3

2020

[401] Automatic Boolean Query Formulation for Systematic Review Literature Search Harrisen Scells, ..., and Justin Clark Proceedings of The Web Conference 2020 2020 - 29 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2023

[402] The Case Records of ChatGPT: Language Models and Complex Clinical Questions T. Poterucha, ..., and C. Haggerty ArXiv 2023 - 0 citations - Show abstract - Cite - PDF Abstract: Background: Artificial intelligence language models have shown promise in various applications, including assisting with clinical decision-making as demonstrated by strong performance of large language models on medical licensure exams. However, their ability to solve complex, open-ended cases, which may be representative of clinical practice, remains unexplored. Methods: In this study, the accuracy of large language AI models GPT4 and GPT3.5 in diagnosing complex clinical cases was investigated using published Case Records of the Massachusetts General Hospital. A total of 50 cases requiring a diagnosis and diagnostic test published from January 1, 2022 to April 16, 2022 were identified. For each case, models were given a prompt requesting the top three specific diagnoses and associated diagnostic tests, followed by case text, labs, and figure legends. Model outputs were assessed in comparison to the final clinical diagnosis and whether the model-predicted test would result in a correct diagnosis. Results: GPT4 and GPT3.5 accurately provided the correct diagnosis in 26% and 22% of cases in one attempt, and 46% and 42% within three attempts, respectively. GPT4 and GPT3.5 provided a correct essential diagnostic test in 28% and 24% of cases in one attempt, and 44% and 50% within three attempts, respectively. No significant differences were found between the two models, and multiple trials with identical prompts using the GPT3.5 model provided similar results. Conclusions: In summary, these models demonstrate potential usefulness in generating differential diagnoses but remain limited in their ability to provide a single unifying diagnosis in complex, open-ended cases. Future research should focus on evaluating model performance in larger datasets of open-ended clinical challenges and exploring potential human-AI collaboration strategies to enhance clinical decision-making. 0.0% topic match

0.0%

8.3

2011

[403] Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review Maura R. Grossman and G. Cormack Richmond Journal of Law and Technology 2011 - 115 citations - Show abstract - Cite 0.0% topic match

0.0%

1.9

2013

[404] ‘Clustering’ documents automatically to support scoping reviews of research: a case study C. Stansfield, ..., and J. Kavanagh Research Synthesis Methods 2013 - 21 citations - Show abstract - Cite 0.0% topic match

0.0%

86.2

2021

[405] How to properly use the PRISMA Statement R. Sarkis-Onofre, ..., and C. Lockwood Systematic Reviews 2021 - 308 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

19.0

2001

[406] Bayesian Approaches to Modeling the Conditional Dependence Between Multiple Diagnostic Tests N. Dendukuri and L. Joseph Biometrics 2001 - 451 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

11.9

2023

[407] MedKPL: A heterogeneous knowledge enhanced prompt learning framework for transferable diagnosis Yuxing Lu, ..., and Guangyu Wang Journal of biomedical informatics 2023 - 17 citations - Show abstract - Cite 0.0% topic match

0.0%

16.4

2022

[408] Improving Chest X-Ray Report Generation by Leveraging Warm-Starting Aaron Nicolson, ..., and B. Koopman Artificial intelligence in medicine 2022 - 46 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[409] Can large language model replace static analysis tools Han Cui https://doi.org/10.1117/12.3031920 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

41.4

2022

[410] Human heuristics for AI-generated language are flawed Maurice Jakesch, ..., and Mor Naaman Proceedings of the National Academy of Sciences of the United States of America 2022 - 100 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

49.0

2023

[411] Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer. M. Fink, ..., and T. Weber Radiology 2023 - 59 citations - Show abstract - Cite Abstract: Background The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. Purpose To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. Materials and Methods This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. Results On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). Conclusion When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Hafezi-Nejad and Trivedi in this issue. 0.0% topic match

0.0%

2.4

2024

[412] Underneath the Numbers: Quantitative and Qualitative Gender Fairness in LLMs for Depression Prediction Micol Spitale, ..., and Hatice Gunes ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2023

[413] Comprehensive analysis of the performance of GPT-3.5 and GPT-4 on the American Urological Association self-assessment study program exams from 2012-2023. Ali Sherazi and David Canes Canadian Urological Association journal = Journal de l'Association des urologues du Canada 2023 - 0 citations - Show abstract - Cite Abstract: INTRODUCTION Artificial intelligence (AI) applications, specifically generative pre-trained transformers, have shown potential in medical education and board-style examinations. To assess this capability, we conducted a study comparing the performance of GPT-3.5 and GPT-4 on the American Urological Association (AUA) 2022 self-assessment study program (SASP) exams from 2012-2023. METHODS We used a standardized prompt to administer questions from the AUA SASP exams spanning 2012-2023, totalling 1679 questions. The performance of the two AI models, GPT-3.5 and GPT-4, was evaluated based on the number of questions answered correctly. Statistical analysis was performed using Fisher's exact test and independent sample t-tests to compare the performance of GPT-4 to that of GPT-3.5 among test years and urology topic areas. Percentile scores were not calculable, however, a score of 50% is required to acquire CME credits on AUA SASP exams. RESULTS The analysis showed significantly superior performance by GPT-4, which scored above 50% across all exam years except 2018, with scores ranging from 48-64%. In contrast, GPT-3.5 consistently scored below this threshold, with scores ranging from 26-38%. The total combined score for GPT-4 was 55%, significantly higher than the 33% achieved by GPT-3.5 (odds ratio [OR] 2.5, 95% confidence interval [CI] 2.2-2.9, p<0.001). GPT-4 significantly outperformed GPT-3.5 among AUA SASP test years from 2012-2023 (mean difference 23, t(22) 14, 95% CI 19-26, p<0.001), as well as among urology topic areas (mean difference 21, t(52)=5.5, 95% CI 13-29, p<0.001). CONCLUSIONS GPT-4 scored significantly higher than GPT-3.5 on the AUA SASP exams in overall performance, across all test years, and in various urology topic areas. This suggests improvement in evolving AI language models in answering clinical urology questions; however, certain aspects of medical knowledge and clinical reasoning remain challenging for AI language models. 0.0% topic match

0.0%

67.3

2023

[414] How AI Responds to Common Lung Cancer Questions: ChatGPT vs Google Bard. A. Rahsepar, ..., and Arash Bedayat Radiology 2023 - 98 citations - Show abstract - Cite Abstract: Background The recent release of large language models (LLMs) for public use, such as ChatGPT and Google Bard, has opened up a multitude of potential benefits as well as challenges. Purpose To evaluate and compare the accuracy and consistency of responses generated by publicly available ChatGPT-3.5 and Google Bard to non-expert questions related to lung cancer prevention, screening, and terminology commonly used in radiology reports based on the recommendation of Lung Imaging Reporting and Data System (Lung-RADS) v2022 from American College of Radiology and Fleischner society. Materials and Methods Forty of the exact same questions were created and presented to ChatGPT-3.5 and Google Bard experimental version as well as Bing and Google search engines by three different authors of this paper. Each answer was reviewed by two radiologists for accuracy. Responses were scored as correct, partially correct, incorrect, or unanswered. Consistency was also evaluated among the answers. Here, consistency was defined as the agreement between the three answers provided by ChatGPT-3.5, Google Bard experimental version, Bing, and Google search engines regardless of whether the concept conveyed was correct or incorrect. The accuracy among different tools were evaluated using Stata. Results ChatGPT-3.5 answered 120 questions with 85 (70.8%) correct, 14 (11.7%) partially correct, and 21 (17.5%) incorrect. Google Bard did not answer 23 (19.1%) questions. Among the 97 questions answered by Google Bard, 62 (51.7%) were correct, 11 (9.2%) were partially correct, and 24 (20%) were incorrect. Bing answered 120 questions with 74 (61.7%) correct, 13 (10.8%) partially correct, and 33 (27.5%) incorrect. Google search engine answered 120 questions with 66 (55%) correct, 27 (22.5%) partially correct, and 27 (22.5%) incorrect. The ChatGPT-3.5 is more likely to provide correct or partially answer than Google Bard, approximately by 1.5 folds (OR = 1.55, P = 0.004). ChatGPT-3.5 and Google search engine were more likely to be consistent than Google Bard by approximately 7 and 29 folds (OR = 6.65, P = 0.002 for ChatGPT and OR = 28.83, P = 0.002 for Google search engine, respectively). Conclusion Although ChatGPT-3.5 had a higher accuracy in comparison with the other tools, neither ChatGPT nor Google Bard, Bing and Google search engines answered all questions correctly and with 100% consistency. 0.0% topic match

0.0%

1004.7

2023

[415] A Survey of Large Language Models Wayne Xin Zhao, ..., and Ji-rong Wen ArXiv 2023 - 1634 citations - Show abstract - Cite - PDF Abstract: Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions. 0.0% topic match

0.0%

7.2

2022

[416] Artificial Intelligence–Enabled Social Media Analysis for Pharmacovigilance of COVID-19 Vaccinations in the United Kingdom: Observational Study Zain U. Hussain, ..., and Amir Hussain JMIR Public Health and Surveillance 2022 - 20 citations - Show abstract - Cite - PDF Abstract: Background The rollout of vaccines for COVID-19 in the United Kingdom started in December 2020. Uptake has been high, and there has been a subsequent reduction in infections, hospitalizations, and deaths among vaccinated individuals. However, vaccine hesitancy remains a concern, in particular relating to adverse effects following immunization (AEFIs). Social media analysis has the potential to inform policy makers about AEFIs being discussed by the public as well as public attitudes toward the national immunization campaign. Objective We sought to assess the frequency and nature of AEFI-related mentions on social media in the United Kingdom and to provide insights on public sentiments toward COVID-19 vaccines. Methods We extracted and analyzed over 121,406 relevant Twitter and Facebook posts, from December 8, 2020, to April 30, 2021. These were thematically filtered using a 2-step approach, initially using COVID-19–related keywords and then using vaccine- and manufacturer-related keywords. We identified AEFI-related keywords and modeled their word frequency to monitor their trends over 2-week periods. We also adapted and utilized our recently developed hybrid ensemble model, which combines state-of-the-art lexicon rule–based and deep learning–based approaches, to analyze sentiment trends relating to the main vaccines available in the United Kingdom. Results Our COVID-19 AEFI search strategy identified 46,762 unique Facebook posts by 14,346 users and 74,644 tweets (excluding retweets) by 36,446 users over the 4-month period. We identified an increasing trend in the number of mentions for each AEFI on social media over the study period. The most frequent AEFI mentions were found to be symptoms related to appetite (n=79,132, 14%), allergy (n=53,924, 9%), injection site (n=56,152, 10%), and clots (n=43,907, 8%). We also found some rarely reported AEFIs such as Bell palsy (n=11,909, 2%) and Guillain-Barre syndrome (n=9576, 2%) being discussed as frequently as more well-known side effects like headache (n=10,641, 2%), fever (n=12,707, 2%), and diarrhea (n=16,559, 3%). Overall, we found public sentiment toward vaccines and their manufacturers to be largely positive (58%), with a near equal split between negative (22%) and neutral (19%) sentiments. The sentiment trend was relatively steady over time and had minor variations, likely based on political and regulatory announcements and debates. Conclusions The most frequently discussed COVID-19 AEFIs on social media were found to be broadly consistent with those reported in the literature and by government pharmacovigilance. We also detected potential safety signals from our analysis that have been detected elsewhere and are currently being investigated. As such, we believe our findings support the use of social media analysis to provide a complementary data source to conventional knowledge sources being used for pharmacovigilance purposes. 0.0% topic match

0.0%

0.0

2024

[417] Text Origin Detection: Unmasking the Source – AI vs Human Vishesh Pandita, ..., and Priyadarshini Patil 2024 5th International Conference for Emerging Technology (INCET) 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

6.5

2024

[418] Large language models overcome the challenges of unstructured text data in ecology Andry Castro, ..., and César Capinha bioRxiv 2024 - 2 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

2.6

2020

[419] You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews Harrisen Scells, ..., and B. Koopman Advances in Information Retrieval 2020 - 12 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

14.7

2020

[420] An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes C. Hamel, ..., and B. Hutton BMC Medical Research Methodology 2020 - 60 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[421] Impact of Large Language Models on Medical Education and Teaching Adaptations Li Zhui, ..., and Ren Wei JMIR Medical Informatics 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

371.4

2023

[422] Artificial Hallucinations in ChatGPT: Implications in Scientific Writing H. Alkaissi and Samy I McFarlane Cureus 2023 - 663 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[423] Rationale Dataset and Analysis for the Commit Messages of the Linux Kernel Out-of-Memory Killer Mouna Dhaouadi, ..., and Michalis Famelis 2024 IEEE/ACM 32nd International Conference on Program Comprehension (ICPC) 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.4

2017

[424] Combining Inter-Review Learning-to-Rank and Intra-Review Incremental Training for Title and Abstract Screening in Systematic Reviews A. Anagnostou, ..., and I. Vlahavas Conference and Labs of the Evaluation Forum 2017 - 11 citations - Show abstract - Cite 0.0% topic match

0.0%

1.1

2023

[425] GPT-4 and Neurologists in Screening for Mild Cognitive Impairment in the Elderly: A Comparative Analysis Study Hao Yang, ..., and Siru Liu medRxiv 2023 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

59.1

2023

[426] Overview of Early ChatGPT’s Presence in Medical Literature: Insights From a Hybrid Literature Review by ChatGPT and Human Experts Omar Temsah, ..., and A. Al-Eyadhy Cureus 2023 - 96 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

13.9

2015

[427] Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module J. Rathbone, ..., and P. Glasziou Systematic Reviews 2015 - 137 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.5

2011

[428] How can we find relevant research more quickly James Thomas and A. O'Mara-Eves Journal Not Provided 2011 - 7 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[429] End-To-End Clinical Trial Matching with Large Language Models Dyke Ferber, ..., and J. N. Kather Journal Not Provided 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.9

1958

[430] Science and religion in seventeenth-century England R. Westfall https://doi.org/10.2307/1845472 1958 - 129 citations - Show abstract - Cite 0.0% topic match

0.0%

157.8

2023

[431] ChatGPT for (Finance) Research: The Bananarama Conjecture M. Dowling and B. Lucey SSRN Electronic Journal 2023 - 295 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

45.6

2010

[432] Expediting systematic reviews: methods and implications of rapid reviews Rebecca Ganann, ..., and H. Thomas Implementation Science : IS 2010 - 653 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.9

2023

[433] ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about Aman Rangapur and Haoran Wang ArXiv 2023 - 3 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

39.8

2012

[434] Deploying an interactive machine learning system in an evidence-based practice center: abstrackr Byron C. Wallace, ..., and T. Trikalinos International Health Informatics Symposium 2012 - 509 citations - Show abstract - Cite 0.0% topic match

0.0%

10.4

2019

[435] CLEF 2019 Technology Assisted Reviews in Empirical Medicine Overview E. Kanoulas, ..., and René Spijker Conference and Labs of the Evaluation Forum 2019 - 61 citations - Show abstract - Cite 0.0% topic match

0.0%

18.7

2020

[436] Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. G. Gartlehner, ..., and F. König Journal of clinical epidemiology 2020 - 90 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.8

2011

[437] Exploiting the systematic review protocol for classification of medical abstracts Oana Frunza, ..., and P. O'Blenis Artificial intelligence in medicine 2011 - 53 citations - Show abstract - Cite 0.0% topic match

0.0%

16.8

1980

[438] Estimating the error rates of diagnostic tests. S. Hui and S. Walter Biometrics 1980 - 753 citations - Show abstract - Cite 0.0% topic match

0.0%

152.1

2023

[439] ChatGPT in education: Strategies for responsible implementation Mohanad Halaweh Contemporary Educational Technology 2023 - 247 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

6.9

2020

[440] A comparison of automatic Boolean query formulation for systematic reviews Harrisen Scells, ..., and B. Koopman Information Retrieval Journal 2020 - 28 citations - Show abstract - Cite 0.0% topic match

0.0%

1.8

2014

[441] Study of bacterial meningitis in children below 5 years with comparative evaluation of gram staining, culture and bacterial antigen detection. Kala Yadhav Ml Journal of clinical and diagnostic research : JCDR 2014 - 19 citations - Show abstract - Cite 0.0% topic match

0.0%

1.2

2024

[442] Current applications and future potential of ChatGPT in radiology: A systematic review H. Temperley, ..., and Ian Brennan Journal of Medical Imaging and Radiation Oncology 2024 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

4.4

2011

[443] Automatic Summarization of Results from Clinical Trials Rodney L. Summerscales, ..., and A. Schwartz 2011 IEEE International Conference on Bioinformatics and Biomedicine 2011 - 57 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

23.3

2023

[444] BioSeq-Diabolo: Biological sequence similarity analysis using Diabolo Hongliang Li and Bin Liu PLOS Computational Biology 2023 - 34 citations - Show abstract - Cite - PDF Abstract: As the key for biological sequence structure and function prediction, disease diagnosis and treatment, biological sequence similarity analysis has attracted more and more attentions. However, the exiting computational methods failed to accurately analyse the biological sequence similarities because of the various data types (DNA, RNA, protein, disease, etc) and their low sequence similarities (remote homology). Therefore, new concepts and techniques are desired to solve this challenging problem. Biological sequences (DNA, RNA and protein sequences) can be considered as the sentences of “the book of life”, and their similarities can be considered as the biological language semantics (BLS). In this study, we are seeking the semantics analysis techniques derived from the natural language processing (NLP) to comprehensively and accurately analyse the biological sequence similarities. 27 semantics analysis methods derived from NLP were introduced to analyse biological sequence similarities, bringing new concepts and techniques to biological sequence similarity analysis. Experimental results show that these semantics analysis methods are able to facilitate the development of protein remote homology detection, circRNA-disease associations identification and protein function annotation, achieving better performance than the other state-of-the-art predictors in the related fields. Based on these semantics analysis methods, a platform called BioSeq-Diabolo has been constructed, which is named after a popular traditional sport in China. The users only need to input the embeddings of the biological sequence data. BioSeq-Diabolo will intelligently identify the task, and then accurately analyse the biological sequence similarities based on biological language semantics. BioSeq-Diabolo will integrate different biological sequence similarities in a supervised manner by using Learning to Rank (LTR), and the performance of the constructed methods will be evaluated and analysed so as to recommend the best methods for the users. The web server and stand-alone package of BioSeq-Diabolo can be accessed at http://bliulab.net/BioSeq-Diabolo/server/. 0.0% topic match

0.0%

1.7

2008

[445] Facilitating biomedical systematic reviews using ranked text retrieval and classification David Martínez, ..., and Timothy Baldwin Journal Not Provided 2008 - 29 citations - Show abstract - Cite 0.0% topic match

0.0%

300.8

2020

[446] GPT-3: Its Nature, Scope, Limits, and Consequences L. Floridi and Massimo Chiriatti Minds and Machines 2020 - 1214 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[447] Leveraging ChatGPT for Effective Research Design and Data Analysis — An Example with Research on Short Video Addiction Mengchan Zhang, ..., and Mi Kang Asian Journal on Addictions 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.1

2021

[448] The rationale behind systematic reviews in clinical medicine: a conceptual framework Hamideh Moosapour, ..., and B. Larijani Journal of Diabetes & Metabolic Disorders 2021 - 11 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2023

[449] Advancing radiology practice and research: harnessing the potential of large language models amidst imperfections Eyal Klang, ..., and Eyal Zimlichman BJR Open 2023 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

180.8

2023

[450] Role of Chat GPT in Public Health Som S. Biswas Annals of Biomedical Engineering 2023 - 302 citations - Show abstract - Cite 0.0% topic match

0.0%

11.6

2023

[451] ChatGPT Surpasses 1000 Publications on PubMed: Envisioning the Road Ahead M. Temsah, ..., and A. Al-Eyadhy Cureus 2023 - 14 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[452] Enhancing interpretation assistance by real-time query resolution in BICR of oncology clinical trials by leveraging ChatGPT bots Manish Sharma, ..., and Ron Korn https://doi.org/10.1117/12.3011602 2024 - 0 citations - Show abstract - Cite Abstract: Blinded Independent Central Review (BICR) is pivotal in maintaining unbiased assessment in oncology clinical trials employing various assessment criteria like Response Evaluation Criteria In Solid Tumors (RECIST) in clinical trials framework. This paper emphasizes the potential of Large Language Models (LLMs) such as OpenAI’s GPT-4 and ChatGPT trained on clinical trials’ documents and other reader training materials, to significantly improve interpretation assistance and real-time query resolution. During central review process these documents are easily accessible to readers but sometimes, given that most readers read on multiple ongoing clinical trials on regular basis, it is a daunting task to search details like trial design, endpoints and specific reader rules. Through various pre-trained frameworks using ChatGPT Application Programming Interface (API), an AI-based chatbot can be used for helping readers saving time by providing accurate study design related questions based on uploaded training documents. If successful, this analysis can open another novel implementation for LLMs (or ChatGPT) in clinical research and medical imaging. This prospective study involved the review of study design and protocol available from clinicaltrials.gov database maintained by The National Library of Medicine (NLM) at the National Institutes of Health (NIH). ClinicalTrials.gov is a registry of clinical trials that contains information on clinical studies funded by the NIH, other federal agencies, and private industry. The database includes over 444,000 trials from 221 countries. The NLM works with the US Food and Drug Administration (FDA) to develop and maintain the database. The ChatGPT based Chatbot was trained on clinical trial design data from respective studies to grasp the intricacies of assessment criteria, patient population, inclusion / exclusion criteria and other assessment nuances, as applicable. Fine-tuning with prompt engineering ensured Chatbot to understand the language and context specific to BICR. The resulting models serve as intelligent assistants to provide a user-friendly interface for reviewers. Reviewers can engage with the chatbot in natural language, obtaining clarifications on assessment guidelines, terminology, and complex cases. To train the Chatbot, we searched for lung cancer studies with following specifications: “Completed Studies | Studies with Results | Interventional Studies | Lung Cancer | Phase 3 | Study Protocols | Statistical Analysis Plans (SAPs)” from the clinicaltrials.gov website. Without any bias, we selected the first three studies in search results for our prospective study and seven questions to ask, representative of commonly encountered queries for readers. The algorithm supported by ChatGPT was evaluated against a Gold Standard medical opinion, determined by a board-certified radiologist with over 20 years of experience in the BICR process. The Chatbot provided immediate, contextually accurate insights for all the questions across three studies. The trick questions which did not have the answers, data or references in the provided text were rightly called out by the Chatbot, suggesting the user to check with document or study team. Though sometimes, in addition to referencing the study team or documents, it did provide a clinical practice or general criteria related feedback. Real-time query resolution reduces response time, preventing delays in assessment and decision-making. LLMs can streamline training by offering on-the-spot explanations and references, enhancing reviewer proficiency and efficiency. By acting as interactive chatbots, LLMs have immense potential to improve quality and efficiency by offering contextual guidance and expedited responses, ultimately enhancing decision-making and study efficiency. 0.0% topic match

0.0%

13.5

2021

[453] Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning Guangyi Liu, ..., and Shuguang Cui IEEE Transactions on Neural Networks and Learning Systems 2021 - 44 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

792.7

2019

[454] BioBERT: a pre-trained biomedical language representation model for biomedical text mining Jinhyuk Lee, ..., and Jaewoo Kang Bioinformatics 2019 - 4601 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1431.3

2018

[455] Improving Language Understanding by Generative Pre-Training Alec Radford and Karthik Narasimhan Journal Not Provided 2018 - 9832 citations - Show abstract - Cite 0.0% topic match

0.0%

308.1

2023

[456] Chatting about ChatGPT: How May AI and GPT Impact Academia and Libraries? B. Lund and Wang Ting SSRN Electronic Journal 2023 - 539 citations - Show abstract - Cite 0.0% topic match

0.0%

4.6

2022

[457] Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study Wojciech Kusa, ..., and Petr Knoth ArXiv 2022 - 13 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

2.9

2017

[458] An Interactive Two-Dimensional Approach to Query Aspects Rewriting in Systematic Reviews. IMS Unipd At CLEF eHealth Task 2 Giorgio Maria Di Nunzio, ..., and Geneviève Henrot Conference and Labs of the Evaluation Forum 2017 - 23 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[459] Assessing Generative Pretrained Transformers (GPT) in Clinical Decision-Making: Comparative Analysis of GPT-3.5 and GPT-4 Adi Lahat, ..., and Eyal Klang Journal of Medical Internet Research 2024 - 0 citations - Show abstract - Cite Abstract: Background Artificial intelligence, particularly chatbot systems, is becoming an instrumental tool in health care, aiding clinical decision-making and patient engagement. Objective This study aims to analyze the performance of ChatGPT-3.5 and ChatGPT-4 in addressing complex clinical and ethical dilemmas, and to illustrate their potential role in health care decision-making while comparing seniors’ and residents’ ratings, and specific question types. Methods A total of 4 specialized physicians formulated 176 real-world clinical questions. A total of 8 senior physicians and residents assessed responses from GPT-3.5 and GPT-4 on a 1-5 scale across 5 categories: accuracy, relevance, clarity, utility, and comprehensiveness. Evaluations were conducted within internal medicine, emergency medicine, and ethics. Comparisons were made globally, between seniors and residents, and across classifications. Results Both GPT models received high mean scores (4.4, SD 0.8 for GPT-4 and 4.1, SD 1.0 for GPT-3.5). GPT-4 outperformed GPT-3.5 across all rating dimensions, with seniors consistently rating responses higher than residents for both models. Specifically, seniors rated GPT-4 as more beneficial and complete (mean 4.6 vs 4.0 and 4.6 vs 4.1, respectively; P<.001), and GPT-3.5 similarly (mean 4.1 vs 3.7 and 3.9 vs 3.5, respectively; P<.001). Ethical queries received the highest ratings for both models, with mean scores reflecting consistency across accuracy and completeness criteria. Distinctions among question types were significant, particularly for the GPT-4 mean scores in completeness across emergency, internal, and ethical questions (4.2, SD 1.0; 4.3, SD 0.8; and 4.5, SD 0.7, respectively; P<.001), and for GPT-3.5’s accuracy, beneficial, and completeness dimensions. Conclusions ChatGPT’s potential to assist physicians with medical issues is promising, with prospects to enhance diagnostics, treatments, and ethics. While integration into clinical workflows may be valuable, it must complement, not replace, human expertise. Continued research is essential to ensure safe and effective implementation in clinical environments. 0.0% topic match

0.0%

37.5

1999

[460] Untangling Text Data Mining Marti A. Hearst Annual Meeting of the Association for Computational Linguistics 1999 - 952 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

3.2

2024

[461] Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases PhD Justin T Reese, ..., and P. N. Robinson medRxiv 2024 - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

35.8

2021

[462] BioSeq-BLM: a platform for analyzing DNA, RNA and protein sequences based on biological language models Hongliang Li, ..., and Bin Liu Nucleic Acids Research 2021 - 112 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

25.6

2022

[463] Legal Prompting: Teaching a Language Model to Think Like a Lawyer Fang Yu, ..., and Frank Schilder ArXiv 2022 - 50 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2023

[464] Abstract 16022: Evaluating the Utility of OpenAI’s GPT-4 as a Diagnostic and Management Aid in Medicine Alan Ge, ..., and Andrew Krumerman Circulation 2023 - 0 citations - Show abstract - Cite Abstract: Introduction: The advent of deep learning in artificial intelligence has led to exponential growth in its capabilities to assist in medical decision making, diagnosis, and clinical reasoning. ChatGPT has recently gained popularity in the media, and while it has the potential to revolutionize modern medicine, its effectiveness in providing clinicians actionable management options is not fully understood. Methods: We investigated the capabilities and limitations of GPT-4, OpenAI’s most advanced model, by comparing its clinical decision making and differential diagnosis to expert clinicians from five NEJM cases published after GPT-4’s training cutoff date (9/2021). The cases were slowly fed to the program, which was prompted to provide management and next steps at each juncture. These recommendations were then compared to the clinicians’ decision making. Results: GPT-4’s final diagnosis was correct for only one out of the five cases. However, it considered the correct final diagnosis in its list of five differential diagnoses in all of the cases. GPT-4’s decisions matched 18-60% of the authors’ decisions (Table 1). Conclusion: GPT-4 considered the correct diagnosis in its differentials for all five cases that were tested. Ultimately it was only able to correctly identify the diagnosis in one case. GPT-4’s management decisions matched that of the authors 18-60% of the time. This initially appears to be lackluster, but GPT-4 was able to offer appropriate decisions for common problems and this match rate of decisions was heavily influenced by discrete lab orders. GPT-4 was able to recommend the final diagnostic test that would reveal the diagnosis in the majority of the cases. Our study was limited by sample size as well as the fact that case distribution included only rare and difficult to diagnose diseases. This brief study shows how large language models such as GPT-4 have the potential to be used to aid in clinical decision making. 0.0% topic match

0.0%

1129.2

2022

[465] Large Language Models are Zero-Shot Reasoners Takeshi Kojima, ..., and Yusuke Iwasawa ArXiv 2022 - 2798 citations - Show abstract - Cite - PDF Abstract: Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding"Let's think step by step"before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars. 0.0% topic match

0.0%

8.7

2023

[466] Distilling Large Language Models for Matching Patients to Clinical Trials Mauro Nievas, ..., and Hrituraj Singh Journal of the American Medical Informatics Association : JAMIA 2023 - 8 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

74.7

2023

[467] The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification D. Chicco and Giuseppe Jurman BioData Mining 2023 - 130 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

65.0

2023

[468] One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era Chaoning Zhang, ..., and Choong-Seon Hong ArXiv 2023 - 105 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2023

[469] Assessing ChatGPT for Text Summarization, Simplification and Extraction Tasks R. Tariq, ..., and Rui Zhang 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) 2023 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[470] Assessing the potential role of ChatGPT in spine surgery research Isabel Herzog, ..., and Michael Vives Journal of Experimental Orthopaedics 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

7.0

2023

[471] An open source corpus and automatic tool for section identification in Spanish health records Iker de la Iglesia, ..., and Aitziber Atutxa Journal of biomedical informatics 2023 - 9 citations - Show abstract - Cite 0.0% topic match

0.0%

5.1

2020

[472] Automating Systematic Literature Review K. Felizardo and Jeffrey C. Carver https://doi.org/10.1007/978-3-030-32489-6_12 2020 - 25 citations - Show abstract - Cite 0.0% topic match

0.0%

3.2

2018

[473] Retrieving and Ranking Studies for Systematic Reviews: University of Sheffield's Approach to CLEF eHealth 2018 Task 2 Amal H. Alharbi, ..., and Mark Stevenson Conference and Labs of the Evaluation Forum 2018 - 20 citations - Show abstract - Cite 0.0% topic match

0.0%

2.3

2024

[474] Exploring the Latest LLMs for Leaderboard Extraction Salomon Kabongo, ..., and Sören Auer ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

93.9

2023

[475] Is ChatGPT Leading Generative AI? What is Beyond Expectations? Ömer Aydın and Enis Karaarslan SSRN Electronic Journal 2023 - 119 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

36.3

2023

[476] How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Tasks Xuanting Chen, ..., and Xuanjing Huang ArXiv 2023 - 62 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

2259.3

2009

[477] The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration A. Liberati, ..., and D. Moher PLoS Medicine 2009 - 34733 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

329.0

2023

[478] Capabilities of GPT-4 on Medical Challenge Problems Harsha Nori, ..., and E. Horvitz ArXiv 2023 - 545 citations - Show abstract - Cite - PDF Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety. 0.0% topic match

0.0%

13.3

2018

[479] Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool A. Gates, ..., and L. Hartling Systematic Reviews 2018 - 89 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1354.0

2016

[480] Rayyan—a web and mobile app for systematic reviews M. Ouzzani, ..., and A. Elmagarmid Systematic Reviews 2016 - 10754 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

463.2

2022

[481] LaMDA: Language Models for Dialog Applications R. Thoppilan, ..., and Quoc Le ArXiv 2022 - 1305 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

5.1

2023

[482] The shift in research trends related to artificial intelligence in library repositories during the coronavirus pandemic Prasetyo Adi Nugroho, ..., and Noraini Ismail Library Hi Tech 2023 - 8 citations - Show abstract - Cite 0.0% topic match

0.0%

6.6

2020

[483] Automating B. Lopatin Django Standalone Apps 2020 - 32 citations - Show abstract - Cite 0.0% topic match

0.0%

26.0

2023

[484] Suicide Risk Assessments Through the Eyes of ChatGPT-3.5 Versus ChatGPT-4: Vignette Study I. Levkovich and Z. Elyoseph JMIR Mental Health 2023 - 30 citations - Show abstract - Cite - PDF Abstract: Background ChatGPT, a linguistic artificial intelligence (AI) model engineered by OpenAI, offers prospective contributions to mental health professionals. Although having significant theoretical implications, ChatGPT’s practical capabilities, particularly regarding suicide prevention, have not yet been substantiated. Objective The study’s aim was to evaluate ChatGPT’s ability to assess suicide risk, taking into consideration 2 discernable factors—perceived burdensomeness and thwarted belongingness—over a 2-month period. In addition, we evaluated whether ChatGPT-4 more accurately evaluated suicide risk than did ChatGPT-3.5. Methods ChatGPT was tasked with assessing a vignette that depicted a hypothetical patient exhibiting differing degrees of perceived burdensomeness and thwarted belongingness. The assessments generated by ChatGPT were subsequently contrasted with standard evaluations rendered by mental health professionals. Using both ChatGPT-3.5 and ChatGPT-4 (May 24, 2023), we executed 3 evaluative procedures in June and July 2023. Our intent was to scrutinize ChatGPT-4’s proficiency in assessing various facets of suicide risk in relation to the evaluative abilities of both mental health professionals and an earlier version of ChatGPT-3.5 (March 14 version). Results During the period of June and July 2023, we found that the likelihood of suicide attempts as evaluated by ChatGPT-4 was similar to the norms of mental health professionals (n=379) under all conditions (average Z score of 0.01). Nonetheless, a pronounced discrepancy was observed regarding the assessments performed by ChatGPT-3.5 (May version), which markedly underestimated the potential for suicide attempts, in comparison to the assessments carried out by the mental health professionals (average Z score of –0.83). The empirical evidence suggests that ChatGPT-4’s evaluation of the incidence of suicidal ideation and psychache was higher than that of the mental health professionals (average Z score of 0.47 and 1.00, respectively). Conversely, the level of resilience as assessed by both ChatGPT-4 and ChatGPT-3.5 (both versions) was observed to be lower in comparison to the assessments offered by mental health professionals (average Z score of –0.89 and –0.90, respectively). Conclusions The findings suggest that ChatGPT-4 estimates the likelihood of suicide attempts in a manner akin to evaluations provided by professionals. In terms of recognizing suicidal ideation, ChatGPT-4 appears to be more precise. However, regarding psychache, there was an observed overestimation by ChatGPT-4, indicating a need for further research. These results have implications regarding ChatGPT-4’s potential to support gatekeepers, patients, and even mental health professionals’ decision-making. Despite the clinical potential, intensive follow-up studies are necessary to establish the use of ChatGPT-4’s capabilities in clinical practice. The finding that ChatGPT-3.5 frequently underestimates suicide risk, especially in severe cases, is particularly troubling. It indicates that ChatGPT may downplay one’s actual suicide risk level. 0.0% topic match

0.0%

0.0

2023

[485] ChatGPT: A Comprehensive Review of a Large Language Model K. Kaswan, ..., and D. Yadav 2023 International Conference on Communication, Security and Artificial Intelligence (ICCSAI) 2023 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

46.2

2023

[486] A medical multimodal large language model for future pandemics Fenglin Liu, ..., and David A. Clifton NPJ Digital Medicine 2023 - 44 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2024

[487] The role of ChatGPT in developing systematic literature searches: an evidence summary Veronica Parisi and Anthea Sutton Journal of EAHIL 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[488] Assessing GPT-4’s Performance in Delivering Medical Advice: Comparative Analysis With Human Experts Eunbeen Jo, ..., and H. J. Joo JMIR Medical Education 2024 - 0 citations - Show abstract - Cite Abstract: Abstract Background Accurate medical advice is paramount in ensuring optimal patient care, and misinformation can lead to misguided decisions with potentially detrimental health outcomes. The emergence of large language models (LLMs) such as OpenAI’s GPT-4 has spurred interest in their potential health care applications, particularly in automated medical consultation. Yet, rigorous investigations comparing their performance to human experts remain sparse. Objective This study aims to compare the medical accuracy of GPT-4 with human experts in providing medical advice using real-world user-generated queries, with a specific focus on cardiology. It also sought to analyze the performance of GPT-4 and human experts in specific question categories, including drug or medication information and preliminary diagnoses. Methods We collected 251 pairs of cardiology-specific questions from general users and answers from human experts via an internet portal. GPT-4 was tasked with generating responses to the same questions. Three independent cardiologists (SL, JHK, and JJC) evaluated the answers provided by both human experts and GPT-4. Using a computer interface, each evaluator compared the pairs and determined which answer was superior, and they quantitatively measured the clarity and complexity of the questions as well as the accuracy and appropriateness of the responses, applying a 3-tiered grading scale (low, medium, and high). Furthermore, a linguistic analysis was conducted to compare the length and vocabulary diversity of the responses using word count and type-token ratio. Results GPT-4 and human experts displayed comparable efficacy in medical accuracy (“GPT-4 is better” at 132/251, 52.6% vs “Human expert is better” at 119/251, 47.4%). In accuracy level categorization, humans had more high-accuracy responses than GPT-4 (50/237, 21.1% vs 30/238, 12.6%) but also a greater proportion of low-accuracy responses (11/237, 4.6% vs 1/238, 0.4%; P=.001). GPT-4 responses were generally longer and used a less diverse vocabulary than those of human experts, potentially enhancing their comprehensibility for general users (sentence count: mean 10.9, SD 4.2 vs mean 5.9, SD 3.7; P<.001; type-token ratio: mean 0.69, SD 0.07 vs mean 0.79, SD 0.09; P<.001). Nevertheless, human experts outperformed GPT-4 in specific question categories, notably those related to drug or medication information and preliminary diagnoses. These findings highlight the limitations of GPT-4 in providing advice based on clinical experience. Conclusions GPT-4 has shown promising potential in automated medical consultation, with comparable medical accuracy to human experts. However, challenges remain particularly in the realm of nuanced clinical judgment. Future improvements in LLMs may require the integration of specific clinical reasoning pathways and regulatory oversight for safe use. Further research is needed to understand the full potential of LLMs across various medical specialties and conditions. 0.0% topic match

0.0%

2.4

2018

[489] Nivolumab for adults with Hodgkin's lymphoma (a rapid review using the software RobotReviewer). M. Goldkuhle, ..., and N. Skoetz The Cochrane database of systematic reviews 2018 - 15 citations - Show abstract - Cite - PDF Abstract: BACKGROUND Hodgkin's lymphoma (HL) is a cancer of the lymphatic system, and involves the lymph nodes, spleen and other organs such as the liver, lung, bone or bone marrow, depending on the tumour stage. With cure rates of up to 90%, HL is one of the most curable cancers worldwide. Approximately 10% of people with HL will be refractory to initial treatment or will relapse; this is more common in people with advanced stage or bulky disease. Standard of care for these people is high-dose chemotherapy and autologous stem cell transplantation (ASCT), but only 55% of participants treated with high-dose chemotherapy and ASCT are free from treatment failure at three years, with an overall survival (OS) of about 80% at three years.Checkpoint inhibitors that target the interaction of the programmed death (PD)-1 immune checkpoint receptor, and its ligands PD-L1 and PD-L2, have shown remarkable activity in a wide range of malignancies. Nivolumab is an anti-(PD)-1 monoclonal antibody and currently approved by the US Food and Drug Administration (FDA) for the treatment of melanoma, non-small cell lung cancer, renal cell carcinoma and, since 2016, for classical Hodgkin's lymphoma (cHL) after treatment with ASCT and brentuximab vedotin. OBJECTIVES To assess the benefits and harms of nivolumab in adults with HL (irrespective of stage of disease). SEARCH METHODS We searched CENTRAL, MEDLINE, Embase, International Pharmaceutical Abstracts, conference proceedings and six study registries from January 2000 to May 2018 for prospectively planned trials evaluating nivolumab. SELECTION CRITERIA We included prospectively planned trials evaluating nivolumab in adults with HL. We excluded trials in which less than 80% of participants had HL, unless the trial authors provided the subgroup data for these participants in the publication or after we contacted the trial authors. DATA COLLECTION AND ANALYSIS Two review authors independently extracted data and assessed potential risk of bias. We used the software RobotReviewer to extract data and compared results with our findings. As we did not identify any randomised controlled trials (RCTs) or non-RCTs, we did not meta-analyse data. MAIN RESULTS Our search found 782 potentially relevant references. From these, we included three trials without a control group, with 283 participants. In addition, we identified 14 ongoing trials evaluating nivolumab, of which two are randomised. Risk of bias of the three included studies was moderate to high. All of the participants were in relapsed stage, most of them were heavily pretreated and had received at least two previous treatments, most of them had also undergone ASCT. As we did not identify any RCTs, we could not use the software RobotReviewer to assess risk of bias. The software identified correctly that one study was not an RCT and did not extract any trial data, but extracted characteristics of the other two studies (although also not RCTs) in a sufficient way.Two studies with 260 participants evaluated OS. After six months, OS was 100% in one study and median OS (the timepoint when only 50% of participants were alive) was not reached in the other trial after a median follow-up of 18 months (interquartile range (IQR) 15 to 22 months) (very low certainty evidence, due to observational trial design, heterogenous patient population in terms of pretreatments and various follow-up times (downgrading by 1 point)). In one study, one out of three cohorts reported quality of life. It was unclear whether there was an effect on quality of life as only a subset of participants filled out the follow-up questionnaire (very low certainty evidence). Three trials (283 participants) evaluated progression-free survival (PFS) (very low certainty evidence). Six-month PFS ranged between 60% and 86%, and median PFS ranged between 12 and 18 months. All three trials (283 participants) reported complete response rates, ranging from 12% to 29%, depending on inclusion criteria and participants' previous treatments (very low certainty evidence).One trial (243 participants) reported drug-related grade 3 or 4 adverse events (AEs) only after a median follow-up of 18 months (IQR 15 to 22 months); these were fatigue (23%), diarrhoea (15%), infusion reactions (14%) and rash (12%). The other two trials (40 participants) reported 23% to 52% grade 3 or 4 AEs after six months' follow-up (very low certainty evidence). Only one trial (243 participants) reported drug-related serious AEs; 2% of participants developed infusion reactions and 1% pneumonitis (very low certainty evidence).None of the studies reported treatment-related mortality. AUTHORS' CONCLUSIONS To date, data on OS, quality of life, PFS, response rate, or short- and long-term AEs are available from small uncontrolled trials only. The three trials included heavily pretreated participants, which had previously undergone regimens of BV or ASCT. For these participants, median OS was not reached after follow-up times of at least 16 months (more than 50% of participants with a limited life expectancy were alive at this timepoint). Only one cohort out of three only reported quality of life, with limited follow-up data so that meaningful conclusions were not possible. Serious adverse events occurred rarely. Currently, data are too sparse to make a clear statement on nivolumab for people with relapsed or refractory HL except for heavily pretreated people, which had previously undergone regimens of BV or ASCT. When interpreting these results, it is important to consider that proper RCTs should confirm these findings.As there are 14 ongoing trials evaluating nivolumab, of which two are RCTs, it is possible that an update of this review will be published in the near future and that this update will show different results to those reported here. 0.0% topic match

0.0%

13607.8

2019

[490] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin, ..., and Kristina Toutanova North American Chapter of the Association for Computational Linguistics 2019 - 79877 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

59.0

1993

[491] Bias, prevalence and kappa. T. Byrt, ..., and J. Carlin Journal of clinical epidemiology 1993 - 1861 citations - Show abstract - Cite 0.0% topic match

0.0%

0.8

2023

[492] Utilizing Artificial Intelligence Tools Using the GPT Chatbot in Medicine - A Review of Flaws, Advantages, and Limitations Krzysztof Edyko, ..., and Urszula Marchewka Journal of Education, Health and Sport 2023 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

15.0

2023

[493] Multiple sequence alignment-based RNA language model and its application to structural inference Yikun Zhang, ..., and Yaoqi Zhou Nucleic Acids Research 2023 - 25 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

10.6

2024

[494] Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomas T. Laohawetwanit, ..., and Sompon Apornvirat Journal of Clinical Pathology 2024 - 9 citations - Show abstract - Cite 0.0% topic match

0.0%

5.3

2024

[495] Artificial Intelligence for Literature Reviews: Opportunities and Challenges Francisco Bolanos, ..., and Enrico Motta ArXiv 2024 - 4 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

0.0

2023

[496] Exploring the User Experience and the Role of ChatGPT in the Academic Writing Process Reem Alqadi, ..., and A. B. Najjar 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE) 2023 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

10.9

2023

[497] Application ChatGPT in conducting systematic reviews and meta-analyses S. Mahuli, ..., and Ansul Kumar British Dental Journal 2023 - 15 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

6.2

2017

[498] Using ontology-based semantic similarity to facilitate the article screening process for systematic reviews Xiaonan Ji, ..., and Po-Yin Yen Journal of biomedical informatics 2017 - 47 citations - Show abstract - Cite 0.0% topic match

0.0%

2.2

2023

[499] Evaluating the Performance of Large Language Models on a Neurology Board-Style Examination M. Schubert, ..., and V. Venkataramani medRxiv 2023 - 3 citations - Show abstract - Cite - PDF Abstract: Summary Background and Objectives Recent advancements in large language models (LLMs) such as GPT-3.5 and GPT-4 have shown impressive potential in a wide array of applications, including healthcare. While GPT-3.5 and GPT-4 showed heterogeneous results across specialized medical board examinations, the performance of these models in neurology board exams remains unexplored. Methods An exploratory, prospective study was conducted between May 17 and May 31, 2023. The evaluation utilized a question bank approved by the American Board of Psychiatry and Neurology and were validated with a small question cohort by the European Board for Neurology. All questions were categorized into lower-order (recall, understanding) and higher-order (apply, analyze, synthesize) questions based on Bloom Taxonomy for learning and assessment. Performance was assessed in relation to overall scores, question type, and topics, along with the confidence level and reproducibility of answers. Results GPT-4 significantly outperformed GPT-3.5 by correctly answering 85% of 1956 questions compared to GPT-3.5 (66.8%) Notably, GPT-4's performance was above the average human score of 73.8%, effectively achieving near-passing and passing grades in the neurology board exam. GPT-4 outperformed human users in Behavioral, Cognitive and Psych-related questions and demonstrated superior performance to GPT-3.5 in six categories. Both models performed better on lower-order than higher-order questions (GPT4: 790 of 893 (88.5%) vs. 872 of 1063 (82%), GPT-3.5: 639 of 893 (71.6%) vs. 667 of 1063 (62.7%)) with GPT-4 also excelling in both lower-order and higher-order questions. Both models consistently used confident language, even when providing incorrect answers (GPT-4: 99.3%, 294 incorrect answers, GPT-3.5: 100%, 650 incorrect answers). Reproducible answers of GPT-3.5 and GPT-4 (more than 75 % same output across 50 independent queries) were associated with a higher percentage of correct answers (GPT-3.5: 66 of 88 (75%), GPT-4: 78 of 96 (81.3%)) than inconsistent answers, (GPT-3.5: 5 of 13 (38.5%), GPT-4: 1 of 4 (25%)). Discussion Despite the absence of neurology-specific training, GPT-4 demonstrated commendable performance, whereas GPT-3.5 performed slightly below the human average. While higher-order cognitive tasks were more challenging for both models, GPT-4's results were equivalent to passing grades in specialized neurology exams. These findings suggests that LLMs like GPT-4 could have significant applications in clinical neurology and healthcare with further refinements. 0.0% topic match

0.0%

51.0

2023

[500] ChatGPT goes to the operating room: evaluating GPT-4 performance and its potential in surgical education and training in the era of large language models N. Oh, ..., and W. Lee Annals of Surgical Treatment and Research 2023 - 79 citations - Show abstract - Cite 0.0% topic match

0.0%

17.8

2023

[501] ChatGPT and Artificial Intelligence in Medical Writing: Concerns and Ethical Considerations A. Doyal, ..., and Ricardo A Serrano Cureus 2023 - 23 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

13645.8

2017

[502] Attention is All you Need Ashish Vaswani, ..., and Illia Polosukhin Neural Information Processing Systems 2017 - 101321 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

7.7

2023

[503] Transformer-based structuring of free-text radiology report databases S. Nowak, ..., and A. Sprinkart European Radiology 2023 - 13 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

1.9

2024

[504] Towards Improved Radiological Diagnostics: Investigating the Utility and Limitations of GPT-3.5 Turbo and GPT-4 with Quiz Cases. Tomohiro Kikuchi, ..., and Takeharu Yoshikawa AJNR. American journal of neuroradiology 2024 - 1 citations - Show abstract - Cite Abstract: BACKGROUND AND PURPOSE The rise of large language models such as generative pre-trained transformers (GPTs) has sparked significant interest in radiology, especially in interpreting radiological reports and image findings. While existing research has focused on GPTs estimating diagnoses from radiological descriptions, exploring alternative diagnostic information sources is also crucial. This study introduces the use of GPTs (GPT-3.5 Turbo and GPT-4) for information retrieval and summarization, searching relevant case reports via PubMed, and investigates their potential to aid diagnosis. MATERIALS AND METHODS From October 2021 to December 2023, we selected 115 cases from the "Case of the Week" series on the American Journal of Neuroradiology website. Their Description and Legend sections were presented to the GPTs for the two tasks. For the Direct Diagnosis task, the models provided three differential diagnoses that were considered correct if they matched the diagnosis in the diagnosis section. For the Case Report Search task, the models generated two keywords per case, creating PubMed search queries to extract up to three relevant reports. A response was considered correct if reports containing the disease name stated in the diagnosis section were extracted. McNemar's test was employed to evaluate whether adding a Case Report Search to Direct Diagnosis improved overall accuracy. RESULTS In the Direct Diagnosis task, GPT-3.5 Turbo achieved a correct response rate of 26% (30/115 cases), whereas GPT-4 achieved 41% (47/115). For the Case Report Search task, GPT-3.5 Turbo scored 10% (11/115), and GPT-4 scored 7% (8/115). Correct responses totaled 32% (37/115) with three overlapping cases for GPT-3.5 Turbo, whereas GPT-4 had 43% (50/115) of correct responses with five overlapping cases. Adding Case Report Search improved GPT-3.5 Turbo's performance (p = 0.023) but not that of GPT-4 (p = 0.248). CONCLUSIONS The effectiveness of adding Case Report Search to GPT-3.5 Turbo was particularly pronounced, suggesting its potential as an alternative diagnostic approach to GPTs, particularly in scenarios where direct diagnoses from GPTs are not obtainable. Nevertheless, the overall performance of GPT models in both direct diagnosis and case report retrieval tasks remains not optimal, and users should be aware of their limitations.ABBREVIATIONS: AI = Artificial Intelligence, GPT = generative pretrained transformer, LLM = large language model. 0.0% topic match

0.0%

3.6

2022

[505] A systematic review and meta-analysis of studies on screening for mild cognitive impairment in primary healthcare L. Karimi, ..., and M. Farahbakhsh BMC Psychiatry 2022 - 10 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

66.6

2023

[506] Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study Arya Rao, ..., and M. Succi Journal of Medical Internet Research 2023 - 82 citations - Show abstract - Cite - PDF Abstract: Background Large language model (LLM)–based artificial intelligence chatbots direct the power of large training data sets toward successive, related tasks as opposed to single-ask tasks, for which artificial intelligence already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as artificial physicians, has not yet been evaluated. Objective This study aimed to evaluate ChatGPT’s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. Methods We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared its accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. Accuracy was measured by the proportion of correct responses to the questions posed within the clinical vignettes tested, as calculated by human scorers. We further conducted linear regression to assess the contributing factors toward ChatGPT’s performance on clinical tasks. Results ChatGPT achieved an overall accuracy of 71.7% (95% CI 69.3%-74.1%) across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI 67.8%-86.1%) and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI 54.2%-66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=–15.8%; P<.001) and clinical management (β=–7.4%; P=.02) question types. Conclusions ChatGPT achieves impressive accuracy in clinical decision-making, with increasing strength as it gains more clinical information at its disposal. In particular, ChatGPT demonstrates the greatest accuracy in tasks of final diagnosis as compared to initial diagnosis. Limitations include possible model hallucinations and the unclear composition of ChatGPT’s training data set. 0.0% topic match

0.0%

6.9

2024

[507] Can ChatGPT-3.5 Pass a Medical Exam? A Systematic Review of ChatGPT's Performance in Academic Testing Anusha Sumbal, ..., and Alina Amir Journal of Medical Education and Curricular Development 2024 - 6 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

4.6

2023

[508] The impact and applications of ChatGPT: a systematic review of literature reviews I. Gabashvili ArXiv 2023 - 7 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

33.3

2014

[509] Systematic review automation technologies G. Tsafnat, ..., and E. Coiera Systematic Reviews 2014 - 345 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

4.3

2023

[510] Outpatient Treatment of Confirmed COVID-19: A Living, Rapid Evidence Review for the American College of Physicians (Version 2) I. Sommer, ..., and G. Gartlehner Annals of Internal Medicine 2023 - 5 citations - Show abstract - Cite Abstract: BACKGROUND Clinicians and patients want to know the benefits and harms of outpatient treatment options for the Omicron variant of SARS-CoV-2. PURPOSE To assess the benefits and harms of 22 different COVID-19 treatments. DATA SOURCES The Epistemonikos COVID-19 L·OVE platform, the iSearch COVID-19 portfolio, and the World Health Organization (WHO) COVID-19 Research Database from 26 November 2021 to 2 March 2023. STUDY SELECTION Two reviewers independently screened abstracts and full texts against a priori-defined criteria. DATA EXTRACTION One reviewer extracted the data and assessed the risk of bias and certainty of evidence (COE). A second reviewer verified the data abstraction and assessments. DATA SYNTHESIS Two randomized controlled trials and 6 retrospective cohort studies were included. Nirmatrelvir-ritonavir was associated with a reduction in hospitalization due to COVID-19 (for example, 0.7% vs. 1.2%; moderate COE) and all-cause mortality (for example, <0.1% vs. 0.2%; moderate COE). Molnupiravir led to a higher recovery rate (31.8% vs. 22.6%; moderate COE) and reduced time to recovery (9 vs. 15 median days; moderate COE) but had no effect on all-cause mortality (0.02% vs. 0.04%; moderate COE) and the incidence of serious adverse events (0.4% vs. 0.3%; moderate COE). Ivermectin had no effect on time to recovery (moderate COE) and resulted in no difference in adverse events compared with placebo (low COE). Sotrovimab resulted in no difference in all-cause mortality compared with no treatment (low COE). No eligible studies for all other treatments of interest were identified. LIMITATION Evidence for nirmatrelvir-ritonavir and sotrovimab is based on nonrandomized studies only. CONCLUSION Nirmatrelvir-ritonavir and molnupiravir probably improve outcomes for outpatients with mild to moderate COVID-19. PRIMARY FUNDING SOURCE American College of Physicians. (PROSPERO: CRD42023406456). 0.0% topic match

0.0%

1.6

2024

[511] Exploring ChatGPT's abilities in medical article writing and peer review Gultekin Kadi and Mehmet ali Aslaner Croatian Medical Journal 2024 - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

1.4

2024

[512] Quality, Accuracy, and Bias in ChatGPT-Based Summarization of Medical Abstracts Joel Hake, ..., and D. J. Parente Annals of Family Medicine 2024 - 1 citations - Show abstract - Cite - PDF Abstract: PURPOSE Worldwide clinical knowledge is expanding rapidly, but physicians have sparse time to review scientific literature. Large language models (eg, Chat Generative Pretrained Transformer [ChatGPT]), might help summarize and prioritize research articles to review. However, large language models sometimes “hallucinate” incorrect information. METHODS We evaluated ChatGPT’s ability to summarize 140 peer-reviewed abstracts from 14 journals. Physicians rated the quality, accuracy, and bias of the ChatGPT summaries. We also compared human ratings of relevance to various areas of medicine to ChatGPT relevance ratings. RESULTS ChatGPT produced summaries that were 70% shorter (mean abstract length of 2,438 characters decreased to 739 characters). Summaries were nevertheless rated as high quality (median score 90, interquartile range [IQR] 87.0-92.5; scale 0-100), high accuracy (median 92.5, IQR 89.0-95.0), and low bias (median 0, IQR 0-7.5). Serious inaccuracies and hallucinations were uncommon. Classification of the relevance of entire journals to various fields of medicine closely mirrored physician classifications (nonlinear standard error of the regression [SER] 8.6 on a scale of 0-100). However, relevance classification for individual articles was much more modest (SER 22.3). CONCLUSIONS Summaries generated by ChatGPT were 70% shorter than mean abstract length and were characterized by high quality, high accuracy, and low bias. Conversely, ChatGPT had modest ability to classify the relevance of articles to medical specialties. We suggest that ChatGPT can help family physicians accelerate review of the scientific literature and have developed software (pyJournalWatch) to support this application. Life-critical medical decisions should remain based on full, critical, and thoughtful evaluation of the full text of research articles in context with clinical guidelines. 0.0% topic match

0.0%

0.0

2024

[513] Addressing High-value Care with Generative Pretrained Transformer 4 Jassimran Singh, ..., and Navya Reddy Perkit Journal of Primary Care Specialties 2024 - 0 citations - Show abstract - Cite Abstract: High-value care emphasizes services offering significant health benefits, aligning with patient preferences while minimizing costs and interventions of little benefit. This approach is increasingly vital in a healthcare environment constrained by finite resources and rising costs. Large Language Models (LLMs) like Generative Pretrained transformer-4 (GPT-4), with their vast data processing capabilities, offer a promising avenue for supporting healthcare providers in making evidence-based, high-value care decisions. This study aims to evaluate the performance of OpenAI’s GPT-4 in providing responses to high-value care clinical scenarios within internal medicine, assessing its accuracy, relevance, and reasoning against established medical guidelines and literature. An observational study was conducted using MKSAP-19’s high-value care questions, comparing GPT-4’s responses to the correct answers based on established studies, trials, and guidelines. The study did not involve real patient data, circumventing the need for Institutional Review Board (IRB) approval. Performance metrics focused on the accuracy, relevance, and consistency of GPT-4’s answers. GPT-4 demonstrated a 74.4% accuracy rate (32 out of 43 questions) across a range of high-value care clinical scenarios, including image-based questions. Errors made by GPT-4 were similar to those by medical residents using MKSAP-19, suggesting areas for model improvement and potential educational applications. The study detailed GPT-4’s decision-making pattern, emphasizing its clinical reasoning capabilities. The findings suggest that GPT-4 can significantly support high-value care in internal medicine by providing accurate, evidence-based responses to complex clinical scenarios. Despite its limitations, including a 25.6% error rate and the scope of its training data, GPT-4’s performance indicates its potential as both a clinical and educational tool in healthcare. 0.0% topic match

0.0%

1.6

2024

[514] LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices Neda Taghizadeh Serajeh, ..., and Mattia De Rosa ArXiv 2024 - 1 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

15.3

2023

[515] The Potential Usefulness of ChatGPT in Oral and Maxillofacial Radiology J. Mago and Manoj Sharma Cureus 2023 - 21 citations - Show abstract - Cite - PDF Abstract: Aim This study aimed to evaluate the potential usefulness of Chat Generated Pre-Trained Transformer-3 (ChatGPT-3) in oral and maxillofacial radiology for report writing by identifying radiographic anatomical landmarks and learning about oral and maxillofacial pathologies and their radiographic features. The study also aimed to evaluate the performance of ChatGPT-3 and its usage in oral and maxillofacial radiology training. Materials and methods A questionnaire consisting of 80 questions was queried on the OpenAI app ChatGPT-3. The questions were stratified based on three categories. The categorization was based on random anatomical landmarks, oral and maxillofacial pathologies, and the radiographic features of some of these pathologies. One oral and maxillofacial radiologist evaluated queries that were answered by the ChatGPT-3 model and rated them on a 4-point, modified Likert scale. The post-survey analysis for the performance of ChatGPT-3 was based on the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis, its application in oral and maxillofacial radiology training, and its recommended use. Results In order of efficiency, Chat GPT-3 gave 100% accuracy in describing radiographic landmarks. However, the content of the oral and maxillofacial pathologies was limited to major or characteristic radiographic features. The mean scores for the queries related to the anatomic landmarks, oral and maxillofacial pathologies, and radiographic features of the oral and maxillofacial pathologies were 3.94, 3.85, and 3.96, respectively. However, the median and mode scores were 4 and were similar to all categories. The data for the oral and maxillofacial pathologies when the questions were not specifically included in the format of the introduction of the pathology, causes, symptoms, and treatment. Out of two abbreviations, one was not answered correctly. Conclusion The study showed that ChatGPT-3 is efficient in describing the pathology, characteristic radiographic features, and describing anatomical landmarks. ChatGPT-3 can be used as an adjunct when an oral radiologist needs additional information on any pathology, however, it cannot be the mainstay for reference. ChatGPT-3 is less detail-oriented, and the data has a risk of infodemics and the possibility of medical errors. However, Chat GPT-3 can be an excellent tool in helping the community in increasing the knowledge and awareness of various pathologies and decreasing the anxiety of the patients while dental healthcare professionals formulate an appropriate treatment plan. 0.0% topic match

0.0%

6.7

2023

[516] An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews Wojciech Kusa, ..., and A. Hanbury Intell. Syst. Appl. 2023 - 12 citations - Show abstract - Cite 0.0% topic match

0.0%

10.7

2023

[517] An AI Dietitian for Type 2 Diabetes Mellitus Management Based on Large Language and Image Recognition Models: Preclinical Concept Validation Study Haonan Sun, ..., and Dongran Han Journal of Medical Internet Research 2023 - 14 citations - Show abstract - Cite - PDF Abstract: Background Nutritional management for patients with diabetes in China is a significant challenge due to the low supply of registered clinical dietitians. To address this, an artificial intelligence (AI)–based nutritionist program that uses advanced language and image recognition models was created. This program can identify ingredients from images of a patient’s meal and offer nutritional guidance and dietary recommendations. Objective The primary objective of this study is to evaluate the competence of the models that support this program. Methods The potential of an AI nutritionist program for patients with type 2 diabetes mellitus (T2DM) was evaluated through a multistep process. First, a survey was conducted among patients with T2DM and endocrinologists to identify knowledge gaps in dietary practices. ChatGPT and GPT 4.0 were then tested through the Chinese Registered Dietitian Examination to assess their proficiency in providing evidence-based dietary advice. ChatGPT’s responses to common questions about medical nutrition therapy were compared with expert responses by professional dietitians to evaluate its proficiency. The model’s food recommendations were scrutinized for consistency with expert advice. A deep learning–based image recognition model was developed for food identification at the ingredient level, and its performance was compared with existing models. Finally, a user-friendly app was developed, integrating the capabilities of language and image recognition models to potentially improve care for patients with T2DM. Results Most patients (182/206, 88.4%) demanded more immediate and comprehensive nutritional management and education. Both ChatGPT and GPT 4.0 passed the Chinese Registered Dietitian examination. ChatGPT’s food recommendations were mainly in line with best practices, except for certain foods like root vegetables and dry beans. Professional dietitians’ reviews of ChatGPT’s responses to common questions were largely positive, with 162 out of 168 providing favorable reviews. The multilabel image recognition model evaluation showed that the Dino V2 model achieved an average F1 score of 0.825, indicating high accuracy in recognizing ingredients. Conclusions The model evaluations were promising. The AI-based nutritionist program is now ready for a supervised pilot study. 0.0% topic match

0.0%

14.2

2023

[518] Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine F. Guillén-Grima, ..., and I. Aguinaga-Ontoso Clinics and Practice 2023 - 14 citations - Show abstract - Cite - PDF Abstract: The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model’s overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. Material and methods: We studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4’s new image analysis capability. Results: GPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (p < 0.001). English translations had a slightly enhanced performance. GPT-4 scored 26.1% of the questions with images in English. The results were worse when the questions were in Spanish, 13.0%, although the differences were not statistically significant (p = 0.250). Among medical specialties, GPT-4 achieved a 100% correct response rate in several areas, and the Pharmacology, Critical Care, and Infectious Diseases specialties showed lower performance. The error analysis revealed that while a 13.2% error rate existed, the gravest categories, such as “error requiring intervention to sustain life” and “error resulting in death”, had a 0% rate. Conclusions: GPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model’s high success rate is commendable, understanding the error severity is critical, especially when considering AI’s potential role in real-world medical practice and its implications for patient safety. 0.0% topic match

0.0%

0.0

2023

[519] [Evidence synthesis 2.0: how artificial intelligence is making systematic reviews more efficient.] Davide Petri Recenti progressi in medicina 2023 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[520] Using ChatGPT to perform a systematic review: a tutorial. Lefteris Teperikidis, ..., and Giuseppe Biondi-Zoccai Minerva cardiology and angiology 2024 - 0 citations - Show abstract - Cite 0.0% topic match

0.0%

6.8

2023

[521] The Potential and Pitfalls of using a Large Language Model such as ChatGPT or GPT-4 as a Clinical Assistant Jingqing Zhang, ..., and Yike Guo ArXiv 2023 - 9 citations - Show abstract - Cite - PDF 0.0% topic match

0.0%

None

[522] Detection of ChatGPT Fake Science with the xFakeBibs Learning Algorithm A. Hamed and Xin Wu Journal Not Provided None - 1 citations - Show abstract - Cite 0.0% topic match

0.0%

29962.3

2014

[523] R: A language and environment for statistical computing. R. Team MSOR connections 2014 - 325668 citations - Show abstract - Cite 0.0% topic match

0.0%

0.0

2024

[524] Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights E. Joe, ..., and Christine J Kirchhoff Journal Not Provided 2024 - 0 citations - Show abstract - Cite - PDF 0.0% topic match

Research topic

Report

References

Select the format you want to download:

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS

Download Citations

Citation Text

BibTeX

RIS