/LLM-pubmed-query-generation-evaluation

LLM PubMed Query Generation Evaluation

LLM PubMed Query Generation Evaluation

This repository contains code and data for evaluating the performance of various language models in assisting with the generation of PubMed search queries. The evaluation is conducted on a set of 9 representative prompts that a user might pass to a model to help generate an advanced search query.

Language Models Evaluated

The following language models are currently being evaluated:

  • anthropic/claude-3-opus
  • openai/gpt-4-0314
  • databricks/dbrx-instruct
  • cohere/command-r-plus
  • mistralai/mixtral-8x7b-instruct
  • mistralai/mistral-7b-instruct:free

This selection includes both open-source and closed models.

Evaluation Metrics

For each prompt, the generated query is passed to the PubMed API, and the initial search results are parsed to provide a crude sense of query performance. The evaluation metrics include:

  • The number of search results returned by PubMed (note: more results do not necessarily indicate a better query, as overly broad queries can return too many results)
  • "Quoted Phrases Not Found Count": the number of terms in the search query not recognized as valid MeSH terms or keywords by PubMed, serving as a rough proxy for model "hallucination"
  • Whether the search query generated by the model caused an error when passed to the PubMed API (Failed)
  • Whether the search query required basic regex processing to be valid for the PubMed API (Regex)

Please note that this evaluation does not closely assess the relevance of the retrieved articles, but rather provides a general indication of query performance.

Results Summary

Here is a summary table of the search result counts for each model across the 9 test prompts:

search_idx anthropic/claude-3-opus cohere/command-r-plus databricks/dbrx-instruct mistralai/mistral-7b-instruct:free mistralai/mixtral-8x7b-instruct openai/gpt-4-0314
0 280 454985 0 326 189 24
1 1078 498 2 0 2200 1531
2 222 71 0 0 0 195
3 276 0 0 0 0 1
4 1788 0 0 0 1063 101
5 2803 696 0 0 49 1292
6 116 0 0 0 0 86
7 4731 4364 0 0 1903 382
8 1525 14562 1332 21750 0 11239

Usage

A Jupyter notebook demonstrating how to run this evaluation on your own will be shared in the future. Please stay tuned for updates.

Results

This section provides detailed results for each of the 9 test prompts, including the search query generated by each model, the number of search results returned by PubMed, and additional evaluation metrics. I'll add some nicer formatting to this shortly!

Search Index: 0

Prompt: Could you construct a PubMed search query to find all meta-analyses and systematic reviews on the efficacy of mindfulness-based stress reduction (MBSR) in treating anxiety disorders? Use appropriate MeSH headings and restrict the search to publications from the last 12 years. Only return the query so I can enter it directly into PubMed. Don't add extra explanation.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 280

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (("Mindfulness"[Mesh] OR "mindfulness-based stress reduction" OR MBSR) AND ("Anxiety Disorders"[Mesh] OR anxiety OR "panic disorder" OR "generalized anxiety disorder" OR "social anxiety disorder" OR agoraphobia OR "specific phobia")) AND ("Meta-Analysis" [Publication Type] OR "meta-analysis as topic"[MeSH Terms] OR "meta-analysis"[All Fields] OR "Systematic Review" [Publication Type] OR "systematic review"[All Fields]) AND (("2011/01/01"[PDAT] : "3000"[PDAT]) AND English[lang])

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 24

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("mindfulness-based stress reduction"[MeSH Terms] OR "MBSR") AND (("anxiety disorders"[MeSH Terms] OR "anxiety"[MeSH Terms]) AND ("systematic review"[Publication Type] OR "meta-analysis"[Publication Type])) AND ("2009/01/01"[PDAT] : "2021/12/31"[PDAT])

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: `; ``(mindfulness-based stress reduction [Title/Abstract]) AND (anxiety disorders [MeSH Terms]) AND ("systematic reviews"[Publication Type] OR "meta-analysis"[Publication Type]) AND ("2011/01/01"[PDAT] : "2023"[PDAT])```

    • PubMed Search URL: PubMed Search

  • Model: cohere/command-r-plus

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 189

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("Mindfulness-Based Stress Reduction"[Mesh] OR "Mindfulness"[Mesh]) AND ("Anxiety Disorders"[Mesh] OR anxiety) AND ("Systematic Review"[ptyp] OR "Meta-Analysis"[ptyp]) NOT ("Animals"[Mesh] NOT "Humans"[Mesh]) AND ("2010/01/01"[PDAT] : "2022/12/31"[PDAT])

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 326

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (*anxiety AND (mindfulness OR mindfulness-based stress reduction)) AND ((systematic review OR meta-analysis) AND ("2011" [DATE] : "2022" [DATE]))

    • PubMed Search URL: PubMed Search

Search Index: 1

Prompt: I need help constructing a PubMed query to locate research on the use of statins for primary prevention of cardiovascular disease in patients over 65. Please return directly the query. Don't add any extra context.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 1078

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (statins OR hmg-coa reductase inhibitors) AND (primary prevention) AND (cardiovascular disease OR CVD OR heart disease OR coronary artery disease OR CAD) AND (elderly OR older adults OR geriatric OR aged[MeSH]) AND (randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized[tiab] OR placebo[tiab] OR clinical trials as topic[mesh:noexp] OR randomly[tiab] OR trial[ti])

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 1531

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (statins[Title/Abstract] OR HMG-CoA reductase inhibitors[Title/Abstract]) AND (primary prevention[Title/Abstract] OR primary[Title/Abstract]) AND (cardiovascular diseases[MeSH Terms] OR cardiovascular disease[Title/Abstract] OR CVD[Title/Abstract]) AND (patients[Title/Abstract] OR patient[Title/Abstract]) AND (over 65[Title/Abstract] OR elderly[Title/Abstract] OR older[Title/Abstract] OR aged[MeSH Terms])

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 2
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `: (("Statin" OR "Hydroxymethylglutaryl-CoA Reductase Inhibitors") AND ("Primary Prevention" OR "Cardiovascular Diseases" OR "Heart Diseases") AND ("Aged" OR "65 and over"))

This is a PubMed query that should help you locate research on the use of statins for primary prevention of cardiovascular disease in patients over 65. It uses a combination of Medical Subject Headings (MeSH) and keywords to search for relevant articles.`

  • Model: cohere/command-r-plus

    • Search Result Count: 498

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("statins"[MeSH Terms] OR "statins"[All Fields] OR "statin"[All Fields]) AND ("primary prevention"[MeSH Terms] OR ("primary"[All Fields] AND "prevention"[All Fields]) OR "primary prevention"[All Fields]) AND ("cardiovascular diseases"[MeSH Terms] OR ("cardiovascular"[All Fields] AND "diseases"[All Fields]) OR "cardiovascular diseases"[All Fields]) AND ("aged, 80 and over"[MeSH Terms] OR ("aged, 80 and over"[All Fields] OR "older adults"[All Fields]))

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 2200

    • Quoted Phrases Not Found Count: 25.0

    • Failed: False

    • Regex: False

    • Search: ("Statins"[Mesh] OR statin* OR HMG-CoA reductase inhibitor*) AND ("Cardiovascular Diseases"[Mesh] OR "Cardiovascular Diseases, Cerebrovascular"[Mesh] OR "Cardiovascular Diseases, Metabolic"[Mesh] OR "Cardiovascular Diseases, Rheumatic"[Mesh] OR "Cardiovascular Diseases, Vascular"[Mesh] OR "Cardiovascular System"[Mesh] OR "Cardiovascular Physiological Phenomena"[Mesh] OR "Cardiovascular System Diseases"[Mesh] OR "Cardiovascular Agents"[Mesh] OR "Cardiovascular Abnormalities"[Mesh] OR "Cardiovascular Function, Heart"[Mesh] OR "Cardiovascular Function, Ventricular"[Mesh] OR "Cardiovascular System, Fetal"[Mesh] OR "Cardiovascular System, Vascular"[Mesh] OR "Cardiovascular System Physiological Phenomena"[Mesh] OR "Cardiovascular Systems, Artificial"[Mesh] OR "Cardiovascular Systems, Blood Vessels"[Mesh] OR "Cardiovascular Systems, Heart-Assist"[Mesh] OR "Cardiovascular Systems, Implantable"[Mesh] OR "Cardiovascular Systems, Mechanical Circulatory Support"[Mesh] OR "Cardiovascular Systems, Microcirculation"[Mesh] OR "Cardiovascular Systems, Myocardial Perfusion"[Mesh] OR "Cardiovascular Systems, Regeneration"[Mesh] OR "Cardiovascular Systems, Sarcomere"[Mesh] OR "Cardiovascular Systems, Simulation"[Mesh] OR "Cardiovascular Systems, Synthetic"[Mesh] OR "Cardiovascular Systems, Transplantation"[Mesh] OR "Cardiovascular Systems, Xenogeneic"[Mesh] OR cardiovascular OR CVD OR heart disease) AND ("Aged"[Mesh] OR elderly OR "65+" OR "older adults") AND ("Primary Prevention"[Mesh] OR "Primary Prevention"[TIAB] OR "preventive"[TIAB] OR "prophylactic"[TIAB] OR "secondary prevention"[TIAB] NOT "secondary prevention"[mesh] NOT "case report"[pt] NOT "case reports"[pt])

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 3.0

    • Failed: False

    • Regex: False

    • Search: ("statins"[Mesh] AND "cardiovascular disease"[Mesh] AND "primary prevention"[Mesh] AND "elderly"[Mesh] AND "patient")[PubMed]

    • PubMed Search URL: PubMed Search

Search Index: 2

Prompt: Could you assist me in creating a PubMed search query to find studies comparing the effectiveness of different antidepressants for treatment-resistant depression? Use pertinent MeSH terms and limit the results to clinical trials published after 2015. I just need the query itself, without any extra information.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 222

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("Depressive Disorder, Treatment-Resistant"[Mesh] OR "depression, treatment-resistant"[tiab] OR "treatment-resistant depression"[tiab] OR "treatment resistant depression"[tiab]) AND ("Antidepressive Agents"[Mesh] OR "Antidepressive Agents" [Pharmacological Action] OR antidepressant[tiab] OR antidepressants[tiab]) AND (comparison[tiab] OR compared[tiab] OR comparing[tiab] OR effectiveness[tiab] OR efficacy[tiab]) AND (Clinical Trial[ptyp] OR Randomized Controlled Trial[ptyp]) AND (("2015/01/01"[PDAT] : "3000/12/31"[PDAT]))

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 195

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("treatment resistant depression"[MeSH Terms] OR "treatment-resistant depression"[All Fields]) AND ("antidepressive agents"[MeSH Terms] OR "antidepressants"[All Fields]) AND (2015/01/01[pdat] : 3000/12/31[pdat]) AND "clinical trial"[Publication Type]

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 2.0
    • Failed: False
    • Regex: True
    • Search: ("Treatment Resistant Depression" [Mesh]) AND ("Antidepressive Agents" [Mesh] OR "Antidepressive Agents, Second-Generation" [Mesh] OR "Antidepressive Agents, Tricyclic" [Mesh]) AND ("Clinical Trials" [Publication Type]) AND ("2015/01/01" [Date - Publication] : "3000" [Date - Publication])

  • Model: cohere/command-r-plus

    • Search Result Count: 71

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (("antidepressive agents"[MeSH Terms] OR "antidepressive agents"[TIAB]) AND ("drug resistance"[MeSH Terms] OR "treatment resistant"[TIAB]) AND ("depression"[MeSH Terms] OR "depression"[TIAB]) AND ("drug therapy, combination"[MeSH Terms] OR "drug therapy"[TIAB] OR "therapy"[TIAB]) AND ("clinical trial"[pt] AND ("2016"[PDAT] : "2023"[PDAT])))

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("Antidepressive Agents"[Mesh] OR "Depression"[Mesh] OR antidepressant*) AND "Treatment Resistant"[Mesh] AND ("Clinical Trial"[Publication Type] OR "Randomized Controlled Trial"[Publication Type]) AND "2016/01/01"[PDAT]: "3000"[PDAT]

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ((SELECTED ANTIDEPRESSANTS) AND (CLINICAL TRIAL) AND (DEPRESSION, TREATMENT RESISTANT)) AND (PUBMED) AND (PUBLISHED AFTER 2015)

    • PubMed Search URL: PubMed Search

Search Index: 3

Prompt: Please help me develop an advanced PubMed query to search for articles discussing the impact of telemedicine on patient outcomes in rural areas. Include relevant MeSH headings and filter the results to include only observational studies from the 2019 onward. Kindly provide the search query alone, sans additional context.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 276

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (("Telemedicine"[Mesh] OR "Telemedicine"[Title/Abstract] OR "Telehealth"[Title/Abstract] OR "Mobile Health"[Title/Abstract] OR "mHealth"[Title/Abstract] OR "eHealth"[Title/Abstract]) AND ("Rural Population"[Mesh] OR "Rural Health"[Mesh] OR "Rural Health Services"[Mesh] OR "Rural Areas"[Title/Abstract] OR "Rural"[Title/Abstract]) AND ("Treatment Outcome"[Mesh] OR "Outcome Assessment, Health Care"[Mesh] OR "Patient Outcome Assessment"[Mesh] OR "Outcome*"[Title/Abstract] OR "Impact*"[Title/Abstract] OR "Effect*"[Title/Abstract])) AND ("Observational Study" [Publication Type] OR "Observational Studies as Topic"[Mesh] OR "Cohort Studies"[Mesh] OR "Case-Control Studies"[Mesh] OR "Cross-Sectional Studies"[Mesh]) AND (("2019/01/01"[PDAT] : "3000/12/31"[PDAT]))

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 1

    • Quoted Phrases Not Found Count: 2.0

    • Failed: False

    • Regex: False

    • Search: ("telemedicine"[MeSH Terms] OR "telemedicine"[Title/Abstract]) AND ("rural"[MeSH Terms] OR "rural"[Title/Abstract]) AND ("patient outcome"[MeSH Terms] OR "patient outcome"[Title/Abstract]) AND ("observational study"[Publication Type] OR "cohort study"[Title/Abstract] OR "case-control study"[Title/Abstract] OR "cross-sectional study"[Title/Abstract]) AND ("2019/01/01"[PDAT] : "3000/12/31"[PDAT])

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ``

    • PubMed Search URL: PubMed Search

  • Model: cohere/command-r-plus

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: nan

    • Failed: True

    • Regex: False

    • Search: (("Telemedicine"[Mesh] OR "telehealth"[Title/Abstract] OR "telemedicine"[Title/Abstract]) AND ("Rural Population"[Mesh] OR "rural"[Title/Abstract] OR "remote"[Title/Abstract]) AND ("Patient Outcome Assessment"[Mesh] OR "patient outcomes"[Title/Abstract] OR "health outcomes"[Title/Abstract]) AND ("Observational Study"[ptyp] AND ("2019/01/01"[PDAT] : "3000/12/31"[PDAT])))

    • PubMed Search URL: [PubMed Search](https://pubmed.ncbi.nlm.nih.gov/?term=(("Telemedicine"[Mesh]+OR+"telehealth"[Title/Abstract]+OR+"telemedicine"[Title/Abstract])+ AND+("Rural+Population"[Mesh]+OR+"rural"[Title/Abstract]+OR+"remote"[Title/Abstract])+ AND+("Patient+Outcome+Assessment"[Mesh]+OR+"patient+outcomes"[Title/Abstract]+OR+"health+outcomes"[Title/Abstract])+ AND+("Observational+Study"[ptyp]+AND+("2019/01/01"[PDAT]+:+"3000/12/31"[PDAT]))))

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 5.0

    • Failed: False

    • Regex: False

    • Search: ("Telemedicine"[Mesh] OR "Telemedicine"[TIAB] OR "Telehealth"[TIAB] OR "Remote Consultation"[TIAB] OR "Virtual Care"[TIAB]) AND ("Rural Health Services"[Mesh] OR "Rural Health"[Mesh] OR "Rural Population"[Mesh] OR "Rural Areas"[Mesh] OR "Telemedicine"[Mesh] OR "Telemedicine"[TIAB] OR "Telehealth"[TIAB] OR "Remote Consultation"[TIAB] OR "Virtual Care"[TIAB]) AND ("Patient Outcome Assessment"[Mesh] OR "Outcome Assessment (Health Care)"[Mesh] OR "Health Status"[Mesh] OR "Health Outcomes"[Mesh] OR "Patient-Centered Care"[Mesh] OR "Randomized Controlled Trial"[Publication Type] OR "Clinical Trial"[Publication Type] OR "Observational Study"[Publication Type] OR "Retrospective Study"[Publication Type] OR "Case-Control Study"[Publication Type]) AND ("2019/01/01"[PDAT] : "3000"[PDAT]) AND "humans"[MeSH Terms] NOT "animals"[MeSH Terms]

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ( MeSH "Telemedicine" [Mesh] AND MeSH "Rural health" [Mesh] AND MeSH "Health outcomes" [Mesh] ) AND 2019:30 AND Type=("Observational study") AND English[lang]

    • PubMed Search URL: PubMed Search

Search Index: 4

Prompt: I require assistance in formulating a PubMed query to identify research on the role of vitamin D supplementation in preventing osteoporosis in postmenopausal women. Apply suitable MeSH terms and confine the results to randomized controlled trials and cohort studies published since 2005. Please supply only the direct search query for the API.

  • Model: anthropic/claude-3-opus
    • Search Result Count: 1788
    • Quoted Phrases Not Found Count: 2.0
    • Failed: False
    • Regex: True
    • Search: `Here is the PubMed search query using appropriate MeSH terms and filters:

("Vitamin D"[Mesh] OR "Cholecalciferol"[Mesh] OR "Ergocalciferols"[Mesh]) AND "Osteoporosis, Postmenopausal"[Mesh] AND ("prevention and control"[Subheading] OR "Prevention"[Mesh] OR "Prophylaxis"[Mesh]) AND (("Randomized Controlled Trial"[Publication Type] OR "Randomized Controlled Trials as Topic"[Mesh] OR "Controlled Clinical Trial"[Publication Type]) OR ("Cohort Studies"[Mesh] OR "Longitudinal Studies"[Mesh] OR "Follow-Up Studies"[Mesh] OR "Prospective Studies"[Mesh])) AND ("2005/01/01"[PDat] : "3000/12/31"[PDat])`

  • Model: openai/gpt-4-0314

    • Search Result Count: 101

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("Vitamin D"[Mesh] OR "Vitamin D Deficiency"[Mesh]) AND ("Osteoporosis, Postmenopausal"[Mesh] OR "Osteoporosis"[Mesh]) AND "supplementation" AND (randomized controlled trial[ptyp] OR cohort studies[ptyp]) AND "2005/01/01"[PDAT] : "3000/12/31"[PDAT]

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 2.0
    • Failed: False
    • Regex: True
    • Search: ` I'm glad to help! Here's a suitable PubMed query for your requirements:

("Vitamin D"[Mesh] OR "Vitamin D Deficiency"[Mesh] OR "Vitamin D3"[Mesh]) AND ("Osteoporosis"[Mesh] OR "Osteoporosis, Postmenopausal"[Mesh]) AND ("Randomized Controlled Trial"[Publication Type] OR "Cohort Studies"[Publication Type]) AND ("2005/01/01"[PDAT] : "3000"[PDAT])

This query uses MeSH (Medical Subject Headings) terms to ensure that the search results are specific to the topic of interest. It also limits the results to randomized controlled trials and cohort studies published since 2005. Please note that the date range is set to 2005-3000 to ensure that all relevant studies published since 2005 are captured.`

  • Model: cohere/command-r-plus
    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Here is the PubMed search query formulated based on your requirements:

('Osteoporosis, Postmenopausal'/Mesh) AND ('Vitamin D'/Mesh) AND ('Dietary Supplements'/Mesh) AND (('Cohort Studies'/pt) OR ('Cohort Studies'/Mesh)) AND (('Randomized Controlled Trial'/pt) OR ('Randomized Controlled Trial'/Mesh)) AND ('2005'/EDAT : '3000'/EDAT)

This query combines MeSH terms for "Osteoporosis, Postmenopausal," "Vitamin D," and "Dietary Supplements" with publication type terms for "Cohort Studies" and "Randomized Controlled Trial." The date filter ensures that only papers published from 2005 onwards are included.`

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 1063

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("Vitamin D"[Mesh] OR "Calcifediol"[Mesh] OR "Calcitriol"[Mesh] OR "Vitamin D"[tiab] OR "Calcifediol"[tiab] OR "Calcitriol"[tiab]) AND ("Osteoporosis"[Mesh] OR "Postmenopause"[Mesh] OR "Osteoporosis"[tiab] OR "Postmenopausal"[tiab] OR "Menopause"[tiab]) AND ("randomized controlled trial"[Publication Type] OR "cohort studies"[Publication Type] OR "clinical trial"[Publication Type] OR "Controlled Clinical Trial"[pt] OR "Randomized Controlled Trial"[pt] OR "multicenter study"[tiab] OR "clinical trial"[tiab] OR "cohort analysis"[tiab]) AND ("2005/01/01"[PDAT] : "2023/12/31"[PDAT])

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 4.0

    • Failed: False

    • Regex: False

    • Search: "vitamin D supplementation"[Mesh] "postmenopausal women"[Mesh] "osteoporosis"[Mesh] "randomized controlled trial"[Mesh] "cohort study"[Mesh] "2005:"

    • PubMed Search URL: PubMed Search

Search Index: 5

Prompt: Can you generate an advanced PubMed query to locate articles exploring the relationship between sleep duration and obesity in children and adolescents? Utilize appropriate MeSH headings and limit the search to studies published within the past 8 years. Just give me the search string, without any extra details.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 2803

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("Sleep Duration"[Mesh] OR "Sleep"[Mesh] OR "Sleep"[Title/Abstract] OR "Sleep Duration"[Title/Abstract]) AND ("Pediatric Obesity"[Mesh] OR "Obesity"[Mesh] OR "Overweight"[Mesh] OR "Obesity"[Title/Abstract] OR "Overweight"[Title/Abstract]) AND ("Child"[Mesh] OR "Adolescent"[Mesh] OR "Child"[Title/Abstract] OR "Children"[Title/Abstract] OR "Adolescent"[Title/Abstract] OR "Adolescents"[Title/Abstract] OR "Teenagers"[Title/Abstract]) AND ("2015/01/01"[PDat] : "2023/12/31"[PDat])

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 1292

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("sleep"[MeSH Terms] OR "sleep"[All Fields]) AND ("duration"[All Fields] OR "time factors"[MeSH Terms] OR "time"[All Fields]) AND ("obesity"[MeSH Terms] OR "obesity"[All Fields]) AND ("child"[MeSH Terms] OR "children"[All Fields] OR "adolescent"[MeSH Terms] OR "adolescents"[All Fields]) AND (("2013/01/01"[PDAT] : "2021/12/31"[PDAT]) AND "humans"[MeSH Terms])

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: ` here! Here is the advanced PubMed query you requested:

((("Sleep"[Mesh]) AND ("Obesity"[Mesh] OR "Body Mass Index"[Mesh] OR "Adiposity"[Mesh])) AND ("Child"[Mesh] OR "Adolescent"[Mesh])) AND ("Publication Date"[Date - Publication] : "8years"[Date - Publication - Range])

This search query will retrieve articles published in the past 8 years that explore the relationship between sleep duration and obesity, body mass index, or adiposity in children and adolescents. The MeSH headings used in this query will help to ensure that the search results are focused and relevant to your research question.`

Search Index: 6

Prompt: Could you help me construct a PubMed query to identify studies investigating the use of probiotics for preventing antibiotic-associated diarrhea? Apply pertinent MeSH headings and limit the search to randomized controlled trials published after 2010. I need only the direct search query for the API.

  • Model: anthropic/claude-3-opus
    • Search Result Count: 116
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Here's the PubMed query for identifying randomized controlled trials investigating the use of probiotics for preventing antibiotic-associated diarrhea, limited to studies published after 2010:

("probiotics"[MeSH Terms] OR "probiotics"[All Fields] OR "probiotic"[All Fields]) AND ("anti-bacterial agents"[MeSH Terms] OR ("anti-bacterial"[All Fields] AND "agents"[All Fields]) OR "anti-bacterial agents"[All Fields] OR "antibiotic"[All Fields]) AND ("diarrhoea"[All Fields] OR "diarrhea"[MeSH Terms] OR "diarrhea"[All Fields]) AND (("prevention and control"[Subheading] OR ("prevention"[All Fields] AND "control"[All Fields]) OR "prevention and control"[All Fields] OR "prevention"[All Fields]) OR "prevent"[All Fields]) AND (("randomized controlled trial"[Publication Type] OR "randomized controlled trials as topic"[MeSH Terms] OR "randomized controlled trial"[All Fields] OR "randomised controlled trial"[All Fields]) OR "rct"[All Fields]) AND ("2010/01/01"[PDAT] : "3000/12/31"[PDAT])

This query includes the following components:

  1. MeSH terms and keywords for probiotics
  2. MeSH terms and keywords for antibiotics
  3. MeSH terms and keywords for diarrhea
  4. Keywords for prevention
  5. Publication type and MeSH terms for randomized controlled trials
  6. Date restriction to articles published after 2010

You can use this query directly in the PubMed API to retrieve the relevant studies.`

  • Model: openai/gpt-4-0314

    • Search Result Count: 86

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("probiotics"[MeSH Terms] OR "probiotics"[All Fields]) AND ("anti-bacterial agents"[Pharmacological Action] OR "anti-bacterial agents"[MeSH Terms] OR "antibiotics"[All Fields]) AND ("diarrhea"[MeSH Terms] OR "diarrhea"[All Fields]) AND ("2011/01/01"[PDAT] : "3000"[PDAT]) AND Randomized Controlled Trial[ptyp]

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `'meSH Terms:"Probiotics"[MeSH] AND "Diarrhea"[MeSH] AND "Antibiotics"[MeSH] AND "Randomized Controlled Trial"[Publication Type]

Limits: Search done in the past 10 years (PubMed's API uses the Entrez date, which is the date the citation was added to PubMed, so you may need to adjust the date range accordingly.)

Direct API query: https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=Probiotics%5BMeSH%5D+AND+Diarrhea%5BMeSH%5D+AND+Antibiotics%5BMeSH%5D+AND+Randomized+Controlled+Trial%5BPublication+Type%5D&retmax=10000&mindate=2010%2F01%2F01&maxdate=2023%2F02%2F22&datetype=edat`

  • Model: cohere/command-r-plus
    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 1.0
    • Failed: False
    • Regex: True
    • Search: `Here is the PubMed search query based on your requirements:

("Probiotics"[Mesh] AND "Antibiotic-Associated Diarrhea"[Mesh]) AND ("2011/01/01"[PDAT] : "3000/12/31"[PDAT]) AND ("Randomized Controlled Trial"[ptyp] AND "humans"[MeSH Terms])

This query employs the MeSH headings "Probiotics" and "Antibiotic-Associated Diarrhea" to focus on relevant studies. It also includes a date filter to retrieve articles published after 2010 and specifically selects for randomized controlled trials involving humans.`

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("Probiotics"[Mesh] OR "Probiotics"[Title/Abstract]) AND ("Diarrhea, Antibiotic-Associated"[Mesh] OR ("Antibiotic-Associated"[Title/Abstract] AND "Diarrhea"[Title/Abstract]) OR "Diarrhea"[Title/Abstract]) AND ("Randomized Controlled Trial"[Publication Type] OR "Controlled Clinical Trial"[Publication Type]) AND "2010/01/01"[PDAT] : NOT "2011/01/01"[PDAT] AND "2021/12/31"[PDAT]

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 1.0

    • Failed: False

    • Regex: False

    • Search: ("probiotics"[Mesh] AND "antibiotic"[Mesh] AND "diarrhea"[Mesh]) AND type:("randomized controlled trial") AND pubmed:(2010:)

    • PubMed Search URL: PubMed Search

Search Index: 7

Prompt: Please assist me in creating an advanced PubMed query to locate articles examining the impact of social support on mental health outcomes in elderly populations. Use appropriate MeSH terms and filter the results to include only prospective cohort studies. Kindly supply the search string alone, without additional information.

  • Model: anthropic/claude-3-opus

    • Search Result Count: 4731

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("Social Support"[Mesh] OR "Family"[Mesh] OR "Friends"[Mesh] OR "Interpersonal Relations"[Mesh]) AND ("Mental Health"[Mesh] OR "Depression"[Mesh] OR "Anxiety"[Mesh] OR "Stress, Psychological"[Mesh]) AND ("Aged"[Mesh] OR "Aging"[Mesh] OR elderly OR older adults OR geriatric) AND ("Cohort Studies"[Mesh] OR "Prospective Studies"[Mesh])

    • PubMed Search URL: PubMed Search

  • Model: openai/gpt-4-0314

    • Search Result Count: 382

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("social support"[MeSH Terms] OR "social support"[All Fields]) AND ("mental health"[MeSH Terms] OR "mental health"[All Fields]) AND (aged[MeSH Terms] OR elderly[All Fields] OR geriatric[All Fields]) AND "prospective studies"[MeSH Terms]

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ``

    • PubMed Search URL: PubMed Search

  • Model: cohere/command-r-plus

    • Search Result Count: 4364

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (("aged"[MeSH Terms] OR "aged"[All Fields] OR "elderly"[All Fields]) AND ("social support"[MeSH Terms] OR ("social"[All Fields] AND "support"[All Fields]) OR "social support"[All Fields]) AND ("mental health"[MeSH Terms] OR ("mental"[All Fields] AND "health"[All Fields]) OR "mental health"[All Fields] OR ("mental"[All Fields] AND "health"[All Fields]) OR "psychological health"[All Fields]) AND ("prospective studies"[MeSH Terms] OR ("prospective"[All Fields] AND "studies"[All Fields]) OR "prospective studies"[All Fields] OR ("cohort"[All Fields] AND "studies"[All Fields]) OR "cohort studies"[All Fields]))

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 1903

    • Quoted Phrases Not Found Count: 3.0

    • Failed: False

    • Regex: False

    • Search: ("Social Support"[Mesh] OR "Social Networking"[Mesh] OR "Social Determinants of Health"[Mesh] OR "Social Isolation"[Mesh] OR "Social Support in Older Adults"[Mesh]) AND ("Mental Health"[Mesh] OR "Mental Disorders"[Mesh] OR "Depression"[Mesh] OR "Anxiety"[Mesh] OR "Psychological Stress"[Mesh] OR "Mental Health Services"[Mesh]) AND ("Elderly"[Mesh] OR "Aged"[Mesh] OR "Aged, 80 and over"[Mesh]) AND ("Prospective Studies"[Mesh] OR "Cohort Studies"[Mesh])

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 0
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Mesh terms: social support, mental health, elderly population, cohort study

Search string: social support AND mental health outcomes AND disabled AND elderly AND cohort study.`

Search Index: 8

Prompt: Can you guide me in formulating a PubMed search query to find research on the effectiveness of lifestyle interventions for managing type 2 diabetes? Include both English and Spanish language articles.Please give me just the query that can be directly passed to the API.

  • Model: anthropic/claude-3-opus
    • Search Result Count: 1525
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Here is the PubMed search query to find research on the effectiveness of lifestyle interventions for managing type 2 diabetes, including both English and Spanish language articles:

("diabetes mellitus, type 2"[MeSH Terms] OR "type 2 diabetes"[All Fields] OR "diabetes type 2"[All Fields]) AND ("life style"[MeSH Terms] OR ("life"[All Fields] AND "style"[All Fields]) OR "life style"[All Fields] OR "lifestyle"[All Fields]) AND ("intervention"[All Fields] OR "interventions"[All Fields]) AND ("effectiveness"[All Fields] OR "efficacy"[All Fields] OR "efficiency"[All Fields]) AND (english[Filter] OR spanish[Filter])

You can directly pass this query to the PubMed API to retrieve relevant search results.`

  • Model: openai/gpt-4-0314

    • Search Result Count: 11239

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: (("lifestyle interventions" OR "behavior change" OR "health behavior" OR "exercise" OR "diet" OR "smoking cessation" OR "alcohol reduction") AND ("type 2 diabetes" OR "T2D" OR "diabetes mellitus type 2" OR "DM2") AND ("effectiveness" OR "efficacy" OR "management" OR "outcome") AND ("english"[Language] OR "spanish"[Language]))

    • PubMed Search URL: PubMed Search

  • Model: databricks/dbrx-instruct

    • Search Result Count: 1332
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: ` Answer: ("lifestyle intervention" OR "lifestyle interventions") AND ("type 2 diabetes" OR "diabetes mellitus type 2") AND ("management" OR "treatment") AND (English[lang] OR Spanish[lang])

This query will search for articles in PubMed that include the terms "lifestyle intervention" or "lifestyle interventions" and "type 2 diabetes" or "diabetes mellitus type 2" and "management" or "treatment" in the title or abstract, and are written in English or Spanish.`

  • Model: cohere/command-r-plus
    • Search Result Count: 14562
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Here is the PubMed search query based on your requirements:

("type 2 diabetes" OR "T2D") AND (lifestyle OR behavior OR intervention) AND (management OR treatment) AND ("effectiveness" OR "efficacy") AND (english[language] OR spanish[language])`

  • Model: mistralai/mixtral-8x7b-instruct

    • Search Result Count: 0

    • Quoted Phrases Not Found Count: 0.0

    • Failed: False

    • Regex: False

    • Search: ("lifestyle interventions"[Title/Abstract] OR "lifestyle modification"[Title/Abstract] OR "dietary interventions"[Title/Abstract] OR "exercise interventions"[Title/Abstract] OR "behavioral interventions"[Title/Abstract]) AND ("type 2 diabetes"[Title/Abstract] OR "diabetes mellitus type 2"[Title/Abstract] OR "non-insulin dependent diabetes"[Title/Abstract]) AND (effectiveness[Title/Abstract] OR efficacy[Title/Abstract] OR outcome[Title/Abstract] OR "randomized controlled trial"[Title/Abstract] OR "clinical trial"[Title/Abstract]) AND (language:English OR language:Spanish)

    • PubMed Search URL: PubMed Search

  • Model: mistralai/mistral-7b-instruct:free

    • Search Result Count: 21750
    • Quoted Phrases Not Found Count: 0.0
    • Failed: False
    • Regex: True
    • Search: `Here is a possible query that can be passed to the API for PubMed search:

"(("type 2 diabetes" AND "lifestyle intervention") OR (dietary interventions AND "type 2 diabetes") OR (exercise interventions AND "type 2 diabetes")) AND (English OR Spanish) AND (article); filter: blewpubtype"`