AI assistant

This is the user documentation for the AI assistant in RevMan. The assistant is currently being rolled out to a limited number of Cochrane authors as a pilot.

Design principles

The AI assistant is a RAG lookup tool that can search across all of Cochrane's official written resources for authors and provide a focused summary of all relevant information. It doesn't take any further actions - it cannot edit a review, and should not give any advice that does not come directly from Cochrane documentation. It does not have access to Cochrane's editorial systems and therefore cannot answer questions about the editorial status of your review.

No personal information is ever exposed to the assistant (unless you send it in your query). This means the third-party LLM provider, Google, will not see personal data. Google cannot train its AI models using any data sent via the assistant.

We are using the smallest LLMs that give sufficient performance, to minimize the environmental impact of the assistant. Once this pilot confirms how the assistant will be used in practice, we may further experiment with smaller LLMs hosted on more environmentally conscious infrastructure. 

The assistant is not intended to replace the Cochrane Support team and is being developed with their collaboration. We predict that the assistant will answer common queries and provide useful context for more complex queries, so that the team has more time for all the work they do behind the scenes other than answering authors' queries! 

Pilot objectives

We want to see how Cochrane authors use the assistant under real-world conditions, and answer the following questions:

  • Does the assistant help authors to access technical and methodological guidance more easily?
  • Does using the assistant impact authors' ability to justify their methodological decisions?
  • Does using the assistant lead to better compliance with Cochrane's methodological expectations, and therefore fewer revisions? 
  • Are support tickets created via the assistant, with context included, quicker for the Cochrane Support team to resolve?
  • Are answers drawn from any particular knowledge sources more frequently?

We will be able to conduct only qualitative analyses of some of these questions, which we will do via user interviews. We will also conduct a manual audit of a random sample of responses, to ensure that they are as good as the response that would otherwise be provided by Cochrane Support; that they are clear, comprehensive and do not make assumptions.

We will publish full methods and results of our evaluation transparently. The development team have no incentive to drive uptake of the assistant, and if through the pilot we find that the assistant is not helpful or has any negative impacts that cannot be mitigated, we will not proceed with full deployment of the assistant. 

If you have any questions, or are a Cochrane author interested in participating in user interviews, please contact rhall@cochrane.org.

Knowledge sources

The assistant searches for answers across the following sources:

  • Cochrane Handbook for Systematic Reviews of Interventions
  • RevMan knowledge base and its linked PDFs
  • Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
  • Template and guidance for writing a Cochrane Plain language summary
  • Cochrane style manual
  • Cochrane author guidelines
  • Cochrane editorial policies
  • Cochrane editorial guidance resource
  • Cochrane Editorial manager knowledge base
  • Methodological Expectations of Cochrane Intervention Reviews (MECIR)
  • Cochrane intervention/prognosis/qualitative review templates (filtered by current review type)
  • Cochrane Learn - About Risk of Bias 2
  • Cochrane Learn - Including non-randomized studies of interventions
  • Article - Defining and determining which quantitative study designs to include in your systematic review of effects of a healthcare intervention
  • Cochrane Library Search Help
  • Cochrane Library - About