Closed Domain Question Answering with LLMs

Background

The following prompt tests an LLM's capabilities to answer closed-domain questions which involves answering questions belonging a specific topic or domain.

Note that due to the challenging nature of the task, LLMs are likely to hallucinate when they have no knowledge regarding the question.

Prompt

Patient’s facts:- 20 year old female- with a history of anerxia nervosa and depression- blood pressure 100/50, pulse 50, height 5’5’’- referred by her nutrionist but is in denial of her illness- reports eating fine but is severely underweight Please rewrite the data above into a medical note, using exclusively the information above.

Code / API

from openai import OpenAIclient = OpenAI() response = client.chat.completions.create(model="gpt-4",messages=[    {    "role": "user",    "content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nPlease rewrite the data above into a medical note, using exclusively the information above."    }],temperature=1,max_tokens=500,top_p=1,frequency_penalty=0,presence_penalty=0)

Open Domain Question Answering with LLMs

Background

The following prompt tests an LLM's capabilities to answer open-domain questions which involves answering factual questions without any evidence provided.

Prompt

In this conversation between a human and the AI, the AI is helpful and friendly, and when it does not know the answer it says "I don’t know". AI: Hi, how can I help you?Human: Can I get McDonalds at the SeaTac airport?

Code / API

from openai import OpenAIclient = OpenAI() response = client.chat.completions.create(model="gpt-4",messages=[    {    "role": "user",    "content": "In this conversation between a human and the AI, the AI is helpful and friendly, and when it does not know the answer it says \"I don’t know\".\n\nAI: Hi, how can I help you?\nHuman: Can I get McDonalds at the SeaTac airport?"    }],temperature=1,max_tokens=250,top_p=1,frequency_penalty=0,presence_penalty=0)

Science Question Answering with LLMs

Background

The following prompt tests an LLM's capabilities to perform science question answering.

Prompt

Answer the question based on the context below. Keep the answer short and concise. Respond "Unsure about answer" if not sure about the answer. Context: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use. Question: What was OKT3 originally sourced from?Answer:

Code / API

from openai import OpenAIclient = OpenAI() response = client.chat.completions.create(model="gpt-4",messages=[    {    "role": "user",    "content": "Answer the question based on the context below. Keep the answer short and concise. Respond \"Unsure about answer\" if not sure about the answer.\n\nContext: Teplizumab traces its roots to a New Jersey drug company called Ortho Pharmaceutical. There, scientists generated an early version of the antibody, dubbed OKT3. Originally sourced from mice, the molecule was able to bind to the surface of T cells and limit their cell-killing potential. In 1986, it was approved to help prevent organ rejection after kidney transplants, making it the first therapeutic antibody allowed for human use.\n\nQuestion: What was OKT3 originally sourced from?\nAnswer:"    }],temperature=1,max_tokens=250,top_p=1,frequency_penalty=0,presence_penalty=0)

Indirect Reasoning with LLMs

Background

Zhang et al. (2024) recently proposed an indirect reasoning method to strengthen the reasoning power of LLMs. It employs the logic of contrapositives and contradictions to tackle IR tasks such as factual reasoning and mathematic proof. It consists of two key steps: 1) enhance the comprehensibility of LLMs by augmenting data and rules (i.e., logical equivalence of contrapositive), and 2) design prompt templates to stimulate LLMs to implement indirect reasoning based on proof by contradiction.

Experiments on LLMs like GPT-3.5-turbo and Gemini-pro show that the proposed method enhances the overall accuracy of factual reasoning by 27.33% and mathematic proof by 31.43% compared to traditional direct reasoning methods.

Below is an example of zero-shot template for proof-by-contradiction.

Prompt

If a+|a|=0, try to prove that a<0.Step 1: List the conditions and questions in the original proposition.Step 2: Merge the conditions listed in Step 1 into one. Define it as wj.Step 3: Let us think it step by step. Please consider all possibilities. If the intersection between wj (defined in Step 2) and the negation of the question is not empty at least in one possibility, the original proposition is false. Otherwise, the original proposition is true.Answer:

Code / API

from openai import OpenAIclient = OpenAI() response = client.chat.completions.create(model="gpt-3.5-turbo",messages=[{  "role": "user",  "content": "If a+|a|=0, try to prove that a<0.\n\nStep 1: List the conditions and questions in the original proposition.\n\nStep 2: Merge the conditions listed in Step 1 into one. Define it as wj.\n\nStep 3: Let us think it step by step. Please consider all possibilities. If the intersection between wj (defined in Step 2) and the negation of the question is not empty at least in one possibility, the original proposition is false. Otherwise, the original proposition is true.\n\nAnswer:"}],temperature=0,max_tokens=1000,top_p=1,frequency_penalty=0,presence_penalty=0)

Physical Reasoning with LLMs

Background

This prompt tests an LLM's physical reasoning capabilities by prompting it to perform actions on a set of objects.

Prompt

Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner.

Code / API

from openai import OpenAIclient = OpenAI() response = client.chat.completions.create(model="gpt-4",messages=[    {    "role": "user",    "content": "Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner."    }],temperature=1,max_tokens=500,top_p=1,frequency_penalty=0,presence_penalty=0)