I was messing around with ragas in python and I kept getting an error I couldn’t figure out. I’m using the amnesty_qa dataset from the datasets library and my embedding function is the HuggingFace embedding from Langchain. I know my model and embedding works because I used them to create a rag application, and I know my Ollama LLM is running because I’ve checked.
My embedding function file looks like this
def get_embedding_function():
embeddings = HuggingFaceEmbeddings()
return embeddings
This is the file where I’m using ragas
from ragas.metrics import (
answer_relevancy,
faithfulness,
context_recall,
context_precision
)
from ragas import evaluate
from langchain_community.llms.ollama import Ollama
from langchain_huggingface import HuggingFaceEmbeddings
from datasets import load_dataset
from get_embedding_function import get_embedding_function
def evaluate_rag():
model = Ollama(model="mistral")
# Run the evaluation
try:
result = evaluate(
amnesty_qa['eval'],
metrics=[
context_precision,
faithfulness,
answer_relevancy,
context_recall,
],
llm=model,
embeddings=get_embedding_function
)
except Exception as e:
print(f"Error during evaluation: {e}")
return
return result
Here’s my error message
Evaluating: 2%|█▋ | 2/80 [02:56<2:02:30, 94.24s/it]Exception raised in Job[54]: TimeoutError()
Exception raised in Job[57]: TimeoutError()
Exception raised in Job[48]: TimeoutError()
Exception raised in Job[51]: TimeoutError()
Exception raised in Job[52]: TimeoutError()
Exception raised in Job[46]: TimeoutError()
Exception raised in Job[61]: TimeoutError()
Exception raised in Job[47]: TimeoutError()
Exception raised in Job[60]: TimeoutError()
Exception raised in Job[53]: TimeoutError()
Exception raised in Job[56]: TimeoutError()
Exception raised in Job[58]: TimeoutError()
Exception raised in Job[49]: TimeoutError()
Exception raised in Job[50]: TimeoutError()
Evaluating: 4%|██▌ | 3/80 [03:00<1:07:54, 52.91s/it]Exception raised in Job[0]: TimeoutError()
Evaluating: 21%|██████████████▉ | 17/80 [03:53<09:06, 8.68s/it]Exception raised in Job[62]: TimeoutError()
Evaluating: 22%|███████████████▊ | 18/80 [05:56<18:52, 18.27s/it]Exception raised in Job[1]: TimeoutError()
Exception raised in Job[68]: TimeoutError()
Exception raised in Job[65]: TimeoutError()
Exception raised in Job[2]: TimeoutError()
Exception raised in Job[64]: TimeoutError()
Exception raised in Job[6]: TimeoutError()
Exception raised in Job[7]: TimeoutError()
Exception raised in Job[4]: TimeoutError()
Exception raised in Job[66]: TimeoutError()
Exception raised in Job[63]: TimeoutError()
Exception raised in Job[5]: TimeoutError()
Exception raised in Job[3]: TimeoutError()
Exception raised in Job[69]: TimeoutError()
Exception raised in Job[67]: TimeoutError()
Evaluating: 24%|████████████████▋ | 19/80 [06:00<16:59, 16.71s/it]Exception raised in Job[8]: TimeoutError()
Evaluating: 42%|█████████████████████████████▊ | 34/80 [08:45<11:14, 14.67s/it]Exception raised in Job[70]: TimeoutError()
Evaluating: 44%|██████████████████████████████▋ | 35/80 [08:56<10:46, 14.36s/it]Exception raised in Job[76]: TimeoutError()
Exception raised in Job[9]: TimeoutError()
Exception raised in Job[12]: TimeoutError()
Exception raised in Job[72]: TimeoutError()
Exception raised in Job[13]: TimeoutError()
Exception raised in Job[14]: TimeoutError()
Exception raised in Job[17]: TimeoutError()
Exception raised in Job[11]: TimeoutError()
Exception raised in Job[71]: TimeoutError()
Exception raised in Job[74]: TimeoutError()
Exception raised in Job[10]: TimeoutError()
Exception raised in Job[73]: TimeoutError()
Exception raised in Job[77]: TimeoutError()
Evaluating: 45%|███████████████████████████████▌ | 36/80 [09:00<09:41, 13.21s/it]Exception raised in Job[15]: TimeoutError()
Evaluating: 61%|██████████████████████████████████████████▉ | 49/80 [09:53<03:41, 7.14s/it]Exception raised in Job[78]: TimeoutError()
Evaluating: 62%|███████████████████████████████████████████▊ | 50/80 [11:45<07:08, 14.28s/it]Exception raised in Job[16]: TimeoutError()
Evaluating: 64%|████████████████████████████████████████████▋ | 51/80 [11:56<06:45, 14.00s/it]Exception raised in Job[29]: TimeoutError()
Exception raised in Job[28]: TimeoutError()
Exception raised in Job[24]: TimeoutError()
Exception raised in Job[20]: TimeoutError()
Exception raised in Job[18]: TimeoutError()
Exception raised in Job[21]: TimeoutError()
Exception raised in Job[26]: TimeoutError()
Exception raised in Job[27]: TimeoutError()
Exception raised in Job[79]: TimeoutError()
Exception raised in Job[25]: TimeoutError()
Exception raised in Job[22]: TimeoutError()
Exception raised in Job[23]: TimeoutError()
Exception raised in Job[19]: TimeoutError()
Evaluating: 65%|█████████████████████████████████████████████▌ | 52/80 [12:00<05:59, 12.84s/it]Exception raised in Job[30]: TimeoutError()
Evaluating: 82%|█████████████████████████████████████████████████████████▊ | 66/80 [13:20<01:56, 8.32s/it]Exception raised in Job[32]: TimeoutError()
Evaluating: 84%|██████████████████████████████████████████████████████████▋ | 67/80 [14:56<03:30, 16.19s/it]Exception raised in Job[42]: TimeoutError()
Exception raised in Job[44]: TimeoutError()
Exception raised in Job[43]: TimeoutError()
Exception raised in Job[35]: TimeoutError()
Exception raised in Job[41]: TimeoutError()
Exception raised in Job[37]: TimeoutError()
Exception raised in Job[39]: TimeoutError()
Exception raised in Job[38]: TimeoutError()
Exception raised in Job[36]: TimeoutError()
Exception raised in Job[33]: TimeoutError()
Exception raised in Job[45]: TimeoutError()
Exception raised in Job[40]: TimeoutError()
Exception raised in Job[34]: TimeoutError()
Evaluating: 100%|██████████████████████████████████████████████████████████████████████| 80/80 [15:00<00:00, 11.25s/it]
{'context_precision': nan, 'faithfulness': nan, 'answer_relevancy': nan, 'context_recall': 1.0000}
New contributor
Futuristic Gamers is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.