we want to Run the Wren AI with the llama3 model so how it is possible ?

[hbanand]

Is it possible to run the WrenAI to be running along with the llama3 on the self hosted environment ? if yes from where we can find the documentation on the same as was unable to find anything relevant in this regards.

[cyyeh]

@hbanand Thanks for reaching out! Please check out this for further details: #277 (comment)

In brief, there might be issues if you use Ollama's models, since LLM output might not always conform to JSON output. I've already added Ollama support in this branch: https:/Canner/WrenAI/tree/feature/ai-service/add-ollama. Welcome to check it out! Also I think we should customize prompts depending on the model we choose. There is still some space for improvement. I would like to hear your thoughts anyway.

For JSON output broken issue, I am thinking to add a component to help solve this issue: https:/noamgat/lm-format-enforcer. I would also like to know more your thoughts here.

In order to try to run Ollama models, please see the demo section of the README in wren-ai-service. You don't need to run make run, you only need to run make prepare, make ui and make ai to start the WrenAI service for local trial usage.

Feel free to ask me if you run into issues, thank you :)

[cyyeh]

@hbanand

Hi, we now support using Ollama and OpenAI API-compatible LLMs now with the latest release: https:/Canner/WrenAI/releases/tag/0.6.0

Setups on how to run Wren AI using custom LLMs: https://docs.getwren.ai/installation/custom_llm#running-wren-ai-with-your-custom-llm-or-document-store

Currently, there is one obvious limitation for custom LLMs: you need to use the same provder(such as OpenAI, or Ollama) for LLM and embedding model. We'll fix that and release a new version soon. Stay tuned 🙂