[BUG] dalle_tool demands OPEN_API_KEY setting when using Azure

[henkbb36org]

Description

I am trying to use the dalle-tool with an OpenAI implementation on Azure. The Crewai Agents in my script work well, WebsiteSearchTool works well but dalle-tool demands an OPEN_API_KEY setting in the environment (which does not work when I set it to my AzureOpenAI key, it will give invalid API key). I configured the dalle-tool like so:
dalle_tool = DallETool(
config = {
"embedder": {
"provider": "azure_openai",
"config": {
"api_key":os.environ.get("AZURE_OPENAI_KEY"),
"model": "dall-e-3",
},
}
}
)
The same type of config works well for the WebsiteSearchTool. I cannot find any documentation or examples on how to get this working Azure. Any help?

Steps to Reproduce

The Agent is configure like so:
illustrator = Agent(
role='Illustrator',
goal='Generate an image based on….,
verbose=True,
memory=True,
backstory=(
"You are an …."
),
tools=[dalle_tool],
llm=default_llm,
allow_delegation=False
)
When running my test script it fails with the error:
I encountered an error while trying to use the tool. This was the error: Error code: 401 - {'error': {'code': 'invalid_api_key', 'message': 'Incorrect API key provided: 696be861********************4a04. You can find your API key at https://platform.openai.com/account/api-keys.', 'param': None, 'type': 'invalid_request_error'}}.

When I don't set any value in the environment variable OPEN_API_KEY it will give the error that the value is not set. Llm is configured using LiteLLM like so:
default_llm = LLM(
model="azure/gpt-4o",
api_key=os.environ.get("AZURE_OPENAI_KEY"),
base_url=os.environ.get("AZURE_OPENAI_ENDPOINT"),
)
The proper values are in the .env which is loaded by load_dotenv(). Agent functions work well, as well the WebsiteSearchTool

Expected behavior

I hoped the DallETool would to use the azure apenai implementation but I cannot get it to do that.

Screenshots/Code snippets

None

Operating System

macOS Sonoma

Python Version

3.11

crewAI Version

0.67.1

crewAI Tools Version

0.12.1

Virtual Environment

Venv

Evidence

I encountered an error while trying to use the tool. This was the error: Error code: 401 - {'error': {'code': 'invalid_api_key', 'message': 'Incorrect API key provided: 696be861********************4a04. You can find your API key at https://platform.openai.com/account/api-keys.', 'param': None, 'type': 'invalid_request_error'}}.
Tool Dall-E Tool accepts these inputs: Dall-E Tool(image_description: 'string') - Generates images using OpenAI's Dall-E model.

Possible Solution

None

Additional context

None

[henkbb36org]

Fixed by writing a custom_dalle_tool.py:
import json
from typing import Type
import requests
from pydantic import BaseModel
from crewai_tools.tools.base_tool import BaseTool
from openai import AzureOpenAI

class ImagePromptSchema(BaseModel):
"""Input for Dall-E Tool."""

image_description: str = "Description of the image to be generated by Dall-E."

class MyCustomDallETool(BaseTool):
name: str = "Dall-E Tool"
description: str = "Generates images using OpenAI's Dall-E model."
args_schema: Type[BaseModel] = ImagePromptSchema

model: str = "dall-e-3"
size: str = "1024x1024"
quality: str = "standard"
n: int = 1

def _run(self, **kwargs) -> str:
    client = AzureOpenAI(
        api_version="YOUR_API_VERSION",
        azure_endpoint=YOUR_AZURE_ENDPOINT,",
        api_key="YOUR_API_KEY",
    )

    image_description = kwargs.get("image_description")

    if not image_description:
        return "Image description is required."

    response = client.images.generate(
        model=self.model, # the name of your DALL-E 3 deployment
        prompt=image_description,
        n=self.n,
    )


    image_data = json.dumps(
        {
            "image_url": response.data[0].url,
            "image_description": response.data[0].revised_prompt,
        }
    )
    response = requests.get(response.data[0].url)
    if response.status_code == 200:
        with open("downloaded_image.jpg", "wb") as file:  # Replace with your desired file name
            file.write(response.content)
        print("Image successfully downloaded and saved as 'downloaded_image.jpg'")
    else:
        print(f"Failed to download image. Status code: {response.status_code}")
    return image_data

Can be used in a from crewai script:
import from custom_dalle_tool import MyCustomDallETool
dalle_tool = MyCustomDallETool(model="dall-e-3", size="1024x1024", quality="standard", n=1)
use tools=[dalle_tool] in Agent definition to call it