程序包 io.agentscope.core.tool.multimodal

类 OpenAIMultiModalTool

java.lang.Object

io.agentscope.core.tool.multimodal.OpenAIMultiModalTool

public class OpenAIMultiModalTool
extends Object

OpenAI multimodal tool.

Supports:

• Text to image(s)
• Image to text (via vision models)
• Text to audio (speech)
• Audio to text (transcription)

This implementation uses the custom OpenAIClient HTTP client instead of the OpenAI Java SDK, keeping the core module lightweight and dependency-free.

Please refer to the OpenAI documentation for more details.

构造器概要

构造器

限定符	构造器	说明
protected	OpenAIMultiModalTool(OpenAIClient client)	Create a new OpenAIMultiModalTool with custom client (for testing).
	OpenAIMultiModalTool(String apiKey)	Create a new OpenAIMultiModalTool with default base URL.
	OpenAIMultiModalTool(String apiKey, String baseUrl)	Create a new OpenAIMultiModalTool with custom base URL.

方法概要

修饰符和类型	方法	说明
reactor.core.publisher.Mono<ToolResultBlock>	openaiAudioToText(String audioUrl, String model, String language, String prompt, String responseFormat, Double temperature)	Convert audio to text (transcription) using OpenAI Whisper models.
reactor.core.publisher.Mono<ToolResultBlock>	openaiImageToText(String imageUrls, String prompt, String model, Integer maxTokens)	Convert image(s) to text using vision models.
reactor.core.publisher.Mono<ToolResultBlock>	openaiTextToAudio(String text, String model, String voice, String responseFormat, Double speed)	Convert text to audio (speech) using OpenAI TTS models.
reactor.core.publisher.Mono<ToolResultBlock>	openaiTextToImage(String prompt, String model, Integer n, String size, String quality, String responseFormat)	Generate image(s) based on the given prompt.

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

构造器详细资料

OpenAIMultiModalTool

public OpenAIMultiModalTool(String apiKey)

Create a new OpenAIMultiModalTool with default base URL.

参数:

apiKey - the OpenAI API key

OpenAIMultiModalTool

public OpenAIMultiModalTool(String apiKey,
 String baseUrl)

Create a new OpenAIMultiModalTool with custom base URL.

参数:

apiKey - the OpenAI API key

baseUrl - the base URL (null for default https://api.openai.com)

OpenAIMultiModalTool

protected OpenAIMultiModalTool(OpenAIClient client)

Create a new OpenAIMultiModalTool with custom client (for testing).

参数:

client - the OpenAI client

方法详细资料

openaiTextToImage

@Tool(name="openai_text_to_image",
      description="Generate image(s) based on the given prompt using OpenAI DALL-E models. Returns image URL(s) or base64 data.")
public reactor.core.publisher.Mono<ToolResultBlock> openaiTextToImage(@ToolParam(name="prompt",description="The text prompt to generate image")
 String prompt,
 @ToolParam(name="model",description="The model to use, e.g., \'dall-e-3\', \'dall-e-2\'",required=false)
 String model,
 @ToolParam(name="n",description="The number of images to generate (1 for dall-e-3, 1-10 for dall-e-2)",required=false)
 Integer n,
 @ToolParam(name="size",description="Size of the image, e.g., \'1024x1024\', \'1792x1024\', \'1024x1792\'",required=false)
 String size,
 @ToolParam(name="quality",description="The quality of the image (\'standard\' or \'hd\' for dall-e-3)",required=false)
 String quality,
 @ToolParam(name="response_format",description="The format of the response (\'url\' or \'b64_json\')",required=false)
 String responseFormat)

Generate image(s) based on the given prompt.

参数:

prompt - the text prompt to generate image

model - the model to use (e.g., "dall-e-3", "dall-e-2")

n - the number of images to generate (1 for dall-e-3, 1-10 for dall-e-2)

size - the size of the image (e.g., "1024x1024", "1792x1024", "1024x1792")

quality - the quality of the image ("standard" or "hd" for dall-e-3)

responseFormat - the format of the response ("url" or "b64_json")

返回:

a ToolResultBlock containing the generated image(s)

openaiImageToText

@Tool(name="openai_image_to_text",
      description="Convert image(s) to text using OpenAI vision models. Analyzes images and returns text descriptions.")
public reactor.core.publisher.Mono<ToolResultBlock> openaiImageToText(@ToolParam(name="image_urls",description="The URLs of the images to analyze (comma-separated)")
 String imageUrls,
 @ToolParam(name="prompt",description="The text prompt describing what to extract from the images",required=false)
 String prompt,
 @ToolParam(name="model",description="The vision model to use, e.g., \'gpt-4o\', \'gpt-4-vision-preview\'",required=false)
 String model,
 @ToolParam(name="max_tokens",description="The maximum number of tokens in the response",required=false)
 Integer maxTokens)

Convert image(s) to text using vision models.

参数:

imageUrls - the URLs of the images to analyze

prompt - the text prompt describing what to extract from the images

model - the vision model to use (e.g., "gpt-4o", "gpt-4-vision-preview")

maxTokens - the maximum number of tokens in the response

返回:

a ToolResultBlock containing the text description of the images

openaiTextToAudio

@Tool(name="openai_text_to_audio",
      description="Convert text to audio (speech) using OpenAI TTS models. Returns audio as base64 data.")
public reactor.core.publisher.Mono<ToolResultBlock> openaiTextToAudio(@ToolParam(name="text",description="The text to convert to speech")
 String text,
 @ToolParam(name="model",description="The TTS model to use, e.g., \'tts-1\', \'tts-1-hd\'",required=false)
 String model,
 @ToolParam(name="voice",description="The voice to use: \'alloy\', \'echo\', \'fable\', \'onyx\', \'nova\', \'shimmer\'",required=false)
 String voice,
 @ToolParam(name="response_format",description="The audio format: \'mp3\', \'opus\', \'aac\', \'flac\'",required=false)
 String responseFormat,
 @ToolParam(name="speed",description="The speed of the speech (0.25 to 4.0)",required=false)
 Double speed)

Convert text to audio (speech) using OpenAI TTS models.

参数:

text - the text to convert to speech

model - the TTS model to use (e.g., "tts-1", "tts-1-hd")

voice - the voice to use ("alloy", "echo", "fable", "onyx", "nova", "shimmer")

responseFormat - the audio format ("mp3", "opus", "aac", "flac")

speed - the speed of the speech (0.25 to 4.0)

返回:

a ToolResultBlock containing the audio as base64 data

openaiAudioToText

@Tool(name="openai_audio_to_text",
      description="Convert audio to text (transcription) using OpenAI Whisper models. Requires audio file URL.")
public reactor.core.publisher.Mono<ToolResultBlock> openaiAudioToText(@ToolParam(name="audio_url",description="The URL of the audio file to transcribe")
 String audioUrl,
 @ToolParam(name="model",description="The transcription model to use, e.g., \'whisper-1\'",required=false)
 String model,
 @ToolParam(name="language",description="The language of the audio (ISO-639-1 code, optional)",required=false)
 String language,
 @ToolParam(name="prompt",description="Optional text to guide the model\'s style",required=false)
 String prompt,
 @ToolParam(name="response_format",description="The format of the response: \'json\', \'text\', \'verbose_json\', etc.",required=false)
 String responseFormat,
 @ToolParam(name="temperature",description="The temperature for sampling (0.0 to 1.0)",required=false)
 Double temperature)

Convert audio to text (transcription) using OpenAI Whisper models.

Note: This requires multipart/form-data upload, which is not yet fully supported. This is a placeholder implementation.

参数:

audioUrl - the URL of the audio file to transcribe

model - the transcription model to use (e.g., "whisper-1")

language - the language of the audio (ISO-639-1 code, optional)

prompt - optional text to guide the model's style

responseFormat - the format of the response ("json", "text", "verbose_json", etc.)

temperature - the temperature for sampling (0.0 to 1.0)

返回:

a ToolResultBlock containing the transcribed text