程序包 io.agentscope.core.tool.multimodal

类 DashScopeMultiModalTool

java.lang.Object
io.agentscope.core.tool.multimodal.DashScopeMultiModalTool
public class DashScopeMultiModalTool extends Object
DashScope multimodal tool.

Support tools:

  • dashscope_text_to_image: Generate image(s) based on the given text.
  • dashscope_image_to_text: Generate text based on the given images.
  • dashscope_text_to_audio: Convert the given text to audio.
  • dashscope_audio_to_text: Convert the given audio to text.
  • dashscope_text_to_video: Generate video based on the given text prompt.
  • dashscope_image_to_video: Generate a video from a single input image and an optional text prompt.
  • dashscope_first_and_last_frame_image_to_video: Generate video transitioning from a first frame to a last frame and an optional text prompt.
  • dashscope_video_to_text: Analyze video and generate a text description or answer questions based on the video content.
convert text to images, convert images to text, convert text to audio, and convert audio to text. Please refer to the `dashscope documentation` for more details.

  • 构造器详细资料

    • DashScopeMultiModalTool

      public DashScopeMultiModalTool(String apiKey)

  • 方法详细资料

    • dashscopeTextToImage

      @Tool(name="dashscope_text_to_image", description="Generate image(s) based on the given prompt, and return image url(s) or base64 data.") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeTextToImage(@ToolParam(name="prompt",description="The text prompt to generate image") String prompt, @ToolParam(name="model",description="The model to use, e.g., \'wanx-v1\', \'qwen-image\', \'wan2.2-t2i-flash\', etc.",required=false) String model, @ToolParam(name="n",description="The number of images to generate",required=false) Integer n, @ToolParam(name="size",description="Size of the image, e.g., \'1024*1024\', \'1280*1280\', \'800*1200\', etc.",required=false) String size, @ToolParam(name="use_base64",description="Whether to use base64 data for images",required=false) Boolean useBase64)
      Generate image(s) based on the given prompt, and return image url(s) or base64 data.
      参数:
      prompt - The text prompt to generate image
      model - The model to use, e.g., 'wanx-v1', 'qwen-image', 'wan2.2-t2i-flash', etc.
      n - The number of images to generate
      size - Size of the image, e.g., '1024*1024', '1280*1280', '800*1200', etc.
      useBase64 - Whether to use base64 data for images
      返回:
      A ToolResultBlock containing the generated image url, base64 data, or error message.

    • dashscopeImageToText

      @Tool(name="dashscope_image_to_text", description="Generate text based on the given images.") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeImageToText(@ToolParam(name="image_urls",description="The URL(s), local file path(s) or Base64 data URL(s)(the format pattern is data:[MIME_type];base64,{base64_image}, e.g., \'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDg...\') of image(s) to be converted into text.") List<String> imageUrls, @ToolParam(name="prompt",description="The text prompt.",required=false) String prompt, @ToolParam(name="model",description="The model to use, e.g., \'qwen3-vl-plus\', qwen-vl-plus\' \'qwen-vl-max\', etc.",required=false) String model)
      Generate text based on the given images.
      参数:
      imageUrls - The URL(s) of image(s) to be converted into text.
      prompt - The text prompt.
      model - The model to use, e.g., 'qwen3-vl-plus', qwen-vl-plus' 'qvq-plus', etc.
      返回:
      A ToolResultBlock containing the generated text or error message.

    • dashscopeTextToAudio

      @Tool(name="dashscope_text_to_audio", description="Convert the given text to audio.") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeTextToAudio(@ToolParam(name="text",description="The text to be converted into audio.") String text, @ToolParam(name="model",description="The TTS model to use. For Qwen TTS: \'qwen3-tts-flash\', \'qwen-tts\'. For Sambert: \'sambert-zhinan-v1\', \'sambert-zhiqi-v1\', \'sambert-zhichu-v1\', etc.",required=false) String model, @ToolParam(name="voice",description="Voice name for Qwen TTS models, e.g., \'Cherry\', \'Serena\'. Ignored for Sambert models.",required=false) String voice, @ToolParam(name="language",description="Language type for Qwen TTS, e.g., \'Chinese\', \'English\'. Ignored for Sambert models.",required=false) String language, @ToolParam(name="sample_rate",description="Sample rate of the audio (e.g., 16000, 24000, 48000).",required=false) Integer sampleRate)
      Convert the given text to audio.

      Supports two types of TTS models:

      • Qwen TTS models (qwen3-tts-flash, qwen-tts) - uses multimodal-generation API
      • Sambert models (sambert-*) - uses speech synthesis SDK
      参数:
      text - The text to be converted into audio.
      model - The TTS model to use. For Qwen TTS: 'qwen3-tts-flash', 'qwen-tts'. For Sambert: 'sambert-zhinan-v1', 'sambert-zhiqi-v1', 'sambert-zhichu-v1', etc.
      voice - Voice name for Qwen TTS models, e.g., 'Cherry', 'Serena'. Ignored for Sambert models.
      language - Language type for Qwen TTS, e.g., 'Chinese', 'English'. Ignored for Sambert models.
      sampleRate - Sample rate of the audio (e.g., 16000, 24000, 48000).
      返回:
      A ToolResultBlock containing the base64 data of audio or error message.

    • dashscopeAudioToText

      @Tool(name="dashscope_audio_to_text", description="Convert the given audio to text.") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeAudioToText(@ToolParam(name="audio_url",description="The file path or URL of audio to be converted into text.") String audioUrl, @ToolParam(name="model",description="The TTS model to use, e.g., \'paraformer-realtime-v1\', \'paraformer-realtime-8k-v1\', \'paraformer-realtime-v2\', \'paraformer-realtime-8k-v2\'.",required=false) String model, @ToolParam(name="sample_rate",description="Sample rate of the audio (e.g., 8000, 16000).",required=false) Integer sampleRate)
      Convert the given audio to text.
      参数:
      audioUrl - The file path or URL of audio to be converted into text.
      model - The speech recognition model to use, e.g., 'paraformer-realtime-v1', 'paraformer-realtime-8k-v1', 'paraformer-realtime-v2', 'paraformer-realtime-8k-v2'.
      sampleRate - Sample rate of the audio (e.g., 8000, 16000).
      返回:
      A ToolResultBlock containing the recognition text or error message.

    • dashscopeTextToVideo

      @Tool(name="dashscope_text_to_video", description="Generate video based on the given text prompt") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeTextToVideo(@ToolParam(name="prompt",description="The text prompt to generate video") String prompt, @ToolParam(name="model",description="The model to use, e.g., \'wan2.6-t2v\', \'wan2.5-t2v-preview\', etc",required=false) String model, @ToolParam(name="negative_prompt",description="The negative prompt to avoid certain elements",required=false) String negativePrompt, @ToolParam(name="audio_url",description="The URL for background audio",required=false) String audioUrl, @ToolParam(name="size",description="Size of the video, e.g., \'1920*1080\', \'1280*720\', etc",required=false) String size, @ToolParam(name="duration",description="Duration of the video in seconds, e.g., \'5\', \'10\', etc",required=false) Integer duration, @ToolParam(name="shot_type",description="Specify the shot type that generates the video. single: default value, output single shot video; multi: output multi-lens video",required=false) String shotType, @ToolParam(name="prompt_extend",description="Whether to automatically extend the prompt (default true)",required=false) Boolean promptExtend, @ToolParam(name="watermark",description="Whether to include watermark (default false)",required=false) Boolean watermark, @ToolParam(name="seed",description="The seed for reproducibility",required=false) Integer seed)
      Generate video based on the given prompt.
      参数:
      prompt - The text prompt to generate video.
      model - The model to use, e.g., 'wan2.6-t2v', 'wan2.5-t2v-preview', etc.
      negativePrompt - The negative prompt to avoid certain elements.
      audioUrl - The URL for background audio.
      size - Size of the video, e.g., '1920*1080', '1280*720', etc.
      duration - Duration of the video in seconds, e.g., '5', '10', etc.
      shotType - Specify the shot type that generates the video. single: default value, output single shot video; multi: output multi-lens video.
      promptExtend - Whether to extend the prompt automatically (default true)
      watermark - Whether to include watermark (default false)
      seed - The seed for reproducibility
      返回:
      A ToolResultBlock containing the generated video url or error message.

    • dashscopeImageToVideo

      @Tool(name="dashscope_image_to_video", description="Generate a video from a single input image and an optional text prompt.Supports optional audio guidance and duration control.") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeImageToVideo(@ToolParam(name="prompt",description="Text prompt describing the video content and motion",required=false) String prompt, @ToolParam(name="model",description="Model to use, e.g., \'wan2.6-i2v-flash\', \'wan2.6-i2v\', etc",required=false) String model, @ToolParam(name="image_url",description="URL, local file path or Base64 data URL((the format pattern is data:[MIME_type];base64,{base64_image}, e.g., \'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDg...\')) of the first frame image") String imageUrl, @ToolParam(name="audio_url",description="URL of the audio file that the model will use to generate video",required=false) String audioUrl, @ToolParam(name="negative_prompt",description="The negative prompt to avoid certain elements",required=false) String negativePrompt, @ToolParam(name="template",description="Name of video effect template, e.g., \'squish\', \'rotation\', etc",required=false) String template, @ToolParam(name="resolution",description="Video resolution, e.g., \'720P\', \'1080P\'",required=false) String resolution, @ToolParam(name="duration",description="Duration of the video in seconds, e.g., 5, 10, 15",required=false) Integer duration, @ToolParam(name="shot_type",description="Specify the shot type that generates the video. single: default value, output single shot video; multi: output multi-lens video",required=false) String shotType, @ToolParam(name="audio",description="Whether to generate audio video (default true)",required=false) Boolean audio, @ToolParam(name="prompt_extend",description="Whether to automatically extend the prompt (default true)",required=false) Boolean promptExtend, @ToolParam(name="watermark",description="Whether to include watermark (default false)",required=false) Boolean watermark, @ToolParam(name="seed",description="The seed for reproducibility",required=false) Integer seed)
      Generate a video based on a single input image (first frame) and an optional text prompt.
      参数:
      prompt - The text prompt describing the video content and motion.
      model - The model to use, e.g., 'wan2.6-i2v-flash', 'wan2.1-i2v-turbo'.
      imageUrl - The URL, local file path, or Base64 data of the input image (first frame).
      audioUrl - URL of the audio file that the model will use to generate video.
      negativePrompt - The negative prompt to avoid certain elements.
      template - Name of video effect template, e.g., 'squish', 'rotation', etc
      resolution - Resolution of the video, e.g., '720P', '1080P'.
      duration - Duration of the video in seconds (e.g., 5, 10, 15). Default depends on model.
      shotType - Specify the shot type that generates the video. single: default value, output single shot video; multi: output multi-lens video.
      audio - Whether to generate audio video (default true).
      promptExtend - Whether to automatically extend the prompt (default true).
      watermark - Whether to include watermark (default false).
      seed - Optional seed for reproducibility.
      返回:
      A ToolResultBlock containing the generated video url or error message.

    • dashscopeFirstAndLastFrameImageToVideo

      @Tool(name="dashscope_first_and_last_frame_image_to_video", description="Generate video transitioning from a first frame to a last frame and an optional text prompt") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeFirstAndLastFrameImageToVideo(@ToolParam(name="prompt",description="Text prompt describing the video content and motion",required=false) String prompt, @ToolParam(name="model",description="Model to use, e.g., \'wan2.2-kf2v-flash\', \'wanx2.1-kf2v-plus\'",required=false) String model, @ToolParam(name="first_frame_url",description="URL, local file path or Base64 data URL(the format pattern is data:[MIME_type];base64,{base64_image}, e.g., \'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDg...\') of the first frame image") String firstFrameUrl, @ToolParam(name="last_frame_url",description="URL, local file path or Base64 data URL(the format pattern is data:[MIME_type];base64,{base64_image}, e.g., \'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDg...\') of the last frame image",required=false) String lastFrameUrl, @ToolParam(name="negative_prompt",description="The negative prompt to avoid certain elements",required=false) String negativePrompt, @ToolParam(name="template",description="Name of video effect template, e.g., \'hanfu-1\', \'solaron\', etc",required=false) String template, @ToolParam(name="resolution",description="Video resolution, e.g., \'720P\', \'1080P\'",required=false) String resolution, @ToolParam(name="prompt_extend",description="Whether to automatically extend the prompt (default true)",required=false) Boolean promptExtend, @ToolParam(name="watermark",description="Whether to include watermark (default false)",required=false) Boolean watermark, @ToolParam(name="seed",description="The seed for reproducibility",required=false) Integer seed)
      Generate video transitioning from a first frame to a last frame and an optional text prompt.
      参数:
      prompt - The text prompt describing the video content and camera movement.
      model - The model to use, e.g., 'wan2.2-kf2v-flash', 'wanx2.1-kf2v-plus'.
      firstFrameUrl - The URL or Base64 data of the first frame image.
      lastFrameUrl - The URL or Base64 data of the last frame image.
      negativePrompt - The negative prompt to avoid certain elements.
      template - Name of video effect template, e.g., 'hanfu-1', 'solaron', etc.
      resolution - Resolution of the video, e.g., '480P', '720P', '1080P'.
      promptExtend - Whether to automatically extend the prompt (default true).
      watermark - Whether to include watermark (default false).
      seed - Optional seed for reproducibility.
      返回:
      A ToolResultBlock containing the generated video url or error message.

    • dashscopeVideoToText

      @Tool(name="dashscope_video_to_text", description="Analyze video and generate a text description or answer questions based on the video content.Supports controlling the frame sampling rate (fps).") public reactor.core.publisher.Mono<ToolResultBlock> dashscopeVideoToText(@ToolParam(name="video_url",description="The URL, local file path or base64 data URL(the format pattern is data:[MIME_type];base64,{base64_video}, e.g., \'data:video/mp4;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAA...\') of the video to analyze.") String videoUrl, @ToolParam(name="prompt",description="The question or instruction regarding the video content",required=false) String prompt, @ToolParam(name="model",description="The vision model to use, e.g., \'qwen3.5-plus\', \'qwen3.5-flash\', \'qwen3-vl-plus\', \'qwen3-vl-flash\', etc",required=false) String model, @ToolParam(name="fps",description="Frames per second to sample from the video for analysis (default 2.0)",required=false) Double fps)
      Analyze video and generate a text description or answer questions based on the video content.
      参数:
      videoUrl - The URL or local path of the video to analyze.
      prompt - The text prompt or question regarding the video content.
      model - The vision model to use, e.g., 'qwen3.5-plus', 'qwen3.5-flash', 'qwen3-vl-plus', 'qwen3-vl-flash', etc.
      fps - Frames per second to sample from the video (e.g., 1, 2, 4). Default is 2.
      返回:
      A ToolResultBlock containing the generated text analysis or error message.