AIDevKit - AI Suite for Unity
API ReferencesDiscordGlitch9
  • Introduction
    • AI Dev Kit 3.7.0
    • Troubleshooting
    • FAQ
    • Update Logs
      • AI Dev Kit v2
      • AI Dev Kit v1
  • Quick Start
    • API Key Setup
      • OpenAI
      • Google Gemini
      • ElevenLabs
      • OpenRouter
    • Adding Models & Voices
      • Quick Add Guide
      • Creating Snippets
    • Self-Hosting with Ollama
  • Editor Tools
    • Editor Chat
    • Asset Generators
    • Asset Managers
      • Prompt History
      • File Manager
      • Chatbot Manager
      • Assistant Manager
  • GEN Tasks
    • Overview
      • Prefixes
      • Sequence
    • Response
    • Image
    • Video
    • SoundFX
    • Speech
    • Transcript
    • Voice Change
    • Audio Isolation
  • Components
    • Chatbot
    • Chatbot (Assistants API)
    • Realtime Assistant
    • Modules
    • Event Receivers
  • Platform API
    • OpenAI
      • 💬Chat completions
      • 🖼️Image operations
      • 🗣️Text to speech
      • 🎙️Speech to text
        • Recording real-time in Unity
      • 💾Files
      • 🔎Embeddings
      • 🛡️Moderations
      • ⚙️Fine-tuning
      • Assistants API
        • How it works
        • Creating custom functions
        • Creating assistants API
    • Google Gemini
      • 📝System instructions
      • 💬Text generation
      • ⚙️Fine-tuning
      • ▶️Fucntion calling
      • 🔎Embeddings
      • 🛡️Safety
      • 💻Code execution
  • Legacy Documents
    • AI Dev Kit 1.0
      • Preperation
      • Scriptable Toolkits
        • Chat Streamer
        • Image Generator
        • Voice Transcriber
        • Voice Generator
      • Editor Tools
      • Troubleshooting (Legacy)
        • ❗Build Error: The name 'UnityMenu' does not exist in the current context
        • ❗The type or namespace name 'AndroidJavaObject' could not be found
        • ❗The type or namaspace name 'Plastic' does not exist
        • ❗Build Error: The name 'Asset Database' does not exist in the current context
        • ❗'ModelData.Create(Provider, string, UnixTime?, string)': not all code paths return a value
      • Code Generators
        • C# Script Generator
        • Unity Component Generator
    • AI Dev Kit 2.0
      • Event Handlers
      • Editor Chat
      • Editor Vision (TTI, ITI)
      • Editor Speech (TTS)
      • Management Tools
        • Prompt History Viewer
        • AI Model Manager
        • TTS Voice Manager
        • OpenAI File Manager
        • OpenAI Assistant Manager
        • ElevenLabs Voice Library
Powered by GitBook
On this page
  • Image Operations Overview:
  • Sample Code for Image Requests:
  1. Platform API
  2. OpenAI

Image operations

PreviousChat completionsNextText to speech

Last updated 11 months ago

To integrate OpenAI's image generation, editing, and variation capabilities into your Unity project, you'll primarily interact with three types of requests using the OpenAI with unity asset. These requests are designed to leverage OpenAI's advanced image models, such as DALL·E, to create, modify, and generate variations of images based on textual prompts or existing images.

For a deep dive into the technical details and options available for these requests, refer to the .

Image Operations Overview:

  1. Image Creation: Generate new images from textual prompts.

  2. Image Editing: Modify existing images based on textual instructions.

  3. Image Variations: Create different variations of an input image.

Sample Code for Image Requests:

1. Image Creation Request:

Generate new images from a textual description using ImageCreationRequest.

var request = new ImageCreationRequest.Builder()
    .SetModel(ImageModel.DallE3)
    .SetPrompt("A cute baby sea otter")
    .SetN(1) // Number of images to generate
    .SetSize(ImageSize._1024x1024)
    .Build();

var result = await request.ExecuteAsync();

Texture2D[] images = result.Textures;
// Now you can use these textures in your Unity project
from openai import OpenAI
client = OpenAI()

client.images.generate(
  model="dall-e-3",
  prompt="A cute baby sea otter",
  n=1,
  size="1024x1024"
)

2. Image Editing Request:

Edit an existing image based on a provided prompt. You'll need to supply the image as a FormFile (for API requests) or a Texture2D object (within Unity).

Using Texture2D from Resources folder.

// Using Texture2D
var otter = Resources.Load<Texture2D>("otter");
var mask = Resources.Load<Texture2D>("mask");

var request = new ImageEditRequest.Builder()
    .SetImage(otter)
    .SetMask(mask)
    .SetPrompt("A cute baby sea otter wearing a beret")
    .SetN(2)
    .SetSize(ImageSize._1024x1024)
    .Build();

var result = await request.ExecuteAsync();

Texture2D[] images = result.Textures;
from openai import OpenAI
client = OpenAI()

client.images.edit(
  image=open("otter.png", "rb"),
  mask=open("mask.png", "rb"),
  prompt="A cute baby sea otter wearing a beret",
  n=2,
  size="1024x1024"
)

Using FormFile with file paths.

// Using FormFile
var otterFile = new FormFile(path to otter.png);
var maskFile = new FormFile(path to mask.png);

var request = new ImageEditRequest.Builder()
    .SetImage(otterFile)
    .SetMask(maskFile)
    .SetPrompt("A cute baby sea otter wearing a beret")
    .SetN(2)
    .SetSize(ImageSize._1024x1024)
    .Build();
    
var result = await request.ExecuteAsync();

Texture2D[] images = result.Textures;
from openai import OpenAI
client = OpenAI()

client.images.edit(
  image=open("otter.png", "rb"),
  mask=open("mask.png", "rb"),
  prompt="A cute baby sea otter wearing a beret",
  n=2,
  size="1024x1024"
)

3. Image Variation Request:

Generate variations of an input image. Similar to image editing, the image should be provided as a FormFile or Texture2D.

var image = Resources.Load<Texture2D>("image_edit_original");

var request = new ImageVariationRequest.Builder()
    .SetImage(image)
    .SetN(2)
    .SetSize(ImageSize._1024x1024)
    .Build();

var result = await request.ExecuteAsync();

Texture2D[] images = result.Textures;
from openai import OpenAI
client = OpenAI()

response = client.images.create_variation(
  image=open("image_edit_original.png", "rb"),
  n=2,
  size="1024x1024"
)

🖼️
Images API Reference