Generative User Interfaces

Generative User Interfaces

Summary

Problem Statement

Currently, creating custom user interfaces for agent interactions requires programmers to define specific tool renderers. This limits the flexibility and adaptability of agent-driven applications.

Motivation

This draft describes an AG-UI extension that addresses generative user interfaces—interfaces produced directly by artificial intelligence without requiring a programmer to define custom tool renderers. The key idea is to leverage our ability to send client-side tools to the agent, thereby enabling this capability across all agent frameworks supported by AG-UI.

Status

Challenges and Limitations

Tool Description Length

OpenAI enforces a limit of 1024 characters for tool descriptions. Gemini and Anthropic impose no such limit.

Arguments JSON Schema Constraints

Classes, nesting, $ref, and oneOf are not reliably supported across LLM providers.

Context Window Considerations

Injecting a large UI description language into an agent may reduce its performance. Agents dedicated solely to UI generation perform better than agents combining UI generation with other tasks.

Detailed Specification

Two-Step Generation Process

flowchart TD
    A[Agent needs UI] --> B["Step 1: <b>What?</b> <br/> Agent calls generateUserInterface <br/>(description, data, output)"]
    B --> C["Step 2: <b>How?</b> <br/> Secondary generator builds actual UI <br/>(JSON Schema, React, etc.)"]
    C --> D[Rendered UI shown to user]
    D --> E[Validated user input returned to Agent]

Step 1: What to Generate?

Inject a lightweight tool into the agent:

Tool Definition:

  • Name: generateUserInterface
  • Arguments:
    • description: A high-level description of the UI (e.g., "A form for entering the user's address")
    • data: Arbitrary pre-populated data for the generated UI
    • output: A description or schema of the data the agent expects the user to submit back (fields, required/optional, types, constraints)

Example Tool Call:

{
  "tool": "generateUserInterface",
  "arguments": {
    "description": "A form that collects a user's shipping address.",
    "data": {
      "firstName": "Ada",
      "lastName": "Lovelace",
      "city": "London"
    },
    "output": {
      "type": "object",
      "required": [
        "firstName",
        "lastName",
        "street",
        "city",
        "postalCode",
        "country"
      ],
      "properties": {
        "firstName": { "type": "string", "title": "First Name" },
        "lastName": { "type": "string", "title": "Last Name" },
        "street": { "type": "string", "title": "Street Address" },
        "city": { "type": "string", "title": "City" },
        "postalCode": { "type": "string", "title": "Postal Code" },
        "country": {
          "type": "string",
          "title": "Country",
          "enum": ["GB", "US", "DE", "AT"]
        }
      }
    }
  }
}

Step 2: How to Generate?

Delegate UI generation to a secondary LLM or agent:

  • The CopilotKit user stays in control: Can make their own generators, add custom libraries, include additional prompts etc.
  • On tool invocation, the secondary model consumes description, data, and output to generate the user interface
  • This model is focused solely on UI generation, ensuring maximum fidelity and consistency
  • The generation method can be swapped as needed (e.g., JSON, HTML, or other renderable formats)
  • The UI format description is not subject to structural or length constraints, allowing arbitrarily complex specifications

Implementation Examples

Example Output: UISchemaGenerator

{
  "jsonSchema": {
    "title": "Shipping Address",
    "type": "object",
    "required": [
      "firstName",
      "lastName",
      "street",
      "city",
      "postalCode",
      "country"
    ],
    "properties": {
      "firstName": { "type": "string", "title": "First name" },
      "lastName": { "type": "string", "title": "Last name" },
      "street": { "type": "string", "title": "Street address" },
      "city": { "type": "string", "title": "City" },
      "postalCode": { "type": "string", "title": "Postal code" },
      "country": {
        "type": "string",
        "title": "Country",
        "enum": ["GB", "US", "DE", "AT"]
      }
    }
  },
  "uiSchema": {
    "type": "VerticalLayout",
    "elements": [
      {
        "type": "Group",
        "label": "Personal Information",
        "elements": [
          { "type": "Control", "scope": "#/properties/firstName" },
          { "type": "Control", "scope": "#/properties/lastName" }
        ]
      },
      {
        "type": "Group",
        "label": "Address",
        "elements": [
          { "type": "Control", "scope": "#/properties/street" },
          { "type": "Control", "scope": "#/properties/city" },
          { "type": "Control", "scope": "#/properties/postalCode" },
          { "type": "Control", "scope": "#/properties/country" }
        ]
      }
    ]
  },
  "initialData": {
    "firstName": "Ada",
    "lastName": "Lovelace",
    "city": "London",
    "country": "GB"
  }
}

Example Output: ReactFormHookGenerator

// ----- Schema (contract) -----
const AddressSchema = z.object({
  firstName: z.string().min(1, "Required"),
  lastName: z.string().min(1, "Required"),
  street: z.string().min(1, "Required"),
  city: z.string().min(1, "Required"),
  postalCode: z.string().regex(/^[A-Za-z0-9\\-\\s]{3,10}$/, "3–10 chars"),
  country: z.enum(["GB", "US", "DE", "AT", "FR", "IT", "ES"]),
});
export type Address = z.infer<typeof AddressSchema>;

type Props = {
  initialData?: Partial<Address>;
  meta?: { title?: string; submitLabel?: string };
  respond: (data: Address) => void; // <-- called on successful submit
};

const COUNTRIES: Address["country"][] = [
  "GB",
  "US",
  "DE",
  "AT",
  "FR",
  "IT",
  "ES",
];

export default function AddressForm({ initialData, meta, respond }: Props) {
  const {
    register,
    handleSubmit,
    formState: { errors },
  } = useForm<Address>({
    resolver: zodResolver(AddressSchema),
    defaultValues: {
      firstName: "",
      lastName: "",
      street: "",
      city: "",
      postalCode: "",
      country: "GB",
      ...initialData,
    },
  });

  const onSubmit = (data: Address) => {
    // Guaranteed to match AddressSchema
    respond(data);
  };

  return (
    <form onSubmit={handleSubmit(onSubmit)}>
      {meta?.title && <h2>{meta.title}</h2>}

      {/* Section: Personal Information */}
      <fieldset>
        <legend>Personal Information</legend>

        <div>
          <label>First name</label>
          <input {...register("firstName")} placeholder="Ada" autoFocus />
          {errors.firstName && <small>{errors.firstName.message}</small>}
        </div>

        <div>
          <label>Last name</label>
          <input {...register("lastName")} placeholder="Lovelace" />
          {errors.lastName && <small>{errors.lastName.message}</small>}
        </div>
      </fieldset>

      {/* Section: Address */}
      <fieldset>
        <legend>Address</legend>

        <div>
          <label>Street address</label>
          <input {...register("street")} />
          {errors.street && <small>{errors.street.message}</small>}
        </div>

        <div>
          <label>City</label>
          <input {...register("city")} />
          {errors.city && <small>{errors.city.message}</small>}
        </div>

        <div>
          <label>Postal code</label>
          <input {...register("postalCode")} />
          {errors.postalCode && <small>{errors.postalCode.message}</small>}
        </div>

        <div>
          <label>Country</label>
          <select {...register("country")}>
            {COUNTRIES.map((c) => (
              <option key={c} value={c}>
                {c}
              </option>
            ))}
          </select>
          {errors.country && <small>{errors.country.message}</small>}
        </div>
      </fieldset>

      <div>
        <button type="submit">{meta?.submitLabel ?? "Submit"}</button>
      </div>
    </form>
  );
}

Implementation Considerations

Client SDK Changes

TypeScript SDK additions:

  • New generateUserInterface tool type
  • UI generator registry for pluggable generators
  • Validation layer for generated UI schemas
  • Response handler for user-submitted data

Python SDK additions:

  • Support for UI generation tool invocation
  • Schema validation utilities
  • Serialization for UI definitions

Integration Impact

  • All AG-UI integrations can leverage this capability without modification
  • Frameworks emit standard tool calls; client handles UI generation
  • Backward compatible with existing tool-based UI approaches

Use Cases

Dynamic Forms

Agents can generate forms on-the-fly based on conversation context without pre-defined schemas.

Data Visualization

Generate charts, graphs, or tables appropriate to the data being discussed.

Interactive Workflows

Create multi-step wizards or guided processes tailored to user needs.

Adaptive Interfaces

Generate different UI layouts based on user preferences or device capabilities.

Testing Strategy

  • Unit tests for tool injection and invocation
  • Integration tests with multiple UI generators
  • E2E tests demonstrating various UI types
  • Performance benchmarks comparing single vs. two-step generation
  • Cross-provider compatibility testing

References

2087950ee