The behavior section of the components docs in the API reference is really hard to understand:

Should be rewritten across all the components to be easier to understand. Maybe something like:

Behavior

Using AnnotatedImage as an input component:

The component will pass its value to your function as:

Type: tuple[str, list[tuple[str, str]]]
Description: a tuple consisting of a str filepath to a base image and list of annotations. Each annotation itself is tuple of a mask (as a str filepath to image) and a str label.

Example code:

import gradio as gr 

def predict(
	value: tuple[str, list[tuple[str, str]]] | None
):
         # process value from the AnnotatedImage component
	return "prediction"

interface = gr.Interface(predict, gr.Annotatedimage(), gr.Textbox())
interface.launch()

Using AnnotatedImage as an output component:

The component will accept its value from function as:

Type: tuple[np.ndarray | PIL.Image.Image | str, list[tuple[np.ndarray | tuple[int, int, int, int], str]]]
Description: a tuple of a base image and list of annotations: a tuple[Image, list[Annotation]]. The Image itself can be str filepath, numpy.ndarray, or PIL.Image. Each Annotation is a tuple[Mask, str]. The Mask can be either a tuple of 4 int's representing the bounding box coordinates (x1, y1, x2, y2), or 0-1 confidence mask in the form of a numpy.ndarray of the same shape as the image, while the second element of the Annotation tuple is a str label.

Example code:

import gradio as gr 

def predict(···) -> tuple[np.ndarray | PIL.Image.Image | str, list[tuple[np.ndarray | tuple[int, int, int, int], str]]] | None
        # process value to return to the AnnotatedImage component
	return value

interface = gr.Interface(predict, gr.Annotatedimage(), gr.Textbox())
interface.launch()