SAM3

Tools 5

• sam3_image_text_inference Segment every instance of a concept in a single image with SAM3 using a natural-language text prompt (concept segmentation). The prompt is a concept to find (e.g. "player in white", "red car"); each matching instance becomes one detection. Returns a JSON list of detection objects, each with: "object_id" (integer index of the instance within this image); "x", "y", "w", "h" (floats); "conf" (float 0-1 detection score); and "mask_b64" (string). The bounding box ("x","y","w","h") is NORMALIZED 0-1: (x, y) is the top-left corner and (w, h) is the box size, each divided by the image width/height -- i.e. the albumentations "coco" layout [x_min, y_min, width, height] but normalized to 0-1 rather than absolute pixels, and NOT the "pascal_voc" [x_min, y_min, x_max, y_max] layout. Bounding-box formats are documented at https://albumentations.ai/docs/3-basic-usage/bounding-boxes-augmentations/#bounding-box-formats . "mask_b64" is a base64-encoded 1-bit PNG of the binary segmentation mask at the in
• sam3_image_visual_inference Segment objects in a single image with SAM3 using visual prompts (bounding boxes and/or points); a drop-in match for SAM2's process_image. Provide at least one of bboxes or points. Each bounding box produces its own mask; all points together describe one object and produce a single mask. The return value is a JSON list of base64-encoded 1-bit-PNG mask strings, ordered as: one mask per bounding box (in the order the boxes were given), then, if points were provided, one final mask for the points prompt. Each string decodes to a binary segmentation mask at the input image resolution; decode it with PIL, e.g. numpy.array(Image.open(io.BytesIO(base64.b64decode(mask_b64)))) to get a 0/255 mask. Box pixel coordinates use (x0, y0) = top-left and (x1, y1) = bottom-right corners of the ORIGINAL input image (albumentations "pascal_voc" [x_min, y_min, x_max, y_max], absolute pixels); point coordinates (x, y) are absolute pixels in the same image. Coordinate formats are documented at https://albume
• sam3_video_annotation Track and segment objects across a video with SAM3 using a natural-language text prompt, and return an annotated video with the segmentation masks overlaid. The prompt is a concept to find (e.g. "player in white", "red car"); every matching instance is tracked across all frames and rendered as a colored mask overlay (color keyed by object). The video can optionally be downsampled before tracking with sample_fps or every_x; the returned video then contains only the sampled frames, muxed at the sampled frame rate so it keeps the original real-time duration. This is the annotated-video counterpart of video_inference: it returns a filesystem path to an mp4 with the mask overlays burned in, rather than JSON detections. Use video_inference instead if you need the structured per-frame bounding boxes and base64 masks.
• sam3_video_inference Track and segment objects across a video with SAM3 using a natural-language text prompt, returning per-object-per-frame detections (or an annotated video). The prompt is a concept to find (e.g. "player in white", "red car"); every matching instance is tracked across all frames with a stable track_id. The video can optionally be downsampled before tracking with sample_fps or every_x; the "frame" field of every detection is a 0-based position in the SAMPLED frame sequence, NOT an original video frame number, so with every_x=5 the detection "frame": 2 corresponds to original frame 10. This tool has two output shapes selected by annotation_mode. When annotation_mode is false (default) it returns a JSON list of detection objects, one per tracked object per frame. Each detection has: "frame" (integer sampled-frame index as defined above); "track_id" (integer, stable across frames for one object); "x", "y", "w", "h" (floats); "conf" (float, always 1 for video); and "mask_b64" (string). The bo
• sam3_video_visual_inference SAM3 video segmentation with visual prompts: track objects through a video given their segmentation masks on a reference frame; a drop-in match for SAM2's process_video. Seed the objects to track by supplying their masks on the reference frame (ref_frame_idx); SAM3 then propagates each object forward -- and, when ref_frame_idx is nonzero, also backward -- through every frame. Unlike video_inference (which finds objects from a text prompt), this tool tracks exactly the objects whose masks you provide, e.g. masks obtained from image_visual_inference or image_text_inference on the reference frame. The video can optionally be downsampled before tracking with sample_fps or every_x; all frame indices (the "frame" field of every detection AND the ref_frame_idx input) are 0-based positions in the SAMPLED frame sequence, NOT original video frame numbers, so with every_x=5 the detection "frame": 2 corresponds to original frame 10. Returns a JSON list of per-frame detections, one entry per tracke