Getting the best results from image-to-3D conversion

Image-to-3D conversion is the feature that gets the most varied results, because the quality of the output depends heavily on the quality of the input image. This article describes what the agent can reliably reconstruct, what it cannot, and how to supplement a weak image with a text prompt to improve the output.

What makes a good input image ¶

The agent reconstructs geometry from a single image by inferring the three-dimensional shape from visual cues: silhouette, shading, perspective, and surface texture. Images that provide clear information on all four of these cues produce the most accurate geometry.

The best input images are: a product photograph on a plain background with soft, even lighting; a hand-drawn sketch with clear, consistent line weight on white paper; or a rendered concept image with no motion blur or depth-of-field effect. The worst input images are: photographs with busy backgrounds, images with strong directional shadows that obscure the silhouette, and low-resolution images where surface detail is lost to compression.

What the agent cannot reconstruct ¶

The agent cannot see the back of an object from a front-facing image. For objects with significant back-surface detail, the agent makes a plausible inference based on the object category, which is usually correct for symmetrical objects and sometimes incorrect for asymmetrical ones.

Very thin features, wire-like elements, and transparent surfaces are consistently difficult. A photograph of a wire-frame chair will produce a solid-looking chair rather than a wire-frame one, because the agent interprets the wires as surface edges rather than structural elements.

Supplementing with a text prompt ¶

The image-to-3D feature accepts an optional text prompt alongside the image. The text prompt is used to fill in information that the image cannot provide: back-surface detail, material properties, and scale. For most objects, adding a short text prompt alongside the image produces noticeably better results than using the image alone.

A useful pattern is to use the image to establish the shape and the text prompt to establish the material and scale. For example, uploading a sketch of a lamp and adding the prompt 'brushed brass base, white linen shade, 45cm tall' produces a more accurate and more useful output than the image alone.

The image gives the agent a shape to work from. The text gives it context. Used together, they cover most of what a single photograph cannot communicate on its own.