Let RGB Be the Language of Vision

Yang, Timing; Yang, Jinrui; Li, Xinlong; Wang, Yuhan; Li, Haoran; Liu, Yanqing; Wei, Guoyizhe; Ying, Jixuan; Wei, Chen; Chellappa, Rama; Zhou, Yuyin; Xie, Cihang; Yuille, Alan; Wang, Feng

Let RGB Be the Language of Vision

RINO — RGB In, RGB Out

Timing Yang¹, Jinrui Yang², Xinlong Li^1*, Yuhan Wang², Haoran Li³, Yanqing Liu², Guoyizhe Wei¹, Jixuan Ying^1*, Chen Wei⁴, Rama Chellappa¹, Yuyin Zhou², Cihang Xie², Alan Yuille¹, Feng Wang^1†

¹Johns Hopkins University ²UC Santa Cruz ³Carnegie Mellon University ⁴Rice University

^†Project lead ^*Visiting students

arXiv Code

Left: conventional vision ties each task to its own representation and architecture. Right: RINO renders every input and output as RGB, so one frozen image editor handles estimation and conditioned generation. — **RINO unifies vision under a single RGB interface.** *Left:* conventional vision ties each task to its own representation and architecture, with task-specific encoders and heads or generators. *Right:* RINO renders every input and output as RGB, so one frozen image editor, driven by a task-specific prompt, handles estimation (depth, normals, segmentation, detection, pose, referring) and the corresponding conditioned generation — all without any task-specific module.

Abstract

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language.

A Unified RGB Interface

Given an input image x_img and a text prompt x_text, a generic multimodal image editor F produces an output image y_img and, optionally, a text response y_text. Everything — natural images, segmentation masks, depth maps, pose maps, edge maps — travels through the same image encoder and decoder inside F. Only lightweight, parameter-free converters translate between RGB and each task's native format, and they run strictly before or after inference.

Understanding: unify the output

The input is a natural image and the prompt names the task. The model repaints the image as an RGB rendering of the target structure — a colored mask, a grayscale depth map, a normal map, a skeleton. A predefined color mapping or decoding rule converts that RGB image back into labels, depth, or keypoints for scoring.

Generation: unify the input

The dual problem. The input is an RGB-formatted condition — a semantic layout, a pose skeleton, a depth map, an edge map — and the prompt describes the desired image. The model synthesizes a natural image that follows the condition, with no ControlNet branch or per-condition adapter attached.

We run RINO-Zero on three open-source image editors, each used as a black-box RGB-to-RGB editor with released weights and no fine-tuning:

Qwen-Image-Edit20B MMDiT · Qwen2.5-VL + VAE

FireRed-Image-Editdouble-stream multimodal diffusion

LongCat-Image-Edit6B bilingual hybrid MMDiT / DiT

Understanding: Unifying Outputs as RGB

Monocular Depth Estimation

We prompt the editor to repaint the input as a grayscale depth visualization — near surfaces bright, far surfaces dark — then recover a dense relative depth map from per-pixel luminance. Because a generative editor produces relative depth with unknown scale and offset, we follow the standard affine-invariant protocol and fit scale and shift to the ground truth per image. Qwen-Image-Edit lands within a few δ₁ points of dedicated depth models on DIODE-indoor (0.938 vs. 0.952), despite having no depth-specific parameters at all.

Zero-shot depth under a disparity-space affine-invariant protocol. Specialist scores are quoted from Depth Anything V2 (ViT-L). δ₁↑ / AbsRel↓; bold marks the best per metric.

Method	NYUv2		KITTI		DIODE-indoor
Method	δ₁↑	AbsRel↓	δ₁↑	AbsRel↓	δ₁↑	AbsRel↓
Relative-depth specialists
MiDaS V3.1	0.980	0.048	0.850	0.127	0.942	0.075
Depth Anything V1	0.981	0.043	0.947	0.076	0.952	0.066
Depth Anything V2	0.979	0.045	0.946	0.074	0.952	0.066
RINO-Zero (generative image editors)
Qwen-Image-Edit	0.927	0.086	0.901	0.093	0.938	0.076
LongCat-Image-Edit	0.840	0.130	0.849	0.130	0.849	0.129

The same editors under the linear depth-space affine-invariant protocol (per-image least-squares scale + shift on GT depth). ^†Vision Banana reports raw metric depth with no alignment, shown only as a reference and not directly comparable. Bold marks the best editor per metric.

Method	NYUv2		KITTI		DIODE-indoor		iBims-1
Method	δ₁↑	AbsRel↓	δ₁↑	AbsRel↓	δ₁↑	AbsRel↓	δ₁↑	AbsRel↓
Metric reference (instruction-tuned)
Vision Banana^†	0.948	0.081	0.915	0.107	0.917	0.108	0.934	0.078
RINO-Zero (linear affine-invariant)
Qwen-Image-Edit	0.792	0.156	0.413	0.329	0.872	0.122	0.824	0.133
LongCat-Image-Edit	0.832	0.135	0.454	0.325	0.825	0.146	0.843	0.129

Qualitative zero-shot depth estimation on DIODE-indoor and KITTI. — Qualitative zero-shot depth estimation on DIODE-indoor (top) and KITTI (bottom).

Surface Normal Estimation

Normals follow the same RGB-to-RGB recipe: the editor repaints the scene as a standard normal map, where surface orientation is encoded by color, and we decode the RGB image back into per-pixel unit normals. FireRed-Image-Edit — a generic editor with no normal-specific training — is the strongest editor on all three datasets and trails the DSINE specialist only slightly in mean angular error, though DSINE stays sharper at the pixel level, as its lower median error shows.

Zero-shot surface-normal estimation. Mean↓ and median↓ angular error in degrees. Bold marks the best editor per metric.

Method	NYUv2		iBims-1		DIODE-indoor
Method	Mean↓	Median↓	Mean↓	Median↓	Mean↓	Median↓
Surface-normal specialists
DSINE	16.4	8.4	17.1	6.1	18.45	13.87
Marigold	20.86	11.13	18.46	8.44	16.67	12.08
StableNormal	19.71	10.53	17.25	8.06	13.70	9.46
Lotus-2	16.9	—	15.4	—	18.58	—
Vision Banana	17.78	8.88	—	—	13.82	11.56
RINO-Zero (ours, zero-shot, RGB → RGB)
Qwen-Image-Edit	20.21	13.46	20.66	12.03	21.99	18.14
FireRed-Image-Edit	17.73	10.99	18.62	10.38	17.25	13.64

Qualitative zero-shot surface normal estimation on NYUv2 and iBims-1. — Qualitative zero-shot surface normal estimation on NYUv2 (top) and iBims-1 (bottom).

Semantic Segmentation

Only the categories present in each image are named in the prompt. After color decoding we apply a lightweight parameter-free refinement — local majority filtering and small connected-component removal — to suppress color jitter and noisy pixels. ADE20K's large label space is handled by a hierarchical strategy: the model first paints 10 super-classes, then fills in fine-grained categories within each region. On PASCAL VOC, RINO's mAcc and aAcc even surpass the zero-shot segmentation specialists; under the coarse 10-class ADE20K setting it outperforms MaskCLIP.

Semantic segmentation on Pascal VOC and Cityscapes. Bold and underline mark the best and second-best among generative image editors.

Method	Pascal VOC			Cityscapes
Method	mIoU↑	mAcc↑	aAcc↑	mIoU↑	mAcc↑	aAcc↑
Zero-shot segmentation specialists
FC-CLIP	—	—	—	56.29	65.42	78.46
MaskCLIP	38.85	45.21	71.63	—	—	—
GroupViT	52.37	56.97	79.28	—	—	—
OVSegmentor	53.82	58.36	80.51	—	—	—
Generative image editors (zero-shot, RGB → RGB)
Vision Banana	—	—	—	69.90	—	—
Direct Editing Baseline	—	—	—	23.02	42.63	64.14
Ours (Qwen-Image-Edit)	44.69	48.00	85.17	24.30	44.38	66.59
Ours (FireRed-Image-Edit)	45.17	61.47	80.82	21.11	42.70	58.00
Ours (LongCat-Image-Edit)	49.68	66.90	83.53	13.42	26.72	49.23

Semantic segmentation on ADE20K under both the 10-class coarse setting and the standard 150-category setting. Bold and underline mark the best and second-best among generative image editors.

Method	ADE20K (10 classes)			ADE20K (150 categories)
Method	mIoU↑	mAcc↑	aAcc↑	mIoU↑	mAcc↑	aAcc↑
Zero-shot segmentation specialists
MaskCLIP	39.26	50.52	65.66	23.78	31.54	50.27
FC-CLIP	42.17	53.11	67.05	34.13	40.86	59.62
RINO-Zero (ours, zero-shot, RGB → RGB)
Qwen-Image-Edit	40.37	53.51	67.35	12.57	20.93	44.85
FireRed-Image-Edit	34.76	46.79	54.77	13.61	22.80	38.95
LongCat-Image-Edit	19.33	30.02	26.37	7.43	14.60	26.79

Zero-shot semantic segmentation on Cityscapes, Pascal VOC, and ADE20K. — Zero-shot semantic segmentation on Cityscapes (top), Pascal VOC (middle), and ADE20K (bottom).

Object Detection & Instance Segmentation

Both tasks run through per-class silhouette painting: for each present class we prompt the editor to paint its instances as solid white blobs on black, specifying the expected instance count. We threshold the output, split touching blobs with morphological opening, and treat each connected component as one instance — the blob is its mask, its tight box is the detection. Classes and counts come from the ground truth (oracle-class, oracle-count) to isolate localization from open-set recognition. Scores stay well below specialists, and the qualitative results show why: the failures are task-specific ones like over-detection and over-segmentation.

Object detection and instance segmentation on COCO (box and mask AP, %). Editors are run zero-shot under an oracle-class, oracle-count per-class silhouette protocol, the same blobs giving boxes and masks (all-points AP, macro-averaged over 80 classes). Bold marks the best editor.

Method	box AP		mask AP
Method	AP₅₀↑	AP↑	AP₅₀↑	AP↑
Supervised detect + segment models (AP quoted from their papers)
Mask DINO	—	50.5	—	46.0
Cascade Mask R-CNN	61.7	43.3	58.6	37.1
Mask R-CNN	63.5	42.8	60.5	38.3
Mask2Former	69.4	49.3	69.4	46.1
RINO-Zero (ours, zero-shot, RGB → RGB)
Qwen-Image-Edit	16.3	9.4	13.9	7.2
FireRed-Image-Edit	14.9	8.9	12.9	6.9
LongCat-Image-Edit	6.3	3.2	4.9	2.4

Qualitative object detection and instance segmentation on COCO val2017.

Panoptic Segmentation

The editor paints each region with a flat color — one named color per class, black for background — and we decode by nearest-color matching, treating stuff classes as single segments and splitting thing classes into connected components. The pattern in the numbers is telling: masks align well with object and region boundaries, giving high SQ (Qwen 65–74), while recognition and instance association lag (Qwen RQ 15–17). Coherent regions are often separated correctly but labeled with the wrong category. The gap is a recognition problem, not a spatial one.

Zero-shot panoptic segmentation. Supervised specialists report Panoptic Quality (PQ, %) as in their papers; editor cells report Segmentation Quality (SQ, %) and additionally give (PQ/RQ) in parentheses (our measurements). Bold marks the best SQ.

Method	Cityscapes	ADE20K	COCO
Supervised specialists (PQ)
EoMT	—	51.7	58.3
ViT-P	70.8	51.9	58.0
OneFormer	67.2	49.8	57.9
Mask2Former	66.6	48.1	57.8
RINO-Zero — SQ (PQ / RQ)
Qwen-Image-Edit	64.8 (13.6 / 17.4)	71.5 (12.3 / 15.9)	74.3 (11.9 / 15.5)
FireRed-Image-Edit	55.9 (8.7 / 11.2)	67.2 (9.4 / 12.5)	70.7 (9.1 / 12.1)
LongCat-Image-Edit	55.4 (1.0 / 1.4)	67.6 (6.7 / 8.4)	75.2 (6.9 / 8.8)

Qualitative zero-shot panoptic segmentation on COCO and ADE20K. — Qualitative zero-shot panoptic segmentation on COCO (top) and ADE20K (bottom).

Human Pose Estimation

Each person is cropped with its ground-truth box and letterboxed to a fixed canvas; the editor repaints the crop as an OpenPose-style colored skeleton on black, and we decode COCO-17 keypoints from the output. Since editors do not reliably reproduce a fixed color palette, the decoder is color-agnostic: joint candidates are peaks of the foreground distance transform, labeled by matching to a canonical upright skeleton template. Precise localization remains hard, but the jump under similarity alignment shows the editors do capture coarse pose structure.

Zero-shot 2D human pose estimation on COCO val2017 single-person crops. PCK is bbox-normalized and reported both in absolute image coordinates (raw) and after per-image similarity alignment (PA), which isolates pose structure by removing scale and translation. Dedicated pose estimators are evaluated under the same top-down protocol.

Method	raw PCK@0.05↑	PA PCK@0.05↑	PA PCK@0.2↑
2D-pose specialists (run under our protocol)
ViTPose	0.868	0.889	0.984
Keypoint R-CNN	0.732	0.765	0.919
YOLO-pose	0.730	0.782	0.953
Mesh-recovery specialist
SAM 3D Body	0.868	—	—
RINO-Zero (generative image editors)
LongCat-Image-Edit	0.233	0.282	0.767
Qwen-Image-Edit	0.060	0.235	0.774
FireRed-Image-Edit	0.047	0.243	0.795

Qualitative zero-shot 2D human pose estimation.

Referring Expression Comprehension & Segmentation

Language grounding through image editing. For each referring expression we prompt the editor to cover the referred object in solid red while leaving all other pixels unchanged, then difference the edited output against the input and keep the largest changed connected component: its pixels are the mask, its tight box is the comprehension result. The editor sees only the image and the expression — no boxes, classes, or instance counts, and no grounding head. This is substantially harder than oracle-class segmentation: the model must parse the expression, pick the target instance, and express the grounded region as an edit.

Referring expression comprehension (REC) and segmentation (RES) on RefCOCOg (UMD split, val). REC: box Prec@0.5; RES: cIoU / mIoU. Bold marks the best editor per metric.

Method	REC	RES
Method	Prec@0.5↑	cIoU↑	mIoU↑
Supervised grounding specialists
LAVT	—	61.2	—
UNINEXT	88.7	74.7	—
OneRef	88.1	—	73.2
Generative image models
Vision Banana	—	73.8	—
Qwen-Image-Edit	51.8	40.3	45.3
LongCat-Image-Edit	47.7	35.0	38.7
FireRed-Image-Edit	47.6	35.7	41.6

Qualitative referring expression comprehension and segmentation on RefCOCOg.

Open-Vocabulary Instance Segmentation (SA-Co)

For each query we prompt the editor once to repaint the image as a binary mask: everything matching the phrase pure black, everything else pure white, and fully white if the phrase is absent. Black pixels become foreground and split into instances by connected components. The model gets no class list and no instance count — just a free-form noun phrase that may or may not be in the image, so it must do both localization and presence detection. Localization works; presence detection is the weak point. The editors paint a mask even when the phrase is absent, which pins IL_MCC near zero and drags the headline cgF1 down with it. Even Vision Banana needs an external MLLM gate to get a meaningful IL_MCC.

Open-vocabulary instance segmentation on the attributes subset of SA-Co/Gold (cgF1 = 100 · pmF1 · IL_MCC). pmF1 is gate-independent, and there RINO is comparable to specialist models. SAM 3 is supervised; Vision Banana adds an MLLM presence gate. Bold marks the best editor.

Method	pmF1↑	IL_MCC↑	cgF1↑
Supervised specialist (quoted, not zero-shot transfer)
SAM 3	72.0	0.76	54.9
Zero-shot transfer
gDino-T	47.3	0.29	13.8
Vision Banana + Gemini	68.7	0.86	58.8
RINO-Zero (ours, zero-shot, RGB → RGB)
Qwen-Image-Edit	39.8	0.04	1.7
FireRed-Image-Edit	30.9	0.01	0.2
LongCat-Image-Edit	17.3	0.09	1.6

Qualitative open-vocabulary instance segmentation on SA-Co/Gold.

Generation: Unifying Inputs as RGB

Depth-Conditioned Image Generation

RINO hands the depth map to the editor as an ordinary image condition and asks for the corresponding natural image — no depth encoder, ControlNet branch, adapter, or task-specific decoder. Among the zero-shot editors Qwen is the most balanced (RMSE-255 33.83, FID 19.44, CLIPScore 30.15). It trails the strongest trained controllers like ControlNet++ and ControlAR, but competes with several trained adapter baselines on depth fidelity while using nothing but a zero-shot RGB editing interface.

Depth-conditioned generation on the MultiGen-20M depth validation split. External rows are paper-reported anchors from ControlNet++ and ControlAR; our editor rows are zero-shot RGB-in/RGB-out runs under a shared prompt, seed 0, steps=8, CFG=2.5. RMSE-255↓ measures DPT-Large depth consistency, FID↓ image quality, CLIPScore↑ text alignment.

Method	N	RMSE-255↓	FID↓	CLIPScore↑
Trained controllable generation models
ControlNet++	5000	28.32	16.66	32.09
ControlAR	5000	29.01	14.61	—
ControlNet (SD1.5)	5000	35.90	17.76	32.45
GLIGEN	5000	38.83	18.36	31.75
T2I-Adapter	5000	48.40	22.52	31.46
Uni-ControlNet	5000	40.65	20.27	31.66
UniControl	5000	39.18	18.66	32.45
RINO-Zero (ours, zero-shot, RGB depth input)
Qwen-Image-Edit	5000	33.83	19.44	30.15
FireRed-Image-Edit	5000	37.55	21.43	30.34
LongCat-Image-Edit	5000	36.50	27.18	28.09

Qualitative depth-conditioned generation on MultiGen-20M. The same RGB depth maps and text prompts are given to all three zero-shot editors. The RGB-in/RGB-out interface preserves coarse scene geometry, while editor backbones differ in texture realism, semantic fidelity, and layout drift.

Semantic Segmentation Map Conditioned Generation

The segmentation map goes in as an ordinary RGB condition, and the editor is asked for a natural image whose semantic layout follows the per-region class colors. Since prompt richness materially affects both image quality and condition fidelity, we re-caption every validation image once with an open-source VLM and use that caption for all editors. All three cluster tightly around 46 mIoU on ADE20K and 37 on COCOStuff, ahead of ControlNet++ on both. Image quality tracks the editor backbone, not the conditioning interface: Qwen and FireRed are essentially tied, LongCat trails on FID.

Semantic segmentation map conditioned generation on ADE20K and COCOStuff. ControlNet++ is included as a task-trained grounded generation reference. Image-editor rows are run zero-shot with RGB segmentation maps and VLM-recaptioned prompts at the native input resolution.

Method	ADE20K					COCOStuff
Method	mIoU↑	mAcc↑	aAcc↑	FID↓	CLIP↑	mIoU↑	mAcc↑	aAcc↑	FID↓	CLIP↑
ControlNet	32.55	—	—	33.28	31.53	27.46	—	—	21.33	13.31
ControlNet++	43.64	—	—	29.49	31.96	34.56	—	—	19.29	13.13
RINO-Zero (ours, zero-shot, RGB → RGB)
Qwen-Image-Edit	46.24	61.92	79.13	32.05	32.62	37.40	50.64	63.50	15.54	34.14
FireRed-Image-Edit	45.82	62.82	78.14	31.70	32.67	37.55	51.20	63.25	15.94	34.07
LongCat-Image-Edit	46.37	61.62	78.91	36.08	32.50	36.38	48.93	61.46	20.95	33.89

Qualitative semantic segmentation map conditioned generation on ADE20K. The RINO interface preserves the semantic layout of the input, while editor backbones differ in texture realism and small-region fidelity.

Human-Pose Conditioned Generation

Ground-truth COCO-17 keypoints are rendered into an OpenPose-style colored skeleton (16 limb segments on a black canvas). The editor receives only that RGB skeleton and the associated caption, and is instructed to align each body part with the corresponding colored bone while leaving the skeleton itself out of the synthesized image. No pose encoder, ControlNet branch, or trainable adapter. RINO's zero-shot results are already comparable to in-domain pose-guided generation models — and qualitatively it often follows the input pose better than ControlNet-based baselines, especially for unusual body configurations and diverse artistic styles.

Human-pose conditioned generation on Human-Art validation. Baselines are quoted from Stable-Pose; * marks released checkpoints. Our editor rows are zero-shot. Bold marks the best baseline per column.

Method	Pose Accuracy		Image Quality		T2I Alignment
Method	CAP↑	PCE↓	FID↓	KID↓	CLIP-score↑
Trained pose-guided generation models
SD*	55.71	2.30	11.53	3.36	33.33
T2I-Adapter	65.65	1.75	11.92	2.73	33.27
ControlNet	69.19	1.54	11.01	2.23	32.65
Uni-ControlNet	69.32	1.48	14.63	2.30	32.51
GLIGEN	69.15	1.46	—	—	32.52
HumanSD	69.68	1.37	10.03	2.70	32.24
Stable-Pose	70.83	1.50	11.12	2.35	32.60
RINO-Zero (generative image editors)
Qwen-Image-Edit	65.97	1.76	36.01	5.83	32.96
FireRed-Image-Edit	68.09	1.82	10.66	5.23	34.24
LongCat-Image-Edit	55.54	2.02	7.91	6.25	32.22

Zero-shot human-pose conditioned generation across styles. — Zero-shot human-pose conditioned generation across styles: input skeleton, GT, our editors (Qwen / LongCat / FireRed), and ControlNet-OpenPose.

Instance Map Conditioned Generation

Unlike a semantic map, this condition distinguishes individual object instances of the same class. RINO renders the instance map as an ordinary RGB condition — no instance encoder, grounding module, detector head, or adapter. The editors follow per-instance layout closely: LongCat reaches the highest AP (28.0) and AR (44.2), surpassing the task-trained InstanceDiffusion reference on both, while Qwen and FireRed stay within a few points. The trained reference still leads at the stricter AP₅₀ threshold, so precise instance boundaries are harder to recover than coarse placement. Image quality is better than the trained reference across all editors.

Instance map conditioned generation on COCO2017 validation. InstanceDiffusion is included as a task-trained grounded generation reference. Image-editor rows are run zero-shot with RGB instance maps and VLM-recaptioned prompts at the native input resolution.

Method	AP↑	AP₅₀↑	Average Recall↑	FID↓
Reported task-trained grounded generation result
InstanceDiffusion	27.1	50.0	38.1	25.5
RINO-Zero (ours, zero-shot, RGB instance map input)
Qwen-Image-Edit	25.4	44.3	40.8	15.3
FireRed-Image-Edit	23.2	42.3	37.4	16.0
LongCat-Image-Edit	28.0	46.3	44.2	18.5

Qualitative instance map conditioned generation on COCO2017 validation. The same RGB instance maps and re-captioned prompts are given to all three zero-shot editors, and the RGB-in/RGB-out interface preserves per-instance placements without an instance encoder.

Canny-Conditioned Image Generation

The hardest regime for RINO: a binary edge map, where zero-shot Qwen-Image-Edit reaches Canny F1 14.90, about 0.40× purpose-trained ControlNet++ (37.04), and trails on FID and CLIP-Score too — as expected for a general instruction editor with no edge-conditioned training. The ablation matters more than the absolute number: dropping the edge map and keeping only the caption collapses fidelity to 6.75, and describing the edge map in words with a VLM instead of passing the image reaches only 7.79. The Canny image carries pixel-level spatial information that a language bottleneck cannot substitute.

Canny-conditioned generation on MultiGen-20M validation (5000 images, 512², 4 samples each). Edge controllability is F1 (cv2.Canny(100,200), ×100); image quality is FID; text alignment is CLIP-Score (CLIP ViT-B/16) — all following the ControlNet++ protocol, with FID and CLIP averaged over the 4 generated groups. Our Qwen-Image-Edit row is zero-shot; trained rows are paper-reported anchors. Bold marks the best zero-shot per column.

Method	F1↑	FID↓	CLIPScore↑
Zero-shot (ours and baselines)
Raw Caption-only	6.75	71.61	27.78
Raw Caption + Canny Caption	7.79	59.39	23.98
Qwen-Image-Edit (canny + caption)	14.90	58.30	27.31
Trained for canny conditioning
T2I-Adapter	23.65	15.96	31.71
GLIGEN	26.94	18.89	31.77
Uni-ControlNet	27.32	17.14	31.84
UniControl	30.82	19.94	31.97
ControlNet (SD1.5)	34.65	14.73	32.15
ControlNet++ (SD1.5)	37.04	18.23	31.87

Qualitative canny-conditioned generation on MultiGen-20M.

Limitations & Outlook

RINO's transfer ability rests on generic image editing backbones whose web-scale pretraining data may already contain diverse visual signals such as segmentation masks and depth maps. Those signals likely occupy only a very small slice of the pretraining distribution, which is one reason a gap to in-domain experts remains on some tasks. Accuracy is bounded both by the visual priors of the editing backbone and by the parameter-free RGB read-out. We call the current fully zero-shot version RINO-Zero.

The natural way to close the gap is to instruction-tune existing image editors on standard editing data mixed with a proportion of RGB-formatted non-natural visual data — segmentation masks, depth maps, and the like — so that the RINO interface is present during training across a wider range of tasks. We hope this work offers useful insight for future unified vision-language systems, and invites extending the paradigm to 3D vision, video, and world models.

BibTeX

@misc{yang2026letrgblanguagevision,
      title={Let RGB Be the Language of Vision},
      author={Timing Yang and Jinrui Yang and Xinlong Li and Yuhan Wang and Haoran Li and Yanqing Liu and Guoyizhe Wei and Jixuan Ying and Chen Wei and Rama Chellappa and Yuyin Zhou and Cihang Xie and Alan Yuille and Feng Wang},
      year={2026},
      eprint={2607.12450},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.12450},
}