Handling test-time unknown degradation is the major challenge in Blind Image Restoration (BIR), necessitating high model generalization. An effective strategy is to incorporate prior knowledge, either from human input or generative model.
We introduce Instant-reference Image Restoration (InstantIR), a novel diffusion-based BIR method which dynamically adjusts generation condition during inference. InstantIR first extracts a compact representation of the input via a pre-trained vision encoder. At each generation step, this representation is used to decode current diffusion latent and instantiate a generative prior. The degraded input is then encoded with this reference, providing robust generation condition.
We observe the variance of generative references fluctuate with degradation intensity, which we further leverage as an indicator for developing a sampling algorithm adaptive to input quality. Extensive experiments demonstrate InstantIR achieves state-of-the-art performance and offering outstanding visual quality. Through modulating generative references with textual description, InstantIR can restore extreme degradation and additionally feature creative restoration.
InstantIR actively aligns the condition encoding with generative prior. This is achieved by a novel previewing mechanism with three key modules: (1) DCP for compact LQ image representation encoding; (2) Previewer for decoding it into generative prior; and (3) Aggregator for integrating the generative reference and LQ input into sampling conditions.
InstantIR pipeline
InstantIR is able to recover rich and realistic texture details in real-world degradation images.
Comparison across SOTA models. LQ stands for Low-Quality input, zoom in for a better view!
Although InstantIR is not explicitly trained on text-image paired data, it demonstrates interesting text-editing ability which is featured by manipulating generative references with textual descriptions.
Creative restoration following text-guidance.
Our work is sponsored by HuggingFace and fal.ai.
@article{huang2024instantir,
title={InstantIR: Blind Image Restoration with Instant Generative Reference},
author={Huang, Jen-Yuan and Wang, Haofan and Wang, Qixun and Bai, Xu and Ai, Hao and Xing, Peng and Huang, Jen-Tse},
journal={arXiv preprint arXiv:2410.06551},
year={2024}
}