Authors:
Junhao Zhuang, Xuan Ju, Zhaoyang Zhang, Yong Liu, Shiyi Zhang, Chun Yuan, Ying Shan
Your star means a lot for us to develop this project!
:star:
🌟 Abstract
Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial application.
To address this, we propose
ColorFlow
, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable
Retrieval Augmented Colorization
pipeline for colorizing images with relevant color references.
Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching.
To evaluate our model, we introduce
ColorFlow-Bench
, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry.
🚀 Getting Started
Follow these steps to set up and run ColorFlow on your local machine:
Clone the Repository
Download the code from our GitHub repository:
git clone https://github.com/TencentARC/ColorFlow
cd ColorFlow
Set Up the Python Environment
Ensure you have Anaconda or Miniconda installed, then create and activate a Python environment and install required dependencies:
You can launch the Gradio interface for PowerPaint by running the following command:
python app.py
Access ColorFlow in Your Browser
Open your browser and go to
http://localhost:7860
. If you're running the app on a remote server, replace
localhost
with your server's IP address or domain name. To use a custom port, update the
server_port
parameter in the
demo.launch()
function of app.py.
🎉 Demo
You can
try the demo
of ColorFlow on Hugging Face Space.
🛠️ Method
The overview of ColorFlow. This figure presents the three primary components of our framework: the
Retrieval-Augmented Pipeline (RAP)
, the
In-context Colorization Pipeline (ICP)
, and the
Guided Super-Resolution Pipeline (GSRP)
. Each component is essential for maintaining the color identity of instances across black-and-white image sequences while ensuring high-quality colorization.
🤗 We welcome your feedback, questions, or collaboration opportunities. Thank you for trying ColorFlow!
📰 News
Release Date:
2024.12.17 - Inference code and model weights have been released! 🎉
📋 TODO
✅ Release inference code and model weights
⬜️ Release training code
📜 Citation
@article{zhuang2024colorflow,
title={ColorFlow: Retrieval-Augmented Image Sequence Colorization},
author={Zhuang, Junhao and Ju, Xuan and Zhang, Zhaoyang and Liu, Yong and Zhang, Shiyi and Yuan, Chun and Shan, Ying},
journal={},
year={2024}
}
More Information About ColorFlow huggingface.co Model
ColorFlow huggingface.co
ColorFlow huggingface.co is an AI model on huggingface.co that provides ColorFlow's model effect (), which can be used instantly with this TencentARC ColorFlow model. huggingface.co supports a free trial of the ColorFlow model, and also provides paid use of the ColorFlow. Support call ColorFlow model through api, including Node.js, Python, http.
ColorFlow huggingface.co is an online trial and call api platform, which integrates ColorFlow's modeling effects, including api services, and provides a free online trial of ColorFlow, you can try ColorFlow online for free by clicking the link below.
TencentARC ColorFlow online free url in huggingface.co:
ColorFlow is an open source model from GitHub that offers a free installation service, and any user can find ColorFlow on GitHub to install. At the same time, huggingface.co provides the effect of ColorFlow install, users can directly use ColorFlow installed effect in huggingface.co for debugging and trial. It also supports api for free installation.