Real-ESRGAN: Training Real-World Blind Super-Resolution
with Pure Synthetic Data
Real-ESRGAN: 使用纯合成数据训练真实世界盲超分辨率

Blind SR aims to restore high-resolution images from low-resolution ones with unknown and complex degradations. The classical degradation model [11, 29] is usually adopted to synthesize the low-resolution input. Generally, the ground-truth image $\bm{y}$ is first convolved with blur kernel $\bm{k}$. Then, a downsampling operation with scale factor $r$ is performed. The low-resolution $\bm{x}$ is obtained by adding noise $\bm{n}$. Finally, JPEG compression is also adopted, as it is widely-used in real-world images.

$$\vspace{-0.2cm}\bm{x}=\mathcal{D}(\bm{y})=[(\bm{y}\circledast\bm{k})\downarrow_{r}+\bm{n}]_{\mathtt{JPEG}},$$

(1)

where $\mathcal{D}$ denotes the degradation process. In the following, we briefly revisit these commonly-used degradations. The detailed settings are specified in Sec. 4.1. More descriptions and examples are in Appendix. A.

Blur. We typically model blur degradation as a convolution with a linear blur filter (kernel). Isotropic and anisotropic Gaussian filters are common choices. For a Gaussian blur kernel $\bm{k}$ with a kernel size of $2t+1$, its $(i,j)\in[-t,t]$ element is sampled from a Gaussian distribution, formally:

$\displaystyle\vspace{-0.4cm}\bm{k}(i,j)$

$\displaystyle=\frac{1}{N}\exp(-\frac{1}{2}\bm{C}^{T}\bm{\Sigma}^{-1}\bm{C}),\quad\bm{C}=[i,j]^{T},$

(2)

where $\bm{\Sigma}$ is the covariance matrix; $\bm{C}$ is the spatial coordinates; $N$ is the normalization constant. The covariance matrix could be further represented as follows:

	$\displaystyle\vspace{-0.2cm}\bm{\Sigma}$	$\displaystyle=\bm{R}\begin{bmatrix}\sigma_{1}^{2}&0\\ 0&\sigma_{2}^{2}\end{bmatrix}\bm{R}^{T},\quad\text{($\bm{R}$ is the rotation matrix)}$		(3)
		$\displaystyle=\begin{bmatrix}cos\theta&-sin\theta\\ sin\theta&cos\theta\end{bmatrix}\begin{bmatrix}\sigma_{1}^{2}&0\\ 0&\sigma_{2}^{2}\end{bmatrix}\begin{bmatrix}cos\theta&sin\theta\\ -sin\theta&cos\theta\end{bmatrix},$		(4)

where $\sigma_{1}$ and $\sigma_{2}$ are the standard deviation along the two principal axes (i.e., eigenvalues of the covariance matrix); $\theta$ is the rotation degree. When $\sigma_{1}=\sigma_{2}$, $\bm{k}$ is an isotropic Gaussian blur kernel; otherwise $\bm{k}$ is an anisotropic kernel.

Poisson noise follows the Poisson distribution. It is usually used to approximately model the sensor noise caused by statistical quantum fluctuations, that is, variation in the number of photons sensed at a given exposure level. Poisson noise has an intensity proportional to the image intensity, and the noises at different pixels are independent.

Resize (Downsampling). Downsampling is a basic operation for synthesizing low-resolution images in SR. More generally, we consider both downsamping and upsampling, i.e., the resize operation. There are several resize algorithms - nearest-neighbor interpolation, area resize, bilinear interpolation, and bicubic interpolation. Different resize operations bring in different effects - some produce blurry results while some may output over-sharp images with overshoot artifacts.

In order to include more diverse and complex resize effects, we consider a random resize operation from the above choices. As nearest-neighbor interpolation introduces the misalignment issue, we exclude it and only consider the area, bilinear and bicubic operations.

JPEG compression. JPEG compression is a commonly used technique of lossy compression for digital images. It first converts images into the YCbCr color space and downsamples the chroma channels. Images are then split into $8\times 8$ blocks and each block is transformed with a two-dimensional discrete cosine transform (DCT), followed by a quantization of DCT coefficients. More details of JPEG compression algorithms can be found in [43]. Unpleasing block artifacts are usually introduced by the JPEG compression.

The quality of compressed images is determined by a quality factor $q\in[0,100]$, where a lower $q$ indicates a higher compression ratio and worse quality. We use the PyTorch implementation - $\mathtt{DiffJPEG}$ [32].

When we adopt the above classical degradation model to synthesize training pairs, the trained model could indeed handle some real samples. However, it still can not resolve some complicated degradations in the real world, especially the unknown noises and complex artifacts (see Fig. 3). It is because that the synthetic low-resolution images still have a large gap with realistic degraded images. We thus extend the classical degradation model to a high-order degradation process to model more practical degradations.

The classical degradation model only includes a fixed number of basic degradations, which can be regarded as a first-order modeling. However, the real-life degradation processes are quite diverse, and usually comprise a series of procedures including imaging system of cameras, image editing, Internet transmission, etc. For instance, when we want to restore a low-quality image download from the Internet, its underlying degradation involves a complicated combination of different degradation processes. Specifically, the original image might be taken with a cellphone many years ago, which inevitably contains degradations such as camera blur, sensor noise, low resolution and JPEG compression. The image was then edited with sharpening and resize operations, bringing in overshoot and blur artifacts. After that, it was uploaded to some social media applications, which introduces a further compression and unpredictable noises. As the digital transmission will also bring artifacts, this process becomes more complicated when the image spreads several times on the Internet.

Such a complicated deterioration process could not be modeled with the classical first-order model. Thus, we propose a high-order degradation model. An $n$-order model involves $n$ repeated degradation processes (as shown in Eq. 5), where each degradation process adopts the classical degradation model (Eq. 1) with the same procedure but different hyper-parameters. Note that the “high-order” here is different from that used in mathematical functions. It mainly refers to the implementation time of the same operation. The random shuffling strategy in [55] may also include repeated degradation processes (e.g., double blur or JPEG). But we highlight that the high-order degradation process is the key, indicating that not all the shuffled degradations are necessary. In order to keep the image resolution in a reasonable range, the downsampling operation in Eq. 1 is replaced with a random resize operation. Empirically, we adopt a second-order degradation process, as it could resolve most real cases while keeping simplicity. Fig. 2 depicts the overall pipeline of our pure synthetic data generation pipeline.

$$\bm{x}=\mathcal{D}^{n}(\bm{y})=(\mathcal{D}_{n}\circ\cdots\circ\mathcal{D}_{2}\circ\mathcal{D}_{1})(\bm{y}).$$

(5)

It is worth noting that the improved high-order degradation process is not perfect and could not cover the whole degradation space in the real world. Instead, it merely extends the solvable degradation boundary of previous blind SR methods through modifying the data synthesis process. Several typical limitation scenarios can be found in Fig. 11.

Ringing artifacts often appear as spurious edges near sharp transitions in an image. They visually look like bands or “ghosts” near edges. Overshoot artifacts are usually combined with ringing artifacts, which manifest themselves as an increased jump at the edge transition. The main cause of these artifacts is that the signal is bandlimited without high frequencies. These artifacts are very common and usually produced by a sharping algorithm, JPEG compression, etc. Fig. 5 (Top) shows some real samples suffering from ringing and overshoot artifacts.

We employ the $sinc$ filter, an idealized filter that cuts off high frequencies, to synthesize ringing and overshoot artifacts for training pairs. The $sinc$ filter kernel can be expressed as¹¹1We use the implementation in this url.:

$\displaystyle\bm{k}(i,j)$

$\displaystyle=\frac{\omega_{c}}{2\pi\sqrt{i^{2}+j^{2}}}J_{1}(\omega_{c}\sqrt{i^{2}+j^{2}}),$

(6)

where $(i,j)$ is the kernel coordinate; $\omega_{c}$ is the cutoff frequency; and $J_{1}$ is the first order Bessel function of the first kind. Fig. 5 (Bottom) shows $sinc$ filters with different cutoff frequencies, and their corresponding filtered images. It is observed that it could well synthesize ringing and overshoot artifacts (especially introduced by over-sharp effects). These artifacts are visually similar to those in the first two real samples in Fig. 5 (Top).

We adopt $sinc$ filters in two places: the blurring process and the last step of the synthesis. The order of the last $sinc$ filter and JPEG compression is randomly exchanged to cover a larger degradation space, as some images may be first over-sharpened (with overshoot artifacts) and then have JPEG compression; while some images may first do JPEG compression followed by sharpening operation.

In the meanwhile, the U-Net structure and complicate degradations also increase the training instability. We employ the spectral normalization regularization  [37] to stabilize the training dynamics. Moreover, we observe that spectral normalization is also beneficial to alleviate the over-sharp and annoying artifacts introduced by GAN training. With those adjustments, we are able to easily train the Real-ESRGAN and achieve a good balance of local detail enhancement and artifact suppression.

We employ Gaussian noises and Poisson noises with a probability of $\{0.5,0.5\}$. The noise sigma range and Poisson noise scale are set to $[1,30]$ and $[0.05,3]$, respectively ($[1,25]$ and $[0.05,2.5]$ for the second degradation process). The gray noise probability is set to 0.4. JPEG compression quality factor is set to $[30,95]$. The final $sinc$ filter is applied with a probability of 0.8. More details can be found in the released codes.

We compare our Real-ESRGAN with several state-of-the-art methods, including ESRGAN [50], DAN [34], CDC [51], RealSR [19] and BSRGAN [55]. We test on several diverse testing datasets with real-world images, including RealSR [5], DRealSR [51], OST300 [49], DPED [18], ADE20K validation [59] and images from Internet. Since existing metrics for perceptual quality cannot well reflect the actual human perceptual preferences on the fine-grained scale [3], we present several representative visual samples in Fig. 7. The quantitative results are also included in the Appendix. B for reference.

It can be observed from Fig. 7 that our Real-ESRGAN outperforms previous approaches in both removing artifacts and restoring texture details. Real-ESRGAN+ (trained with sharpened ground-truths) can further boost visual sharpness. Specifically, the first sample contains overshoot artifacts (white edges around letters). Directly upsampling will inevitably amplify those artifacts (e.g., DAN and BSRGAN). Real-ESRGAN takes such common artifacts into consideration and simulates them with $sinc$ filter, thus effectively removing ringing and overshoot artifacts. The second sample contains unknown and complicated degradations. Most algorithms can not effectively eliminate them while Real-ESRGAN trained with second-order degradation processes could. Real-ESRGAN is also capable of restoring more realistic textures (e.g., brick, mountain and tree textures) for real-world samples, while other methods either fail to remove degradations or add unnatural textures (e.g., RealSR and BSRGAN).

Though Real-ESRGAN is able to restore most real-world images, it still has some limitations. As shown in Fig. 11, 1) some restored images (especially building and indoor scenes) have twisted lines due to aliasing issues. 2) GAN training introduces unpleasant artifacts on some samples. 3) It could not remove out-of-distribution complicated degradations in the real world. Even worse, it may amplify these artifacts. These drawbacks have great impact on the practical application of Real-ESRGAN, which are in urgent need to address in future works.