Test Time Adaptation for Blind Image Quality Assessment
测试时自适应用于无参考图像质量评估

One of the first pieces of work on TTA [35] introduces a joint training framework using a loss for the main task and a self-supervised auxiliary task loss. The choice of a relevant auxiliary task is a challenging part of the design. Sun et al. [35] use rotation prediction as a pre-text task for image classification applications. Researchers also explore simpler tasks that are highly correlated with the main task, such as feature alignment with the batch and a simple contrastive loss framework [24] for adaptation. However, these methods require the source data to train the source model all over again to enable TTA.

TENT [37] studies source-free TTA, where entropy minimization-based adaptation even outperforms some methods that use the source data. Moreover, only the batch normalization parameters of the model are adapted in TENT. There are several works on batch norm statistics adaptation [29, 32] to improve the robustness of the model at test time. SHOT [20] presents a clustering-based pseudo-labeling method to align features from the target domain to the source domain using an information maximization loss. While a plethora of methods exists as above, the auxiliary tasks in these methods are not relevant for IQA.

Most classical NR IQA algorithms are mainly based on natural scene statistics (NSS) [26, 42, 23]. In [27], the naturalness of the distorted image in the wavelet domain is modeled based on NSS. Saad et al. [31] also design the NSS model in the discrete cosine transform (DCT) domain. CORNIA [39], and HOSA [38] are among the earliest codebook learning based methods to predict quality. With the emergence of deep learning and the availability of large subject-rated IQA databases, various general-purpose NR IQA methods have been designed based on convolutional neural network (CNN) architectures [40, 22, 2, 25]. Some of these methods require end-to-end training [2, 25] of deep neural networks (DNN), while others [43, 34, 9, 15] are based on updating pre-trained models with some modifications.

More recently, Zhang et al. [43] propose a method to train bilinear DNNs to simultaneously model both authentic and synthetic distortions. Su et al. [34] design a self-adaptive hyper network to provide weights for the quality prediction module parameters and handle various types of distortions and content in the images. Several methods [9, 15] explore transformer-based architectures along with a CNN to capture dependencies between local and global features. MetaIQA [44] proposes meta-learning on synthetic distortions by using a shared quality prior knowledge model to adapt to any kind of distortion.

All the existing methods assume that the train and test data come from the same distribution. If there is a distribution shift across different databases, we need adaptation of the pre-trained model to learn about the target distributional information.

Let the model trained on the source data be $f_{\theta}$ where $\theta=(\theta_{e},\theta_{c})$ corresponds to the parameters of the network, and $\theta_{e}$ and $\theta_{c}$ correspond to the parameters of the feature extractor and regression layers. Thus, for an input image $x$, we denote $f_{\theta}(x)=f_{\theta_{c}}(f_{\theta_{e}}(x))$. Since our goal is to learn the distribution shift between train and test data, we only update the parameters of the feature extractor, $\theta_{e}$, to align the features between the train and test distributions in a lower dimensional space.

The key challenge in TTA is the choice of a self-supervised auxiliary task that is highly correlated with the main task of IQA. However, relying too much on the auxiliary task can affect the performance of the main task. To prevent loss of learnt information induced by the auxiliary task [4], we first project the features to a lower dimensional space using a non-linear projection head $f_{\theta_{s}}$, parameterized by $\theta_{s}$. The auxiliary task drives the adaptation of the feature extractor through the projection head. Thus our model now has three parts parametrized by $(\theta_{e},\theta_{s},\theta_{c})$ to resemble a Y-shape as shown in Figure 1.

When a batch of test instances $D$ arrives, we extract features from the feature extractor, followed by the projection head, and update the set of parameters ($\theta_{e},\theta_{s})$ by optimizing a self-supervised objective function $\mathcal{L}_{s}(D)$. Updating all the model parameters of the feature extractor $\theta_{e}$ can cause the model to diverge too much from training, and the performance can drop drastically. Inspired by prior work on test time entropy minimization [37] and improving robustness for test data [32], we only update the linear and lower-dimensional feature modulation parameters. In a neural network, normalization layers satisfy these properties. So we only adapt the batch normalization layers by updating the affine parameters to mitigate distributional shifts. Thus we update these parameters by optimizing the auxiliary task loss given by

After optimizing the above loss function, we use the updated feature extractor parameters $\theta_{e}^{*}$ and pre-trained quality regressor parameters $\theta_{c}$ to predict the quality for a batch of test data. Since the distribution across batches can vary significantly, we ignore the earlier updated model and start the TTA based on source model weights for new test instants. Thus the updated target model always only depends on the incoming test data.

Our goal is to carefully choose self-supervised auxiliary tasks that capture quality-aware distributional information to adapt the feature encoder. We formulate two novel and complementary self-supervised learning techniques which help to learn the distribution shift between train and test data. These self-supervised objective functions are - 1) Group Contrastive Loss and 2) Rank Loss. While the GC loss works well when there is a reasonable separation of quality among the samples in a batch, the rank loss works better even when the quality of the batch samples is similar. On the other hand, the rank loss is meaningful only when the quality of the input image is not extremely low, while the GC loss is independent of the quality of a given image. Thus, a combination of the two losses renders our TTA extremely effective across various scenarios.

We evaluate TTA on four popular IQA databases using four different quality-aware models. In particular, we consider state-of-the-art deep IQA models such as TReS [9], MUSIQ [15] , HyperIQA [34], and MetaIQA [44]. These models contain a ResNet backbone with batch normalization layers, which we model as the feature extractor $f_{\theta_{e}}$ and only update its batch normalization parameters. We model the rest of the network as the quality regressor part, $f_{\theta_{c}}$. TReS and MUSIQ use transformers as a part of their architecture, which we include as a part of the quality regressor in all our main experiments. We also explore the adaptation of transformers by including them as part of the feature extractor and updating their layer normalization statistics in the supplementary material.

We project the features extracted from the last layer of ResNet through a 256-dimensional fully connected (FC) layer corresponding to the self-supervised projection head $f_{\theta_{s}}$. These lower dimensional features are used to adapt the model at test time. Three IQA models, TReS, MUSIQ, and HyperIQA are trained on the LIVEFB database [41] containing 39,810 images. MetaIQA is trained on two synthetically distorted databases, TID2013 [30] and KADID-10k [21].

We choose challenging databases to evaluate the generalization capability of our TTA-IQA method. The test datasets are described as follows:

KonIQ-10k [11] is a popular in-the-wild authentically distorted database consisting of 10,073 quality-scored images.

PIPAL [13] is a large-scale IQA database for evaluating perceptual image restoration. It contains 29k distorted images, including 19 different GAN-based algorithms.

CID2013 [36] consists of 480 images in six image sets captured by 79 imaging devices.

LIVE-IQA [33] consists of 779 synthetically distorted images from 29 reference images.

We evaluate our results using Spearman’s rank-order correlation coefficient (SROCC) and Pearson’s linear correlation coefficient (PLCC).

We implement our setup in PyTorch and conduct all the experiments with an 16 GB NVIDIA RTX A4000 GPU. During TTA, we randomly select a patch of size $224\times 224$ from the input image and apply quality preserving augmentations such as horizontal flip and vertical flip before passing it through the network for TTA. We use the ADAM [16] optimizer with a learning rate of 0.001 and set the number of iteration as 3. All the experiments were run on five different seeds using a batch size of 8, and the final results were obtained by averaging.

With regard to the self-supervised auxiliary tasks, we use a combination of rank loss and GC loss as our objective function using $\lambda=1$ given in Equation (7). We choose $p=0.25$ to determine the groups. We calculate the GC loss using $\tau=1$. For the rank loss, the test image is synthetically blurred using a Gaussian blur filter of size $5\times 5$ with two sets of standard deviations $\sigma$, for the Gaussian blur kernel. We keep $\sigma\in[40,80]$ for highly distorted images and $\sigma\in[1,20]$ for less blurred images. For compression distortions, we specify the quality factor in $[80,95]$ for lower compression and $[30,60]$ for higher compression rates. Similarly, we add zero mean Gaussian white noise to the test image with variance in $[0.05,0.1]$ for higher distortion and in $[0.005,0.01]$ for lower distortion.

Table 1 shows the performance of our method on all four test datasets using all four quality-aware models. In addition to the source model, we also compare with rotation prediction [35] as the auxiliary task during TTA. A comparison with such a task helps understand the role of quality-aware losses for the TTA of IQA models.

We observe that TTA-IQA using the combination of rank and GC loss outperforms the source models in all the cases. Note that the PIPAL dataset has a huge distribution shift from the authentically distorted LIVEFB dataset on which three of the models were trained. While most source models perform very poorly on PIPAL, we achieve 10%-20% improvement over the source models. On the KonIQ-10k dataset, TTA-IQA gives around 1%-10% improvement over the source model. As both KonIQ-10k and LIVEFB are authentically distorted datasets, the distribution shift is reasonably small, leading to smaller improvements using adaptation. CID2013 is also an authentically distorted dataset and gives similar improvements of around 2%-13% over the source models. Our experiments on the LIVE-IQA database provide a significantly greater improvement of 10%-33% owing to the shift from authentic to synthetic distortions for three models.

We also observe from Table 1 that our approach outperforms the rotation prediction task in most cases. Other TTA ideas, such as TENT [37] and training using masked autoencoders [7] are not applicable for the IQA task. In particular, the notion of entropy in regression tasks is not clear. On the other hand, masked reconstructions tend to lead to quality degradation. Thus, we do not compare with them.

Need for Rank Loss and GC Loss for TTA. We perform an ablation study with respect to the two losses in Table 2. We see from the results that the rank loss and the GC loss individually always improve on the source models. Further, the combination of the rank and GC loss provides an even better performance over the individual losses in most evaluation scenarios. Even in scenarios, where the combination loss achieves the second-best performance, its performance is very close to the best. For the rest of the experiments in this section, we present results on all four datasets using the TReS method alone.

We now discuss scenarios where the GC loss is particularly more useful than the rank loss. If the test image is extremely distorted, none of the three kinds of distortion types can create much difference in the perceptual quality of the two degraded versions. To understand this further, we consider highly distorted images (corresponding to a differential mean opinion score (DMOS) greater than 70) of different distortion types in the LIVE-IQA dataset. We apply TTA only for these images using different losses. The SROCC performance for these images in Figure 5 reveal that the rank loss is not very effective. However, the GC loss works well in such scenarios.

Conversely, we also discuss scenarios where the rank loss is more useful than the GC loss. To illustrate this idea, we select multiple images having similar quality with DMOS in $[28,44]$ from the LIVE-IQA dataset. We adapt the source model for this set of images using both the GC and rank losses. From Figure 4, we observe that the rank loss leads to much better adaptation in terms of the resulting model correlating with the ground truth scores. Intuitively, when the input images have a similar quality, the pseudo-labels given by the source model may be noisy, leading to an inaccurate grouping of the images into the lower and higher quality groups. This can adversely affect the GC loss. So the rank loss is more effective than the GC loss for test batches with similar quality images.

Effect of Selecting Multiple Distortion Types in the Rank Loss. We experiment with the impact of different kinds of distortion types while incorporating the rank loss. In particular, we consider three scenarios: one where only one of the distortion types is used in the rank loss, another where all the three types are used to obtain three loss terms which are summed together to update the model, and a third, where only one of the distortion types is chosen based on the discussion in Section 3.2.2. We observe from Table 3 that choosing the distortion type with the maximal difference in quality gives the best performance for the rank loss. This experiment validates our hypothesis that only a rank loss that can discriminate between the degraded image pairs is useful for TTA.

Selection of Group Size by Varying $p$. We explore different values of $p$ for constructing the GC loss. For a batch size of 8, possible choices of the group size are $2,3,4$ corresponding to $p=0.25,0.375,0.5$ respectively. From Table 4, we observe that the performances are roughly similar for different values of $p$ with a slightly superior performance for $p=0.25$. We note that as $p$ increases, the groups are larger and probably less discriminative in terms of quality, leading to a slightly poorer performance.

Selection of Number of Groups for GC Loss. In our formulation, we define the GC loss only between two contrastive groups. In principle, we can extend our idea of GC loss to multiple groups. In particular, we can cluster the images in a batch into multiple groups and apply the GC loss between every pair of groups and sum the loss terms. Thus, images coming from the same group act as positive pairs, and images from two different groups are considered as negative pairs. From Table 5, we observe that the number of groups $G$ does not impact the performance much.

Choice of Number of Iterations for Learning Auxiliary Task. We also examine the effect of the number of iterations of parameter updates during TTA. Figure 6 shows that the performance on CID2013 dataset is fairly robust until 4 iterations. Beyond that, the model overfits the auxiliary task and leads to poor performance at test time.