A. AGIs Collection A. AGIs 收集

Considering the success of stable diffusion models, we select two text-to-image diffusion models stable-inpainting-v1 and stable-diffusion-v2 (sub-models derived from [1]) as the AGI models. To ensure content diversity and catch up with the popular trends, we use the hot keywords from the PNGIMG website ¹ for AGIs generation, and the employed keywords are exhibited in Table I, which contains the main objects, the second objects, places, and styles. Some sample AGIs are further illustrated in Fig. 2.
考虑到稳定扩散模型的成功，我们选择了两个文本到图像的扩散模型 stable-inpainting-v1 和 stable-diffusion-v2（源自 [1] 的子模型）作为 AGI 模型。为了确保内容的多样性并跟上流行趋势，我们使用了来自 PNGIMG 网站的热门关键词进行 AGI 生成，所使用的关键词展示在表 I 中，包含主要对象、次要对象、地点和风格。一些示例 AGI 进一步在图 2 中说明。

In order to evaluate the statistical discrepancy between NSIs and AGIs, we present the distributions of five quality-related attributes for comparison. The NSIs are sourced from the in-the-wild KonIQ-10k IQA database [12], while the AGIs are collected through the proposed AGIQA-1K database. The quality-related attributes under consideration are light, contrast, colorfulness, blur, and spatial information (SI). Detailed descriptions of these attributes can be found in [13]. As shown in Fig. 3, the quality-related attribute distributions of NSIs and AGIs are quite similar and tend to be Gaussian-like. Specifically, AGIs are relatively blurrier and contain more spatial information than NSIs.
为了评估国家标准图像（NSIs）与人工生成图像（AGIs）之间的统计差异，我们展示了五个与质量相关的属性分布以供比较。NSIs 来源于野外的 KonIQ-10k IQA 数据库 [12]，而 AGIs 则通过提议的 AGIQA-1K 数据库收集。考虑的质量相关属性包括光照、对比度、色彩丰富度、模糊度和空间信息（SI）。这些属性的详细描述可以在 [13] 中找到。如图 3 所示，NSIs 和 AGIs 的质量相关属性分布非常相似，并且趋向于高斯分布。具体而言，AGIs 相对更模糊，并且包含比 NSIs 更多的空间信息。

Fig. 4.  图 4。

Exhibition for some common AGI distortions, where the generation keywords are marked in the top right. The content of (a) is poorly visible due to the blur and inexplicit texture. Unexpected artifacts are introduced to the bottom left of (b). (c) Contains unnatural content such as the hands of the woman do not cope with common sense. (d) Is too simple and does not fit the text keyword of “wearing coat”.
展示了一些常见的 AGI 失真，其中生成关键词标记在右上角。由于模糊和不明确的纹理，(a) 的内容可见性较差。意外的伪影出现在(b) 的左下角。(c) 包含不自然的内容，例如女性的手与常识不符。(d) 过于简单，不符合“穿外套”的文本关键词。

Show All 显示全部

B. Subjective Experiment B. 主观实验

To evaluate the quality of AGIs, a subjective experiment is conducted following the guidelines of ITU-R BT.500-13 [14]. The subjects are asked to rate the overall quality levels of exhibited AGIs from the technical issues, AI artifacts, unnaturalness, discrepancy, and aesthetic aspects. Some typical distortion examples are shown in Fig. 4. The AGIs are presented in random order on an iMac monitor with a resolution of up to 4096×2304, using an interface designed with Python Tkinter, as shown in Fig. 5. The interface allows viewers to browse the previous and next AGIs and rate them using a quality scale that ranges from 0 to 5, with a minimum interval of 0.1. A total of 22 graduate students (10 males and 12 females) participate in the experiment, and they are seated at a distance of around 1.5 times the screen height (45cm) in a laboratory with normal indoor lighting.
为了评估 AGI 的质量，按照 ITU-R BT.500-13 [14]的指导原则进行了一项主观实验。受试者被要求从技术问题、人工智能伪影、不自然性、差异性和美学方面对展示的 AGI 的整体质量水平进行评分。一些典型的失真示例如图 4 所示。AGI 以随机顺序在分辨率高达 4096×2304 的 iMac 显示器上展示，使用 Python Tkinter 设计的界面，如图 5 所示。该界面允许观众浏览前一个和下一个 AGI，并使用从 0 到 5 的质量评分标准进行评分，最小间隔为 0.1。共有 22 名研究生（10 名男性和 12 名女性）参与了实验，他们坐在距离屏幕高度约 1.5 倍（45 厘米）的地方，在一个正常室内照明的实验室中。

To limit the experiment time for each session to less than half an hour, the experiment is split into 5 sessions, each of which includes the subjective quality evaluation for about 200 AGIs. This results in more than 22×1,080=23,760 quality ratings.
为了将每个实验会话的时间限制在半小时以内，实验分为 5 个会话，每个会话包括对约 200 个 AGI 的主观质量评估。这导致产生超过 22×1,080=23,760 个质量评分。

Fig. 5.

An example of the quality assessment interface, where the AGI and corresponding keywords are shown at the same time. The subject can then evaluate the quality of AGIs and record the quality scores with the scroll bar on the right.

Show All 显示全部

C. Subjective Data Analysis
C. 主观数据分析

After the subjective experiment, all quality ratings from the subjects are collected. The raw rating judged by the i-th subject on the j-th image is denoted by rij Z-scores are obtained from the raw ratings using the following formula:
在主观实验后，收集所有受试者的质量评分。第 i 位受试者对第 j 幅图像的原始评分用 rij 表示。Z 分数是通过以下公式从原始评分中获得的：

View Source 查看源代码 \begin{equation*}z_{ij}=\frac{r_{ij}-\mu_{i}}{\sigma_{i}},\tag{1}\end{equation*}

MOSj=1M∑i=1Mz′ij,(2)

View Source 查看源代码 \begin{equation*}MOS_{j}= \frac{1}{M}\sum\limits_{i=1}^{M}z_{ij}^{\prime},\tag{2}\end{equation*}

A. Benchmark Models A. 基准模型

Due to the absence of pristine reference images in the proposed AGIQA-1k database, only no-reference (NR) IQA models are selected for comparison. The selected models can be classified into three groups:
由于在提议的 AGIQA-1k 数据库中缺乏原始参考图像，因此仅选择无参考（NR）图像质量评估模型进行比较。所选模型可以分为三组：

• Handcrafted-based models: This group includes BMPRI [17], CEIQ [18], DSIQA [19], NIQE [20], and SISBLIM [21]. These models extract handcrafted features based on prior knowledge about image quality.
  基于手工特征的模型：该组包括 BMPRI [17]、CEIQ [18]、DSIQA [19]、NIQE [20]和 SISBLIM [21]。这些模型基于对图像质量的先验知识提取手工特征。

• Handcrafted &SVR-based models: This group includes friquee [22], GMLF [23], HIGRADE [24], NFERM [25], and NFSDM [26]. These models combine handcrafted features from a Support Vector Regression (SVR) to represent perceptual quality.
  手工制作的基于 SVR 的模型：该组包括 friquee [22]、GMLF [23]、HIGRADE [24]、NFERM [25]和 NFSDM [26]。这些模型结合了来自支持向量回归（SVR）的手工特征，以表示感知质量。

• Deep learning-based models: This group includes ResNet50 [27], StairIQA [28], and MGQA [29]. These models characterize quality-aware information by training deep neural networks from labeled data.
  基于深度学习的模型：该组包括 ResNet50 [27]、StairIQA [28]和 MGQA [29]。这些模型通过从标记数据中训练深度神经网络来表征质量感知信息。

Fig. 6.  图 6。

Illustration of the MOS probability distribution.
MOS 概率分布的示意图。

Show All 显示全部

Notably, the models mentioned above have exhibited strong performance in previous IQA tasks for natural scenes.
值得注意的是，上述模型在之前的自然场景图像质量评估任务中表现出色。

B. Evaluation Criteria B. 评估标准

In this study, three primary metrics are utilized to evaluate the consistency between the predicted scores and Mean Opinion Scores (MOSs): Spearman Rank Correlation Coefficient (SRoCC), Pearson Linear Correlation Coefficient (PLCC), Kendall's Rank Correlation Coefficient (KRoCC). The SRoCC metric measures the similarity between two sets of rankings, while the PLCC metric computes the linear correlation between two groups of rankings. The KRoCC metric, on the other hand, estimates the ordinal relationship between two measured quantities.
在本研究中，使用了三个主要指标来评估预测分数与平均意见分数（MOSs）之间的一致性：斯皮尔曼等级相关系数（SRoCC）、皮尔逊线性相关系数（PLCC）和肯德尔等级相关系数（KRoCC）。SRoCC 指标测量两组排名之间的相似性，而 PLCC 指标计算两组排名之间的线性相关性。另一方面，KRoCC 指标则估计两个测量量之间的序关系。

To map the predicted scores to MOSs, a five-parameter logistic function is applied, which is a standard practice suggested in [30]:
为了将预测分数映射到 MOS，应用了五参数逻辑函数，这是[30]中建议的标准做法：

View Source 查看源代码 \begin{equation*}\hat{X}=\alpha_{1}\left(0.5-\frac{1}{1+e^{\alpha_{2}(X-\alpha_{3})}}\right)+\alpha_{4}X+\alpha_{5},\tag{3}\end{equation*}

C. Experimental Setup C. 实验设置

All the benchmark models in III-A are validated on the proposed AGIQA-1k database. The database is split randomly in an 80/20 ratio for training/testing while ensuring the image with the same object label falls into the same set. The partitioning and evaluation process is repeated several times for a fair comparison while considering the computational complexity, and the average result is reported as the final performance. For &SVR-based models, the repeating time is 1,000, implemented by LIBSVM [31] with radial basis function (RBF) kernel. For deep learning-based models, the repeating time is 10, using ResNet50 [27] as the network backbone. The Adam optimizer [32] (with an initial learning rate of 0.00001 and batch size 40) is used for 100-epochs training on an NVIDIA GTX 4090Ti GPU.
所有在 III-A 中的基准模型都在提议的 AGIQA-1k 数据库上进行了验证。该数据库以 80/20 的比例随机分割为训练/测试集，同时确保具有相同对象标签的图像落入同一组。分区和评估过程重复多次以进行公平比较，同时考虑计算复杂性，最终结果以平均值报告。对于基于&SVR 的模型，重复次数为 1,000 次，使用 LIBSVM [31]实现，采用径向基函数（RBF）核。对于基于深度学习的模型，重复次数为 10 次，使用 ResNet50 [27]作为网络主干。Adam 优化器[32]（初始学习率为 0.00001，批量大小为 40）用于在 NVIDIA GTX 4090Ti GPU 上进行 100 个周期的训练。

Table II Performance results on the AGIQA-1k database and two different generative model subsets. The best performance results are marked in RED and the second performance results are marked in BLUE.
表 II 在 AGIQA-1k 数据库和两个不同生成模型子集上的性能结果。最佳性能结果用红色标记，第二性能结果用蓝色标记。

D. Performance Discussion
D. 性能讨论

The performance results on the proposed AGIQA-1K database and corresponding two different generative model subsets are exhibited in Table II, from which we can make several conclusions. 1) The handcrafted-based methods achieve poor performance on the whole database and two subsets, which indicates the extracted handcrafted features are not effective for modeling the quality representation of AGIs. This is because most employed handcrafted features of these methods are based on the prior knowledge learned from NSIs, which apparently do not hold for the AGIs. 2) The deep learning-based methods achieve relatively more competitive performance results on the whole database and two subsets. However, they are still far away from satisfactory. 3) Nearly all the IQA models achieve the best performance on the whole database and undergo significant performance drops on the stable-diffusion-v2 subsets. We attempt to give the reasons for such a phenomenon. More keywords are utilized for the stable-diffusion-v2 model, therefore making the AGIs generated by a such model more diverse and complicated. This makes it more challenging for the IQA models to extract quality-aware features from AGIs, which inevitably leads to performance drops.
在所提出的 AGIQA-1K 数据库及其对应的两个不同生成模型子集上的性能结果如表 II 所示，从中我们可以得出几个结论。1）基于手工特征的方法在整个数据库和两个子集上的表现较差，这表明提取的手工特征对于建模 AGI 的质量表示并不有效。这是因为这些方法所采用的大多数手工特征是基于从 NSI 中学习的先验知识，而这些知识显然不适用于 AGI。2）基于深度学习的方法在整个数据库和两个子集上取得了相对更具竞争力的性能结果。然而，它们仍然远未令人满意。3）几乎所有的 IQA 模型在整个数据库上表现最佳，但在 stable-diffusion-v2 子集上性能显著下降。我们尝试给出这种现象的原因。stable-diffusion-v2 模型使用了更多的关键词，因此使得由该模型生成的 AGI 更加多样化和复杂。 这使得 IQA 模型从 AGI 中提取质量感知特征变得更加困难，这不可避免地导致性能下降。

We further validate the performance of the IQA models on the AGIQA-1K database with the anime and realistic styles. The experimental results are listed in Table III. It seems that the IQA models gain similar performance across different styles, which suggests that the styles have a limited impact on the performance of current IQA models.
我们进一步验证了 IQA 模型在 AGIQA-1K 数据库上对动漫和现实风格的表现。实验结果列在表 III 中。似乎 IQA 模型在不同风格下的表现相似，这表明风格对当前 IQA 模型的性能影响有限。

Table III Performance results on the AGIQA-1k database with different styles. The best performance results are marked in red and the second performance results are marked in blue.
表 III 在 AGIQA-1k 数据库上不同风格的性能结果。最佳性能结果用红色标记，第二性能结果用蓝色标记。