2.1. Image Modeling

As the foundation of many computer vision tasks, the image classification has been studied for decades, from heavily rely on manually-crafted features [59], [28], [38] to the deep neural network [30] era, the deep learning dominates the image modeling since 2012 [29]. For the past decade, the networks go deeper, wider and more complex [44], [47], [21], [56] to fit into various tasks including classification [21], detection [40], [32], segmentation [20], [8], pose estimation [45] and more. Besides the network architectures, the convolution layers also evolve from the basic convolution to the depth-wise convolution [57], nonlocal convolution [53] and deformable convolution [10]. In parallel with convolution networks, the recent researches show that the attention based architecture, previously commonly used for NLP tasks [51], transfers to image modeling well. The pioneer work ViT [12] and the following works DEiT [50], SWIN [33], CoaT [58], and more recent Mixer [49] all achieve comparable or superior performances compared with convolution networks. The majority of works on image modeling focus on the performance while in this paper, we focus on both efficiency and effectiveness trade-off.

2.2. Frequency-domain Learning

Frequency domain learning gains much less attention compared with RGB domain modeling in past decades. Only a few works propose to make use of JPEG encoding for faster image classification [17], [14]. Although efficient, these works are less effective than the SOTA image classification models at their times. Some recent works try to incorporate the frequency components from DCT transformation to the channel for better modeling [2], [1], however, the effectiveness gap still exists. Besides, the frequency domain representation has also been used in compression [55], [35], pruning [35] and convnet compression [54], [13]. Although works, the frequency domain modeling generally suffers from the low accuracy and low efficiency, which make them less favored by many image downstream tasks. Our proposed DCFormer with image compression achieves parallel performance compared with SOTA RGB networks at much low computational costs, which stands out from previous frequency domain modeling works. The recent work WaveViT [60] achieves strong performance with discrete wavelet transformation based representation. We share the similar scope on frequency representation based modeling, but instead leverage DCT-based representation because of its flexibility to support strategical down-sampling that improves efficiency without performance compromise.

2.3. Efficient Image Modeling

There are several attempts for efficient image modeling. The convolution kernel or network compression [18], [22] is a straight-forward way to reduce model FLOPs but often leads to noticeable performance drop. Later the carefully designed compact networks with very small memory footage are proposed to work on edge devices including squeezeNet [24], MobileNets [23], [42], and ShuffleNets [61], [36]. More recently, the neural architecture search is widely used as a tool for searching the efficient and accurate network architectures e.g. Proxylessnas [5] and EfficientNet [48]. Different from these approaches that try to build a smaller network, we propose to reduce computation based on frequency domain image compression.

3.1. Frequency Domain Modeling

In this paper, we adopt DCT based frequency domain representation because it is commonly used for image compression [41], image encoding [43], and various computer vision tasks.

3.1.1 Domain Converting Preliminaries

For an RGB image I_RGB ∈ R^3×W×H, we first convert the image color space from RGB to yCrCb color space (I_YCrCb) and patchfy the image as:

P=[P0,P1,…,Pi]=patchfy(IYCrCb)(1)

View Source\begin{equation*}P = \left[ {{P_0},{P_1}, \ldots ,{P_i}} \right] = \operatorname{patchfy} \left( {{I_{YCrCb}}} \right)\tag{1}\end{equation*} where P∈R(3×Hps×Wps×ps×ps) is a sequence of patches, ps denotes the side length of each patch. DCT [4] is applied on each patch to generate the frequency domain representation:

Di=DCT(Pi)(2)

View Source\begin{equation*}{D_i} = {\text{DCT}}\left( {{P_i}} \right)\tag{2}\end{equation*} where the DCT map of each patch D_i ∈ R^3×ps×ps has the same dimension as the original patch P_i.

The patchfy operation preserves the relative spatial information, while each point in a patch D_i carries certain frequency information. Patch size selection involves a tradeoff. Smaller patches lead to higher spatial resolution but less fine-grained frequency information and opposite for the larger patch size. We empirically select 8 × 8 as patch size for the best efficiency-effectiveness trade-off, the same as JPEG’s encoding process. Such design potentially allows us to directly obtain the DCT components from raw JPEG images for faster training and inference.

3.1.2 Frequency Domain Encoder

For a compressed frequency map S, the pixels no longer hold spatial relationships inside each patch. Different from previous works that try to shift frequency components to channels for the convolution based modeling [1], [2], we propose to build a network that directly works on the frequency map. The 2D Inverse-DCT (IDCT) transformation for each frequency patch can be mathematically formulated as:

IDCT(Si)=A⊺(Si)A(3)

View Source\begin{equation*}{\text{IDCT}}\left( {{S_i}} \right) = {A^ \intercal }\left( {{S_i}} \right)A\tag{3}\end{equation*} where A denotes the DCT transform matrix. The above equation can be further illustrated as below and is consistent with the transformer layer’s formulation:

IDCT(Si)=A⊤(Si)A=(WqΛW⊤k)(WvSi)(W)(4)

View Source\begin{equation*}\operatorname{IDCT} \left( {{S_i}} \right) = {A^ \top }\left( {{S_i}} \right)A = \left( {{W_q}\Lambda W_k^ \top } \right)\left( {{W_v}{S_i}} \right)(W)\tag{4}\end{equation*} where W_k,W_q,W_v denotes the learnable linear projection for key, query and value. W denotes the learnable weights for linear layers after attention. Although it is not guaranteed that W is strictly the transpose of (WqΛW⊤k), it is possible for transformers to learn the approximation of the IDCT. The observation in Equation 4 makes the transformer architecture a good fit for our compressed image modeling. Note that, it is possible for convolution networks to simulate IDCT through carefully designed kernel size and strides, but it will be less effective compared with transformer networks (ablations in Table 4d). The commonly used transformers, including sequence transformer (e.g. ViT[12]) and hierarchical vision transformer (e.g. SWIN[33], NAT [19]) work directly in the frequency domain with minor changes to the patch size and embedding.

Frequency Embedding: Each frequency point S_i,j on S_i carries the relative positional information as well as certain frequency band information. To maintain the frequency information, we propose frequency embedding (FE) in addition to the commonly used positional embedding as:

⎧⎩⎨⎪⎪FE(j,2k)=sin(j/100002k/dm)FE(j,(2k+1))=sin(j/10000(2k+1)/dm)(5)

View Source\begin{equation*}\begin{cases} {F{E_{(j,2k)}} = \sin \left( {j/{{10000}^{2k/dm}}} \right)} \\ {F{E_{(j,(2k + 1))}} = \sin \left( {j/{{10000}^{(2k + 1)/dm}}} \right)} \end{cases}\tag{5}\end{equation*} where j ∈ [0,ps²] denotes the position in the downsampled frequency patch S_i. k denotes the k-th dimension of total dm feature dimensions. We apply the frequency embedding by adding it to the frequency map S.

Classification: We unpatchfy the compressed patches based on their relative locations as:

S~=unpatchfy(S)(6)

View Source\begin{equation*}\tilde S = {\text{unpatchfy}}(S)\tag{6}\end{equation*} where the unpatchfy operation reorganize a sequence of compressed DCT map based on their relative spatial location. The DCFormer encoder extras feature embedding from the compressed frequency domain representations as:

XE=ϕ(S~)(7)

View Source\begin{equation*}{X^E} = \phi (\tilde S)\tag{7}\end{equation*} where X^E stands for the DCFormer encoder feature map. The classification can be done by adding a [CLS]-token [12] or use a linear layer [33].

3.2. Efficient Frequency Domain Modeling

3.2.1 Frequency Domain Compression

The famous JPEG compression infers that compression can be achieved by discarding the non-visually significant values via quantization [52]. Following similar intuition, we aim maintain only the informative frequency components from each DCT map D_i for efficient image modeling. We explored three types of compression strategies as follows:

Figure 2:

An overview of our model, DCFormer. The model first takes RGB image as input, converting it to DCT-based frequency representation, follows by an optional frequency compression module. The compression module (when τ>1) offers significant efficiency boost, with slight performance trade-off. The frequency based representation is then augmented with positional and frequency embedding and feed into a set of DCFormer blocks. The frequency attention is compatible with various transformer attentions (e.g. SWIN attention, neighbour attention). An linear projection with CE loss is used for classification, and a MSE reconstruction loss can be used as aux loss when frequency compression is applied.

Show All

Averaging: An average pooling with τ × τ kernel over the DCT map D_i as:

{Si(jτ,kτ)=1τ2[Di(j,k)+…+Di(j,k+τ)+…+Di(j+τ,k+τ)](8)

View Source\begin{equation*}\begin{cases} {{S_i}\left( {\frac{j}{\tau },\frac{k}{\tau }} \right) = \frac{1}{{{\tau ^2}}}\left[ {{D_i}(j,k) + \ldots + {D_i}(j,k + \tau )} \right.} \\ {\left. { + \ldots + {D_i}(j + \tau ,k + \tau )} \right]} \end{cases}\tag{8}\end{equation*} where j and k are the coordinates of D_i. S_i denotes the compressed DCT map D_i and τ is the compression ratio.

Soft-selection: A cross-attention based soft-selection method on D_i as:

Si=MHCA(Conv2D(Di),qemb)(9)

View Source\begin{equation*}{S_i} = {\text{MHCA}}\left( {Conv2D\left( {{D_i}} \right),{q_{emb}}} \right)\tag{9}\end{equation*} where MHCA denotes the multi-head cross-attention module qemb∈Rps2τ2×c is the query embedding, Conv2D is an 1 × 1 2D convolution that expands the channel of Dⁱ, e.g. c = 128, to support multi-head attentions.

Hard-selection: A hard-selection follows zigzag pattern that focus on low frequency components as:

Si=[ψ(Di)]k,k∈[1,ps2τ2](10)

View Source\begin{equation*}{S_i} = {\left[ {\psi \left( {{D_i}} \right)} \right]_k},k \in \left[ {1,\frac{{p{s^2}}}{{{\tau ^2}}}} \right]\tag{10}\end{equation*} where ψ is the zigzag encoding used in [37], ψ(Di)∈Rpsτ×pgT stands for the sequence of frequency components after zigzag encoding, which are then sorted by their frequency band from low to high. To maintain visually significant information, we hard select the first 1τ2 elements from ψ(D_i), which covers the DC component and most of lowfrequency and part of middle frequency information.

We empirically choose zigzag-baed hard-selection for compression as it works best without introducing additional computation. The cross-attention based soft-selection is deprecated since it is computationally heavy. Averaging based approaches perform the worst since averaging frequency responses over different bands does not make sense.

3.2.2 Reconstruction Aux Loss

Because frequency compression causes information lose, we further introduce a reconstruction decoder adopting auxiliary loss during training which encourages the DCFormer encoder to generate comprehensive and semantic-related feature embeddings. The decoder is not used in the inference and hence does not introduce additional inference computations. It is worth mentioning that the decoder has to be lightweight with limited capacities, so that the decoder utilizes the encoded features as much as possible instead of learning the new semantics features by itself. We propose a simple eight-layer convolution neural network with 3 × 3 kernels as the decoder. Because convolution is less effective on frequency-to-RGB domain converting, the decoder has to rely on semantic information generated by DCFormer encoder to reconstruct the RGB image. The reconstruction process thus enforces the encoder to generate more comprehensive representation during training. The decoder has four up-sampling stages, each stage has two convolution layers and a spatial up-sampling layer defined as:

XDi={XEconv2D(conv2D(U(XDi−1)))i=1i∈[2,4](11)

View Source\begin{equation*} X_i^D = \begin{cases} {{X^E}}&{i = 1} \\ {conv2D\left( {conv2D\left( {U\left( {X_{i - 1}^D} \right)} \right)} \right)}&{i \in [2,4]} \end{cases} \tag{11}\end{equation*} where U denotes the 4× bilinear interpolation up-sampling. The XDi denotes the feature from i-th decoder stage.

3.2.3 Losses

We apply the categorical cross-entropy to the DCFormer encoder output as classification loss (Lcls). For the reconstructed images, we calculate the MSE loss as a measure of reconstruction quality (LMSE). We also adopt the perceptual loss (Lperceptual), which is commonly used in image super-resolution [25] tasks, to encourage the DCFormer encoder to generate semantic related representations. The final loss is thus defined as:

L=Lcls+αLMSE+βLperceptual(12)

View Source\begin{equation*}\mathcal{L} = {\mathcal{L}_{cls}} + \alpha {\mathcal{L}_{MSE}} + \beta {\mathcal{L}_{perceptual}}\tag{12}\end{equation*}

We empirically determined the α = 0.1 and β = 0.01.

Table 1: Frequency modeling results on ImageNet-1K validation set. τ = 1 and τ = 2 denotes frequency input without and with 4X compression.

4.1. Frequency Domain Modeling Results

4.1.3 Instance Segmentation on COCO

Setting: We also evaluate the instance segmentation performance on the COCO dataset, and our training follows the same protocol used in the detection experiments.

Table 2: Comparison on COCO Object detection and instance segmentation on 5k validation set, with 800 × 1333 input images. DCFormer-SW and DCFormer-NA denots DCFormer with SWIN [33] and NAT [19] as backbones, respectively.

Table 3: Comparing with RGB-based works on image classification, object detection and instance segmentation. Detection and instance segmentation tasks run on 800 × 1333 input images, 1.5× denotes 1.5 times larger input images. DCFormerSW/NA denotes DCFormer with SWIN [33] and NAT [19] as backbones, respectively.

Results: DCFormer achieves consistent performance improvements on instance segmentation (Table 2) compared with other frequency domain modeling approaches [1], [2].

4.2. Comparing with RGB-based SOTA

Classification Table 3 (a) shows our results on ImageNet1K validation set compared with the previous works based on convolution [21], [34], transformer [33], [50]. All the models listed are only trained on ImageNet-1K from scratch. We find that the proposed DCFormer is able to operate at lower computational budget to maintain similar performance comparing with commonly used RGB models. For example, DCFormer-SWIN-S (= 2) achieves 80.9% top1 accuracy with only 2.7G FLOPs, significantly more efficient than SWIN-T [33]. The DCFormer-NA-T also achieves slightly better performance at lower FLOPs comparing with recent works ConvNeXT-T [34] and SWINT [33]. Furthermore, the DCFormer is able to achieve performance comparable to RGB SOTA at same computational budget. For example, the DCFormer-SWIN-T (τ = 2) with 512×512 input resolution slightly outperforms SWINT [33] at same FLOPs. The DCFormer-NA-B also achieves performance comparable to SOTA NAT-B [19] with same input resolution and same FLOPs. The results demonstrate the outstanding efficiency and effectiveness of the proposed approach.

Table 4: Ablation studies on ImageNet-1K. All the experiments use DCFormer-SW-T and images of 256 × 256 as backbone without the reconstruction decoder, unless specified.

Object Detection To compare with SOTA RGB models, we trained DCFormer with SWIN backbone on cascade mask RCNN pipeline. The proposed DCFormer is able to reduce the FLOPs and latency on object detection tasks (Table 3 (b)). Giving the same input image resolution, the DCFormer-SWIN-S achieves slightly worse performance compared with the SOTA SWIN-T and ConvNeXt-T models but with 15% less FLOPs. Similar to image classification, the performance gap can be compensated by higher input resolution without significant FLOPS increase. By scaling up the input image by 1.5× (1200 × 1666), the DCFormer-SWIN-S is able to achieve comparable performance with 11% less FLOPs.

Instance Segmentation The DCFormer with large input resolution achieves similar performance compared with SOTA SWIN [33] and ConvNeXT [34] based approaches (Table 3 (c)). We notice that the DCFormer maintains good efficiency due to the proposed frequency compression. However, the DCFormer performs slightly worse on instance segmentation tasks. This is probably due to the reduced feature map size and lack of high-frequency (texture) information. There are researches showing that texture information helps with instance segmentation [26]. To better preserve these textures during the compression as well as maintaining the low computation will be our future work.

4.3. Ablation Study

We justify the important design choices, effectiveness and generalization of proposed model on ImageNet-1K image classification task. All the experiments are performed on DCFormer-SWIN-T. The images are 256 × 256 resolution and have 8 × 8 frequency patch size, with τ = 2 are used, unless specified.

Building components breakdown. Table 4a analyzes the contribution of each proposed components, by adding them one at a time, to a standard SWIN transformer. We use a SWIN transformer on an RGB image down-sampled to 112 × 112 as the baseline (same FLOPs). By performing hard-selection on the frequency components, we boost the performance by 0.34% without introducing additional computation. This also verifies our intuition that the frequency domain down-sampling better preserves information that is visually significant. The proposed frequency embedding slightly enhance the performance by 0.2%. Additionally, the reconstruction decoder achieves a slight improvement by 0.16% which shows that our decoder works as expected. Note that the decoder only introduces additional FLOPs in training but it is not used during inference.

Compression ratio. Table 4b compares the classification performance under different frequency compression ratios. The larger compression ratio, e.g. τ = 4, leads to lower FLOPs but lower accuracy since more information was dropped; same for the opposite. Based on the ablation, we choose τ = 0.5 for the best efficiency and effectiveness trade-off. It is also interesting to see that the DCFormer with no frequency down-sampling (τ = 1) achieves the same accuracy and FLOPs as SWIN-T with RGB image input. This indicates the frequency domain representation is as effective as the RGB representation, as we argue that transformer is a good fit for frequency domain modeling.

DCT Patch Size. Table 4c studies the impact of using different DCT patch sizes on images of different resolutions. In general, smaller DCT patch size, e.g. 8² works better on smaller images (e.g. 256², 384²). Further increasing the input resolution with same DCT patch size does not consistently enhance the performance, because small DCT patches with limited DCT bases only contain limited information. Larger DCT patches convey more frequency information and yield better performance on inputs with high-resolution. Dynamically adjusting DCT patch size will be our future research.

Generalization. Table 4d explores the generalization of proposed image compression with different backbones. Our approach generalizes to different backbones. The transformer-based encoders generally yield less performance drop by using frequency domain inputs, which proves our illustration that the attention can simulate inverse-DCT operation more effectively. It is also worth mentioning that the ViT-B has high FLOPs due to its lack of multi-scale feature hierarchy.

Compression method. Table 4e compares different frequency compression methods. We first notice that average pooling which is commonly used in spatial down-sampling causes significant performance drop when applied to the frequency domain. This is because averaging data points that belong to different frequency bands does not make sense as they do not have direct spatial associations. We then try to use the cross-attention to learn the weighted average based compression. However, cross-attention requires applying extra convolution layers on input DCT map, which introduces additional computation and makes the compression less efficient and contradict to our motivations. The zigzag selection works best at no additional computations in our case. Similar approach was used in JPEG compression and similar patterns were observed [2].

Effectiveness. Table 4f compares and analyzes several alternatives to the proposed image modeling at similar FLOPs, including: directly down-sample the input image by 2×; and after proposed frequency domain compression, reconstruct the RGB image with IDCT and feed it into the standard SWIN transformer. For better comparison, the SWIN-T with RGB input of 256 × 256 is used as the baseline. The results indicate that the proposed method is more effective than the alternative, as the reconstruction may suffer from noises introduced during padding and converting.