Learning Deep MRI Reconstruction Models from Scratch in Low-Data Regimes

MRI reconstruction is an inverse problem that aims to recover an image from a respective undersampled acquisition:
MRI 重建是一个逆问题，旨在从相应的欠采样采集中恢复图像：

$\displaystyle MFx=y$

(1)

where $F$ is the Fourier transform, $M$ is the sampling mask defining acquired k-space locations, $x$ is the multi-coil image to be reconstructed and $y$ are acquired multi-coil k-space data. To improve problem conditioning, additional prior information regarding the expected distribution of MR images is incorporated in the form of a regularization term:

$\displaystyle\hat{x}=\underset{x}{\arg\min}\quad\lambda||MFx-y||_{2}^{2}+R(x)$

(2)

where the first term enforces DC between reconstructed and acquired k-space data, $R(x)$ reflects the MRI prior, and $\lambda$ controls the balance between the DC and regularization terms.

The DC term can be implemented by injecting the acquired values of k-space data into the reconstruction [13]. Thus, mapping through a DC block is given as:

$\displaystyle f_{DC}(x)=F^{-1}\Lambda Fx+\dfrac{\lambda}{1+\lambda}F^{-1}y$

(3)

where $\Lambda$ is a diagonal matrix with diagonal entries set to $\frac{1}{1+\lambda}$ at acquired k-space locations and set to 1 in unacquired locations.

In traditional methods, the regularization term is based on a hand-constructed transform domain where data are assumed to have a sparse representation [9]. For improved conformation to the distribution of MRI data, recent frameworks instead adopt deep network models to capture either SG priors learned from a large MRI database with hundreds of subjects, or SS priors learned from individual test scans. Learning procedures for the two types of priors are discussed below.

SG priors: In MRI, SG priors are typically adopted to suppress aliasing artifacts in the zero-filled reconstruction (i.e., inverse Fourier transform) of undersampled k-space acquisitions [27]. A deep network model that performs de-aliasing can be learned from a large training dataset of undersampled and corresponding fully-sampled k-space acquisitions, and then employed to implement $R(.)$ in Eq. 2 during inference. The regularization term based on an SG prior is given as:
SG 先验：在 MRI 中，通常采用 SG 先验来抑制欠采样 k 空间采集的零填充重建（即逆傅里叶变换）中的混叠伪影 [27]。执行去混叠的深度网络模型可以从欠采样和相应的全采样 k 空间采集的大型训练数据集中学习，然后在推理过程中用于等式 2 中的实现 $R(.)$ 。基于 SG 先验的正则化项如下：

$\displaystyle R_{SG}\left(x\right)=\underset{x}{\arg\min}||C_{SG}(F^{-1}y;\hat{\theta}_{SG})-x||^{2}_{2}$

(4)

where $C_{SG}$ is an image-domain deep network with learned parameters $\hat{\theta}_{SG}$. The formulation in Eq. 4 assumes that $C_{SG}$ recovers multi-coil output images provided multi-coil input images. The parameters ${\theta}_{SG}$ for $C_{SG}$ can be learned based on a pixel-wise loss between reconstructed and ground-truth images. Training is conducted offline via an empirical risk minimization approach based on Monte Carlo sampling [13]:

$\displaystyle\mathcal{L}_{SG}(\theta_{SG})=\sum_{n=1}^{N}||C_{SG}(F^{-1}y^{n};\theta_{SG})-\breve{x}^{n}||_{p}$

(5)

where $N$ is the number of training scans, $n$ is the training scan index, $||.||_{p}$ denotes $\ell_{p}$ norm, $\breve{x}^{n}$ is the ground-truth multi-coil image derived from the fully-sampled acquisition for the $n$th scan, and $y^{n}$ are respective undersampled k-space data.

A common approach to build $C_{SG}$ is based on unrolled architectures that perform cascaded projections through CNN blocks to regularize the image and DC blocks to ensure conformance to the physical signal model [27]. Given a total of $K$ cascades with tied CNN parameters across cascades, the mapping through the $k$th cascade is [13, 68, 69]:

$\displaystyle x^{r}_{k}=f_{DC}\left(f_{SG}\left(x^{r}_{k-1};{\theta}_{SG}\right)\right)$

(6)

where $x_{k}^{r}$ is the image for the $r$th scan (that could be a training or test scan) at the output of the $k$th cascade ($k\in[1,2,...,K]$), and $x_{0}^{r}=F^{-1}y^{r}$ where $y^{r}$ are the acquired undersampled data for the $r$th scan. Meanwhile, $f_{SG}$ is the CNN block embedded in the $k$th cascade with parameters ${\theta}_{SG}$.
其中 $x_{k}^{r}$ 是 $r$ 第 $k$ Th 级联输出端的第 Th 扫描（可能是训练或测试扫描）的图像 （ $k\in[1,2,...,K]$ ），其中 $x_{0}^{r}=F^{-1}y^{r}$ $y^{r}$ 是第 Th 扫描获取的 $r$ 欠采样数据。同时， $f_{SG}$ 是嵌入在 $k$ 带有参数 ${\theta}_{SG}$ 的 th 级联中的 CNN 块。

As the parameters of SG priors are trained offline and then frozen during inference, deeper network architectures can be used for enhanced reconstruction performance along with fast inference. However, learning deep networks requires substantial training datasets that may be difficult to collect. Moreover, since SG priors learn aggregate representations of MRI data across training subjects, they may show poor generalization to subject-specific variability in anatomy [19].

SS priors: Unlike SG priors, SS priors are not learned from a dedicated training dataset but instead they are learned directly for individual test scans to improve generalization [15]. The SS prior can also be used to implement $R(.)$ in Eq. 2 with the respective regularization term expressed as:

$\displaystyle R_{SS}\left({x}\right)=\underset{x}{\arg\min}||C_{SS}(F^{-1}y;\hat{\theta}_{SS})-x||^{2}_{2}$

(7)

where $C_{SS}$ is an image-domain network with parameters $\hat{\theta}_{SS}$. In the absence of ground-truth images, the parameters ${\theta}_{SS}^{q}$ for the $q$th test scan can be learned based on proxy k-space losses between reconstructed and acquired undersampled data [22]. Learning is conducted online to minimize this proxy loss:
其中 $C_{SS}$ 是具有参数 $\hat{\theta}_{SS}$ 的影像域网络。在没有真实图像的情况下，可以根据重建和采集的欠采样数据之间的代理 k 空间损失来学习 $q$ 第 x 次测试扫描的参数 ${\theta}_{SS}^{q}$ [22]。学习在线进行，以尽量减少这种代理损失：

$\displaystyle\mathcal{L}_{SS}(\theta_{SS}^{q})=||MFC_{SS}(F^{-1}y^{q};\theta_{SS}^{q})-y^{q}||_{p}$

(8)

where $y^{q}$ are acquired undersampled k-space data for the $q$th scan. An unrolled architecture can be adopted to build $C_{SS}$ by performing cascaded projections through network and DC blocks, resulting in the following mapping for the $k$th cascade:

$\displaystyle x^{q}_{k}=f_{DC}\left(f_{SS}\left(x^{q}_{k-1};\theta_{SS}^{q}\right)\right)$

(9)

$f_{SS}$ can be operationalized as a linear or nonlinear network [23, 22]. As the parameters of SS priors are learned independently for each test scan, they promise enhanced generalization to subject-specific anatomy. However, since training is performed online during inference, SS priors can introduce substantial computational burden, particularly when deep nonlinear networks are used that also increase the risk of overfitting [70].

Here, we propose to combine SS and SG priors to maintain a favorable trade-off between generalization performance and computational efficiency under low-data regimes. In the conventional unrolling framework, this requires computation of serially alternated projections through the SS, SG and DC blocks:

$\displaystyle x^{r}_{k}=f_{DC}\left(f_{SG}\left(f_{SS}\left(x^{r}_{k-1};\theta^{r}_{SS}\right);{\theta}_{SG}\right)\right)$

(10)

The unrolled architecture with $K$ cascades can be learned offline using the training set. Note that scarcely-trained SG blocks under low-data regimes can perform suboptimally, introducing residual errors in their output. In turn, these errors will accumulate across serial projections to degrade the overall performance.

To address this limitation, here we introduce a novel architecture, PSFNet, that performs parallel-stream fusion of SS and SG priors as opposed to the serial combination in conventional unrolled methods. PSFNet utilizes a nonlinear SG prior for high performance, and a linear SS prior to enhance generalization without excessive computational burden. The two priors undergo parallel-stream fusion with learnable fusion parameters $\eta$ and $\gamma$, as displayed in Figure 1. These parameters adaptively control the relative weighting of information extracted by the SG versus SS streams during the course of training in order to alleviate error accumulation. As such, the mapping through the $k$th cascade in PSFNet is:

$$x^{r}_{k}=\eta_{k}f_{DC}(f_{SS}(x^{r}_{k-1};\theta_{SS}^{r}))+\gamma_{k}f_{DC}(f_{SG}(x^{r}_{k-1};{\theta}_{SG}))$$

(11)

In Eq. 11, the learnable fusion parameters for the SS and SG blocks at the $k$th cascade are $\eta_{k}$ and $\gamma_{k}$, respectively. To enforce fidelity to acquired data, DC projections are performed on the outputs of SG and SS blocks. In PSFNet, the SG prior is learned collectively from the set of training scans and then frozen during inference on test scans. In contrast, the SS prior is learned individually for each scan, during both training and inference.

Training: PSFNet involves a training phase to learn model parameters for the SG prior as well as its fusion with the SS prior. For each individual scan in the training set, PSFNet learns a dedicated SS prior for the given scan. Since learning of a nonlinear SS prior has substantial computational burden, we adopt a linear SS prior in PSFNet. In particular, the SS block performs dealiasing via convolution with a linear kernel [71]:

$\displaystyle f_{SS}(x^{n}_{k-1};\theta^{n}_{SS})=F^{-1}\{\theta^{n}_{SS}\circledast Fx^{n}_{k-1}\}$

(12)

where $\theta^{n}_{SS}\in\mathbb{C}^{(z\times z\times w\times w)}$ with $n$ denoting the training scan index, $z$ denoting the number of coil elements, and $w$ denoting the kernel size in k-space. The SS blocks contain unlearned Fourier and inverse Fourier transformation layers as their input and output layers, respectively, and convolution is computed over the spatial frequency dimensions in k-space. Meanwhile, the SG prior is implemented as a deep CNN operating in image domain:
其中 $\theta^{n}_{SS}\in\mathbb{C}^{(z\times z\times w\times w)}$ with $n$ 表示训练扫描索引， $z$ 表示线圈元件的数量，并 $w$ 表示 k 空间中的内核大小。SS 块包含未学习的傅里叶和逆傅里叶变换层分别作为其输入和输出层，并在 k 空间中的空间频率维度上计算卷积。同时，SG 先验被实现为在图像域中运行的深度 CNN：

$\displaystyle f_{SG}(x^{n}_{k-1};\theta_{SG})=CNN(x^{n}_{k-1})$

(13)

Across the scans in the training set, the training loss for PSFNet can then be expressed in constrained form as:

$$\mathcal{L}_{PSFNet}(\theta_{SG},\boldsymbol{\gamma},\boldsymbol{\eta})=\sum_{n=1}^{N}||\eta_{K}f_{DC}(f_{SS}(x^{n}_{K-1};\hat{\theta}_{SS}^{n}))\\ +\gamma_{K}f_{DC}(f_{SG}(x^{n}_{K-1};\theta_{SG}))-\breve{x}^{n}||_{p}\\ \mbox{s.t. }\hat{\theta}_{SS}^{n}=\underset{\theta_{SS}^{n}}{\arg\min}||F^{-1}W^{n}y^{n}-f_{SS}(F^{-1}W^{n}y^{n};\theta^{n}_{SS})||_{2}^{2}$$

(14)

The constraint in Eq. 14 corresponds to the scan-specific learning of the SS prior $\hat{\theta}^{n}_{SS}$, which is then adopted to calculate the loss. Assuming that the linear relationships among neighboring spatial frequencies are similarly distributed across k-space [71], $\hat{\theta}^{n}_{SS}$ is learned by solving a self-regression problem on the subset of fully-sampled data in central k-space, where $W^{n}$ is a mask operator that selects data within this calibration region.

Note that, unlike deep reconstruction models purely based on SG priors, the SG prior in PSFNet is not directly trained to remove artifacts in zero-filled reconstructions of undersampled data. Instead, the SG prior is trained to concurrently suppress artifacts in reconstructed images along with the SS prior; and the relative importance attributed to the two priors is determined by the fusion parameters at each cascade. As such, the SS prior can be given higher weight during initial training iterations where the SG prior is scarcely trained, whereas its weight can be relatively reduced during later iterations once the SG prior has been sufficiently trained. This adaptive fusion approach thereby lowers reliance on the availability of large training sets.

Inference: During inference on the $q$th test scan, the respective SS prior is learned online as:

$\displaystyle\hat{\theta}_{SS}^{q}=\underset{\theta_{SS}^{q}}{\arg\min}||F^{-1}W^{q}y^{q}-f_{SS}^{q}(F^{-1}W^{q}y^{q};\theta^{q}_{SS})||_{2}^{2}$

(15)

Afterwards, the learned $\hat{\theta}_{SS}^{q}$ is used along with the previously trained $\hat{\theta}_{SG}$ to perform repeated projections through $K$ cascades as described in Eq. 11. The multi-coil image recovered by PSFNet at the output of the $K$ cascade is:

$$\hat{x}^{q}=\eta_{K}f_{DC}(f_{SS}(x^{q}_{K-1};\hat{\theta}_{SS}^{q}))+\gamma_{K}f_{DC}(f_{SG}(x_{K-1}^{q};\hat{\theta}_{SG}))$$

(16)

where $\hat{x}^{q}$ denotes the recovered image. The final reconstruction can be obtained by performing combination across coils:

$$\hat{x}^{q}_{combined}=A^{*}\hat{x}^{q}$$

(17)

where $A$ are coil sensitivities, and $A^{*}$ denotes the conjugate of $A$.

In each cascade, PSFNet contained two parallel streams with SG and SS blocks. The SG blocks comprised an input layer followed by a stack of 4 convolutional layers with 64 channels and 3x3 kernel size each, and an output layer with ReLU activation functions. They processed complex images with separate channels for real and imaginary components. The SS blocks comprised a Fourier layer, 5 projection layers with identity activation functions, and an inverse Fourier layer. They processed complex images directly without splitting real and imaginary components. The linear convolution kernel used in the projection layers was learned from the calibration region by solving a Tikhonov regularized self-regression problem [11]. The DC blocks comprised 3 layers respectively to implement forward Fourier transformation, restoration of acquired k-space data and inverse Fourier transformation. PSFNet was implemented with 5 cascades, $K$=5. The weights of SG, SS, and DC blocks were tied across cascades to limit model complexity [27]. The only exception were fusion coefficients that determine the relative weighting of the SG and SS blocks at each stage ($\gamma_{1},..,\gamma_{k},...,\gamma_{5}$ $\eta_{1},...\eta_{k},...,\eta_{5}$). These fusion parameters were kept distinct across cascades. Coil-combination on the recovered multi-coil images was performed using sensitivity maps estimated via ESPIRiT [71].
在每个级联中，PSFNet 包含两个带有 SG 和 SS 块的并行流。SG 块包括一个输入层，后跟一个由 4 个卷积层组成的堆栈，每个卷积层有 64 个通道，每个卷积层的大小为 3x3 内核，以及一个具有 ReLU 激活函数的输出层。他们使用单独的通道处理实部和虚部的复杂图像。SS 块包括一个傅里叶层、5 个具有身份激活函数的投影层和一个逆傅里叶层。他们直接处理复杂的图像，而无需拆分实部和虚部。投影层中使用的线性卷积核是通过解决 Tikhonov 正则化自回归问题 [11] 从校准区域学习的。DC 模块由 3 层组成，分别实现正向傅里叶变换、获取的 k 空间数据恢复和逆傅里叶变换。PSFNet 使用 5 个级联 $K$ （=5） 实现。SG、SS 和 DC 块的权重在级联之间捆绑在一起，以限制模型复杂性 [27]。唯一的例外是确定每个阶段 SG 和 SS 块的相对权重的融合系数 （ $\gamma_{1},..,\gamma_{k},...,\gamma_{5}$ $\eta_{1},...\eta_{k},...,\eta_{5}$ ）。这些融合参数在级联中保持不同。使用通过 ESPIRiT [71] 估计的灵敏度图对恢复的多线圈图像进行线圈组合。

Experimental demonstrations were performed using brain MRI scans from the NYU fastMRI database [72]. Here, contrast-enhanced T₁-weighted (cT₁-weighted) and T₂-weighted acquisitions were considered. The fastMRI dataset contains volumetric MRI data with varying image and coil dimensionality across subjects. Note that a central aim of this work was to systematically examine the learning capabilities of models for varying number of training samples. To minimize potential biases due to across-subject variability in MRI protocols, here we selected subjects with matching imaging matrix size and number of coils. To do this, we only selected subjects with at least 10 cross-sections and only the central 10 cross-sections were retained in each subject. We further selected subjects with an in-plane matrix size of 256x320 for cT₁ acquisitions, and of 288x384 for T₂ acquisitions. Background regions in MRI data with higher dimensions were cropped. Lastly, we restricted our sample selection to subjects with at least 5 coil elements, and geometric coil compression [73] was applied to unify the number of coils to 5 in all subjects.

Fully-sampled acquisitions were retrospectively undersampled to achieve acceleration rates of R=4x and 8x. Random undersampling patterns were designed via either a bi-variate normal density function peaking at the center of k-space, or a uniform density function across k-space. The standard deviation of the normal density function was adjusted to maintain the expected value of R across k-space. The fully-sampled calibration region spanned a 40x40 window in central k-space.

PSFNet was compared against several state-of-the-art approaches including SG methods, SS methods, and traditional PI reconstructions. For methods containing SG priors, both supervised and unsupervised variants were implemented.

PSFNet: A supervised variant of PSFNet was trained using paired sets of undersampled and fully-sampled acquisitions.

PSFNet_US: An unsupervised variant of PSFNet was implemented using self-supervision based on only undersampled training data. Acquired data were split into two non-overlapping sets where 40% of samples was reserved for evaluating the training loss and 60% of samples was reserved to enforce DC [64].

MoDL: A supervised SG methods based on an unrolled architecture with tied weights across cascades was used [27]. MoDL serially interleaves SG and DC blocks. The number of cascades and the structure of SG and DC blocks were identical to those in PSFNet.

MoDL_US: An unsupervised variant of MoDL was implemented using self-supervision. A 40%-60% split was performed on acquired data to evaluate the training loss and enforce data consistency, respectively [64].

sRAKI-RNN: An SS method was implemented based on the MoDL architecture [67]. Learning was performed to minimize DC loss on the fully-sampled calibration region. Calibration data were randomly split with 75% of samples used to define the training loss and 25% of samples reserved to enforce DC. Multiple input-output pairs were produced for a single test sample by utilizing this split.

SPIRiT: A traditional PI reconstruction was performed using the SPIRiT method [11]. Reconstruction parameters including the regularization weight for kernel estimation ($\kappa$), kernel size ($w$), and the number of iterations ($N_{iter}$) were independently optimized for each reconstruction task via cross-validation.

SPARK: An SS method was used to correct residual errors from an initial SPIRiT reconstruction [17]. Learning was performed to minimize DC loss on the calibration region. The learned SS prior was then used to correct residual errors in the remainder of k-space.

For all methods, hyperparameter selection was performed via cross-validation on a three-way split of data across subjects. There was no overlap among training, validation and test sets in terms of subjects. Data from 10 subjects were reserved for validation, and data from a separate set of 40 subjects were reserved for testing. The number of subjects in the training set was varied from 1 to 50. Hyperparameters that maximized peak signal-to-noise ratio (PSNR) on the validation set were selected for each method.

Training was performed via the Adam optimizer with learning rate $\zeta$=$10^{-4}$, $\beta_{1}$=0.90 and $\beta_{2}$=0.99 [74]. All deep learning methods were trained to minimize hybrid $\ell_{1}$-$\ell_{2}$-norm loss between recovered and target data (e.g., between reconstructed and ground truth images for PSFNet, between recovered and acquired k-space samples for PSFNet_US) [64]. For PSFNet and MoDL, the selected number of epochs was 200, batch size was set to 2 for the limited number of training samples ($N_{samples}<$10), and to 5 otherwise. In DC blocks, $\lambda=\infty$ was used to enforce strict data consistency. For PSFNet and SPIRiT, the kernel width ($w$) and regularization parameter ($\kappa$) values were set as ($\kappa$, $w$) = ($10^{-2}$, 9) at R= 4 and ($10^{-2}$, 9) at R=8 for cT₁-weighted reconstructions, and as (10⁰, 17) at R=4 and ($10^{-2}$, 17) at R=8 for T₂-weighted reconstructions. For SPIRiT, the number of iterations $N_{iter}$ was set as 13 at R=4 and 27 at R=8 for cT₁-weighted reconstructions, 20 at R=4 and 38 at R=8 for T₂-weighted reconstructions. For sRAKI-RNN, the selected number of epochs was 500 and batch size was set to 32. All other optimization procedures were identical to MoDL. For SPARK, network architecture and training procedures were adopted from [17], except for the number of epochs ($N_{epoch}$) and learning rate ($\zeta$) which were optimized on the validation set as ($N_{epoch}$, $\zeta$)= (100, $10^{-2}$) For cT₁-weighted reconstructions, and ($N_{epoch}$, $\zeta$)= (250, $10^{-3}$) for T₂-weighted reconstructions.

All competing methods were executed on an NVidia RTX 3090 GPU, and models were coded in Tensorflow except for SPARK which was implemented in PyTorch. SPARK was implemented using the toolbox at https://github.com/YaminArefeen/spark_mrm_2021. The code to implement PSFNet will be available publicly at https://github.com/icon-lab/PSFNet upon publication.
所有竞争方法都在 NVidia RTX 3090 GPU 上执行，除了 SPARK 在 PyTorch 中实现外，模型都是用 Tensorflow 编码的。SPARK 是使用 https://github.com/YaminArefeen/spark_mrm_2021 的工具箱实现的。实施 PSFNet 的代码将在发布后于 https://github.com/icon-lab/PSFNet 公开提供。

Performance assessments for reconstruction methods were carried out by visual observations and quantitative metrics. PSNR and structural similarity index (SSIM) were used for quantitative evaluation. For each method, metrics were computed on coil-combined images from the reconstruction and from the fully-sampled ground truth acquisition. Statistical differences between competing methods were examined via non-parametric Wilcoxon signed-rank tests.

Several different experiments were conducted to systematically examine the performance of competing methods. Assessments aimed to investigate reconstruction performance under low training data regimes, generalization performance in case of mismatch between training and testing domains, contribution of the parallel-stream design to reconstruction performance, sensitivity to hyperparameter selection, performance in unsupervised learning, and computational complexity.

Performance in low-data regimes: Deep SG methods for MRI reconstruction typically suffer from suboptimal performance as the size of the training dataset is constrained. To systematically examine reconstruction performance, we trained supervised variants of PSFNet and MoDL while the number of training samples ($N_{samples}$) was varied in the range [2-500] cross sections. To attain a given number of samples, sequential selection was performed across subjects and across cross-sections within each subject. Thus, the number of unique subjects included in the training set roughly corresponded to $N_{samples}/10$ (since there were 10 cross-sections per subject). SS reconstructions were also performed with sRAKI-RNN, SPIRiT and SPARK. In the absence of fully-sampled ground truth data to guide the learning of the prior, unsupervised training of deep reconstruction models may prove relatively more difficult compared to supervised training. In turn, this may elevate requirements on training datasets for unsupervised models. To examine data efficiency for unsupervised training, we compared the reconstruction performance of PSFNet_US and MoDL_US as $N_{samples}$ was varied in the range of [2-500] cross sections. Comparisons were also provided against sRAKI-RNN, SPIRiT and SPARK.

Generalization performance: Deep reconstruction models can suffer from suboptimal generalization when the MRI data distribution shows substantial variation between the training and testing domains. To examine generalizability, PSFNet models were trained on data from a source domain and tested on data from a different target domain. The domain-transferred models were then compared to models trained and tested directly in the target domain. Three different factors were altered to induce domain variation: tissue contrast, undersampling pattern, and acceleration rate. First, the capability to generalize to different tissue contrasts was evaluated. Models were trained on data from a source contrast and tested on data from a different target contrast. Domain-transferred models were compared to target-domain models trained on data from the target contrast. Next, the capability to generalize to different undersampling patterns was assessed. Models were trained on data undersampled with variable-density patterns and tested on data undersampled with uniform-density patterns. Domain-transferred models were compared to target-domain models trained on uniformly undersampled data. Lastly, the capability to generalize to different acceleration rates was examined. Models were trained on acquisitions accelerated at R=4x and tested on acquisitions accelerated at R=8x. Domain-transferred models were compared to target-domain models trained at R=8x.

Sensitivity to hyperparameters: SS priors are learned from individual test scans as opposed to SG priors trained on larger training datasets. Thus, SS priors might show elevated sensitivity to hyperparameter selection. We assessed the reliability of reconstruction performance against suboptimal hyperparameter selection for SS priors. For this purpose, analyses were conducted on SPIRiT, SPARK and PSFNet that embody SS methods to perform linear reconstructions in k-space. The set of hyperparameters examined included regularization parameters for kernel estimation ($\kappa$) and kernel size ($w$). Separate models were trained using $\kappa$ in range [10^-3-10⁰] and $w$ in range [5-17].

Computational complexity: Finally, we assessed the computational complexity of competing methods. For each method, training and inference times were measured for a single subject with 10 cross-sections. Each cross-section had an imaging matrix size of 256x320 and contained data from 5 coils. For all methods including SS priors, hyperparameters optimized for cT₁-weighted reconstructions at R=4 were used.

Ablation analysis: To assess the contribution of the parallel-stream design in PSFNet, a conventional unrolled variant of PSFNet was formed, named as PSFNet_Serial. PSFNet_Serial combined the SG and SS priors via serial projections as described in Eq. 10. Modeling procedures and the design of SG and SS blocks were kept identical between PSFNet and PSFNet_Serial for fair comparison. Performance was assessed as $N_{samples}$ was varied in the range of [2-500] cross sections.

Common SG methods for MRI reconstruction are based on deep networks that require copious amounts of training data, so performance can substantially decline on limited training sets [28, 59]. In contrast, PSFNet leverages an SG prior to concurrently reconstruct an image along with an SS prior. Therefore, we reasoned that its performance should scale favorably under low-data regimes compared to SG methods. We also reasoned that PSFNet should yield elevated performance compared to SS methods due to residual corrections from its SG prior. To test these predictions, we trained supervised variants of PSFNet and MoDL along with SPIRiT, sRAKI-RNN, and SPARK while the number of training samples ($N_{samples}$) was systematically varied. Figure 2 displays PSNR performance for cT₁-weighted and T₂-weighted image reconstruction as a function of $N_{samples}$. PSFNet outperforms the scan-general MoDL method for all values of $N_{samples}$ ($p<0.05$). As expected, performance benefits with PSFNet become more prominent towards lower values of $N_{samples}$. PSFNet also outperforms traditional SPIRiT and scan-specific sRAKI-RNN and SPARK methods broadly across the examined range of $N_{samples}$ ($p<0.05$). Note that while MoDL requires $N_{samples}=30$ (3 subjects) to offer on par performance to SS methods, PSFNet yields superior performance with as few as $N_{samples}=2$. Representative reconstructions for cT₁- and T₂-weighted images are depicted in Figures 3 and 4, where $N_{samples}=10$ from a single subject were used for training. PSFNet yields lower reconstruction errors compared to all other methods in this low-data regime, where competing methods either show elevated noise or blurring.
用于 MRI 重建的常见 SG 方法基于需要大量训练数据的深度网络，因此在有限的训练集上性能可能会大幅下降 [28， 59]。相比之下，PSFNet 在同时重建映像之前利用 SG 以及 SS 先验。因此，我们推断，与 SG 方法相比，它的性能在低数据制度下应该具有良好的扩展性。我们还推断，与 SS 方法相比，PSFNet 应该产生更高的性能，因为其 SG 先验的残差校正。为了测试这些预测，我们训练了 PSFNet 和 MoDL 的监督变体以及 SPIRiT、sRAKI-RNN 和 SPARK，而训练样本的数量 （ $N_{samples}$ ） 是系统变化的。图 2 显示了 cT₁ 加权和 T₂ 加权图像重建的 PSNR 性能与 的函数关系。 $N_{samples}$ 对于 （ $p<0.05$ ） 的所有 $N_{samples}$ 值，PSFNet 的性能都优于扫描通用 MoDL 方法。正如预期的那样，PSFNet 的性能优势在较低的 . $N_{samples}$ PSFNet 在检查的范围内也广泛优于传统的 SPIRiT 和扫描特异性 sRAKI-RNN 和 SPARK 方法 $N_{samples}$ （ $p<0.05$ ）。请注意，虽然 MoDL 需要 $N_{samples}=30$ （3 个主题）提供与 SS 方法相当的性能，但 PSFNet 只需 $N_{samples}=2$ 即可产生卓越的性能。图 3 和 4 描述了 cT₁ 和 T₂ 加权图像的代表性重建，其中 $N_{samples}=10$ 使用单个受试者进行训练。与这种低数据范围内的所有其他方法相比，PSFNet 产生的重建误差较低，其中竞争方法要么显示更高的噪声，要么显示模糊。

Naturally, the performance of PSFNet increases as more training samples are available. Since the SS prior is independently learned for individual samples, it should not elicit systematic performance variations depending on $N_{samples}$. Thus, the performance gains can be attributed to improved learning of the SG prior. In turn, we predicted that PSFNet would put more emphasis on its SG stream as its reliability increases. To examine this issue, we inspected the weightings of the SG ($\gamma$) and SS ($\eta$) streams as the training set size was varied. Figure 5 displays weightings at the last cascade as a function of $N_{samples}$. For lower values of $N_{samples}$ where the quality of the SG prior is relatively limited, the SG and SS priors are almost equally weighted. In contrast, as the learning of the SG prior improves with higher $N_{samples}$, the emphasis on the SG prior increases while the SS prior is less heavily weighted.

We then questioned whether the performance benefits of PSFNet are also apparent during unsupervised training of deep network models. For this purpose, unsupervised variants PSFNet_US and MoDL_US were trained via self-supervision [64]. PSFNet_US was compared against MoDL_US, SPIRiT, sRAKI-RNN, and SPARK while the number of training samples ($N_{samples}$) was systematically varied. Figure 6 displays PSNR performance for cT₁-weighted and T₂-weighted image reconstruction as a function of $N_{samples}$. Similar to the supervised setting, PSFNet_US outperforms MoDL_US for all values of $N_{samples}$ ($p<0.05$), and the performance benefits are more noticeable at lower $N_{samples}$. In this case, however, MoDL_US is unable to reach the performance of the best performing SS method (SPARK) even at $N_{samples}=500$. In contrast, PSFNet_US starts outperforming SPARK with approximately $N_{samples}=50$ (5 subjects). The enhanced reconstruction quality with PSFNet_US is corroborated in representative reconstructions for cT₁- and T₂-weighted images depicted in Figures 7 and 8, where $N_{samples}=100$ were used for training. Taken together, these results indicate that the data-efficient nature of PSFNet facilitates the training of both supervised and unsupervised MRI reconstruction models.

An important advantage of SS priors is that they allow model adaptation to individual test samples, thereby promise enhanced performance in out-of-domain reconstructions [22]. Yet, SG priors with fixed parameters might show relatively limited generalizability during inference [23, 75]. To assess generalization performance, we introduced domain variations by altering three experimental factors: tissue contrast, undersampling pattern, and acceleration rate. For methods comprising SG components, we built both target-domain models that were trained in the target domain, and domain-transferred models that were trained in a non-target domain. We then compared the reconstruction performances of the two models in the target domain.

First, we examined generalization performance when the tissue contrast varied between training and testing domains (e.g., trained on cT₁, tested on T₂). Table 1 lists performance metrics for competing methods with $N_{samples}=500$. While performance losses are incurred for domain-transferred PSFNet-DT and MoDL-DT models that contain SG components, these losses are modest. On average, MoDL-DT shows a loss of 0.3dB PSNR and 0.1% SSIM ($p<0.05$), and PSFNet-DT shows a loss of 0.2dB PSNR and 0.1% SSIM ($p<0.05$). Note that PSFNet-DT still outperforms the closest competing SS method by 2.2dB PSNR and 1.8% SSIM ($p<0.05$).

Second, we examined generalization performance when models were trained with variable-density and tested on uniform-density undersampling patterns. Table 2 lists performance metrics for competing methods. On average across tissue contrasts, MoDL-DT suffers a notable performance loss of 3.6dB PSNR and 2.5% SSIM ($p<0.05$). In contrast, PSFNet-DT shows a relatively limited loss of 0.4dB PSNR and 0.2% SSIM ($p<0.05$). Note that PSFNet-DT again outperforms the closest competing SS method by 3.4dB PSNR and 3.7% SSIM ($p<0.05$).

Third, we examined generalization performance when models were trained at R=4x and tested on R=8x. Table 3 lists performance metrics for competing methods. On average across tissue contrasts, MoDL-DT suffers a notable performance loss of 1.0dB PSNR and performs slightly better in SSIM by 0.2%SSIM ($p<0.05$), whereas PSFNet-DT shows a lower loss of 0.6dB PSNR ($p<0.05$) and performs similarly in SSIM ($p>0.05$). PSFNet-DT outperforms the closest competing SS method by 1.2dB PSNR and 1.9% SSIM ($p<0.05$). Taken together, these results clearly suggest that the SS prior in PSFNet contributes to its improved generalization performance over the scan-general MoDL method, while the SG prior in PSFNet enables it to outperform competing SS methods.

Parameters of deep networks that implement SS priors are to be learned from a single test sample, so the resultant models can show elevated sensitivity to the selection of hyperparameters compared to SG priors learned from a collection of training samples. Thus, we investigated the sensitivity of PSFNet to key hyperparameters of its SS prior. SPIRiT, SPARK and PSFNet methods all embody a linear k-space reconstruction, so the relevant hyperparameters are the regularization weight and width for the convolution kernel. Performance was evaluated for models were trained in the low-data regime (i.e., $N_{samples}=10$, 1 subject) for varying hyperparameter values.

Figure 9 displays PSNR measurements for SPIRiT, SPARK and PSFNet across $\kappa$ in range (10^-3-10⁰). While the performance of SPIRiT and SPARK is notably influenced by $\kappa$, PSFNet is minimally affected by sub-optimal selection. On average across contrasts, the difference between the maximum and minimum PSNR values is 8.4dB for SPIRiT, 4.5dB for SPARK, and a lower 0.7dB for PSFNet. Note that PSFNet outperforms competing methods across the entire range of $\kappa$ ($p<0.05$). Figure 10 shows PSNR measurements for competing methods across $w$ in range (5-17). In this case, all methods show relatively limited sensitivity to the selection of $w$. On average across contrasts, the difference between the maximum and minimum PSNR values is 1.5dB for SPIRiT, 0.5dB for SPARK, and 0.2dB for PSFNet. Again, PSFNet outperforms competing methods across the entire range of $w$ ($p<0.05$). Overall, our results indicate that PSFNet yields improved reliability against sub-optimal hyperparameter selection than competing SS methods.

Next, we assessed the computational complexity of competing methods. Table 4 lists the training times of methods with SG priors, MoDL and PSFNet. Note that the remaining SS based methods do not involve a pre-training step. As it involves learning of an SS prior on each training sample, PSFNet yields elevated training time compared to MoDL. In return, however, it offers enhanced generalization performance and data-efficient learning. Table 4 also lists the inference times of SPIRiT, SPARK, sRAKI-RNN, MoDL and PSFNet. MoDL and PSFNet that employ SG priors with fixed weights during inference offer fast run times. In contrast, SPARK and sRAKI-RNN that involve SS priors learned on individual test samples have a high computational burden. Although PSFNet also embodies an SS prior, its uses a relatively lightweight linear prior as opposed to the nonlinear priors in competing SS methods. Therefore, PSFNet benefits from data-efficient learning while maintaining computationally-efficient inference.

To demonstrate the value of the parallel-stream fusion strategy in PSFNet over conventional unrolling, PSFNet was compared against a variant model PSFNet_Serial that combined SS and SG priors through serially alternated projections. Separate models were trained with number of training samples in the range $N_{samples}$=[2-500]. Performance in cT₁- and T₂ -weighted image reconstruction is displayed in Figure 11. PSFNet significantly improves reconstruction performance over PSFNet_Serial across the entire range of $N_{samples}$ considered ($p<0.05$), and the benefits grow stronger for smaller training sets. On average across contrasts for $N_{samples}<10$, PSFNet outperforms PSFNet_Serial by 1.8dB PSNR and 0.6% SSIM ($p<0.05$). These results indicate that the parallel-stream fusion of SG and SS priors in PSFNet is superior to the serial projections in conventional unrolling.