这是用户在 2024-9-8 23:31 为 http://localhost:8888/notebooks/host-starter-solution.ipynb 保存的双语快照页面,由 沉浸式翻译 提供双语支持。了解如何保存?

ADC 2024 starter notebook
ADC 2024 入门笔记本

This baseline notebook is designed to offer a starting point for the competiters. Please note that the approach we've taken is not THE solution — it's simply ONE possible approach. Our aim is to assist participants in exploring different ways to preprocess and model the data. Please feel free to fork the notebook and save the model/data for your own exploration.
这款基准笔记本旨在为参赛者提供一个起点。请注意,我们采取的方法并不是解决方案——它只是一种可能的方法。我们的目标是帮助参与者探索对数据进行预处理和建模的不同方法。请随意分叉笔记本并保存模型/数据以供您自己探索。

This notebook was prepared by Virginie Batista and Angèle Syty from the Institut d'Astrophysique de Paris, and Orphée Faucoz from Centre National d’Etudes Spatiales (CNES), with support from Gordon Yip and Tara Tahseen from University College London.
本笔记本由巴黎天体物理学研究所的 Virginie Batista 和 Angèle Syty 以及国家空间研究中心 (CNES) 的 Orphée Faucoz 编写,并得到了伦敦大学学院的 Gordon Yip 和 Tara Tahseen 的支持。

READ THIS BEFORE YOU PROCEED
在继续之前请阅读本文

This training procedure uses the light dataset produced from this notebook (Version 5). We applied all the calibration steps EXCEPT Linearity Correction with Chunksize = 1. The binned dataset is available to download here. If you want to carry out all the correction, you will have to do so yourself.
此训练过程使用从此笔记本(版本 5)生成的光数据集。我们应用了除块大小 = 1 时的线性校正之外的所有校准步骤。分箱数据集可在此处下载。如果您想执行所有更正,则必须亲自执行。

This notebook will only provide the model checkpoints, you are welcomed to use these checkpoints with your own script and submit to the leaderboard.
本笔记本仅提供模型检查点,欢迎您将这些检查点与您自己的脚本一起使用并提交到排行榜。

Task overview
任务概述

The challenge's primary objective is to process these exposures to produce a single, clean spectrum for each exoplanet, summarizing the rp/rs values across all wavelengths.
该挑战的主要目标是处理这些曝光,为每个系外行星生成单一、干净的光谱,总结所有波长的 rp/rs 值。

The exposure are subject to noises and the images or spectrum are not perfect. The Jitter noise has a complex signature that the ML model should recognize to produce a better spectra.
曝光会受到噪声的影响,图像或光谱并不完美。抖动噪声具有复杂的特征,ML 模型应识别该特征以产生更好的频谱。

Different techniques are possible and are up to the participant imagination to produce a novel (and hopefully better) solution to this task.
不同的技术是可能的,并且取决于参与者的想象力来为该任务产生新颖的(并且希望是更好的)解决方案。

Here outline our baseline approach :
这里概述了我们的基线方法:

We first fit a 1D CNN to fit the mean value of the transmission spectra, taking as input the transit white curve (total flux of each image taken as a function of time).
我们首先拟合一维 CNN 来拟合透射光谱的平均值,将传输白曲线(每幅图像的总通量作为时间的函数)作为输入。

For the second part of the baseline, to retrieve the atmopsheric features, we make the data lighter by summing up the fluxes along the y-axis, for each wavelength, resulting in 2D images of dimension (N_times, N_wavelengths). We also cut the signal to remove the out of transit in order to enhance transit depth variations between wavelengths. For the same reason, we substract the mean flux, corresponding to the average transit depth, to keep only wavelength variations around this mean. We use a 2D CNN to fit the atmospheric features.
对于基线的第二部分,为了检索大气特征,我们通过对每个波长沿 y 轴的通量求和来使数据更轻,从而产生尺寸为(N_times,N_wavelengths)的二维图像。我们还切割信号以消除传输之外的信号,以增强波长之间的传输深度变化。出于同样的原因,我们减去与平均传输深度相对应的平均通量,以仅保留该平均值周围的波长变化。我们使用 2D CNN 来拟合大气特征。

In [1]:

Import library
导入库

In [2]:
2024-08-28 10:29:23.387138: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-08-28 10:29:23.387261: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-08-28 10:29:23.493763: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Setup Paths and Read Data
设置路径和读取数据

In [3]:
In [4]:

We create a directory to save the outputs of this notebook, and define the hyperparameters of the model
我们创建一个目录来保存此笔记本的输出,并定义模型的超参数

In [5]:
Directory ./output created.

1D-CNN for mean transit depth

Preprocessing for 1D CNN

In [6]:

We create the dataset by adding the FGS frame, crushed in one column, at the end of the AIRS data cube.
我们通过在 AIRS 数据立方体的末尾添加压缩为一列的 FGS 框架来创建数据集。

The images are normalized using the star spectrum extracted from the images themselves.
使用从图像本身提取的星光谱对图像进行标准化。

In [7]:

we sum up the pixels on the y-axis to transform the data into 2D images
我们对 y 轴上的像素求和,将数据转换为 2D 图像

In [8]:

We divide the images by the star flux assuming the first and last 50 instants belong to the out of transit.
我们将图像除以恒星通量,假设第一个和最后 50 个瞬间属于凌日之外。

In [9]:

Split the targets and observations between valid and train
在有效和训练之间分割目标和观察结果

We start by computing a "white curve", that is actually the sum of the signal over the all image, as a function of time. We split the data and normalize the train/valid/test data.
我们首先计算一条“白色曲线”,它实际上是整个图像上的信号之和,作为时间的函数。我们分割数据并对训练/有效/测试数据进行标准化。

In [10]:
In [11]:
In [12]:

Train 1D CNN

The model to estimate the mean of the target spectrum using the white light-curve is a 1D-CNN with Dropout layers to make a MC Dropout prediction.
使用白光曲线估计目标光谱平均值的模型是具有 Dropout 层的 1D-CNN,用于进行 MC Dropout 预测。

In [13]:
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                     Output Shape                  Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ input_layer (InputLayer)        │ (None, 187, 1)         │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv1d (Conv1D)                 │ (None, 185, 32)        │           128 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling1d (MaxPooling1D)    │ (None, 92, 32)         │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization             │ (None, 92, 32)         │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv1d_1 (Conv1D)               │ (None, 90, 64)         │         6,208 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling1d_1 (MaxPooling1D)  │ (None, 45, 64)         │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv1d_2 (Conv1D)               │ (None, 43, 128)        │        24,704 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling1d_2 (MaxPooling1D)  │ (None, 21, 128)        │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv1d_3 (Conv1D)               │ (None, 19, 256)        │        98,560 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling1d_3 (MaxPooling1D)  │ (None, 9, 256)         │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 2304)           │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 500)            │     1,152,500 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 500)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 100)            │        50,100 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_1 (Dropout)             │ (None, 100)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │           101 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 1,332,429 (5.08 MB)
 Trainable params: 1,332,365 (5.08 MB)
 Non-trainable params: 64 (256.00 B)
In [14]:
Running ...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1724841149.976147      68 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
W0000 00:00:1724841149.998138      68 graph_launch.cc:671] Fallback to op-by-op mode because memset node breaks graph update
W0000 00:00:1724841154.223044      69 graph_launch.cc:671] Fallback to op-by-op mode because memset node breaks graph update
Done.

1D CNN Inference

In [15]:

Then, we perform the MC Dropout to obtain the mean prediction and the uncertainty associated. We choose to compute 1000 instances.
然后,我们执行 MC Dropout 以获得平均预测和相关的不确定性。我们选择计算 1000 个实例。

In [16]:
Running ...
Done.
In [17]:
In [18]:
MSE :  98.67602253308468 ppm
In [19]:

2D CNN for atmospheric features
用于大气特征的 2D CNN

We now remove the mean value (transit depth) of the spectra to keep the atmospheric features only
我们现在删除光谱的平均值(凌日深度)以仅保留大气特征

Preprocessing for 2D CNN

In [20]:

We normalize the targets so that they range between -1 and 1, centered on zero
我们对目标进行标准化,使其范围在 -1 到 1 之间,以零为中心

In [21]:
In [22]:
In [23]:
(538, 187, 283)

We cut the transit to keep the in-transit. We assume an arbitrary transit duration of 40 instants with a transit occuring between 75 and 115.
我们削减了过境以保持在途。我们假设任意的凌日持续时间为 40 个瞬间,其中凌日发生在 75 到 115 之间。

In [24]:

We remove the mean value of the in-transit to get relative data like the targets
我们删除传输中的平均值以获得像目标这样的相关数据

In [25]:

We use the same normalization as for the targets, i.e. between -1 and 1 centered on zero
我们对目标使用相同的归一化,即以零为中心的 -1 和 1 之间

In [26]:
In [27]:

Train 2D CNN

In [28]:
Model: "functional_1"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                     Output Shape                  Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ input_layer_1 (InputLayer)      │ (None, 40, 283, 1)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d (Conv2D)                 │ (None, 40, 283, 32)    │           128 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d (MaxPooling2D)    │ (None, 20, 283, 32)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_1           │ (None, 20, 283, 32)    │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D)               │ (None, 20, 283, 64)    │         6,208 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_1 (MaxPooling2D)  │ (None, 10, 283, 64)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_2 (Conv2D)               │ (None, 10, 283, 128)   │        24,704 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_2 (MaxPooling2D)  │ (None, 5, 283, 128)    │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_3 (Conv2D)               │ (None, 5, 283, 256)    │        98,560 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_4 (Conv2D)               │ (None, 5, 283, 32)     │        24,608 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_3 (MaxPooling2D)  │ (None, 5, 141, 32)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization_2           │ (None, 5, 141, 32)     │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_5 (Conv2D)               │ (None, 5, 141, 64)     │         6,208 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_4 (MaxPooling2D)  │ (None, 5, 70, 64)      │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_6 (Conv2D)               │ (None, 5, 70, 128)     │        24,704 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_5 (MaxPooling2D)  │ (None, 5, 35, 128)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_7 (Conv2D)               │ (None, 5, 35, 256)     │        98,560 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_6 (MaxPooling2D)  │ (None, 5, 17, 256)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten_1 (Flatten)             │ (None, 21760)          │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 700)            │    15,232,700 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout_2 (Dropout)             │ (None, 700)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 283)            │       198,383 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
 Total params: 15,715,019 (59.95 MB)
 Trainable params: 15,714,891 (59.95 MB)
 Non-trainable params: 128 (512.00 B)
In [29]:

Postprocessing and visualisation

We obtain uncertainties on the predictions by computing a MCDropout.
我们通过计算 MCDropout 获得预测的不确定性。

In [30]:
In [31]:
MSE :  40.931186673657066 ppm
In [32]:
In [33]:
In [34]:

Combine 1D and 2D CNN output for FINAL SPECTRA
组合 1D 和 2D CNN 输出以获得最终光谱

In [35]:
In [36]:
In [37]:
MSE :  106.7461112000446 ppm
In [38]:
x1.00
>
<
>>
<<
O
x1.00