Machine vision for low-cost remote control of mosquitoes by power laser
用于通过功率激光器低成本远程控制蚊子的机器视觉

Abstract 抽象

In this paper, we present an innovative and effective method for remote monitoring of mosquitoes and their neutralization. We explain in detail how we leverage modern advances in neural networks to use a powerful laser to neutralize mosquitoes. The paper presented the experimental low-cost prototype for mosquito control, which uses a powerful laser to thermally neutralize the mosquitoes. The developed device is controlled by a single-board computer based on the neural network. The paper demonstrated experimental research for mosquito neutralization during which, to maximize approximation to natural conditions, simulation of various working conditions was conducted. The manuscript showed that a low-cost device can be used to kill mosquitoes with a powerful laser.
在本文中，我们提出了一种创新有效的蚊子远程监测及其中和方法。我们详细解释了我们如何利用神经网络的现代进步来使用强大的激光来中和蚊子。该论文介绍了用于蚊子控制的低成本实验原型，它使用强大的激光对蚊子进行热中和。开发的设备由基于神经网络的单板计算机控制。该论文展示了蚊子中和的实验研究，在此期间，为了最大限度地接近自然条件，对各种工作条件进行了模拟。手稿表明，可以使用低成本设备通过强大的激光杀死蚊子。

1 Introduction 1 介绍

The purpose of this manuscript is to demonstrate the possibility of using a computer vision system for remote monitoring of the position of mosquitoes and their subsequent neutralization. The relevance of this work is confirmed by the presence of problems associated with diseases that are transmitted by mosquito bites [1,2,3,4]. Due to the fact that the problem with mosquitoes is becoming more relevant every year, it is advisable to use technological progress for solving this problem, in particular: neural networks.
本手稿的目的是展示使用计算机视觉系统远程监测蚊子位置并随后中和蚊子的可能性。与蚊虫叮咬传播的疾病相关的问题证实了这项工作的相关性[1,2,3,4]。由于蚊子问题每年都变得越来越重要，因此建议使用技术进步来解决这个问题，特别是：神经网络。

For remote object detection, the characteristic of a camera is very important. A typical camera can correctly track small moving objects at several centimeters. This is a limiting factor when using machine vision to monitor mosquitoes. The speed of a mosquito is no more than 5 m per second. For a linear scan camera, the color of a mosquito is a moving few pixel. Since the surrounding color scheme plays a role, it is necessary to use color cameras. Monochrome cameras are faster, but they are suitable for laboratory tests, where artificial conditions of an experiment are made: the background is white, the mosquito is dark. The temperature of the mosquito is like the environment, which is why infrared cameras cannot be used. The size of a mosquito can vary from 1 to 5 mm, this is the main criterion for choosing the size of the sensor matrix along with the number of pixels, and for choosing a camera lens.
对于远程目标检测，相机的特性非常重要。典型的相机可以正确跟踪几厘米处的小型移动物体。这是使用机器视觉监测蚊子时的限制因素。蚊子的速度不超过每秒 5 m。对于线性扫描相机，蚊子的颜色是移动的几个像素。由于周围的配色方案起着作用，因此有必要使用彩色相机。单色相机速度更快，但它们适用于实验室测试，其中人工实验条件是：背景是白色的，蚊子是深色的。蚊子的温度就像环境一样，这就是为什么不能使用红外摄像机的原因。蚊子的大小可以从 1 到 5 毫米不等，这是选择传感器矩阵大小和像素数以及选择相机镜头的主要标准。

The following articles have a similar purpose—mosquitoes control. In the article [5], the accurate determination of the presence of a mosquito by its acoustic signature by the neural network was achieved. However, this research indicates only the presence of the mosquito and not its exact location. Mosquito image processing in the work [6] allowed researchers to detect mosquitoes and their location. In the paper, the photographs are in high resolution were used. The proposed neural network model was intended for classification and not for monitoring. In the manuscript [7], automatic conveyor of acoustic data collection is introduced. The machine-learning algorithm allows you to detect several types of mosquitoes offline by sound recording. The proposed method is intended for mosquito’s classification. The neural network for the classification of mosquitoes was presented in the manuscript [8]. Mosquito monitoring was not considered. The next manuscript is the close topic of our research [9], which presents the results of mosquito detection using a camera with high resolution. In the paper, monitoring for a close distance was considered. The results for determining the position of z are not presented.
以下文章具有类似的目的 — 蚊子控制。在文章 [5] 中，通过神经网络的声学特征准确确定了蚊子的存在。然而，这项研究仅表明蚊子的存在，而不是它的确切位置。这项工作中的蚊子图像处理 [6] 使研究人员能够检测蚊子及其位置。在论文中，使用了高分辨率的照片。所提出的神经网络模型用于分类，而不是用于监控。在手稿 [7] 中，介绍了声学数据收集的自动传送带。机器学习算法允许您通过录音离线检测多种类型的蚊子。所提出的方法旨在对蚊子进行分类。手稿中介绍了用于蚊子分类的神经网络 [8]。未考虑蚊子监测。下一份手稿是我们研究的热门主题 [9]，它展示了使用高分辨率相机检测蚊子的结果。在本文中，考虑了近距离监测。没有给出确定 z 位置的结果。

Mosquitoes are several millimeters in size, but in the research, as a rule, for detection, larger objects are used—pedestrians, ships, cars, and crops [10,11,12]. Among insects, researches on tracking bees are popular. In these studies, a swarm is tracked and not a single bee [13].
蚊子有几毫米大，但在研究中，为了进行检测，通常会使用更大的物体——行人、轮船、汽车和农作物[10,11,12]。在昆虫中，追踪蜜蜂的研究很受欢迎。在这些研究中，跟踪的是一群蜜蜂，而不是一只蜜蜂 [13]。

It is worth adding that the Bill and Melinda Gates Foundation research on the introduction of laser mosquito neutralization technology as a promising method of combating malaria was sponsored. As a result, the company Photonic Sentry was created (https://photonicsentry.com/). This company demonstrated a video on mosquito neutralization by laser but did not provide any technical characteristics of the device on the website.
值得一提的是，比尔和梅琳达·盖茨基金会 （Bill and Melinda Gates Foundation） 赞助了关于引入激光蚊子中和技术作为一种有前途的抗疟疾方法的研究。因此，Photonic Sentry 公司应运而生 （https://photonicsentry.com/）。该公司演示了一段关于激光中和蚊子的视频，但没有在网站上提供该设备的任何技术特性。

In the paper [14], a system that uses a combination of optical sources, detectors, and sophisticated software to locate and destroy mosquitoes using a laser is presented. This research was implemented at the junction of the emergence of deep neural networks, and therefore, their capabilities have not yet been considered in this article. The authors studied the effect of mosquito speed on tracking accuracy. At the same time, sufficient information about the equipment was not provided. We used a similar method to control the position of the laser with mirrors. Auxiliary methods for determining the position of mosquitoes, due to the combination of optical sources and detectors that were considered in the paper, are very valuable and can be used by us in the future to improve the tracking accuracy.
在论文 [14] 中，提出了一种结合使用光源、探测器和复杂软件来使用激光定位和消灭蚊子的系统。这项研究是在深度神经网络出现的交界处实施的，因此，本文尚未考虑它们的能力。作者研究了蚊子速度对跟踪精度的影响。同时，没有提供有关设备的足够信息。我们使用类似的方法用镜子控制激光器的位置。由于论文中考虑了光源和探测器的组合，确定蚊子位置的辅助方法非常有价值，将来可以被我们用来提高跟踪精度。

One of the latest and most relevant works in this area was presented by Intellectual Ventures Laboratory, Bellevue, WA, USA [15]. The authors have achieved impressive results in the use of the laser and identified several key parameters required for accurate tracking and targeting of objects during their flight. Remarkably, several species of mosquitoes have been considered. It was not possible to take full advantage of the results, since the presented installation is difficult to implement, but we plan to introduce the ideas laid down in their implementation into our research in the future. The main goal of this research is to eradicate mosquitoes. To do this, you can choose two ways—to increase the tracking accuracy—which requires more processing time and more expensive equipment, or, as we did, choose low tracking accuracy. We used more laser shots and cheap equipment, which ultimately gives the same positive result. Since with accuracy (10%) of 10 shots per second, we can destroy one mosquito. At the same time, in the future, the accuracy of the device will be increased when using STM32 and deep neural networks. This improvement will reduce processing time and installation cost.
该领域最新和最相关的工作之一是由美国华盛顿州贝尔维尤的 Intellectual Ventures 实验室 [15]。作者在使用激光方面取得了令人印象深刻的结果，并确定了在飞行过程中准确跟踪和瞄准物体所需的几个关键参数。值得注意的是，已经考虑了几种蚊子。由于所提出的安装难以实现，因此无法充分利用结果，但我们计划将实施中提出的想法引入我们未来的研究中。这项研究的主要目标是根除蚊子。为此，您可以选择两种方法 — 提高跟踪精度 — 这需要更多的处理时间和更昂贵的设备，或者像我们一样，选择较低的跟踪精度。我们使用了更多的激光射击和廉价的设备，最终产生了同样的积极结果。因为每秒 10 次射击的准确率 （10%） 我们可以消灭一只蚊子。同时，在未来使用 STM32 和深度神经网络时，设备的准确性将得到提高。这种改进将减少处理时间和安装成本。

It is worth noting that the problem with vectors of diseases with the help of mosquitoes is primarily pronounced in countries with weak economies, so the solution to this gap should be in a low-price range.
值得注意的是，在蚊子的帮助下传播疾病媒介的问题主要在经济薄弱的国家很明显，因此解决这一差距的方法应该是低价。

2 Materials and methods
阿拉伯数字 材料和方法

A mosquito is comparable in size with a few pixels for the low-cost camera (10 MP), an increase in the mosquito size of the camera leads to an obvious improvement in the result, but significantly reduces the area of online monitoring. According to flight dynamics, it is possible to predict what type of insect flies we are monitoring, for example, when flying, Drosophila melanogaster has several atypical movements [16]. Since the position of the camera can be arbitrary, the camera was calibrated by a coordinate system. The process of camera calibration is described in detail in the manuscript [17].
蚊子的大小与低成本相机 （10 MP） 的几个像素相当，相机蚊子大小的增加会导致结果的明显改善，但显着减少了在线监控的面积。根据飞行动力学，可以预测我们正在监测的昆虫蝇类型，例如，在飞行时，黑腹果蝇有几种非典型运动 [16]。由于相机的位置可以是任意的，因此相机是通过坐标系校准的。手稿 [17] 中详细描述了相机校准的过程。

To use the pre-trained deep-learning network on the Raspberry Pi to monitor mosquitoes, it was originally intended to use deep-learning methods. But due to the limited amount of RAM on the Raspberry Pi 4 (1–4 GB) and because of the low processor speed of 1.5 GHz, the use of deep neural networks is almost impossible (ResNet > 100 MB, VGGNet > 550 MB, AlexNet > 200 MB, GoogLeNet > 30 MB). In an attempt to use SqueezeNet, we managed to get a model with a weight of 5 MB, but even in this case, the result of processing the image for the presence of the desired object was about 1 s. Real-time detection with R-CNN, Fast R-CNN, Faster R-CNN, Yolo, and RetinaNet has the same recognition speed problem. The solution is to use NVIDIA Jetson TX1 and TX2—a special platform for computing neural networks. The main disadvantage is the high cost. Therefore, we focused on the methods that can be implemented on the Raspberry Pi, which allowed us to create a low-cost device.
为了在 Raspberry Pi 上使用预先训练的深度学习网络来监测蚊子，它最初打算使用深度学习方法。但由于 Raspberry Pi 4 上的 RAM 数量有限 （1-4 GB） 并且由于 1.5 GHz 的低处理器速度，使用深度神经网络几乎是不可能的（ResNet > 100 MB，VGGNet > 550 MB，AlexNet > 200 MB，GoogLeNet > 30 MB）。在尝试使用 SqueezeNet 时，我们设法获得了一个权重为 5 MB 的模型，但即使在这种情况下，处理图像以存在所需对象的结果也约为 1 秒。使用 R-CNN、Fast R-CNN、Faster R-CNN、Yolo 和 RetinaNet 进行实时检测存在相同的识别速度问题。解决方案是使用 NVIDIA Jetson TX1 和 TX2 — 一个用于计算神经网络的特殊平台。主要缺点是成本高。因此，我们专注于可以在 Raspberry Pi 上实现的方法，这使我们能够创建一个低成本的设备。

One of the most common methods for detecting moving objects is frame difference, background subtraction, and analysis of the optical flux field. However, this method showed the low result for tracking.
检测移动物体的最常用方法之一是帧差、背景减去和光通量场分析。但是，此方法显示跟踪结果较低。

The type is common mosquito (lat. Culex pipiens). The size of mosquitoes did not exceed 5 mm. Gender and age of the mosquito was not considered. Photos of mosquitoes were taken by camera from distances of 300 mm. Resolution of mosquito’s image composed of − 250 × 250 pixels. In this work, the cascade of Haar was used. In the research, 1460 photos with mosquitoes were used as positive examples and 1500 photos as negative examples.
类型是普通蚊子（拉丁语 Culex pipiens）。蚊子的大小不超过 5 毫米。未考虑蚊子的性别和年龄。蚊子的照片是用相机从 300 毫米的距离拍摄的。蚊子图像的分辨率由 − 250 × 250 像素组成。在这项工作中，使用了 Haar 的级联。在研究中，1460 张有蚊子的照片被用作正面例子，1500 张照片被用作负面例子。

Power laser (15 W) is dangerous for people. Therefore, the command—“use the laser” should be given with maximum confidence, when the target is identified correctly. It is also necessary to know the mathematics of mosquito flight. Descriptions of the flight and the compilation of a mathematical model of the mosquito was described in the papers [18,19,20]. Similar research is presented in the paper [21], where the speed of a mosquito is determined by the expression, which was used in this work:
功率激光器 （15 W） 对人体有害。因此，当目标被正确识别时，应该以最大的置信度给出命令——“使用激光”。还需要了解蚊子飞行的数学知识。论文中描述了蚊子的飞行情况和数学模型的汇编[18,19,20]。论文 [21] 中提出了类似的研究，其中蚊子的速度由该表达式决定，该表达式在这项工作中使用：

$$S=S_{max}-\left(S_{max}-S_{min}\right)\times F\left(b,b0\right),$$

(1)

where S_max and S_min are the maximum and minimum speed of a mosquito, F is a ramp function that takes values between 0 and 1.
其中 S_max 和 S_min 是蚊子的最大和最小速度，F 是采用 0 到 1 之间的值的斜坡函数。

In operation, the flight direction and speed of each mosquito are updated every time with units. The updated position of mosquitoes is calculated by
在操作中，每只蚊子的飞行方向和速度每次都会用单位更新。蚊子的更新位置计算公式为

$$\left(\begin{array}{c}{x}_{n+1}\\ y_{n+1}\end{array}\right)=\left(\begin{array}{c}{x}_n\\ y_n\end{array}\right)+s\left(\mathrm{\Delta }T\right)\times \overrightarrow{d}+\overrightarrow{V}\times \mathrm{\Delta }T,$$

(2)

where (x_n, y_n) is the position of the mosquito at time step n. Number d is a direction vector that varies between tracking and tracing. V is the wind speed.
其中 （x_n， y_n） 是蚊子在时间步 n 的位置。数字 d 是在跟踪和跟踪之间变化的方向向量。V 是风速。

There are two main types of distortion: radial distortion and tangential distortion. In our case, radial distortion does not bring an error. However, there is tangential distortion—because of which we have image distortions caused by instability in the camera position parallel to the image plane, due to the mobility of the developed prototype.
变形主要有两种类型：径向变形和切向变形。在我们的例子中，径向变形不会带来误差。然而，存在切向失真，因此，由于开发原型的移动性，平行于图像平面的相机位置的不稳定会导致图像失真。

As a result, the coordinates of the pixels were recalculated by the following formula:
因此，像素的坐标由以下公式重新计算：

$$u_{\text{corrected}}=u\left(1+k_1\times r^2+k_2\times r^4+k_3\times r^6\right)+2\times p_1\times u\times v+p_2\left(r^2+2\times u^2\right)$$

(3)

$$v_{\text{corrected}}=u\left(1+k_1\times r^2+k_2\times r^4+k_3\times r^6\right)+p_1\times \left(r^2+2\times u^2\right)+2\times p_2\times u\times v$$

(4)

$$r^2=u^2+v^2,$$

(5)

where ($u_{\text{corrected}}$, $v_{\text{corrected}}$) is the pixel location after removing geometric distortions; k₁, k₂, and k₃ are the radial distortion coefficients; p₁ and p₂ are the coefficients of tangential distortion.
其中 （ $u_{\text{corrected}}$ ， $v_{\text{corrected}}$ ） 是去除几何变形后的像素位置;k₁、k₂ 和 k₃ 是径向畸变系数;p₁ 和 p₂ 是切向畸变的系数。

The next tools in the research were used:
使用了研究中的下一个工具：

- Raspberry Pi 3 Model B +, Broadcom BCM2837B0 with a 64-bit quad-core processor (ARM Cortex-A53);
- Raspberry Pi 3 Model B +，Broadcom BCM2837B0，配备 64 位四核处理器 （ARM Cortex-A53）;

- Programming language—python 3.6, library of machine vision OpenCV 3.4.1;
- 编程语言 — python 3.6、机器视觉 OpenCV 3.4.1 库;

- PI camera, Sony IMX219 Exmor;
- PI 相机，Sony IMX219 Exmor;

- Telephoto lens Focal length: 70–200 mm;
- 长焦镜头焦距：70–200 毫米;

- Servomotor with operating speed (7.4 V): 0.065 s/60°;
- 运行速度 （7.4 V） 的伺服电机：0.065 s/60°;

- Galvanometer with speed—20 kpps;
- 带速度的检流计 — 20 kpps;

- Audio Sensor, MAX9814; - 音频传感器，MAX9814;

- Power laser, 15 W, wavelength—450 nm;
- 功率激光器，15 W，波长 - 450 nm;

- Microcontroller esp8266.
- 微控制器 esp8266。

All experiments were carried out during the daytime. Therefore, the monitoring was carried out with a widely closed diaphragm. Vignetting and other aberrations are reduced to the minimum, and the depth of field was increased. The main problem for identifying a mosquito is the background on which the mosquito is located. The problem with the environments can be solved by correctly pre-processing the image, which allows us to identify useful features from an image.
所有实验均在白天进行。因此，监测是在宽密的隔膜下进行的。暗角和其他像差减少到最低限度，并增加了景深。识别蚊子的主要问题是蚊子所在的背景。环境问题可以通过正确预处理图像来解决，这使我们能够从图像中识别有用的特征。

In this research, the task was to show the prospects of using this method to fight mosquitoes. To simplify the task, the background was taken monophonic, which made it possible to determine the position of the mosquito in the image accurately, Fig. 1 (up to row). We considered with a diverse background, when the mosquito is next to the plant, Fig. 1 (low row). In this case, the standard means of computer vision are not able to extract useful features.
在这项研究中，任务是展示使用这种方法对抗蚊子的前景。为了简化任务，背景是单声道的，这使得可以准确地确定蚊子在图像中的位置，图 3。1（最多一行）。我们考虑了不同的背景，当蚊子在植物旁边时，图 31 （低行）。在这种情况下，计算机视觉的标准方法无法提取有用的特征。

Fig. 1 图 1

Image processing operation: a received image with mosquito, b edges detection by cv2.Canny (img, 10,500), c image thresholding by cv2.threshold (img, 127,155, cv2.THRESH_BINARY_INV), d image thresholding by cv2.threshold (img, 127,155, cv2.THRESH_TOZERO_INV), e morphological transformations by cv2.morphologyEx (img, cv2.MORPH_BLACKHAT, kernel1), f morphological transformations by cv2.morphologyEx (img, cv2.MORPH_GRADIENT, kernel2), g segmentation by cv2.kmeans (pixel_values, k, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS, h color detection by cv2.inRange (hsv, h_min, h_max), where hsv, h_min, and h_max for up row—from 0, 0, 19 to 128, 35, 168. For low row, hsv, h_min, and h_max—from 23, 3, 128 to 65, 44, 180
图像处理操作：A 接收到带有蚊子的图像，通过 cv2 进行 B 边缘检测。Canny （img， 10,500）， c 图像阈值 cv2.threshold （img， 127,155， cv2.THRESH_BINARY_INV），d 通过 cv2.threshold （img， 127,155， cv2.THRESH_TOZERO_INV），e cv2.morphologyEx （img， cv2.MORPH_BLACKHAT， kernel1）， f cv2.morphologyEx （img， cv2.MORPH_GRADIENT， kernel2）， g segmentation by cv2.kmeans （pixel_values， k， None， criteria， 10， cv2.KMEANS_RANDOM_CENTERS，通过 cv2.inRange （hsv， h_min， h_max） 进行 h 颜色检测，其中 hsv、h_min 和 h_max 为向上行 — 从 0、0、19 到 128、35、168。对于低排、hsv、h_min 和 h_max - 从 23、3、128 到 65、44、180

As can be seen from Fig. 1 (up to row), if one tone behind the mosquito is obtained, it is quite easy to identify its signs, because of which it is possible to calculate mosquito coordinates without using deep neural networks. However, at the same time, the use of these filters with a diverse background (low row) does not give positive results. In the natural environment, the use of various filters and cascades of Haar will not allow to correctly track the object. In this case, the use of deep neural networks is feasible, for example—Mask R-CNN.
从图 1 中可以看出。1（最多一行），如果获得蚊子后面的一个音调，则很容易识别它的迹象，因此可以在不使用深度神经网络的情况下计算蚊子坐标。然而，与此同时，使用这些具有不同背景（低行）的过滤器并没有产生积极的结果。在自然环境中，使用各种过滤器和 Haar 级联将无法正确跟踪对象。在这种情况下，使用深度神经网络是可行的，例如 Mask R-CNN。

3 Technical description of the project
3 项目技术描述

For the implementation of machine vision, we used OpenCV-4.1.1 library. The next researches have similar methods for insect identification [22,23,24].
为了实现机器视觉，我们使用了 OpenCV-4.1.1 库。接下来的研究有类似的昆虫鉴定方法[22,23,24]。

To facilitate navigation, we developed an electronic board with determining the distance to the source of the object. The electronic has an audio sensor for mosquito sound control and an analog amplifier for an increased signal from the audio sensor, Fig. 2.
为了方便导航，我们开发了一个电子板，用于确定到物体源的距离。该电子设备有一个用于蚊子声音控制的音频传感器和一个用于增加音频传感器信号的模拟放大器，图 1。2.

Fig. 2 图 2

A device for converting an analog signal: 1—analog devices for converting and filtering an audio signal into voltage from 1 to 4.5 V, 2—an audio sensor, 3—terminals for connecting the board to esp8266 microcontroller
一种用于转换模拟信号的设备： 1—用于将音频信号转换并过滤为 1 至 4.5 V 电压的模拟设备， 2—一个音频传感器， 3—用于将电路板连接到 esp8266 微控制器的端子

The buzzing of a mosquito is accompanied by a subtle sound since the oscillation frequency of its wings is from 1000 Hz and with a sound from 36 dB—depending on the species this is most likely to be the second harmonic, mosquito fundamental tones can range approximately between 300 and 700 Hz [25].
蚊子的嗡嗡声伴随着微妙的声音，因为它翅膀的振荡频率为 1000 Hz，声音为 36 dB——根据物种的不同，这很可能是二次谐波，蚊子的基本音调范围约为 300 至 700 Hz[25]。

A board with a sound sensor can detect a mosquito sound of 30–50 dB in a radius of 70 cm with an accuracy of up to 8 cm. For research purposes, this is enough, since in practice you can use a sensor with a higher resolution. Initially, we wanted to add additional noise or record in real conditions. However, these technical issues have already been solved in the works [26,27,28]. The buzzing of a mosquito is accompanied by a subtle sound since the oscillation frequency of its wings is from 1000 Hz and with a sound from 36 dB. We measure the sound of the commander with Audio Sensor, MAX9814. With an external audio noise generator, we set the setting about the presence of a mosquito at 30 dB, 800 Hz. For checking, we measured the noise with Noise Level Meter PCE-MSL with frequency range 31.5 Hz and accuracy 2 dB (Manufacturer: PCE Instruments, USA).
带有声音传感器的板可以检测到半径为 70 cm 的 30-50 dB 的蚊子声音，精度高达 8 cm。出于研究目的，这就足够了，因为在实践中您可以使用更高分辨率的传感器。最初，我们想添加额外的噪声或在真实条件下进行录制。然而，这些技术问题已经在工作中得到解决 [26,27\u201228]。蚊子的嗡嗡声伴随着微妙的声音，因为它翅膀的振荡频率为 1000 Hz，声音为 36 dB。我们使用 Audio Sensor， MAX9814 测量指挥官的声音。使用外部音频噪声发生器，我们将蚊子存在的设置设置为 30 dB、800 Hz。为了进行检查，我们使用频率范围为 31.5 Hz、精度为 2 dB 的噪声电平计 PCE-MSL（制造商：PCE Instruments，USA）测量了噪声。

In our case, the focus of the camera is configured in such a way that after focusing the window for monitoring is 5 cm by 5 cm in size, which, given the speed of the mosquito, will give the program in the Raspberry Pi3 time to identify it. When the sensor is triggered, the program usually has 0.1 s to detect the mosquito before it has time to leave the camera’s viewing area. To implement the protection of a specific area against mosquitoes, Fig. 3b shows a diagram. The analog signal from the audio sensor is used to control the focus of the camera. It should be noted that today there are cameras capable of reading a car number for 1 km, so the distance for control largely depends on the resolution of the camera. The experimental design is shown in Fig. 3.
在我们的例子中，相机的焦距配置为对焦后用于监控的窗口为 5 厘米 x 5 厘米，考虑到蚊子的速度，这将使 Raspberry Pi3 中的程序有时间识别它。当传感器被触发时，程序通常有 0.1 秒的时间来检测蚊子，然后蚊子才有时间离开摄像机的观察区域。为了实现对特定区域的蚊子保护，图 13b 显示了一个图表。来自音频传感器的模拟信号用于控制摄像机的焦距。应该注意的是，今天有能够读取 1 公里车号的相机，因此控制距离在很大程度上取决于相机的分辨率。实验设计如图 1 所示。3.

Fig. 3 图 3

a Scheme of sensor action: 1—three boxes with mosquitoes, 2—laser machine with the telephoto lens, 3—servomotor for changing the position on the x-axis, 4—the distance between the camera and the boxes changed from 0.5 to 1.5 m, and 5—laser, 15 W, 450-nm wavelength; b schematic representation of the control area, where radius it is the length of a control area
a 传感器动作方案：1—三个带蚊子的盒子，2—带长焦镜头的激光机，3—用于改变 x 轴位置的伺服电机，4—相机与盒子之间的距离从 0.5 m 变为 1.5 m，以及 5—激光，15 W，450 nm 波长;B 控制区域的示意图，其中 radius 是控制区域的长度

In the experiment, the system at the expense of a camera with the servomotor monitors mosquitoes in three boxes. When the audio sensor is triggered, the camera focuses on a predetermined area due to the telephoto lens. The audio sensor, when triggered, transmits an analog signal to the esp8266 which via Wi-Fi transmits a signal to a Raspberry Pi3. In the next step, Raspberry Pi3 sends the coordinates of the mosquito to the galvanometer and sends a signal for turning on the laser. The servomotor changes the position along the Z-axis—360°. This can allow one device to control a fully specified radius, Fig. 3b.
在实验中，该系统以牺牲带有伺服电机的摄像头为代价，在三个盒子中监控蚊子。当音频传感器触发时，由于长焦镜头，相机会聚焦在预定区域。触发时，音频传感器将模拟信号传输到 esp8266，esp8266 通过 Wi-Fi 将信号传输到 Raspberry Pi3。下一步，Raspberry Pi3 将蚊子的坐标发送到检流计，并发送打开激光器的信号。伺服电动机沿 Z 轴 - 360° 改变位置。这可以允许一个设备控制完全指定的半径，图 3。3b.

For the experimental scheme, a mathematical expression is derived that determines the dependence of the number of sensors on the area control:
对于实验方案，推导出了一个数学表达式，用于确定传感器数量对区域控制的依赖性：

$$y={\text{e}}^{0.3906+0.1707x}.$$

(6)

Average approximation error for this formula is 13.4%.We measured 115 times for each box and calculated the mean error. This dependence is shown in Fig. 4.
此公式的平均近似误差为 13.4%。我们对每个盒子测量了 115 次并计算了平均误差。这种依赖性如图 1 所示。4.

Fig. 4 图 4

Dependence of the number of audio sensors on the radius of the object to be monitored
音频传感器数量与待监控对象半径的相关性

Due to the small size of the mosquito and the slight expansion of the image, the Haar cascades cannot identify the mosquito by exact features, such as wings and legs, therefore, the cascades determine only the general contours of the mosquito and the background. Due to this reason, we used OpenCV to make a bright background with filter—segmentation by cv2.kmeans. The algorithm of the developed system present in Fig. 5
由于蚊子的体型小且图像略有扩展，Haar 级联无法通过确切的特征（例如翅膀和腿）来识别蚊子，因此，级联仅确定蚊子的一般轮廓和背景。由于这个原因，我们使用 OpenCV 制作带有滤镜的明亮背景——通过 cv2.kmeans 进行分割。图 1 中所示的开发系统的算法。5

Fig. 5 图 5

Mosquito detection algorithm
蚊子检测算法

Figure 6 explains the principle of operation of the mirror galvanometer.
图 6 解释了镜面检流计的工作原理。

Fig. 6 图 6

Galvanometer working principle. a Photo of the galvanometer, b principle of operation
振镜工作原理。a 检流计照片，b 工作原理

The speed of the galvanometer is 20 kpps, that is why this element is the fastest in the device, it has enough milliseconds to hit the target. To control the galvanometer, we use Raspberry Pi3. Raspberry Pi3 receives the image from the camera and after that calculates the position of the mosquito. After that, the protocol computer sends 22 bits signal to the MCP3553 by SPI where the electronic signal processing board (Fig. 7, position 8) converts this signal in voltage from − 12 to + 12 V. The DC motor uses this signal to control the position of the mirrors, and therefore, the direction of the laser. Figure 7 shows the developed prototype device, principle of operation and his dimension.
检流计的速度为 20 kpps，这就是为什么这个元件是设备中最快的，它有足够的毫秒来击中目标。为了控制检流计，我们使用 Raspberry Pi3。Raspberry Pi3 从相机接收图像，然后计算蚊子的位置。之后，协议计算机通过 SPI 向MCP3553发送 22 位信号，其中电子信号处理板（图 D）。7，位置 8）将此信号的电压从 − 12 V 转换为 + 12 V。直流电机使用此信号来控制反射镜的位置，从而控制激光的方向。图 7 显示了开发的原型设备、工作原理和他的尺寸。

Fig. 7 图 7

The developed device: a photo of the device, b structural scheme, c device dimensions in mm: 1—PI cameras, 2—galvanometer, 3—Raspberry Pi, 4—laser rangefinder for checking the distance, 5—laser device, 6—power supply, 7—galvanometer driver boards, 8—analog conversion boards, 9—object detection, 10—laser beam
开发的设备：设备照片，b 结构方案，c 设备尺寸（毫米）：1 — PI 相机，2 — 振镜，3 — Raspberry Pi，4 — 用于检查距离的激光测距仪，5 — 激光设备，6 — 电源，7 — 振镜驱动板，8 — 模拟转换板，9 — 物体检测，10 — 激光束

Figure 8 demonstrates mosquito detection process.
图 8 展示了蚊子检测过程。

Fig. 8 图 8

Mosquito detection moment
蚊虫检测时刻

Figure 9 shows the complete installation of the equipment.
图 9 显示了设备的完整安装。

Fig. 9 图 9

Installation during the experiment: 1—telephoto lens, 2—laser setup, 3—microcontroller esp8266, 4—servo motor to change the position along the axis-z, 5—three sections with mosquitoes and audio sensors
实验期间的安装：1 - 长焦镜头，2 - 激光设置，3 - 微控制器 ESP8266,4 - 伺服电机，用于沿 z 轴改变位置，5 - 带蚊子和音频传感器的三个部分

It should be noted that the number of boxes with mosquitoes does not affect the project’s efficiency. Significant factors are audio sensors, the accuracy of which directly determines the performance of the project.
需要注意的是，装有蚊子的箱子数量不会影响项目的效率。重要的因素是音频传感器，其准确性直接决定了项目的性能。

4 Experimental results
4 实验结果

We did 150 attempts for each experiment (0.5 m, 1 m, 1.5 m), and Tables 1, 2, and 3 present the arithmetic mean for each case. Tracking is successful is positive when the camera detects the mosquito and tracks it with success. Detected by the camera—when the mosquito was in sight of the camera and was detected. Detected by the audio sensor—when the mosquito was closer than 10 cm away from the audio sensor and was detected. Neutralized by laser—when mosquito was neutralized. The state of the mosquito was determined visually. The delay between attempts was 5 s. The box contained one mosquito each—to exclude the option with the accidental destruction of the mosquito. Each time when mosquito was neutralized, the experiment was stopped and continued after the mosquito was placed in the box. The experiment took place in the daylight, lux—150 lx. External light on the camera was excluded and external noise sources were not simulated. Mosquitoes are not classified by gender. Type—Common mosquito (Latin Culex pipiens). The mosquitoes did not exceed 5 mm in size. The readings are averaged; the average is taken from 150 readings with different samples from the array, when the distance between the camera and the box with a mosquito was 0.5 m, 1 m, and 1.5 m.
我们对每个实验 （0.5 m、1 m、1.5 m） 进行了 150 次尝试，表 1 、 2 和 3 显示了每种情况的算术平均值。当相机检测到蚊子并成功跟踪时，跟踪成功是积极的。由摄像机检测到 - 当蚊子在摄像机的视线范围内并被检测到时。由音频传感器检测到 - 当蚊子距离音频传感器不到 10 厘米并被检测到时。被激光中和——当蚊子被中和时。蚊子的状态是通过目测确定的。尝试之间的延迟为 5 秒。盒子里各有一只蚊子——排除了意外破坏蚊子的选项。每次中和蚊子后，停止实验，将蚊子放入盒子后继续实验。实验在日光下进行，勒克斯为 150 lx。相机上的外部光线被排除在外，外部噪声源没有被模拟。蚊子不按性别分类。类型 - 普通蚊子（拉丁语 Culex pipiens）。蚊子的大小不超过 5 毫米。读数是平均的;平均值取自阵列中不同样本的 150 个读数，当时相机与装有蚊子的盒子之间的距离为 0.5 m、1 m 和 1.5 m。

Table 1 Results of the research when the distance between the camera and the box with a mosquito was 0.5 m
表 1 相机与装有蚊子的盒子之间的距离为 0.5 m 时的研究结果

Table 2 Results of the research when the distance between the camera and the box with a mosquito was 1 m
表 2 当相机与装有蚊子的盒子之间的距离为 1 m 时的研究结果

Table 3 Results of the research when the distance between the camera and the box with a mosquito was 1.5 m
表 3 相机与装有蚊子的盒子之间的距离为 1.5 m 时的研究结果

The relationship between the increase in distance between the tracking system and the mosquito was not noticed, which allows us to conclude that neural networks can be used for remote neutralization of mosquitoes. The camera identification of mosquitoes has high rates, which proves our suggestion about the possibility of using this method to neutralize mosquitoes. At the same time, the work has not yet succeeded in attaining high indices when precisely operating a laser, which relates to the technical execution of the project. There are difficulties with focusing the beam of the laser, because of which, even with the correct guidance to the mosquito of the laser, about 50% of the energy of laser passes by due to the small size of the mosquito.
跟踪系统与蚊子之间距离的增加之间的关系没有被注意到，这使我们能够得出结论，神经网络可用于远程中和蚊子。蚊子的相机识别率很高，这证明了我们关于使用这种方法中和蚊子的可能性的建议。同时，在精确操作激光器时，这项工作尚未成功获得高指标，这与项目的技术执行有关。聚焦激光束存在困难，因此，即使正确引导激光的蚊子，由于蚊子的体积小，大约 50% 的激光能量也会通过。

5 Deep learning for mosquito detect
5 用于蚊子检测的深度学习

It is needed to improve concentration on the size of the laser beam on the mosquito. To test the feasibility of using deep neural networks on a stationary computer, the Mask R-CNN network was used. The Mask R-CNN network mask is one of the most advanced and highly efficient networks in the classification and tracking tasks. The model was trained following the recommendations of the author’s Mask R-CNN [29], the initial weights were used from the developers, who are publicly available at https://github.com/facebookresearch/Detectron. Accurate determination of the contour of the mosquito will allow us to aim the laser more accurately—to hit exactly at the center of the mosquito. In addition, we will be able to shoot at the wings and make it impossible for mosquitoes to fly, and this will allow using lower laser power.
需要提高对蚊子身上激光束大小的注意力。为了测试在固定计算机上使用深度神经网络的可行性，使用了 Mask R-CNN 网络。Mask R-CNN 网络掩码是分类和跟踪任务中最先进、最高效的网络之一。该模型按照作者的 Mask R-CNN [29] 的建议进行训练，初始权重来自开发人员，这些开发人员在 https://github.com/facebookresearch/Detectron 上公开提供。准确确定蚊子的轮廓将使我们能够更准确地将激光对准蚊子的中心。此外，我们将能够射击翅膀，使蚊子无法飞行，这将允许使用较低的激光功率。

For training, 400 positive images of mosquitoes and 1000 negative examples were used (for negative examples, other insects and plants were used). Positive results were obtained in tracking problems when expanding from 50,000 pixels on an area of not more than 25 cm². In view of the small area, it is extremely inefficient in terms of mosquito control. The model was trained following the recommendations of the author’s Musk R-CNN [30], the initial weights were used from the developers, who are publicly available at https://github.com/facebookresearch/Detectron. Accurate determination of the contour of the mosquito will allow us to aim the laser more accurately—to hit exactly at the center of the mosquito. In addition, we will be able to shoot at the wings and make it impossible for mosquitoes to fly, and this will allow using lower laser power. For example, if we take the mosquito position above the center. The center of an area with a mosquito can be determined in real-time using an OpenCV—search for color contrast. The model calculation error is calculated using pixels. The model subtracts pixels from the image that came to the input of the model from the pixels in the image that came out at the output (as on cycle GAN), as a result, after that, it counts how many pixels are classified incorrectly. Mask R-CNN develops the Faster R-CNN architecture by adding one more branch that predicts the position of the mask covering the found object, and thus solves the instance segmentation problem The mask is just a rectangular matrix, in which 1 at some position means that the corresponding pixel to an object of the specified class, 0—that the pixel does not belong to the object. In our case, for Mask R-CNN, only one object was used for tracking—a mosquito, so the neural network quite simply found solutions for determining the contours of an object, focusing on the color edge contrast this reason in spite of so small amount image for training we received results with high accuracy. Finally, Mask R-CNN showed an accuracy of recognition of 90%, Fig. 10.
为了进行训练，使用了 400 张蚊子的正面图像和 1000 个负面示例（对于负面示例，使用了其他昆虫和植物）。当在不超过 25 cm² 的区域上从 50,000 像素扩展时，在跟踪问题上获得了积极的结果。鉴于面积小，在蚊虫控制方面效率极低。该模型按照作者的 Musk R-CNN [30] 的建议进行训练，初始权重来自开发人员，这些开发人员在 https://github.com/facebookresearch/Detectron 上公开可用。准确确定蚊子的轮廓将使我们能够更准确地将激光对准蚊子的中心。此外，我们将能够射击翅膀，使蚊子无法飞行，这将允许使用较低的激光功率。例如，如果我们采取中心上方的蚊子位置。可以使用 OpenCV 实时确定蚊子区域的中心，搜索颜色对比度。模型计算误差使用像素计算。该模型从输出处出来的图像中减去来自模型输入的图像中的像素（如在循环 GAN 上），因此，在此之后，它会计算有多少像素被错误分类。掩码 R-CNN 通过添加一个分支来开发更快的 R-CNN 体系结构，该分支预测覆盖找到对象的掩码的位置，从而解决了实例分割问题掩码只是一个矩形矩阵，其中某个位置的 1 表示与指定类的对象对应的像素，0 表示该像素不属于该对象。 在我们的例子中，对于 Mask R-CNN，只有一个对象用于跟踪——蚊子，因此神经网络非常简单地找到了确定对象轮廓的解决方案，专注于颜色边缘对比度，尽管用于训练的图像量如此之小，但我们收到了高精度的结果。最后，Mask R-CNN 显示识别准确率为 90%，图 3。10.

Fig. 10 图 10

Mosquito marking by Mask R-CNN: a labeling with VGG image annotator, b the result of detection by Mask R-CNN
使用 Mask R-CNN 进行蚊子标记：a 使用 VGG 图像注释器进行标记，b 使用 Mask R-CNN 进行检测的结果

In tracking tasks in the recorded video with a mosquito with the same background, the accuracy was comparable with the results obtained on the Haar cascades, Fig. 11.
在用具有相同背景的蚊子跟踪录制视频中的任务时，准确性与在 Haar 级联上获得的结果相当，图 3。11.

Fig. 11 图 11

Mosquito detection process: a labeling with VGG image annotator, b the result of detection by Mask R-CNN
蚊子检测过程：a 使用 VGG 图像注释器进行标记，b 通过 Mask R-CNN 检测结果

The results of using deep neural networks can be significantly improved using more images to train the model.
使用更多图像来训练模型可以显著改善使用深度神经网络的结果。

6 Discussion 6 讨论

The accuracy of laser effectivity is not more than 10%. However, the developed installation within 1 s can make 10 shots. As a result, this installation can destroy one mosquito per second with a probability close to 100%. The improvement of this result is possible due to the use of more accurate equipment.
激光有效性的准确率不超过 10%。但是，在 1 秒内开发的装置可以拍摄 10 张照片。因此，这种装置每秒可以消灭一只蚊子，概率接近 100%。由于使用了更精确的设备，这一结果是可能的。

We proposed an innovative method for remote monitoring of mosquitoes using a camera, where sound sensors were used to determine the initial position of mosquitoes.
我们提出了一种使用摄像头远程监测蚊子的创新方法，其中使用声音传感器来确定蚊子的初始位置。

The article conducted theoretical studies aimed at monitoring mosquitoes at a remote distance. As a result, mathematical expressions were obtained that determine the position of the mosquito when it is located at a remote distance from the camera. A formula is derived by which the necessary number of sound sensors are calculated to protect certain territories from mosquitoes. The main problem with mosquito monitoring is its small size. At a meter, a mosquito for the camera appears as a few pixels, which makes it difficult to extract any useful characteristics from the image. In this study, we concluded that to identify a mosquito in an image with an area of 100 cm², it is enough to use an image with 24,025 pixels. These settings will allow you to process images at a speed of 20 frames per second.
该文章进行了旨在远程监测蚊子的理论研究。结果，获得了数学表达式，当蚊子位于距离相机较远的位置时，它决定了蚊子的位置。得出一个公式，通过该公式计算出保护某些地区免受蚊子侵害所需的声音传感器数量。蚊子监测的主要问题是体积小。在一米处，相机的蚊子显示为几个像素，这使得很难从图像中提取任何有用的特征。在这项研究中，我们得出结论，要在面积为 100 cm² 的图像中识别蚊子，使用 24,025 像素的图像就足够了。这些设置将允许您以每秒 20 帧的速度处理图像。

We presented mathematical modeling of mosquito flight that allows us to predict the flight mathematically. From the moment, a mosquito’s position is detected by the camera and the computer calculates its position, the mosquito’s position on the image changes within 1 cm. This change is critical for the galvanometer. However, the mathematical modeling of mosquito flight allows the galvanometer to direct the laser beam in advance to the point where a mosquito will be.
我们提出了蚊子飞行的数学模型，使我们能够从数学上预测飞行。从相机检测到蚊子位置的那一刻起，计算机计算出它的位置，蚊子在图像上的位置变化了 1 厘米以内。这种变化对于检流计至关重要。然而，蚊子飞行的数学建模允许检流计提前将激光束对准蚊子所在的位置。

We solved problems with remote monitoring, for this, we adjusted the focus of the lens, and its position in the x, y, and z-axes. Which allowed us to monitor the mosquito’s position, regardless of the distance between the object and mosquitoes.
我们解决了远程监控的问题，为此，我们调整了镜头的焦距及其在 x、y 和 z 轴上的位置。这使我们能够监测蚊子的位置，而不管物体和蚊子之间的距离如何。

This algorithm managed to unite such components as mechanics, optics, mathematical modeling, and machine vision. From the point of view of autonomous systems, this study can be used in various fields, such as monitoring different remote objects.
该算法设法将力学、光学、数学建模和机器视觉等组件联合起来。从自主系统的角度来看，这项研究可以用于各个领域，例如监控不同的远程对象。

The article conducted theoretical studies aimed at monitoring a mosquito at a remote distance. As a result, mathematical expressions were obtained determining the position of the mosquito when it is located at a remote distance from the camera. A formula is derived by which the necessary number of sound sensors are calculated to protect certain territories from mosquitoes.
这篇文章进行了旨在远程监测蚊子的理论研究。结果，获得了数学表达式，当蚊子位于距离相机较远的位置时，它的位置。得出一个公式，通过该公式计算出保护某些地区免受蚊子侵害所需的声音传感器数量。

As an alternative to using sound sensors to identify the approximate location of mosquitoes, continuous monitoring of a given area can be used, Fig. 12a. The room is divided into sectors, where the camera monitors the box with x- and y-axes, then the focus of the camera goes deep into the room along the z-axis (1,2,3 panels, Fig.12a) and continues scanning along the x- and y-axes. At the same time, to attest to the depth of the mosquito’s location along the z-axis of the mosquito, we can use the Gauss filter, in Fig. 12b, c.
作为使用声音传感器识别蚊子大致位置的替代方法，可以使用对给定区域进行连续监测，图 3。12个。房间被划分为多个区域，摄像头用 x 轴和 y 轴监控盒子，然后摄像头的焦点沿着 z 轴深入房间（1,2,3 个面板，图 D）。12a） 并继续沿 x 轴和 y 轴扫描。同时，为了证明蚊子沿蚊子 z 轴的位置深度，我们可以使用高斯滤波器，如图 1 所示。12b， c.

Fig. 12 图 12

Room-monitoring scheme, b the mosquito is focused, and c mosquito is out of focus 
房间监控方案，b 蚊子聚焦，c 蚊子失焦

A promising direction in the development of the developed installation is the use of STMicroelectronics STM32 microcontrollers. It is given that X-CUBE-AI has an expansion pack for STM32CubeMX. This extension can work with various deep-learning environments such as Keras, TensorFlow, Caffe, and ConvNetJs. Thanks to this, the neural network can be trained on a desktop computer with the possibility of computing on the GPU. After integration, the optimized library for the 32-bit STM32 microcontroller is used. Machine learning is planned to be implemented on the edge computing of the STM32 microcontroller. This will reduce the cost of the device and reduce the size of the device. At the same time, peripheral calculations will reduce the speed of image processing, and in our case, this is the most time-consuming procedure. Therefore, by reducing the image processing time and increasing the processing power of the processor for working with deep neural networks, we can achieve results in which dozens of mosquitoes will be destroyed within 1 s.
开发装置的一个有前途的方向是使用 STMicroelectronics STM32 微控制器。鉴于 X-CUBE-AI 有一个 STM32CubeMX 的扩展包。此扩展可与各种深度学习环境配合使用，例如 Keras、TensorFlow、Caffe 和 ConvNetJs。多亏了这一点，神经网络可以在台式计算机上进行训练，并可以在 GPU 上进行计算。集成后，使用针对 32 位 STM32 微控制器的优化库。计划在 STM32 微控制器的边缘计算上实现机器学习。这将降低设备的成本并减小设备的尺寸。同时，外围计算会降低图像处理的速度，在我们的例子中，这是最耗时的过程。因此，通过减少图像处理时间和提高处理器处理深度神经网络的处理能力，我们可以实现在 1 秒内消灭数十只蚊子的结果。

All lasers from 1 mW start to pose a hazard to the eyes. In the future, we will consider the possibility of using a device with tracking a person in the laser field of view for safety tasks.
所有 1 mW 以上的激光都开始对眼睛造成危害。将来，我们将考虑使用在激光视野中跟踪人员的设备进行安全任务的可能性。

7 Conclusions 7 结论

The article proved that modern advances in machine vision and machine learning are enough to use a powerful laser to neutralize mosquitoes, which, given the scales of poverty, caused by transmission diseases, mosquito bites, is a promising direction for the protection of certain areas from the presence of mosquitoes. The results of this work can be significantly improved when using another type of equipment since the accuracy depends on the technical characteristics of the camera, servo motor, and sound sensors.
文章证明，机器视觉和机器学习的现代进步足以使用强大的激光来中和蚊子，鉴于传播疾病、蚊子叮咬造成的贫困规模，这是保护某些地区免受蚊子存在的一个有前途的方向。使用其他类型的设备时，这项工作的结果可以得到显着改善，因为精度取决于相机、伺服电机和声音传感器的技术特性。