关于这个项目 (About this project)

This project is part of the Algorithms for Massive Data course organized by the University of Milan, that I recently had the chance to attend. The task is to develop the Deep Learning model able to recognize eye diseases, from eye-fundus images using the TensorFlow library. An important requirement is to make the training process scalable, so create a data pipeline able to handle massive amounts of data points. In this article, I summarize my findings on convolutional neural networks and methods of building efficient data pipelines using the Tensorflow dataset object. Entire code with reproducible experiments is available on my Github repository: https://github.com/GrzegorzMeller/AlgorithmsForMassiveData

该项目是我最近有幸参加的由米兰大学组织的“海量数据算法”课程的一部分。 任务是开发使用TensorFlow库从眼底图像识别眼睛疾病的深度学习模型。 一个重要的要求是使培训过程具有可扩展性，因此创建一个能够处理大量数据点的数据管道。 在本文中，我总结了有关卷积神经网络和使用Tensorflow数据集对象构建有效数据管道的方法的发现。 我的Github存储库中提供了具有可重复实验的整个代码： https ： //github.com/GrzegorzMeller/AlgorithmsForMassiveData

介绍 (Introduction)

Early ocular disease detection is an economic and effective way to prevent blindness caused by diabetes, glaucoma, cataract, age-related macular degeneration (AMD), and many other diseases. According to World Health Organization (WHO) at present, at least 2.2 billion people around the world have vision impairments, of whom at least 1 billion have a vision impairment that could have been prevented[1]. Rapid and automatic detection of diseases is critical and urgent in reducing the ophthalmologist’s workload and prevents vision damage of patients. Computer vision and deep learning can automatically detect ocular diseases after providing high-quality medical eye fundus images. In this article, I show different experiments and approaches towards building an advanced classification model using convolutional neural networks written using the TensorFlow library.

早期眼病检测是预防由糖尿病，青光眼，白内障，年龄相关性黄斑变性(AMD)和许多其他疾病引起的失明的经济有效方法。 根据世界卫生组织(WHO)的目前，全世界至少有22亿人有视力障碍，其中至少有10亿人本来可以预防[1]。 快速和自动检测疾病对于减轻眼科医生的工作量并防止患者视力损害至关重要。 提供高质量的医学眼底图像后，计算机视觉和深度学习可以自动检测眼部疾病。 在本文中，我展示了使用使用TensorFlow库编写的卷积神经网络构建高级分类模型的不同实验和方法。

数据集 (Dataset)

Ocular Disease Intelligent Recognition (ODIR) is a structured ophthalmic database of 5,000 patients with age, color fundus photographs from left and right eyes, and doctors’ diagnostic keywords from doctors. This dataset is meant to represent the ‘‘real-life’’ set of patient information collected by Shanggong Medical Technology Co., Ltd. from different hospitals/medical centers in China. In these institutions, fundus images are captured by various cameras in the market, such as Canon, Zeiss, and Kowa, resulting in varied image resolutions. Annotations were labeled by trained human readers with quality control management[2]. They classify patients into eight labels including normal (N), diabetes (D), glaucoma (G), cataract (C), AMD (A), hypertension (H), myopia (M), and other diseases/abnormalities (O).

眼病智能识别(ODIR)是一个结构化的眼科数据库，包含5,000名年龄的患者，左眼和右眼的彩色眼底照片以及医生的医生诊断关键字。 该数据集旨在代表由上工医疗技术有限公司从中国不同医院/医疗中心收集的“真实”患者信息集。 在这些机构中，眼底图像由市场上的各种相机(例如佳能，蔡司和Kowa)捕获，从而产生不同的图像分辨率。 注释由经过培训的人类读者进行质量控制管理来标记[2]。 他们将患者分为八个标签，包括正常(N)，糖尿病(D)，青光眼(G)，白内障(C)，AMD(A)，高血压(H)，近视(M)和其他疾病/异常(O) 。

After preliminary data exploration I found the following main challenges of the ODIR dataset:

经过初步的数据探索，我发现了ODIR数据集的以下主要挑战：

· Highly unbalanced data. Most images are classified as normal (1140 examples), while specific diseases like for example hypertension have only 100 occurrences in the dataset.

·高度不平衡的数据。 大多数图像被归类为正常图像(1140个示例)，而特定疾病(例如高血压)在数据集中仅出现100次。

· The dataset contains multi-label diseases because each eye can have not only one single disease but also a combination of many.

·数据集包含多标签疾病，因为每只眼睛不仅可以患有一种疾病，而且可以患有多种疾病。

· Images labeled as “other diseases/abnormalities” (O) contain images associated to more than 10 different diseases stretching the variability to a greater extent.

·标记为“其他疾病/异常”(O)的图像包含与10多种不同疾病相关的图像，这些图像在更大程度上扩展了变异性。

· Very big and different image resolutions. Most images have sizes of around 2976x2976 or 2592x1728 pixels.

·非常大且不同的图像分辨率。 大多数图像的大小约为2976x2976或2592x1728像素。

All these issues take a significant toll on accuracy and other metrics.

所有这些问题都会对准确性和其他指标造成重大损失。

数据预处理 (Data Pre-Processing)

Firstly, al