ИСТИНА

Pre- and post-harvest monitoring of fruit crops using AI-assisted hyperspectral imaging Alexei Solovchenko Faculty of Biology, Lomonosov Moscow State University, 119234 Moscow, Russia, e-mail: solovchenkoae@my.msu.ru For any fruit producer and eventually for consumer, maintaining the highest quality of fruit produce at all stages of the production chain, from farm to table is the top priority. This goal calls for the development of affordable and efficient, preferably non-invasive techniques for monitoring the quality of the fruit during pre- and postharvest period. Apart from sorting out damaged fruit, it is important to monitor the fruit size, ripeness, coloration, possible damages. This is essential to choose a correct harvest window and pick the fruit at a similar ripening stages to simplify their postharvest handling and extend their shelf life. Hyperspectral reflectance imaging is an emerging method for rapid non-invasive quantitative screening of plant traits. This method is essential for accelerated breeding of crop plants as well as for precision agriculture practices. However, extraction of sensible information from reflectance images is hindered by the complexity of plant optical properties, especially when they are measured in the field. Rapid advances in computer vision over the past decade have resulted in unprecedented increase in precision and overall performance of machine learning (ML) models, now rivaling that of human experts for certain problems. On the other end of the scale reside “classic” methods. They are grounded in fundamental biophysical properties of plants and aimed at uncovering relationships between the physiological state and its manifestations through noninvasively observable variables. Of particular note is spectroscopy, the approach spanning at least several decades of plant research. During that time, a wealth of knowledge about pigments, plant interactions with light, and non-destructive phenology assessment has been accumulated. Specific examples include reflectance-based monitoring changes in pigment composition induced by ripening, environmental stresses, and phytopathogen attacks. One of the most simple, yet powerful techniques emerging from this development is vegetation indices (VI), transforming quantification of incident radiation into highly biologically relevant parameters (see e.g. [1]). Reflectance indices such as Plant Senescence Reflectance Index, PSRI or Anthocyanin Reflectance Index, ARI previously developed for remote sensing of vegetation and point-based reflectometers to infer the spatially resolved information on plant development and biochemical composition using ripening apple fruit as the model [1]. Specifically, the proposed approach enables capturing data on distribution of chlorophylls and primary carotenoids as well as secondary carotenoids (both linked with fruit ripening and leaf senescence during plant development) as well as the information on spatial distribution of anthocyanins (known as stress pigments) over the plant surface (Fig. 1). The choice of optimal method of fruit imaging and efficient image processing method is still a subject of debate. Here, we have dissected the information content of hyperspectral images focusing on either spectral component, spatial component, or both. We have employed random forest (RF) classifiers using different parameters as inputs: reflectance spectra, vegetation indices (VIs), and spatial texture descriptors (local binary patterns, or LBP), comparing their performance in the task of damage detection in apple fruit. The amount of information in raw hypercubes was found to be over an order of magnitude excessive for the end-to-end problem of classification. Converting spectra to vegetation indices has resulted in a 60-fold compression with no significant loss of information relevant for phenotyping and more robust performance with respect to varying illumination conditions. Accordingly, the advanced machine learning approaches could be more efficient if complemented by spectral information about the objects in question (Fig. 2). The abovementioned techniques are essentially complemented by methods for fruit detection in images of apple trees in industrial orchards (Fig. 3). One of the recent developments in this field is based on the use of a two-stage neural network designed for the localization of objects. The neural network generates a segmentation mask and a bounding box for each fruit found on the input image. This information can then be used to count the number of fruits in the images and to construct yield estimate. It is shown that by training the neural network with the Adam optimizer and a smaller learning rate, the quality of fruit detection of all sizes, and especially of small fruits, can be improved [3]. Fig. 3. Examples on the images of apple trees from industrial orchards. On the left — a fragment, in the middle — a reference markup, on the right — the proposed method. (According to [3]) The fruit detection methods can be used for targeting further ML-based data processing methods precisely at regions of interest representing fruits. Thus, the information content of hyperspectral images can be dissected for non-invasive assessment of fruit health focusing either on spectral component, spatial component, or both [2]. To this end, popular “classic” machine learning approaches: random forest (RF) and support vector classifiers (SVC) can be employed and compared their end-to-end performance on a relatively large hyperspectral image dataset as a measure of the information retention after dimensionality reduction was performed. This approach can be extended by combining vegetation indices (VI) and spatial texture descriptors (local binary patterns, LBP) [2]. It became obvious that even the advanced machine learning approaches could be more efficient if they are complemented by spectral information about the objects in question. As a result, a balanced approach to obtaining the image data for computer vision-based fruit grading seems to be most productive and cost-effective. For the implementation of such an approach, capturing images at a few carefully selected spectral channels would be sufficient, although one should be aware of the limitations outlined above. A knowledge-based dimensionality reduction would drastically shorten the adoption time for the newest “mainstream” deep learning architectures to spectral proximal sensing, removing the need to compromise on the learning of spatial features. Powerful but costly and relatively slow hyperspectral sensors will occupy the R&D niche for the development of novel and improved non-invasive methods of the assessment of fruit quality. To conclude, the processing of hyperspectral reflectance images of fruit both in pre- and post-harvest with the VI developed for remote sensing of vegetation can potentially yield a plethora of information useful for automated non-invasive phenotyping of fruit crop plants, crop sizing and quality control. This information would be a welcome complement to currently widespread methods of conventional lab methods, morphological analysis of plant images with machine learning algorithms enriching their results with quantitative information on plant physiological condition and biochemical composition. This approach would find a broad application also in precision agriculture e.g. in gauging fruit quality and ripening conveying more valuable to the consumers worldwide. References [1] Solovchenko A.E., Shurygin B.M., Kuzin A.I., Solovchenko O.V., Krylov A.S. Extraction of Quantitative Information from Hyperspectral Reflectance Images for Noninvasive Plant Phenotyping. Russian Journal of Plant Physiology. 2022. V. 69. 144. DOI: 10.1134/S1021443722601148 [2] Boris Shurygin, Igor Smirnov, Andrey Chilikin, Dmitry Khort, Alexey Kutyrev, Svetlana Zhukovskaya, Alexei Solovchenko. Mutual augmentation of spectral sensing and machine learning for non-invasive detection of apple fruit damages. MDPI Horticulturae. 2022. 8(12), 1111; https://doi.org/10.3390/horticulturae8121111. [3] D. A. Nesterov, B. M. Shurygin, A. E. Solovchenko, A. S. Krylov, D. V. Sorokin. A CNN-based method for fruit detection in apple tree images. Computational Mathematics and Modeling, Vol. 33, No. 3, July, 2022