A Multispectral-Oriented Semantic Computing System applied to Agriculture and Ocean Analysis

Jinmika Wijitdechakul, Student ID: 81749249

Graduate School of Media and Governance, Keio University Doctoral student, Cyber Informatics (CI) Program

Abstract: This research presents a multispectral-oriented semantic computing system for environmental changes and detection with a new method to integrate two different semantic spaces for realizing deep semantic interpretation. Three significant approaches are introduced for realizing (1) a multispectral image space for semantic computing analysis, (2) dictionary creation for deep environment interpretation (coral reef analysis), and (3) space integration between multispectral image (colour) space and physical parameters (sea-water and land quality) space applied to agriculture and ocean analysis. We clarify the feasibility and effectiveness of our method and system by showing several experimental results on agriculture and ocean condition interpretation. Keywords: Semantic Computing, Multispectral Image Analysis, Agriculture Analysis, Ocean Analysis, SDGs Implementation

I. RESEARCH OVERVIEW

This study presents a new environmental-semantic computing system for automated agriculture and ocean analysis using the intelligent sensing system which includes natural and physical sensors. To realize this method, 4 main approaches can be identified: 1) To create semantic computing system which can be supported multi-disciplinary data for global environmental analysis, 2) To integrate various heterogeneous local semantic spaces for deeper and wider interpretation system, 3) To create “the multispectral semantic-image space” which is created for dynamically computing semantic equivalence, similarity and difference between multispectrum images and environmental situations, and 4) To create new highly-abstract queries for supporting global environmental semantic memory system. As one of the analyzing functions, our system applies a semantic associative computing based on a Mathematical Model of Meaning(MMM) and multidimensional space to create “Global Environment Analysis” for analysing and interpreting environmental phenomena and changes occurring in the physical world. Moreover, This research also applied the Sensing, Processing, and Actuation (SPA) concept for realizing the actual implementation. The discussion of this research covers the key technologies to realize our system. Multispectral semantic-image space and water/land quality space were integrated to interpret the agricultural sustainability and ocean health contexts which is the new method to implement the SDGs from local to global aspect. The results demonstrated that our proposed method can utilize the advanced technology to interpret context-dependent agriculture/ocean knowledge sharply.

Fig. 1 Research overview

II. RESEARCH BACKGROUND

Multispectral imaging is becoming an increasingly useful tool in the environmental research field, which have been developed for various applications in environmental monitoring to detect changes area, remote monitoring or ground-based surveys. Usually, this powerful technique is commonly used for the earth observation by using satellite. The sensing data or satellite image is defined as the collection of reflected, emitted, or backscattered energy from an object or area in multiple bands of spectrum. Not only remote sensing analysis but multispectral imaging system also widely used for a landscape survey, and demonstrated for agricultural applications by capturing from small airborne or drone.

In order to monitor environmental phenomena, it is common to use a satellite to capture sensing data acquired by a multispectral sensor in ground-based surveys or earth observations. In general, the multispectral sensor is able to collect the reflectance, emittance or backscattered energy from an object or area in multiple bands of spectrum that is usefulness for discriminating among land-cover classes or substances (e.g., water, crop, forestry, landslides, urban areas, etc.) Though the multispectral sensor or satellite data are usually used in research fields, to expand the application of multispectral sensor, several researchers have been developing a small sensor or camera to capture the data by using airborne or UAV. Based on the background of multispectral imaging technologies is motivated this research to apply useful techniques between multispectral imaging and semantic computing for analyzing environmental phenomena and applied for agriculture and ocean.

This research picked up the environmental problem in terms of agricultural sustainability and ocean health that becoming a critical issue in our society which are related to the sustainable development goals (SDGs). Environmental sustainability in agriculture means good stewardship of the natural systems and resources that farms rely on. Among other things, this involves: building and maintaining healthy

soil, managing water wisely, minimizing air, water, and climate pollution, and promoting biodiversity. While, the ocean health play a central and positive role in ecosystem and human life, including through the climate system. Damage to the ocean is highly effect to human beings and nature in widespread. Therefore, this research is concerned this two environmental issues by utilizing the performance of semantic computing for detecting, monitoring, and interpreting environmental phenomena and changes occurring in the real world. III. PROBLEM STATEMENT We proposed a multispectral-oriented semantic computing system to interpret agricultural sustainability and ocean health by integrating various heterogeneous local semantic spaces for deep analysis. There is knowledge gap to construct the global environmental interpretation system that can integrate multi discipline data. The knowledge gap and technical gap of agriculture and ocean analysis lead us to formulate our research question and statement. Research Question

1. How to utilize the multispectral-oriented semantic computing system for agriculture and ocean analysis?

2. How to build a new platform? for support highly-abstract queries and retrieval for Global Environment which can be decomposed to local environmental phenomena.

3. How to create heterogeneous semantic space integration? that able to support various multi discipline data.

4. How to utilize nature as a sensing for Global environmental analysis? 5. How to create new methodology for implementing sustainable development goals (SDGs)?

Expected Result 1. Semantic Computing: Semantic space integration modelling to support multi-disciplinary data for

global environmental analysis in agriculture and ocean context. 2. Function: Interpretation and retrieval function for highly-abstract queries such as global warming,

SDGs, agricultural sustainability, ocean acidification and so on. 3. Methodology: Agricultural sustainability and Ocean health analysis for implementing sustainable

development goals (SDGs). 4. Actuation: From Local analysis to Global Action.

Research Limitation 1. We cover current implementation of agricultural sustainability and ocean health only in

environmental aspect. 2. Addressing only for marine-experts, agricultural-experts and non-experts user. 3. The analysis was specially made for multispectral image data (5 filter color) 4. Agricultural sustainability contexts covered by this study are: Land quality and Crop quality. 5. Ocean health contexts covered by this study are: recreation activities and ecosystem.

IV. RELATED WORKS

1. The Mathematical Model of Meaning (MMM)

The Mathematical Model of Meaning (MMM) is applied as the semantic associative computing method to realize dynamic context recognition that can be used for obtaining multimedia information by giving the user’s impression and the contents of the information as context. A semantic space is created as a space for representing various contexts which correspond to its subspaces. The essential of this space is that the method to provide dynamic context recognition as the context-dependent interpretation is realized by dynamically selecting a certain subspace from the entire semantic space [6].

The main feature of MMM is to create space for memorizing and recalling various multimedia information (image, text, sound and video) that are mapped on semantic space and also performing the interpretation function by calculating the correlation of the retrieval semantic space.

2. Multispectral Semantic-Image Space

The most essential and significant point of “multispectral-semantic computing method” is that it realizes the interpretation of “substances (materials)” appearing and reflected in the multi-spectral image by using multispectral semantic-image space and semantic projection function [3]. In the environmental research fields, the computational tools of the multispectral image are widely used to analyze observed nature environment data and to explain environmental changes.

As a previous study, the concept of “multi-spectrum semantic-image space” has been proposed by Y. Kiyoki [3]. Multi-spectrum semantic-image space is a new semantic computing method to realize semantic associative search for the multi-colors spectrum images in the multidimensional semantic space. This space is created for dynamically computing semantic equivalence, similarity, and difference between multispectral image and environmental situations [3].

Multi-spectrum semantic-image space consists of 6 dimensions or 6 axes each of which corresponds to a single channel image filtered by color filter ((a) IR cut-off filtered axis, (b) Blue filtered axis, (c) Blue-Green filtered axis, (d) Green filtered axis, (e) Red filtered axis, and (f) Infra-Red pass filtered axis). In this space, an orthogonal semantic space is defined and each pixel which corresponds to the brightness in each filtered images is map onto space for computing semantic correlations between a pixel and environmental-objects semantic objects, according to various contexts (context image or context words) which are represented environment situations and phenomenon.

V. SOLUTION AND PROPOSED IDEA

Taking inspiration from my study in Global Environmental System Leaders (GESL) Program, I realized the current environmental issue that needs quick response and solution. This research utilize advance technologies to develop computational models to understand environmental phenomena. Start by utilizing natural sensing combined with multi-disciplinary data sources to construct the dimensional semantic computing system which represents a memory recall mechanism. This research proposed a new way to implement agricultural sustainability and ocean health in environmental aspect based on semantic computing techniques. The ultimate goal is that our system able to interpret and retrieve highly-abstract queries (eg. global warming, SDGs, agricultural sustainability, ocean acidification, etc.) which are these

queries can decomposed to local environmental phenomena. Three significant approaches are introduced in our research;

1. Multispectral Semantic-Image Space for Environmental Analysis An essential point of multispectral semantic-image space is that it realizes the interpretation of substance appearing and reflected in the multispectral image by using multi-dimensional multispectral semantic-image space and semantic projection function. In the area of environment research field especially agriculture and ocean analysis, several methods have been developed for analyzing and interpreting multispectral images to realize the environmental phenomena and changes. However, those methods presented the interpreting result as image or graph (visualization). This research introduces the method of multispectral semantic-image space by analyzing multimedia data and interpreting analyzed data to meaningful words.

This research proposes the concept of multi-spectrum semantic-image space for environmental analysis by defining 5 filter colors (dimensions). Each dimension corresponds to spectral data or brightness value in the corresponding pixel in each filtered images. The spectral data or pixel value can be mapped to the corresponding axis ((a) Red filtered axis, (b) Green filtered axis, (c) Blue filtered axis, (d) Near Infrared filtered axis, and (e) Red Edge filtered axis.

Fig. 2 semantic space for agriculture analysis Fig. 3 semantic space for ocean analysis

According to the agriculture and ocean analysis, we utilize the multispectral semantic-image

space to recognize the actual phenomena in various contexts by applying the subspace selection function to select the most highly correlated semantic elements and form a semantic projection for dynamic semantic interpretation. Figure 2 and 3 show semantic space for agriculture and ocean analysis.

2. Coral Health Dictionary Creation

Because the research question for our study looks to understand how to utilize nature as a sensing or bioindicator to recognize the actual phenomena. Based on knowledge integration on marine science and computer science, we utilize coral species to recognize the ocean phenomena. Each coral species are sensitive to various environmental factors which highly advantage to detect or monitor the ocean health in

many contexts. This research utilizes five coral species (Acropora, Pavona maldivensis, Yellow Gorgonian, Heliopora coerulea, and Lobophytum) that applicable to monitor ocean health in three contexts (Climate change, Pollution, and Recreation). We defined contexts based on threat information provided by the International Union for Conservation of Nature Red lists. The need of coral health dictionary is for supporting the dynamic ocean health interpretation. We applied the image processing technique to construct the coral health dictionary. Each words correspond to health level (Healthy, Risky, Bleaching, Mortality, and Dead) and spectral characteristics.

Table 1 Context setting Fig. 4 The most significant coral species in each area

3. Semantic Space Integration for Agriculture Sustainability and Ocean Health

Implementation For implementing the SDGs in environmental aspect, this research proposed a semantic space integration method (SSI) for integrating heterogeneous local semantic spaces that obtaining multi-discipline data related to environmental (agriculture science and marine science) and computer science fields. This method realizes the global environmental semantic memory system. The procedure for semantic space integration and semantic search consists of the following process: Process 1: Creation for individual matrics (original matrix creation) Process 2: Semantic Space Integration (SSI) for matrics (Space integration) Process 3: Orthogonal semantic space for MMM

Our research integrate multispectral image space and agriculture/ocean space for interpreting agricultural sustainability and ocean health context. This integrated system will support highly-abstract queries such as SDGs, Global Warming, Climate Change, which allows users to understand the actual local phenomena.

4. Current Implementation: Semantic Space Integration for Ocean Health Analysis (Okinawa case study)

This section discussed on one of our current implementation for ocean health analysis (marine recreation context). We show the feasibility and effectiveness of our proposed method to realize the ocean analysis by integrating multispectral semantic-image space and reef water quality space to recognize the marine recreation phenomena. In this implementation method, we applied the Sensing, Processing, and Actuation (SPA) concept which is highly effective to detect environmental phenomena as multispectral image and reef water quality data in physical space and mapped these multi-discipline data to cyber space to make an analytical and semantic computing, while actuation process aims to actuate the analytically computed results to real space by visualization for expressing environmental phenomena with casualties and influence. Sensing Process (S): In this phrase we collected multispectral image data (Lobophytum coral) and reef water data for monitoring marine recreation activities.

- Multispectral image data The dataset used in this study is multispectral image data captured by MAPIR multi-spectral

survey cameras which consist of five filter color. Dataset were acquired on May 5th, 2018 at Okinawa, Japan. Sensor calibration and image preprocessing were apply before the implementation.

Fig. 5 Lobophytum coral image from 9 locations in Okinawa Prefecture, Japan

- Reef water quality data

The reef water quality data consist of five parameters (temperature, pH, Salinity, and Suspended Solids) which are most highly significant for coral and ocean conditions. The following figure (Fig.6) shows reef water data in 9 locations.

Fig. 6 Reef water quality data

Processing process Step1: Determining stage of coral condition based on multispectral analysis To determine coral condition, semantic elements are selected and combined based on user context or analysis propose. This experiment focused on marine recreation context which highly related to three semantic elements; Near Infrared, Red, and Red Edge. We apply two environmental indices (Normalized Coral Index and Chlorophyll Index) to interpret coral health level. The following figure (Fig.7) show spectral (pixel) values extracted from multispectral image include five filter colors (red, green, blue, near-infrared, and red edge).

Fig. 7 Spectral value extracted from multispectral image

Step2: Semantic Space Integration for Ocean Health To interpret ocean health in marine recreation context, we applied semantic space integration between multispectral semantic-image space and reef water quality space. The following figure (Fig.8) show an integrated matrix consists of Normalized Coral Index, Chlorophyll Index, Temperature, pH, Salinity, and Suspended solids.

Fig.8 Integrated matric (multispectral image and reef water parameters)

Step 3: Interpretation As the experimental results of semantic analysis, we show two high marine recreation activity areas in West and East Ishigaki Island. By applying semantic analysis to determine ocean health, we are able to interpret marine recreation activities and understand the actual ocean conditions. From these results, the system answer to the abstract questions related to Global Environmental Analysis such as SDGs, Global Warming, Ocean Health, etc. and allow users to retrieve the local scale analysis for Global Action.

Fig. 9 Experimental results (Marine recreation context) Actuations Process (A) As the results of ocean health analysis system for the context of marine recreation activities, we visualized the analytical results and show the actual utilization of real-time image processing in regional sensing and analysis of nature, and mapping the sensing data onto 5D World Map System (Fig. 10 and 12)

Fig. 10 Experimental results visualized on 5D World Map System (Ishigaki Island, Taketomi Island, and Kohama Island)

In addition, we compared our analytic results (semantic computing) with the ground truth data provided by the Sango plafo (okinawa coral reefs information platform). Our system is able to show high accuracy interpreted results of marine recreation activities in the same location. The advantage of our method is that we can provide the quick response system for ocean health analysis rather than fundamental analysis method such as Manta-Tow Survey (snorkel diver) or Line Intercept Transect (LIT)method. The following figure shows the comparison of our analytical results on 5D World Map and Sango plafo (Fig 11 and 13)

(12-a) The result on 5DWM

(12-b) Coral cover survey information (Sango Plafo)

Fig. 11 The comparison of our analytical results on 5D World Map and Sango plafo (South Ishigaki Island)

Fig. 12 Experimental results visualized on 5D World Map System (Okinawa Island, Iejima Island, and Nago Island)

(13-a) The result on 5DWM

(13-b) Coral cover survey information (Sango Plafo)

Fig. 13 The comparison of our analytical results on 5D World Map and Sango plafo (Okinawa Island)

5. Conclusion This research presents a new environmental-semantic computing system for automated agriculture and ocean analysis using the intelligent sensing system which includes natural and physical sensors. To realize agriculture and ocean analysis system, we proposed new methods to transform environmental data into semantic words. The proposed methods include; (1) coral health dictionary creation, we utilized five coral species that applicable to monitor ocean phenomena and (2) semantic space integration, we integrated various heterogeneous local semantic spaces which support multi-disciplinary data and defined the semantic subspace selection (i.e., extraction of essential dimensions of data) to analyze the agricultural sustainability and ocean health, which reflects the actual agriculture and ocean conditions. As the result of research progress, we demonstrate that our proposed methods can realize the global environmental semantic memory system to interpret context-dependent agriculture/ocean knowledge sharply and create new highly-abstract queries (semantic words) that can be decomposed to local phenomena. Our findings also indicate that our proposed methods can be tools to implement sustainable development goals (SDGs) for agriculture (SDG 2.4) and ocean (SDG 14.1).

VI. RESEARCH PROGRESS

VII. PUBLICATION Journal papers

1. Wijitdechakul, J., Kiyoki, Y., Sasaki, S. and Koopipat, C., “A Multispectral Imaging and Semantic Computing System for Agricultural Monitoring and Analysis”, Information Modelling and Knowledge Bases, Vol.XXVIII(2017), IOS Press, pp.314-333.

2. Wijitdechakul, J., Kiyoki Y., Sasaki S., “An Application of Multispectral Semantic Image Space for Global Farming Analysis and Crop Condition Comparisons”, Information Modelling and Knowledge Bases, Vol.XXIX(2018), IOS Press, pp.176-187.

3. Wijitdechakul, J., Kiyoki, Y., and Koopipat, C., “An environmental semantic computing system of multispectral imagery for coral health monitoring and analysis”, Information Modelling and Knowledge Bases, Vol.XIX(2019), IOS Press

International Conference 1. Wijitdechakul, J., Kiyoki, Y., Sasaki, S. and Koopipat, C., “UAV-based Multispectral Image

Analysis System with Semantic Computing for Agricultural Health Conditions Monitoring and Real-time Management”, International Electronics Symposium (IES) 2016, IEEE, vol. 18, pp.463-468. (September 29-30, 2016 at Bali, Indonesia) BEST PAPER AWARD

2. Wijitdechakul, J., Kiyoki, Y., Sasaki, S. and Koopipat, C., “UAV-Based Multispectral Aerial Image Retrieval Using Spectral Feature and Semantic Computing,” 2017 International Electronics Symposium on Knowledge Creation and Intelligent Computing (IES-KCIC 2017), IEEE, vol. 1, pp.101-107. (September 26-27, 2017 at Surabaya, Indonesia)

VIII. BIBLIOGRAPHICAL REFERENCE

1. Wijitdechakul, J., Kiyoki, Y., Sasaki, S.and Koopipat, C. “A Multispectral Imaging and Semantic Computing System for Agricultural Monitoring and Analysis” Information Modelling and Knowledge Bases XXVIII(2017), pp.314-333.

2. Kiyoki, Y., Chen, X., Sasaki, S.and Koopipat, C. “A Globally-Integrated Environment Analysis and Visualization System with Multi-Spectral & Semantic Computing in Multi-Dimensional World Map,” Information Modelling and Knowledge Bases XXVIII (2017), pp.106-122.

3. Kiyoki, Y., Chen, X., Sasaki, S.and Koopipat, C. “Multi-Dimensional Semantic Computing with Spatial-Temporal and Semantic axis for Multi-spectrum Images in Environment Analysis,” Information Modelling and Knowledge Bases XXVII (2016), pp. 27-47.

4. McFeeters, S. K. “The use of Normalized Difference Water Index (NDWI) in the delineation of open water features,” International Journal of Remote Sensing (1996), pp.1425–1432.

5. Huete, A.R. “A soil-adjusted vegetation index (SAVI)” Remote Sensing of Environment(1998), vol. 25, pp. 259-309.

6. Kiyoki, Y., Kitagawa, T. and Hayama, T., “A metadatabase system for semantic image search by a mathematical model of meaning,” ACM SIGMOD Record(1994), Vol.23, No. 4, pp.34-41

7. Kiyoki, Y. and Kitagawa, T., “A semantic associative search method for knowledge acquisition,” Information Modelling and Knowledge Bases VI (1995), pp. 121-130.

8. Chen, X. and Kiyoki, Y., “A Visual and Semantic Image Retrieval Method Based on Similarity Computing with Query-Context Recognition,” Information Modelling and Knowledge Bases XVIII (2007), pp. 245-252.

9. M. W. J. Carper, T. M. Lillesand, and R. W. Kiefer, “The use of intensity-Hue-Saturation transformations for merging SPOT Panchromatic and multispectral image data,'' Photogrammetric Engineering & Remote Sensing, vol.56(4), pp. 459-467, 1990.

10. P. S. Chavez, S. C. Sides, and J. A. Anderson, “Comparison of three different methods to merge multiresolution and multispectral data: LANDSAT ™ and SPOT Panchromatic”, Photogrammetric Engineering & Remote Sensing, vol. 57(3), pp. 295-303, 1991.

11. P. W. Mausel, W. J. Kramber, and J. K. Lee, “Optimum band selection for supervised classification of multispectral data,'' Photogrammetric Engineering & Remote Sensing, vol. 56(1), pp. 55-60, 1990.

12. Y. Kiyoki, and et al., “An Environmental-Semantic Computing System for Coral-Analysis in Water-Quality and Multispectral Image Space with Multi-Dimensional World Map,” EJC International Conference on Information Modeling and Knowledge Bases, June 2017, pp.66-80.

13. Hou-Ping W., Ping L., Yu W., Ying-Wei H. and Cheng L. “Measurement for opto-electronic conversion functions (OECFs) of digital still-picture camera” SPIE Proceedings Vol. 7850 (2010)

14. A. ARDESHIR GOSHTASBY., “2-D and 3-D Image Registration: for Medical, Remote Sensing, and Industrial Applications” ISBN: 978-0-471-64954-0, 284 pages (Feb, 2005).

15. http://wwf.panda.org/about_our_earth/blue_planet/coasts/coral_reefs/coral_threats/ 16. Gleason, A. C. R., R. P. Reid, and K. J. Voss. "Automated classification of underwater

multispectral imagery for coral reef monitoring." OCEANS 2007. IEEE, 2007. Pp.1-8.

17. M. Soriano, S. Marcos, C. Saloma, “Image Classification of Coral Reef Components from Underwater Color video”, MTS/IEEE Oceans 2001. An Ocean Odyssey. Conference Proceedings, 2001, pp.1008-1013. (IEEE Cat. No.01CH37295)

18. Ragunathan (Raj) Rajkumar, Insup Lee, Lui Sha, John Stankovic, "Cyber-Physical Systems: The Next Computing Revolution", DAC '10 Proceedings of the 47th Design Automation Conference, ACM New York, NY, USA, 2010, Pages 731-736, 2010.

19. Shiori Sasaki, Yasushi Kiyoki, "Real-time Sensing, Processing and Actuating Functions of 5D World Map System: A Collaborative Knowledge Sharing System for Environmental Analysis", IOS press,Series Frontiers in Artificial Intelligence and Applications, Volume 280: Information Modelling and Knowledge Bases XXVII, pp. 220 - 239, DOI 10.3233/978-1-61499-611-8-220, 2016.

20. http://www.teachoceanscience.net/teaching_resources/education_modules/coral_reefs_and_climate_change/how_do_we_monitor_coral_reefs/

21. Meng Ji-hua, Wu Bing-fang, “A Study on the crop condition monitoring methods with remote sensing”, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Vol.XXXVll. Part B8. Beijing 2008, pp.945-990

22. Joshua Kelsey, Arko Lucieer, “Sensor Correction of 6-Band Multispectral Imaging Sensor for UAV Remote Sensing” Remote Sens. 2012,4,1462-1493; doi:10.3390/rs4051462

23. Dunford, R.; Michel, K.; Gagnage, M.; Piegay, H.; Tremolo, M.L Potential and constraints of Unmanned Aerial Vehicle technology for the characterization of Mediterranean riparian forest. Int.J. Remote Sees. 2009, 30, 4915-4935

24. Pastor, E.; Lopez, J.; Royo, P. UAV payload and mission control hardware/software architecture. IEEE Aerosp. Electron. Syst. Mag. 2007, 3-8

25. Glenn, E.P.; Huete, A.R.; Nagler, P.L.; Nelson, S.G. Relationship between remotely-sensed vegetation indices, canopy attributes and plant physiological process: What vegetation indices can and cannot tell us about the landscape. Sensors 2008,8,2136-2160

26. Lacava, T.; Brocca, L.; Calice, G.; Melone, F.; Moramarco, T.; Pergola, N.; Tramutoli, V. Soil moisture variations monitoring by AMSU-based soil wetness indices: A long-term inter-comparison with ground measurements. Remote Sees. Environ. 2010, 114, 2317-2325.