Non-Invasive Data Extraction from Machine Display Units Using Video Analytics

Artificial Intelligence (AI) has the potential to transform manufacturing by improving shop floor processes such as production, maintenance and quality. However, industrial datasets are notoriously difficult to extract in a real-time, streaming fashion thus, negating potential AI benefits. The main example is some specialized industrial controllers that are operated by custom software which complicates the process of connecting them to an Information Technology (IT) based data acquisition network. Security concerns may also limit direct physical access to these controllers for data acquisition. To connect the Operational Technology (OT) data stored in these controllers to an AI application in a secure, reliable and available way, we propose a novel Industrial IoT (IIoT) solution in this paper. In this solution, we demonstrate how video cameras can be installed in a factory shop floor to continuously obtain images of the controller HMIs. We propose image pre-processing to segment the HMI into regions of streaming data and regions of fixed meta-data. We then evaluate the performance of multiple Optical Character Recognition (OCR) technologies such as Tesseract and Google vision to recognize the streaming data and test it for typical factory HMIs and realistic lighting conditions. Finally, we use the meta-data to match the OCR output with the temporal, domain-dependent context of the data to improve the accuracy of the output. Our IIoT solution enables reliable and efficient data extraction which will improve the performance of subsequent AI applications.

Experimental Simulation Set-Up for Validating Out-Of-The-Loop Mitigation when Monitoring High Levels of Automation in Air Traffic Control

An increasing degree of automation in air traffic will also change the role of the air traffic controller (ATCO). ATCOs will fulfill significantly more monitoring tasks compared to today. However, this rather passive role may lead to Out-Of-The-Loop (OOTL) effects comprising vigilance decrement and less situation awareness. The project MINIMA (Mitigating Negative Impacts of Monitoring high levels of Automation) has conceived a system to control and mitigate such OOTL phenomena. In order to demonstrate the MINIMA concept, an experimental simulation set-up has been designed. This set-up consists of two parts: 1) a Task Environment (TE) comprising a Terminal Maneuvering Area (TMA) simulator as well as 2) a Vigilance and Attention Controller (VAC) based on neurophysiological data recording such as electroencephalography (EEG) and eye-tracking devices. The current vigilance level and the attention focus of the controller are measured during the ATCO’s active work in front of the human machine interface (HMI). The derived vigilance level and attention trigger adaptive automation functionalities in the TE to avoid OOTL effects. This paper describes the full-scale experimental set-up and the component development work towards it. Hence, it encompasses a pre-test whose results influenced the development of the VAC as well as the functionalities of the final TE and the two VAC’s sub-components.

Study on Liquid Nitrogen Gravity Circulation Loop for Cryopumps in Large Space Simulator

Gravity circulation loop for the cryopumps of the space simulator is introduced, and two phase mathematic model of flow heat transfer is analyzed as well. Based on this model, the liquid nitrogen (LN2) gravity circulation loop including its equipment and layout is designed and has served as LN2 feeding system for cryopumps in one large space simulator. With the help of control software and human machine interface, this system can be operated flexibly, simply, and automatically under four conditions. When running this system, the results show that the cryopumps can be cooled down and maintained under the required temperature, 120 K.

Determination of the Concentrated State Using Multiple EEG Channels

Analysis of EEG brainwave provides information on mental or emotional states. One of the particular states that can have various applications in human machine interface (HMI) is concentration. 8-channel EEG signals were measured and analyzed. The concentration index was compared during resting and concentrating periods. Among eight channels, locations the frontal lobe (Fp1 and Fp2) showed a clear increase of the concentration index during concentration regardless of subjects. The rest six channels produced conflicting observations depending on subjects. At this time, it is not clear whether individual difference or how to concentrate made these results for the rest six channels. Nevertheless, it is expected that Fp1 and Fp2 are promising locations for extracting control signal for HMI applications.

Control Signal from EOG Analysis and Its Application

A game using electro-oculography (EOG) as control signal was introduced in this study. Various EOG signals are generated by eye movements. Even though EOG is a quite complex type of signal, distinct and separable EOG signals could be classified from horizontal and vertical, left and right eye movements. Proper signal processing was incorporated since EOG signal has very small amplitude in the order of micro volts and contains noises influenced by external conditions. Locations of the electrodes were set to be above and below as well as left and right positions of the eyes. Four control signals of up, down, left and right were generated. A microcontroller processed signals in order to simulate a DDR game. A LCD display showed arrows falling down with four different head directions. This game may be used as eye exercise for visual concentration and acuity. Our proposed EOG control signal can be utilized in many other applications of human machine interfaces such as wheelchair, computer keyboard and home automation.

Analytical Model Based Evaluation of Human Machine Interfaces Using Cognitive Modeling

Cognitive models allow predicting some aspects of utility and usability of human machine interfaces (HMI), and simulating the interaction with these interfaces. The action of predicting is based on a task analysis, which investigates what a user is required to do in terms of actions and cognitive processes to achieve a task. Task analysis facilitates the understanding of the system-s functionalities. Cognitive models are part of the analytical approaches, that do not associate the users during the development process of the interface. This article presents a study about the evaluation of a human machine interaction with a contextual assistant-s interface using ACTR and GOMS cognitive models. The present work shows how these techniques may be applied in the evaluation of HMI, design and research by emphasizing firstly the task analysis and secondly the time execution of the task. In order to validate and support our results, an experimental study of user performance is conducted at the DOMUS laboratory, during the interaction with the contextual assistant-s interface. The results of our models show that the GOMS and ACT-R models give good and excellent predictions respectively of users performance at the task level, as well as the object level. Therefore, the simulated results are very close to the results obtained in the experimental study.

Knowledge Representation and Retrieval in Design Project Memory

Knowledge sharing in general and the contextual access to knowledge in particular, still represent a key challenge in the knowledge management framework. Researchers on semantic web and human machine interface study techniques to enhance this access. For instance, in semantic web, the information retrieval is based on domain ontology. In human machine interface, keeping track of user's activity provides some elements of the context that can guide the access to information. We suggest an approach based on these two key guidelines, whilst avoiding some of their weaknesses. The approach permits a representation of both the context and the design rationale of a project for an efficient access to knowledge. In fact, the method consists of an information retrieval environment that, in the one hand, can infer knowledge, modeled as a semantic network, and on the other hand, is based on the context and the objectives of a specific activity (the design). The environment we defined can also be used to gather similar project elements in order to build classifications of tasks, problems, arguments, etc. produced in a company. These classifications can show the evolution of design strategies in the company.