3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft...

15
1 • Article • 3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU 1* Shuntao LIU 1 Deshuai KONG 1 Qichang HE 2 Xue WANG 1 1. Chengdu Aircraft Industrial (Group) Co., Ltd. , China 2. School of Mechanical Engineering, Shanghai JiaoTong University, Shanghai, China * Corresponding author, [email protected] Supported by Sichuan Civil-Military Inosculation Special Fund (ZYF -2017-66). Abstract Aiming at the lack of closed-loop feedback and optimization enabling tools in aircraft automatic spraying system at present, 3D virtual-real mapping technique, namely digital twin, of the automatic spraying system is studied systematically in this paper. With the sensors installed in the spraying system, the spraying working parameters are collected on-line and are used to drive the three-dimension virtual spraying system to realize the total-factor monitoring of the spraying operation. Furthermore, the Operation Evaluation Model is applied to the analysis and management of the key indexes of spraying quality. That is, once the data value of the key indexes is over the threshold, the operation will be optimized automatically. The results of a case study show that the above approach can well support the high-efficiency analysis, evaluation and optimization of the spraying operation process. Keywords Digital Twin; Aircraft; Automatic Spraying; Virtual Reality; Virtual Environment 1 Introduction Surface spraying or painting, this’s the last step in the manufacture of aircraft, is one of the most time-consuming parts of modern aircraft manufacturing. As part of the special process, the spraying process requires high skill for operators. On the one hand, the spraying operators are supposed to master certain knowledge of spraying through training and practical operations. On the other hand, excellent practical ability is also required. Manual spraying which is unstable in spraying quality, is also harmful to

Transcript of 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft...

Page 1: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

1

• Article •

3D Virtual-Real Mapping of aircraft automatic spray

operation and online simulation monitoring

Shiguang QIU1*,Shuntao LIU1,Deshuai KONG1,Qichang HE2,Xue WANG1

1. Chengdu Aircraft Industrial (Group) Co., Ltd. , China

2. School of Mechanical Engineering, Shanghai JiaoTong University, Shanghai, China

* Corresponding author, [email protected]

Supported by Sichuan Civil-Military Inosculation Special Fund (ZYF -2017-66).

Abstract Aiming at the lack of closed-loop feedback and optimization enabling tools in aircraft automatic

spraying system at present, 3D virtual-real mapping technique, namely digital twin, of the automatic

spraying system is studied systematically in this paper. With the sensors installed in the spraying system,

the spraying working parameters are collected on-line and are used to drive the three-dimension virtual

spraying system to realize the total-factor monitoring of the spraying operation. Furthermore, the Operation

Evaluation Model is applied to the analysis and management of the key indexes of spraying quality. That is,

once the data value of the key indexes is over the threshold, the operation will be optimized automatically.

The results of a case study show that the above approach can well support the high-efficiency analysis,

evaluation and optimization of the spraying operation process.

Keywords Digital Twin; Aircraft; Automatic Spraying; Virtual Reality; Virtual Environment

1 Introduction

Surface spraying or painting, this’s the last step in the manufacture of aircraft, is one of the most

time-consuming parts of modern aircraft manufacturing. As part of the special process, the spraying

process requires high skill for operators. On the one hand, the spraying operators are supposed to master

certain knowledge of spraying through training and practical operations. On the other hand, excellent

practical ability is also required. Manual spraying which is unstable in spraying quality, is also harmful to

Page 2: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

2

people’s health. While, surface spraying by robots has the unique advantages in spraying efficiency, quality

consistency, safety and environmental protection, and it has potential in broader market and future

development [1]. The size of an aircraft is usually far beyond the working space of common industrial

robots. Therefore, they need to be specially designed, modified or integrated, which has high technical

complexity. The robotic spraying system has been introduced into the aviation industry and has initially

realized the surface painting of the whole machine by robots. The automatic surface painting process of the

aircraft surface involves many processes, and the reasonable arrangement of each processes can effectively

improve the automation level of the spraying system. Dongjing Miao et al. [2] studied the key technologies

related to the spraying operation planning, such as aircraft pose calibration and spray gun trajectory

planning. Based on the secondary development technology, the operation planning platform in the CATIA

environment was developed.

Due to the large size and complex shape of the aircraft, multi-robot cooperation is required in the automatic

surface spraying system. Besides, the spraying process parameters are complicated and dynamically change

with the working space, time and environment, thus, it is a typical and complex automated operating

system which has high requirements for spraying process and robot-collaboration. At present, the research

in this field focuses on the development of automatic spraying systems and the planning of spraying

operations. It has not yet involved the implementation of closed-loop feedback optimization in the field of

automatic spraying systems. The digital twin technology, also called virtual-real mapping technology, can

link the physical world with the virtual model to realize online monitoring, simulation analysis and

automatic optimization of the production process [3]. Grieves M et al. describe the digital twin concept and

its development, show how it applies across the product lifecycle [4]. Fei T et al. presents a new method for

product design based digital twin method and proposes a framework of digital twin-driven product design

[5]. Arne Bilberg et al. discusses an object-oriented event-driven simulation as a digital twin of a flexible

assembly cell coordinated with a robot to perform assembly tasks alongside human [6]. Tao F et al. a novel

concept of digital twin workshop was proposed. to solve the communication and interaction between the

physical world and the virtual world of manufacturing [7]. Lehmann C et al. presents guidelines for the

implementation of the digital twin in production systems [8].

This paper systematically studies the 3D digital twin modeling technology of the aircraft automatic

spraying system to realize the total-factor, full-view 3D monitoring of the spraying operation process, and

Page 3: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

3

comprehensively support the efficient analysis and evaluation optimization of the automatic spraying

operation.

2 Overall schemes

The automatic spraying system for aircraft mainly consists of three IRB-5500 robots and three sets of

three-degree-of-freedom motion platform. Before spraying operation, the whole outer surface of the aircraft

is divided into several surface blocks according to the size of the robot workspace, each surface block size

needs to be smaller than robot workspace. Then, each moving platform position is designed to ensure that

the spraying range of the robot at each designated station should cover the corresponding surface block

completely. Finally, each robot is delivered to the designated position by the motion platform in advance,

and then each robot starts spraying operation.

The overall scheme of the digital twin technology for the automatic surface spraying system of the aircraft

is shown in Figure 1. Firstly, based on the process flow, the whole spraying process is simulated with all

related factors under consideration in the virtual environment to verify the rationality of the robot's motion

path. It means to avoid the interference between robots and aircraft, robots and surrounding shop floor

environments, as well as robots and robots to prevent major accidents during the spraying process. After

the simulation verification, the operation planning data is transmitted to the on-site industrial computer to

drive the robot to perform the spraying operation. During the spraying operation, the spraying system

parameters are collected by sensors installed in the spraying system, including the spraying state dataset ,

the robots’ motion parameter set, the process parameter set, and the working environment dataset, and all

these dataset are transmitted in real time to the virtual monitoring system. By this means, the whole

operation process is mapped to the virtual environment, so as to realize total-factor monitoring of the

spraying operation. The key indicators of the spraying quality are monitored online through the operation

evaluation model. Once the value of indicators exceeds the set threshold, based on the spraying operation

optimization model, the system will retrieve the process parameter knowledge base to find the optimal

process parameters, and adjust the process parameters of the on-site operation through control commands.

If there is no suitable process parameter or the value of the indicator still exceeds the threshold value after

adjustment, the spraying is stopped by the control command and an alarm is issued to notify the technician

to perform on-site processing. After the spraying is completed, if the spraying quality of a local area is still

not good enough, all process parameter information, environment (temperature, humidity) and spraying

characteristics (thickness) can be traced through the spraying process. All this information is used to

comprehensively analyze and re-adjust the process parameters. The optimal process parameters are then

Page 4: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

4

stored in knowledge database to achieve closed-loop optimization, which fully support the construction of

spray optimization model under complex multi-factors.

Figure 1 The overall scheme of the aircraft automatic spraying digital twin system

In summary, the key technologies of the digital twin for aircraft automatic spraying system mainly include

the virtual-real mapping technology in the spraying operation process, the online evaluation and

optimization of the spraying operation process, and the multi-robot co-simulation in the large complex

spraying environment.

3 Key technology

3.1 Virtual-real mapping of spraying operation process

3.1.1 Total-factor modeling of spraying operation

Seven data sets included in the total-factor information of the spraying operation are listed in Table 1. The

spraying operation process is monitored online to ensure the quality of the spraying, and at the same time

support the virtual restoration of the real situation of the spraying operation to optimize the spraying

process. The specific meanings of each parameter are as follows.

Table 1 Total-factor information of the spraying operation

Page 5: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

5

Type Symbol Description Specific Parameters

Quality parameter set Q

Characterize quality-related variables such

as spray thickness, consistency, etc.

ΔHa(t),σHa ,ΔFHa,

ΔHb(t),σHb ,ΔFHb

Production parameter

set

Rate

Characterize production-related variables

such as spraying schedule, spraying area,

spraying ratio, etc.

S,Sa,Sb,PctS,Ta,Tb,PctT,

Sp,Spa,Spb,PctSp

Process parameter set OPrt,P

Including spraying objects and spraying

process parameters which are important

variables that affect the quality of spraying.

Type,Da,Db,α,v,f,Pa,

Pb,W,PrtNo,PrtCur,PrtMat

Device parameter set K,ODEV

Information that is closely related to the

spray equipment itself, such as motion

information of spray system, operating status

signals, etc.

RbtA,RbtB,RbtC,SigRbtA,

SigRbtB,SigRbtC

Environmental

parameter set

E

The environmental variables that affect the

quality of the spray and the selection of the

spray process parameters.

Te,Hr

For spraying process parameter set P= {Type, Da, Db, α, v, f, Pa, Pb, W}, this data set is the key

parameter to optimize the spraying process and improve the spraying quality, where:

Type denotes the type of painting, due to differences in composition, viscosity, etc., the process

parameters of automatic spraying with different paint are also quite different. Da denotes the spraying

distance, α denotes the angle between the spray direction and the aircraft skin’s normal. The calculation

method of Da and α is shown in Section 3.2.1. Db denotes the lap distance, v denotes the spraying speed, f

denotes the paint flow rate, and Pa denotes the atomization pressure. Pb represents the fan pressure; W

represents the spray width, W = C * λ, where λ is determined by Da, Pa, Pb.

For spraying motion parameter set K= {RbtA, RbtB, RbtC}, this data set is the basic motion data for

realizing the synchronous mapping from the real robot to the virtual robot, where:

RbtA represents the motion parameter of the robot A, RbtA={Pos , R, Ctrl}, Pos denotes the position of

the robot, Pos = { x, y, z }, R denotes the joint posture of the robot, R = {R1, R2, R3, R4, R5, R6, R7}.

Page 6: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

6

For spraying environment parameter set E= {Te,Hr }, this data set is the environmental variable that

affects the spray quality, which will affect the selection of the spray process parameters. In the data set, Te

represents the temperature and Hr represents the humidity in the shop floor.

For spraying object parameter set OPrt= {PrtNo, PrtCur, PrtMat }, this parameter set is the attribute

data of the sprayed object, where:

PrtCur represents the curvature of the sprayed skin, and PrtMat represents the material, including

composite, metal and so on.

For spraying system operating state set ODEV= {SigRbtA, SigRbtB, SigRbtC, ...}, this parameter set is

the running status of the spraying system, such as running, pausing, breakdown, etc., where: SigRbtA,

SigRbtB, SigRbtC is the running state of the robot, as well as other status information.

For spraying progress parameter set Rate = {S, Sa, Sb, PctS, Ta, Tb, PctT, Sp, Spa, Spb, PctSp}, this

parameter set characterizes the spraying progress, where:

S represents the planned total spraying area; Sa denotes the total area that has been sprayed, which is

calculated in real time according to the robot's motion trajectory and the range of spray gun; Sb denotes the

remaining spray area, Sb = S-Sa; PctS denotes the ratio of the sprayed area to the total spraying area, PctS =

(Sa / S) * 100%; Ta denotes the spraying time, Tb denotes the estimated spraying time left, Tb = Sb / v;

PctT denotes the ratio of the spraying time to the total time, PctT = (Ta / (Ta + Tb)) * 100%, PctT helps to

understand the spraying progress from time dimension; Sp represents the spraying area of the current

sprayed skin, automatically calculated during spraying planning; Spa denotes the area that has been sprayed

of the current skin. When spraying a new skin, spraying time t is recorded, Spa = W * t * v; Spb denotes the

remaining spraying area of the current skin, Spb = Sp-Spa; PctSp denotes the ratio of spraying area to total

area of the current skin, PctSp = (Spa / SP) * 100%. The spraying process of the current skin can be

intuitively understood from PctSp.

The meaning of the spraying quality parameter set is described in detail in 3.2.1.

3.1.2 Data-driven modeling of the spray robot

The process of the data-driven modeling of the spray robot mainly involves three levels: (1) establishing

the parent-child relationship between joints of the robot model; (2) collecting and analyzing the spraying

data; (3) driving the robot according to the data. Figure 2 illustrated the process of data-driven modeling of

the robot. First, an ABB IRB-5500 spray robot is modeled as a 6-joints tandem robot with three

independent auxiliary axis. In order to model the kinematic chain of the end joint of the robot arm, a

Page 7: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

7

parent-child relationship is created by connecting adjacent joint arms, whereby the transformation applied

to the parent object can be simultaneously transferred to the child object, and the motion of the child object

will not affect the parent object.

Figure 2 Process of the data-driven modeling of spray robot

After that, the data of the spraying process is collected by the OPC UA [9] standard protocol and the data

file is generated after processing. The spraying operation data includes system running time, coordinate

XYZ of the three-degree-of-freedom platform, and angles of the seven joints of the robot R1-R6. The state

of the spray includes the opening and closing of the spray gun and the process parameters. The coordinates

and the angles are absolute values, that is, the coordinates relative to the zero position of the platform, and

the rotation angles relative to the initial positions of joints of the robot. The directly collected data files

need to be analyzed through a customized interface. Then corresponding data in the file is extracted and

sorted into a single format instruction according to time. Each instruction includes the coordinate

information of the platform, angle information of the robot joints, and status information of the spraying

process at the current time.

Finally, the real scene of the aircraft is mapped into the virtual environment. There is a certain difference

between the position of the aircraft in the real scene and that in the virtual environment, as shown in the

following figure. The position transformation relationship Mb between the planning environment and the

actual environment can be found through calibration. The position of the aircraft in the actual scene can be

mapped to the virtual environment by setting the position matrix Mb*Ma1. The coordinate information and

Page 8: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

8

angle information in the extracted format instruction are in one-to-one correspondence with the objects in

the established three independent auxiliary axis model and the robot model. According to the system

running time, the corresponding motion instruction is called, and then the position of the auxiliary axis and

the corresponding rotation angle of each joint of the spray robot are controlled according to the position

information and the angle information in the instruction, thereby realizing the data driving of the painting

robot.

Figure 3 Calibration of spray robot in the virtual scene

3.1.3 Spraying visualization technology

In the real process of spraying, the paint is sprayed out by the spray gun mounted on the robot to form a

mist cone, which can be regarded as consisting of small particles. Each particle has the following

characteristics: (1) It has a certain life cycle, starting from the spray gun, disappearing after collision with

the aircraft skin, and ending the life cycle; (2) It has its own motion state, having a certain emission angle,

initial velocity and acceleration after spraying; (3) The motion is generally linear, regardless of the

rotational motion of the droplet itself; (4) It may collide with the surrounding environment; (5) It has a

certain appearance state.

In order to realize the visualization of the spray in a three-dimensional virtual environment, a particle

system is used to simulate the mist cone. The particle system is mainly used to simulate the generation and

display problem of a large number of tiny substances moving or changing according to certain rules on a

computer [10]. Each particle in the particle system has its own set of properties, such as the life cycle,

Page 9: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

9

velocity and acceleration, color, position, etc., which are updated over time. The emission shape of the

particle system is set to a conical shape, and the emission position is set at the nozzle of the spray gun. The

radius and cone angle of the emission cone are set at the same time. The life cycle of particles needs to be

adjusted according to the spraying distance and the running speed of the particles. If the life cycle of

particles is too long, it will take up a lot of memory and GPU. If it is too short, it will not be able to collide

with the fuselage to complete the spraying. The particles can only disappear after colliding with the

fuselage. By setting the mist cone parameters, the spray visualization scene is shown in Figure 4.

Figure 4 Simulation result of the mist cone

The format instruction extracted from the spray data contains information of the spraying process status,

including the start and pause time of the spray, the color, the cone angle of the mist cone, and so on. In the

particle system, the on and off status of the particle emission, the color of the particle, and the radius and

angle of the cone in the conical emission shape can all be controlled. Relating the extracted spray state

information with the properties of the particle system in one-to-one correspondence, the data can be

effectively used to control the spraying process. In the 3D virtual environment, the mist cone consists of a

particle system, and through the collision of each particle with the fuselage model, the position of the

collision point is obtained by collision detection, and the color of the collision point is changed by using

vertex coloring method to realize the visualization of the spraying process of fuselage.

3.2 Online evaluation and optimization of spraying operation process

3.2.1 Online evaluation of spraying operation process

In the spraying operation, in order to ensure the quality of the spraying, it is necessary to evaluate the

quality of the spraying process efficiently. Traditional manual spraying relies mainly on operators’

experience to observe the painted surface, identify and solve problems in time. Therefore, in the automatic

Page 10: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

10

spraying process of robots, it is urgent to monitor the quality of spraying online to prevent large-scale

spraying accidents. Spray uniformity is a key indicator to characterize the quality of spray. It is proposed to

use online calculation of paint coating thickness to quantify the uniformity of spraying, and to construct an

online evaluation model of spray uniformity. The specific modeling process is as follows.

For the coating thickness Ha(t) of a skin at any time, it is calculated as follows:

Ha(t)=f*(t-t0)/ Spa (1)

Then, establishing a single skin uniformity online monitoring model based on eq.(1) :

ΔHa(t)=| Ha(t)-H |,ΔHa(t)≤K1

σHa (t)=STDEVP(Ha (t),Ha (t-N),Ha (t-2N),……,Ha (t0)),σHa (t) ≤K2

ΔFHa (t)=Fit (ΔHa(t),ΔHa(t-N),ΔHa(t-2N),……,ΔHa(t-MN)),|ΔFHa’(t) |≤K3

Where: ΔHa(t) is the deviation between the coating thickness and the design thickness H at any point in

time. The deviation should meet the design value K1, which is the basic requirement for spraying; σB(t)

is the fluctuation of the coating thickness, which characterizes the stability of the spray quality, σ

B(t)should be less than K2, K2 is obtained from a series of process experiments; ΔFHa (t) is a linear

fitting function of 0-t, and ΔFHa’(t) is the derivation of ΔFHa (t), ΔFHa’(t) characterizes the variation

trend of the spray thickness, theoretically ΔFHa’(t)=0, and actually |ΔFHa’(t) | should be less than K3

according to the process test. In the above model, N is the sampling frequency. In order to reduce the online

calculation of online fitting, according to the actual needs of the process, MN is generally 600s.

For all sprayed skins at any time, the thickness Hb (t) is calculated as follows:

Hb(t)=f*(t-t0)/ Sa (2)

Based on eq.(2) to establish a full-aircraft spray uniformity online monitoring model :

ΔHb (t)=| Hb (t)-H |,ΔHb (t)≤K1

σHb (t)=STDEVP(Hb (t),Hb (t-N),Hb (t-2N),……,Hb (t0)),σHb (t) ≤K2

ΔFHb (t)=Fit (ΔHb (t),ΔHb (t-N),ΔHb (t-2N),……,ΔHb (t-MN)),|ΔFHb’(t) |≤K3

The parameters of the full-aircraft spray uniformity online monitoring model have the same meanings

compared with the parameters of the single skin spray uniformity online monitoring model. The sampling

frequency is larger, and the MN is generally three times that of the single skin.

In addition to the uniformity of the spray as a quantitative indicator of the quality of the spray, it is

also necessary to monitor the key parameters of the spray quality online. Among them, the distance and

angle between the spray robot nozzle and the surface of the aircraft surface, closely affect the quality of the

Page 11: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

11

spray. In theory, the distance between the spray head and the fuselage should be consistent, and the angle is

perpendicular to the surface of the aircraft. In the virtual environment, detection is performed by means of

emitting radiation. The model of the robot nozzle is cylindrical. Starting from the center of the bottom, the

vertical line is perpendicular to the bottom surface, and the direction is outward, and this forms a ray.

During the spraying operation, regardless of the posture of the nozzle, the ray is always perpendicular to the

plane of the nozzle, intersecting the collision model of the fuselage. The aircraft collision model is

composed of a mesh of the simplified body. When the ray intersects the model, it is equivalent to

calculating the intersection of the straight line and the triangle. The distance between the collision point and

the ray’s base point is the real spraying distance Da. The true spray angleαis calculated by multiplying the

vector of the ray and the normal vector of collision point with the skin. By monitoring the value of Da and

αonline, the warning will timely alert when the threshold is exceeded.

3.2.2 Online optimization of spraying operation process

During the actual spraying operation, there are some factors that cannot be completely considered in the

planning stage, so the actual spraying process execution always has a gap with the ideal state. For example,

the planning of the spraying path t and the spraying parameters are all done under ideal conditions, but the

real aircraft will has deformations in different areas due to its own weight during the assembly process. In

addition, the shopfloor environment, the robot motion accuracy, the paint properties etc., will affect the

final spray quality. Therefore, it is necessary to comprehensively judge the abnormality of spraying through

the total-factor online monitoring model and quality online monitoring model. Then, the system

automatically analyzes and selects the process parameters in the spraying process knowledge base

according to the analyzed result and the characteristic attributes of the current spraying object, for example,

to adjust the injection distance, spray angle, paint flow rate, spray speed and other parameters for timely

online optimization. Online optimization of the spray operation process is as shown in the Figure 5.

Page 12: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

12

Figure 5 Online optimization of spraying operation process

3.3 Multi-robot efficient co-simulation in large scenes

In the actual spraying, the collision among the spray robot, the fuselage skin and the shop floor facilities

should be avoided. Since the entire virtual scene model consists of tens of millions of geometric patches, it

is impossible to perform multi-robot real-time co-simulation in software tools such as DELMIA [11]. A

simplified collision detection model [12] is established in Unity3D to replace the original object geometry

model for collision detection. Using the spray robot as an example, the robot consists of multi-joints, and

the suitable bounding box model is added to each joint of the robot. Figure 6 shows the bounding box

model added to the robot. As the aircraft model is much more complicated, if bounding box components for

collision detection is used, the collision detection will not be accurate enough, so the mesh body component

is added to the aircraft for collision detection.

Figure 6 Mesh model for collision detection

Then, a highly efficient collision detection method is needed. The octree segmentation method is used to

complete the segmentation of the collision model. From the root node, the nodes of the octree are traversed.

Page 13: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

13

If the nodes intersect, the traversal is continued. If they do not intersect, abandon the traversal of the subtree

to achieve real-time collision detection, and finally the collision result can be obtained [13]. The results

returned from the collision detection include the occurrence time of the collision, the location of the

collision point, the direction, etc. The system records the collected data and used for later analysis of the

spray process.

4 Development and application of system

The aircraft automatic spraying digital twin system is developed with the .NET framework, and the

underlying spraying process data is collected by the OPC UA standard protocol. The human-computer

interaction interface uses virtual reality technology to establish a 3D virtual scene, providing user friendly

operations such as rotation, positioning and scaling. And using the particle system to develop spraying

visualization model, the spraying process is real-time rendered according to the spraying process

parameters. The collision detection algorithm is used to detect the interference between the robot and the

aircraft parts in real time. The main functions of each module are shown in Figure 8.

Figure 7 The modules of the aircraft automatic spraying digital twin system

The virtual-real mapping system of aircraft automatic spraying uses WPF [14] to integrate the system

function modules, and uses the Unity3D engine to realize the 3D visualization. The system integrates

equipment management, human-computer interaction, visual display, logic calculation and spray process

Page 14: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

14

knowledge base module, and realizes timely communication between each module through message

mechanism. The platform visualization interface is shown in Figure 8.

Figure 8 The interface of the aircraft automatic spraying digital twin system

5 Conclusion

Aiming at the lack of closed-loop feedback optimization enabling tool for aircraft automatic spraying

system, a digital twin model of aircraft automatic spraying system is proposed. The key technology is

studied systematically. The full-factor information model and spray visualization model of aircraft

automatic spraying operation are constructed. The online virtual-real mapping was realized with OPC UA

protocol; The online evaluation and closed-loop optimization of the spraying process based on knowledge

engineering is realized. The collision detection method for tens of millions of virtual spraying particle in

virtual scenes is studied. The practical application verifies that the digital twin technology of the aircraft

automatic spraying system can greatly improve the planning efficiency and quality of the spraying

operation.

References

1 Wang G L, Wu D, Chen K. Current Status and Development Trend of Aviation Manufacturing Robot [J]. Aeronautical

Manufacturing Technology, 2015(10).

2 Miao D J, Wu L, Xu J, et al. Automatic spraying robot system for aircraft surfaces and spraying operation planning [J].

Journal of Jilin University (Engineering and Technology Edition), 2015, 45(2).

3 Fei T, Cheng J, Qi Q, et al. Digital twin-driven product design, manufacturing and service with big data[J]. International

Journal of Advanced Manufacturing Technology, 2018, 94(9-12):3563-3576.

Page 15: 3D Virtual-Real Mapping of aircraft automatic spray …...3D Virtual-Real Mapping of aircraft automatic spray operation and online simulation monitoring Shiguang QIU1*,Shuntao LIU1,Deshuai

15

4 Grieves M., Vickers J. Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems. In:

Kahlen FJ., Flumerfelt S., Alves A. (eds) Transdisciplinary Perspectives on Complex Systems. Springer, Cham. 2017.

5 Fei T, Sui F, Liu A, et al. Digital twin-driven product design framework[J]. International Journal of Production Research,

2018(1):1-19.

6 Arne Bilberg, Ali Ahmad Malik. Digital twin driven human–robot collaborative assembly[J]. CIRP Annals -

Manufacturing Technology,2019,68(1). Volume 68, Issue 1, 2019, Pages 499-502

7 Tao F, Zhang M, Cheng J, et al. Digital twin workshop: a new paradigm for future workshop[J]. Computer Integrated

Manufacturing Systems, 2017, 23(1): 1-9

8 Lehmann C. The digital twin: Realizing the cyber-physical production system for industry 4.0[J]. Procedia Cirp, 2017,

61:335-340.

9 Wan J, Chen B, Imran M, et al. Toward Dynamic Resources Management for IoT-Based Manufacturing[J]. IEEE

Communications Magazine, 2018, 56(2):52-59.

10 Messaoudi F, Simon G, Ksentini A. Dissecting games engines: the case of Unity3D[C]// International Workshop on

Network & Systems Support for Games. 2015.

11 Zhao L Z, Zhang Y H, Wu X H, et al. Virtual Assembly Simulation and Ergonomics Analysis for the Industrial

Manipulator Based on DELMIA[M]// Proceedings of the 6th International Asia Conference on Industrial Engineering

and Management Innovation. 2016.

12 Liu Y X. Research and Application of Collision Detection Technology Based on Hybrid Bounding Box in Virtual

Simulation of Industrial Robot[D].

13 Liu X P, Weng X Y, Chen H, et al. An Improved Algorithm for Octree-Based Exact Collision Detection [J]. Journal of

Computer-Aided Design and Computer Graphics (in chinese), 2005, 17(12):2631-2635.

14 Macdonald M. Pro WPF 4.5 in C#[M]. 2012.