Robotic path planning using NDT ultrasonic data for autonomous inspection

. Robot deployed ultrasonic inspection for Non-Destructive Testing (NDT) offers several advantages including time ef ﬁ ciency gains, the reducing of repetitive manual workloads for operators and the enabling of inspection of environments hazardous to human health. Due to accuracy requirements, NDT robotic inspection has traditionally used the concept of digital twins for path planning activities. Recent development has sought to automate this process through visual feedback using low-cost camera sensors. However, these methods do not take into account the use of NDT data itself as part of the robot path planning process. As a consequence, poor path planning accuracy can result due to the inability of conventional cameras to capture internal defects or geometric features. This paper introduces a novel concept of using the NDT ultrasonic data as part of a robotic path planning feedback loop. Firstly, the robot is manually positioned near the start of a weld, and the ultrasonic data is collected. Next, algorithms are implemented to monitor changes in the weld geometry, to determine the robot ’ s movement and pose based on real-time monitoring data, and to enable the robot to autonomously scan a weld with a minimum of operators input, path planning or digital twin. This is advantageous to NDT as visual sensors are unable to monitor geometric features within the weld. The ability to use the NDT data ensures the inspection continues at the optimal con ﬁ guration (e.g. correct stand off and limiting probe skew), and achieves optimal path planning for NDT robots. The experimental results have shown that the tracking algorithm can effectively and accurately track defects in the sample during the ultrasonic probe detection process with an error rate within±1 mm.


Introduction
With the rapid development of modern industrial technology, more and more industries require the use of Non-Destructive Testing (NDT) for quality inspection.In order to meet the needs of different NDT applications, the use of automated equipment to replace manual inspection of workpieces offers a more advanced inspection method leading to less operator fatigue, and improvements in repeatability [1].Robots can be programmed to perform specific operations, thus replacing manual labour for heavy, complex and dangerous inspection tasks.Deployment of NDT by robots has the advantages of high accuracy of movement, high efficiency and lower operating costs.As a result, robotic inspection has been a growing area of research in the manufacturing and remanufacturing industry in recent years [2].
Robotic ultrasonic inspection has been used to speed up the NDT process of critical components, although typically such systems rely on precise digital twins to support the path planning activity.Autonomous systems have been developed based on visual feedback to remove the requirement of a digital twin, and automate the path planning process leading to improved inspection speeds.These autonomous systems are based on robots that manipulate ultrasonic sensors based on predefined or pre-computed toolpaths [3].As a result, such robotic inspection systems typically lack a strategy for flexible and agile automation, particularly where visual systems do not provide the necessary feedback.There is an increasing need to automate the process of robotic dynamic toolpath generation to enable efficient NDT inspection of components where visual information is not available À such as in the case of grounded flush bevel welds where no visual cap is present.
The key to generating automated scanning paths for NDT robots lies in the automated movement of the ultrasound probe under different scenarios.A variety of control modes have been used in current research on industrial robotic NDT systems to automate the movement of ultrasonic probes in different scenes, mainly based on path planning and marker-based guidance control methods [4,5].Among them, path planning based on 3-dimensional scene reconstruction and marker-based object pose estimation are typical ultrasound robot imaging control methods.Based on this paradigm, previous robotic nondestructive systems use 3-dimensional cameras or other high-precision acquisition devices to acquire and reconstruct the scene surface and plan the robot motion path based on the geometric properties of the target obtained after analysis with reported 80% success rate of capturing the ultrasonic pulse-echo data for the desired surface points on the sample and an accuracy of 0.43 mm for the ultrasonic testing tool path produced using paraboloid spiral image acquisition approach [6].Some studies have combined force sensors to adjust the contact force by adjusting the position of the ultrasound probe, with the reported accuracy of 3D ultrasound reconstruction as 0.72 ± 0.24 mm [7].However, these methods have certain limitations; firstly, the use of a conventional camera and laser feedback intelligence to locate the surface of the workpiece does not allow for the identification of internal defects in the work piece.Secondly, these control methods contain multiple, highly interrelated motions, and aspects of each motion need to be highly characterised to match the specific imaging task.
The main purpose of this paper is to consider the use of real-time ultrasonic inspection data feedback as a basis for NDT dynamic robot path planning.Based on the inspection data there are two main objectives: firstly analysis of the ultrasonic NDT data to determine positional information of geometric features and secondly to use this information to derive a motion for the robotic system to take.The proposed novel dynamic path planning method overcomes the limitations of conventional vision systems such as the need for complex and potentially inaccurate point cloud correction.In addition, the requirement for a priori knowledge of the surface (e.g., imposed primitives) is eliminated.This is done while making full use of the available data to minimise the risk of in-process conflicts.The proposed approach enables real-time robot path correction that was previously unavailable in visionenabled path planning regimes.The proposed full path planning execution process is illustrated in Figure 1.

Theoretical background 2.1 Ultrasonic weld inspection
The weld inspection deployed in this work has utilised Full Matrix Capture (FMC) for ultrasonic data acquisition and collection and the Total Focusing Method (TFM) as the image reconstruction algorithm.
FMC is an advanced data acquisition process that uses ultrasonic array transducers to capture the complete time domain signal from every possible transmitting and receiving element combination.Traditional weld inspection approaches have used single-element ultrasonic probes and phased array methods.FMC has been chosen for this work as it provides improved image quality and resolution over other methods and is commonly referred to as the 'gold standard' [8].Additionally, FMC data is often processed with an image reconstruction algorithm allowing for computer vision methods to be exploited.The principle of FMC is shown in Figure 2. Generally modern array ultrasound transducer excitation receiver boards have parallel independent receiver channels.Thus, the full matrix acquisition process can be described as assuming the ultrasound array contains N elements from which a matrix is populated {A ij (t)} representing the amplitude responses of the echo signal excited by the ith element and received by the jth element (1 i N, 1 j N) .If elements 1 to N are excited in sequence and all arrays are received, the final N Â N sets of ultrasonic A-scan data are collected known as the full matrix.The collected data is arranged in the following way: data in column i represents the echo amplitude emitted from the i th element, while the rows represent the echo amplitudes received by the jth element.The full matrix data structure format is shown in equation ( 1).
TFM uses the FMC data to provide an image within the component that allows for increase in sensitivity to small defects through an image reconstruction algorithm [9].This algorithm is based on the Synthetic Aperture Focusing Technique [10] but extended specifically for FMC data.The inspection region with respect to the transducer is defined as the Region of Interest (ROI).Each pixel within this image represents a finite inspection resolution with the x-axis being the direction of the transducer's active axis (i.e., the direction in which the elements are stacked), while the image y-axis refers the depth within the component (commonly referred to as the z-axis during inspection).For each pixel and each transmit-receive combination, the arrival time along the respective A-Scan is computed after consideration of the component geometry and velocity À referred to as the Time of Flight (ToF).This process is repeated for all pixels and all transmit and receive combinations with the pixel intensity values being summed.The method is also referred to as the Delay and Sum (DAS) approach [11].
The TFM algorithm is expressed in equation ( 2) and supported by Figure 3, where i(x,z) k is the time of flight from the transmission element k and j(x,z) n is the receive time of flight to reception element n of the array for a given pixel at position (x,z), h is the Hilbert transformed signal allowing for both the real and imaginary parts to be computed.For a TFM reconstruction this calculation is computed for every pixel within the image.
Compared to other traditional ultrasonic methods, the TFM algorithm has the advantages of being flexible and highly efficient.Not only that, TFM can improve the coverage of corner detection and increase the sensitivity to small defects.The algorithm makes use of valid data information at each point for computational imaging and can obtain ultrasound images with high resolution and contrast.
The main visualisation modes (A-scan, B-scan, C-scan, D-scan) offered by NDT based on ultrasonic signal transmission/reception are shown in Figure 4 [12].
The A-Scan shows a one-dimensional raw ultrasonic signal, i.e., the amount of ultrasound energy received versus time.Traditionally the B-scan provides a twodimensional image with the vertical axis representing time converted to distance and the horizontal axis representing the next A-Scan in the sequence.The TFM reconstructed image is also referred to as the B-Scan (a synthetically generated one).The C-scan is a two-dimensional top view showing the test part.In the C-scan no front and rear surface reflections are used, instead the reflection from the defect is the only projection.The 3-dimensional position of the defect in the weld can be determined from both the C-scan and B-Scan.The D-scan shows a profile where both the acoustic beam plane and the scanned surface are perpendicular.In the D-scan view, the height of the defect and the length of the defect displayed in the direction of the scan axis can be observed, allowing quick differentiation of defects in the weld.

Computer vision
Typically when analysing an ultrasonically generated image, the operator will make fundamental decisions as to the classification of a defect based on its position, size, orientation, and amplitude.In many ways, this decision is based on intuition and experience, which can be difficult to quantify.One of the objectives of this research is to lock in on to a key geometric feature and track changes in its position relative to time.To support this, Computer Vision (CV) algorithms are explored within this paper, specifically the meanshift algorithm [13,14].
The intuition behind the meanshift algorithm is as follows: consider a collection of pixels within a window (typically a user defined box).The goal is to move the window relative to mean pixel location.This is repeated for the next image and the centre location of the previous window and newly computed window is extracted and used as a measure as to how much and in what direction the user-selected feature has moved.This concept is shown in Figure 5 using images of a motorway by way of example.
The meanshift algorithm is based on target tracking.All the target tracking is done through a cycle of three operations: firstly, generate the target model and update the target model.Secondly, the target position is predicted, and finally the matching criterion of the target model is calculated.Among them, the initialisation of the tracking target in the first frame is done manually as follows.
-Consider a collection of pixels within a window (typically a user defined box), which is the region where the object to be tracked is located, the dimensions of the region and the kernel function window width h.-The probabilities of each feature value in the corresponding feature space are calculated for the selected region for the objective description and modelling.
-The description of the candidate template is calculated for each feature value in the feature space of the possible target regions in the subsequent frames.-The Bhattacharyya similarity coefficient is used as the similarity function to calculate the similarity between the candidate region in the current frame and the target model in the previous frame to find the meanshift vector that maximizes the similarity function for the purpose of tracking.
Since the starting point of this vector is the position of the target in the previous frame and the ending point is the position of the target in the current frame.According to the convergence property of the meanshift algorithm, the meanshift vector will eventually converge at the real position of the target through continuous cyclic computation.The pseudo-code of the meanshift algorithm based on this study is shown in Table 1 and a flowchart described in Figure 6.
Through the use of ultrasonically generated images, and computer vision, the next goal is to use this information to determine a direction the robotic dynamic path should take.

Robotic control
In order for the robot to be able to adjust its path dynamically according to the meanshift coordinates during the inspection process, it is first necessary to convert the local coordinates of the end-effector to the world coordinate system by means of robot coordinate conversion.The B-scan image coordinate system is set by the system to be consistent with the world coordinate system.The transformation process is to determine the coordinates of the weld feature point P in the robotic arm coordinate system O B X B Y B Z B as x B P ; y B P ; z B P À Á .In general, the coordinates of the weld feature under the ultrasonic coordinate system O U X U Y U Z U as x U P ; y U P ; z U P À Á can be easily extracted from the B-scan data and transformed to the coordinates of the locating point under the rotating mechanism coordinate system O R X R Y R Z R of the probe as x R P ; y R P ; z R P À Á with the transformation matrix as equation ( 3): where: R R U is a 3À3 dimensional rotational transformation matrix; P R O U is a 3 Â 1 dimensional translational transformation matrix.
The ultrasound probe scans the weld platform for geometric features by travelling initially in a straight line at a speed of 15 mm/s using the SpeedL command as supported by the robotic platform (Universal Robotics UR3e).Based on the feedback from the internal algorithm for tracking of the weld root geometric feature in the Bscan, the motion is adjusted in the +x, Àx direction.Scanning along +y to complete the scan of the weld seam of the welding platform.The coordinate system of robot and weld platform is shown in Figure 7.An experimental data acquisition and data processing system has been designed and assembled to verify the feasibility of enabling robots to perform autonomous weld scanning based on ultrasonic inspection data.The system architecture is illustrated in Figure 8.The system components are presented below.
-A Universal Robotics UR3e platform was deployed, where the robot UR3e was able to reach a load of 6.6lbs     Image reconstruction from the acquired ultrasonic data was carried out using TFM at a resolution of 125 mm per pixel.TFM was selected over conventional UT and Phased Array UT due to its ability to provide greater lateral resolution, thus providing optimal image resolution for tracking of geometric features.Acquisition and data processing was performed in real-time focused on an area 30 mm 2 (horizontally À15 mm to +15 mm and vertically from 20 mm to 50 mm) depicted in Figure 9.This allowed for the volume of the weld to be imaged including the weld root and weld cap at a pixel density of 240 Â 240 per image mode.The image modes of half-skip and full-skip were volumetrically merged and used as the input image for the meanshift algorithm, this is to provide a single image for computation.
The probe on the robot end-effector was manually placed on one side of the weld plate and adjusted to the optimum position based on the imaging results of the Bscan.The dynamic path planning was then initiated and monitored, allowing for collection of multiple B-Scans and the generation of a C-Scan view to assist in later interpretation.

Experimental results
The reliability of the meanshift tracking algorithm used in this paper was experimentally verified through testing on a calibration block with a thickness of 25 mm.The experiment evaluated the accuracy of the algorithm by comparing the actual moving distance of the ultrasonic probe with the moving distance of the tracking box on the ultrasonic detection image.The benchmark table demonstrates that the algorithm achieved a high level of accuracy, with an error rate within ±1mm as shown in Table 2.These findings indicate that the tracking algorithm can effectively and accurately identify and track defects in the sample during the ultrasonic probe detection process.The accuracy is comparable to those achieved with additional vision camera [6] or force measurement [7], but our method did not use additional sensors other than the ultrasound probe.
Initial experimental results demonstrated the robot was able to adjust the path according to the positioning of the geometric feature within the B-scan image acquired by ultrasonic inspection.And keeping the window always within the ultrasonic scan imaging area, enabling fine tuning of the robotic path in the ÀX, +X direction in the ÀY direction of travel.
The method has been shown to be feasible in application to both linear and sinusoidal shaped welds.The robot scan trajectory can be seen in the shape of the ultrasonic gel imprint generated on the weld platform.The linear weld sweep trajectory forms a serrated shape within the ultrasonic gel, as shown in Figure 10.The sinusoidal shaped weld scan trajectory results in a smooth sine shape within the ultrasonic gel which was marked by red curve lines, once again, proving and supporting the feasibility of the method.There is a very high confidence level for the fillet weld type, based on a close examination of the C-scan results.
The developed visualisation platform of the system is shown in Figure 11, where the different window areas show the different imaging results.The top left corner shows the B-scan imaging, which is used to adjust the robot starting position as well as to observe the inspection process of the robot.In the upper right corner is the A-scan image of the defect scan.Figure 11 shows a C-scan image in the bottom window which visualises the horizontal position of the weld.
The imaging results of the C-scan are shown in more detail in Figure 12.The shape and size of the geometric feature can be clearly seen, indicating that the robot is able to obtain a good geometric detection rate when automatically inspecting the weld with TFM imaging, something not possible by human operator.In addition, by adjusting the scale line in the figure, its defect imaging is basically level with the horizontal scale line.This indicates that using the image tracking algorithm can accurately manoeuvre the robot with respect to this response, as well as that the robot's autonomous sweeping path has a high degree of accuracy.
The ability of the robot to automatically scan the weld seam and extract defect features provides a significant advantage to NDT.This paper presents a novel method for real-time adjustment of the scan path of an NDT robot, which uses real-time data collected by phased array probes, after conversion of the robot end-effector ultrasound probe to Bscan 2-D image coordinates, to adjust the sweep path of the robot in real-time.The proposed method allows the operator to obtain highly accurate ultrasonic sweep data without the need for digital twinning and without lengthy pre-scan and path planning steps when the part weld distribution is not known.
A potential limitation of the method is the accuracy of the first B-scan image obtained, i.e., the ability to detect the location of the target feature and ensure it is in the region of interest, which requires the operator to autonomously approach the precise placement of the UT probe to ensure that the most comprehensive defect imaging map possible is obtained so that the resulting path is accurately placed within the robot work cell.
The method established in this paper for the autonomous scanning of weld seams by NDT robots is effective and accurate, and can provide a theoretical basis for subsequent NDT robots to achieve intelligent path planning for sweeping weld seams.The future plan is to upgrade the scanning method to include probe skew based on the ultrasonic response.This will improve the accuracy and reliability of the.robot's automated weld seam inspection results.

Fig. 4 .
Fig. 4. Visualisation of the basic pattern diagram of ultrasound test results.
(3 kg)  during automatic tasks, Reach radius of up to 19.7 inches (500 mm), Weight of 24.7lbs (11.2 kg), Small footprint of just Ø128 mm, 6-axis motion with 360-degree wrist-joint rotation and infinite end-joint rotation.-AnOlympus 5L64-A2 transducer with a 0.6mm pitch, containing a 64-element linear array with a 5 MHz central frequency.Data was sampled at a rate of 25 MHz with a 16-bit amplitude resolution for 64 active elements.This probe was attached to a Rexolite wedge of 36 0 incidence angle.

Fig. 9 .
Fig. 9. Depiction of FMC data acquisition region of interest.

Table 2 .
Comparison of actual and tracked probe movement distances on B-scan image.29mm in the +Y direction(3mm), while the tracking box moved from 9mm to 12mm in the +Y direction(3mm).(Error ~0mm)