Page 28 - Read Online

Zhang et al. J Mater Inf 2024;4:34 https://dx.doi.org/10.20517/jmi.2024.64 Page 21 of 28

ability of AI agent and LLMs is conducted. The deep learning architecture for welding image defect
recognition is generated and optimized by LLMs to explore the workflow to program assisted by artificial
intelligence, which can be the pioneer for automatic programming in the welding automation field.

Welding defect recognition through automated programming technology
The procedure for using AI agents for automated programming in welding defect recognition tasks can
follow the steps below. The first and most crucial step is to clarify the requirements that the programming is
expected to fulfill. The automatic programming process is conducted by inputting the requirement into the
LLMs and obtaining the outputting programs generated by LLMs. To make sure the LLMs can understand
the requirement completely, clear steps for the programs are required for the input content to LLMs. In the
input content, the program functions should be involved, including dataset loading, image preprocessing,
dataset splitting, pre-trained weights loading, and training loss storage. Additionally, the development
environment, including details such as the programming language (Python) and frameworks (PyTorch),
should be clearly defined. It is worth noting that the outputs and user interfaces of four LLMs including
ChatGPT-3.5, Claude, Microsoft Copilot, and Baidu ERNIE Bot were compared only in the simple task of
extracting arc contours. For programming deep learning models aimed at welding defect recognition,
ChatGPT was exclusively utilized, as it demonstrated the best ability to understand input requirements and
generate highly applicable code.

When requirements are input into ChatGPT, the programs are generated automatically. However,
automated programming cannot be accomplished in a single attempt and requires multiple interactions
between humans and artificial intelligence. Occasionally, the code generated by ChatGPT may not be
runnable in a local integrated development environment, resulting in compiler errors. These error messages
can be fed back into ChatGPT, which will then generate solutions. There may also be new requirements for
the programming. Developers can input the current programming along with the new requirements into
the ChatGPT, which will then generate new programming with the added functionality automatically.
Through the methodologies above, the programming can be finalized to ensure stable execution and
successful fulfillment of the specified tasks. The ability of ChatGPT to iteratively refine outputs based on
repeated inputs was most valued. Expecting a single attempt by ChatGPT to fully comprehend all
requirements and generate flawless code was not the focus of this approach. Compared to human-written
programs, those generated by ChatGPT tend to be more standardized, comprehensible and characterized by
clear logic and comments.

Welding image stitching through automated programming technology
Another application of automated programming in welding defect recognition is the development of image
stitching technology. The input for ChatGPT involves generating an executable program in a Python
environment to stitch high-resolution welding tissue images. The program is automatically generated based
on the following function modules: preprocessing, feature detection, feature matching, image fusion, and
post-processing. The initial version of the program may not perfectly meet the stitching requirements.
Adjustments can be made according to the specific function modules. For instance, various methods can be
applied to the feature detection function, including scale-invariant feature transform (SIFT), speeded-up
robust features (SURF), and oriented FAST and rotated BRIEF (ORB). In this research, LLMs are employed
to generate programs for feature detection based on these methods, with each method tested within the full
program to compare their performance in the image stitching task. After comparison, the best feature
detector is selected, employing the SIFT method to handle the high-resolution characteristics inherent in
welding images.