Increasing the reliability of industrial equipment using computer vision

Pavel Krinitsyn

Works as an analyst at a large metallurgical enterprise and applies machine learning and digital vision algorithms to optimize production processes.

But before we move on to the content of the study, I will say a few words about the equipment itself and its role in the production process.

Transport equipment occupies a large share of all industrial facilities at metallurgical plants. The functioning of the entire production process depends on the performance of belt conveyors. Conveyors ensure the continuity of the supply of raw materials to the next production site for further processing and move finished products to the next stages of production or for shipment.

The device that moves the loads can be metal plates – tracks – or a conveyor belt. Their scope of application is different, each has its own advantages, disadvantages and technical features: complexity of repair and installation, high cost of spare parts, restrictions on the installation location. Conveyors with a conveyor belt made on a PVC or rubber base are often used in production. Their share among conveyor equipment is more than 90%.

Figure 1. The metal chain of the conveyor, assembled from separate steel plates - tracks - is not afraid of the impact of foreign solid objects that can damage the soft conveyor belt. — Figure 1. The metal chain of the conveyor, assembled from separate steel plates – tracks – is not afraid of the impact of foreign solid objects that can damage the soft conveyor belt.

The service life of a rubber conveyor belt is on average one to two years, while metal tracks last for tens of years. Given the high failure rate of equipment for transporting and processing petroleum coke, increasing the reliability of the conveyor belt is a priority and particularly urgent task in production.

Figure 2. In terms of conveyor belt service life, problems most often arise with conveyor equipment

Here are the main problems that arise when operating a conveyor belt:

Belt wear – over time, the belt rubs against the materials being transported and wears out, which leads to the formation of through wear and deformation.
The ingress of foreign objects, which mechanically damages the tape, creates breaks and through defects on it.
Damage to the rubber layer of the belt by extreme temperatures from exposure to petroleum coke after the calcining oven and cooling drum – cracks form on the working surface, their depth and intensity gradually increase, and the belt breaks.

Figure 3. Stages of development of a conveyor belt defect as a result of damage to the working layer

For protection against high temperatures, grade 2T3 tape is used, but it also cannot withstand prolonged contact with heated calcined coke.

The service life of conveyor belts can also be affected by other factors, such as the fractional and chemical composition of the raw materials: calcined and raw petroleum coke, anode mass. When distilling raw petroleum coke in winter, the conveyor belt is exposed to chemical reagents that reduce the freezing of the raw materials. To reduce their harmful effects, oil-resistant conveyor belt 2MS is used. But gradually its working surface softens, loses its elastic properties and deforms.

The service life of conveyor belts is also affected by various operating conditions of the equipment. Some equipment operates continuously, while others operate according to a set schedule. The load of conveyors with raw materials also varies: constant load or periodically changing during operation, operation at maximum productivity or in a gentle mode. Because of this, it is impossible to establish a standard service life for conveyor belts: it varies greatly for similar equipment even within a single workshop.

Specially trained service personnel perform regular classic visual diagnostics of the technical condition of belts during operation. But this procedure also has its drawbacks. During inspection, the belt is usually under a layer of material, and with a large amount of equipment, it is impossible to simultaneously stop all transport systems. The human factor problem is also of great importance: every defect missed by a specialist can stop the conveyor and the entire production line, which will lead to extremely negative consequences.

That is why it is so important to implement a differentiated approach to equipment maintenance, which takes into account the degree of influence of a specific conveyor on the production process as a whole. Belt damage can be different: some require immediate repair, while others allow for repairs to be planned within predictable timeframes. Therefore, we were faced with the task of developing and implementing an automated system for belt diagnostics during operation. Such a system would significantly increase the accuracy and timeliness of defect detection, and would ensure more stable and safe operation of production lines.

In order to cope with typical belt faults in our production: surface cracks, deformations and local breaks or through holes, we studied the possibilities of independently implementing computer vision algorithms, as we wanted to assess the condition of transport equipment without involving repair personnel, i.e. in automatic mode.

Computer vision is a technology that enables computers to “see” and interpret visual data by analyzing images or video. It applies image processing, machine learning, and artificial intelligence techniques to perform object detection and segmentation tasks that only humans could previously perform.

Thus, using video cameras and image analysis algorithms, we tracked the condition of the conveyor belt in real time, and the system automatically identified signs of wear – cracks, breaks.

Feed state classifier based on pre-trained ResNet-18 neural network model

We processed the video stream from the camera with a recording speed of 250 fps and obtained images of conveyor belts with defects. We assigned classes to the images in accordance with the dominant defect on them. For example, if there were breaks and cracks on the belt, the image was assigned the class “Belt break” – as the most important from the point of view of further trouble-free operation of the equipment.

To increase the versatility of the model, we performed image augmentation on the training sample:

transformed_dataset = []
transform = transforms.Compose([
    transforms.RandomHorizontalFlip(p=0.5),  # Случайное отражение по горизонтали с вероятностью 50%
    transforms.RandomRotation(degrees=(-15, 15)),  # Случайный поворот на угол от –15 до 15 градусов
    transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1)  # Изменение цветовых характеристик
])

The position of the images and their color gamut were distorted.

Figure 4. We change the position of the images - the camera can be installed at different angles above the conveyor - and simulate different illumination of the object — Figure 4. We change the position of the images – the camera can be installed at different angles above the conveyor – and simulate different illumination of the object

Next, we imported the pre-trained ResNet-18 model from the Torchvision package of the PyTorch library, froze all the layers of the model, and replaced the last fully connected layer with a new one — with four output classes. We defined the loss and optimizer functions:

model = models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
num_classes = 4  # Количество классов в нашем датасете
model.fc = nn.Linear(model.fc.in_features, num_classes)
# Определение функции потерь и оптимизатора
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Then we tested the classifier model on 20 epochs and got a Loss error value of 7%. Considering that the training sample only had 224 images, we got good results.

To test the model's performance, we loaded an image from the training sample. After processing by the model, we received a conclusion about the state of the tape.

Thus, we have found out that thanks to the classifier it is possible to detect the presence or absence of defects on the belt and determine what kind of visual diagnostics is needed: with the involvement of specialists of the service organization or with the help of detection and segmentation algorithms. Such an algorithm of work is easy to implement, it can be used to diagnose most conveyor equipment and control its serviceability.

Conveyor Belt Defect Detector Using Fast R-CNN from PyTorch Library

To train the detector model, we needed a set of images with labeled defects. We labeled them in the CVAT program. The classes of recognizable defects are cracks, tape tears, and tape joint condition.

Figure 6. The results of marking conveyor belt defects are saved in .xml format

In the program, each image was assigned its own ID number, defect name and its coordinates. There could be several defect marks for one image.

Then, using Python methods, we created a dataframe based on the XML markup. To do this, we defined a function def parse_annotation: it parsed an XML file with annotations for images. Function parse_annotation converted the XML data into a list of dictionaries, each of which contained detailed information about a single image object: file name, image size, class label, and coordinates of polygon line points.

def parse_annotation(xml_file):
    tree = ET.parse(xml_file)
    root = tree.getroot()

    objects = []
    for image in root.findall('image'):
        filename = image.get('name')
        width = int(image.get('width'))
        height = int(image.get('height'))

        for polyline in image.findall('polyline'):
            label = polyline.get('label')
            points = polyline.get('points')

            objects.append({
                'name': filename,
                'width': width,
                'height': height,
                'class': label,
                'points': points
            })
    return objects

We checked the image markup based on the created dataframe.

Figure 7. Three classes of markings on the image of a section of conveyor belt

Next, we created a class for our dataset (class CustomDataset(Dataset)), thus preparing the data in a convenient format for loading in batches using DataLoader from PyTorch:

class CustomDataset(Dataset):
    def __init__(self, root, df, transforms=None):
        self.root = root
        self.transforms = transforms
        self.dataframe = df
        self.imgs = list(self.dataframe['filename'].unique())
    def __getitem__(self, idx):
        img_name = self.imgs[idx]
        img_path = os.path.join(self.root, img_name)
        img = Image.open(img_path).convert("RGB")
        current_annotations = self.dataframe[self.dataframe['filename'] == img_name]
        num_objs = len(current_annotations)
        boxes = []
        labels = []
        for _, row in current_annotations.iterrows():
            boxes.append([row["xmin"], row["ymin"], row["xmax"], row["ymax"]])
            labels.append(row["class"])

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        labels = torch.as_tensor(labels, dtype=torch.int64)
        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        target = {"boxes": boxes, "labels": labels, "image_id": image_id, "area": area, "iscrowd": iscrowd}
        if self.transforms:
            img, target = self.transforms(img, target)
        return img, target
    def __len__(self):
        return len(self.imgs)

And we created a function for training the detector model. We trained the model:

def train_model(num_classes, num_epochs, batch_size, device):
    dataset = CustomDataset(img_path, df, transforms=get_transform(train=True))
    data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0, collate_fn=collate_fn)
    class_weights = calculate_class_weights(df)
    criterion = torch.nn.CrossEntropyLoss(weight=class_weights)
    weights = FasterRCNN_ResNet50_FPN_Weights.COCO_V1
    model = fasterrcnn_resnet50_fpn(weights=weights)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    model = model.to(device)
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

    for epoch in range(num_epochs):
        model.train()
        epoch_loss = 0
        progress_bar = tqdm(data_loader, desc=f'Epoch {epoch + 1}/{num_epochs}', leave=False)
        for images, targets in progress_bar:
            images = list(image.to(device) for image in images)
            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
            loss_dict = model(images, targets)
            losses = sum(loss for loss in loss_dict.values())
            epoch_loss += losses.item()
            optimizer.zero_grad()
            losses.backward()
            optimizer.step()
            progress_bar.set_postfix(loss=epoch_loss / (progress_bar.n + 1))
        lr_scheduler.step()
        print(f"Epoch {epoch + 1} completed with loss {epoch_loss / len(data_loader)}")
    print("Training completed.")
    return model

The trained model recognized objects in images from the training set.

Figure 8. Results of the detector model operation

The detector model improved the diagnostics of the conveyor belt: it allowed not only to detect faults, but also to assess their scale. For example, it was now possible to calculate the area of detected damage. This information is especially important for determining the further strategy for repairing conveyor equipment.

The use of such a model is suitable for equipment that is installed in places where it is difficult to service – cramped conditions, insufficient ceiling height – and allows you to completely eliminate diagnostics by service specialists.

Segmentation of conveyor belt sections images using Mask R-CNN from PyTorch library

Image segmentation is usually used to highlight objects and their boundaries in images. So we defined a function for creating a segmentation mask of objects and their subsequent visualization:

def create_segmentation_mask(height, width, points):
    mask = np.zeros((height, width), dtype=np.uint8)
    points = points.split(';')
    points = [list(map(float, p.split(','))) for p in points]
    points = np.array(points, dtype=np.int32)
    cv2.fillPoly(mask, [points], 1)
    return mask
def show_images_with_masks(images, masks, num_images=3):
    combined = list(zip(images, masks))
    random.shuffle(combined)
    images, masks = zip(*combined)

    fig, axes = plt.subplots(num_images, 2, figsize=(10, 15))
    for i in range(num_images):
        axes[i, 0].imshow(images[i])
        axes[i, 0].set_title(f'Image {i+1}')
        axes[i, 1].imshow(masks[i], cmap='gray')
        axes[i, 1].set_title(f'Mask {i+1}')
        
    plt.tight_layout()
    plt.show()

Figure 9. Masks of objects on the image of a defective section of the tape

We created an image segmentation model to process the dataset and trained it:

# Загрузка модели
weights = MaskRCNN_ResNet50_FPN_Weights.COCO_V1
model = maskrcnn_resnet50_fpn(weights=weights)
# Количество классов (например, 1 класс + фон)
num_classes = 3
# Замена box_predictor
model.roi_heads.box_predictor = torchvision.models.detection.faster_rcnn.FastRCNNPredictor(
    model.roi_heads.box_predictor.cls_score.in_features, num_classes)
# Замена mask_predictor
model.roi_heads.mask_predictor = torchvision.models.detection.mask_rcnn.MaskRCNNPredictor(
    model.roi_heads.mask_predictor.conv5_mask.in_channels,
    model.roi_heads.mask_predictor.conv5_mask.out_channels,
    num_classes
)
# Проверка модели
print(model)

To evaluate the quality of the model's training, we visualized the model's predictions on new images.

Figure 10. Segmented defects on the conveyor belt

Segmentation of defects on the tape expands diagnostic capabilities, allowing more accurate calculation of the area of damage to the tape and prediction of the remaining service life. The nature of the location of objects in the image indicates the causes of defects, helps to develop corrective measures to eliminate or reduce the impact of these defects.

Conveyor equipment is usually installed and operated in places with increased dust and humidity, so the cameras must have a sufficient degree of dust and moisture protection. We used video cameras with an IP64 protection rating.

After training the algorithm, the labeled dataset with images of conveyor belt sections shows good results and perfectly identifies defects on the belt. This is enough to determine in detail the technical condition of the belt and decide whether it is necessary to conduct a visual inspection with the involvement of service specialists.

Algorithms trained using the coordinates of the vertices of the bounding rectangles provide much more information about the defect itself: its approximate size, location, and nature of the damage. Using this information, it is possible to predict the service life of the tape, roughly estimate the area of its damage, and record the impact of high temperatures or chemical reagents on it.

Segmentation of objects on the tape provides a better idea of the damage area and accurately records damaged areas. It is advisable to use segmentation for especially important equipment, since its trouble-free operation ensures the operability of the entire production line.

In the future, we will optimize the equipment maintenance system according to categories of importance for the production process based on the information received about the operation of the models and integrate this algorithm into the enterprise management system – MES.