LibTorch实战三：C++版本YOLOV5.4的部署

一、环境配置

二、.torchscript.pt版本模型导出

三、C++版本yolov5.4实现

四、问题记录

4.1、注释 detector.h中，注释如下头文件

4.2、错误： “std”: 不明确的符号

4.3、建议常被debug版本libtorch

4.4、问题：编译成功后，运行代码，发现torch::cuda::is_available()返回false

4.5、导出模型，命令行有警告

五、reference

一、环境配置

win10
vs2017
libtorch-win-shared-with-deps-debug-1.8.1+cpu
opencv349

　　由于yolov5代码，作者还在更新(写这篇博客的时候，最新是5.4，没错这是几年前写的)，模型结构可能会有改变，所以咱们使用的libtorch必须满足其要求，最好是一致。我这里提供本博客采用的yolov5版本python源码。

在源码中的requirments.txt中要求依赖库版本如下；在c++环境中，咱们这里用的libtorch1.8.1（今天我也测试了环境：libtorch-win-shared-with-deps-1.7.1+cu110，也能够正常检测，和本博客最终结果一致）；同时用opencv&c++作图像处理，不需要c++版本torchvision：

# pip install -r requirements.txt

# base ----------------------------------------
matplotlib>=3.2.2
numpy>=1.18.5
opencv-python>=4.1.2
Pillow
PyYAML>=5.3.1
scipy>=1.4.1
torch>=1.7.0
torchvision>=0.8.1# 以下内容神略

为了便于调试，我这里下载的是debug版本libtorch，而且是cpu版本，代码调好后，转GPU也很简单吧。opencv版本其实随意，opencv3++就行。

二、.torchscript.pt版本模型导出

　　打开yolov5.4源码目录下models文件夹，编辑export.py脚本，如下，将58行注释，新增59行（GPU版本还需要修改一些内容，GPU版本后续更新，这篇博客只管CPU版本）

"""Exports a YOLOv5 *.pt model to ONNX and TorchScript formats

Usage:
    $ export PYTHONPATH="$PWD" && python models/export.py --weights ./weights/yolov5s.pt --img 640 --batch 1
"""

import argparse
import sys
import time

sys.path.append('./')  # to run '$ python *.py' files in subdirectories

import torch
import torch.nn as nn

import models
from models.experimental import attempt_load
from utils.activations import Hardswish, SiLU
from utils.general import set_logging, check_img_size
from utils.torch_utils import select_device

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--weights', type=str, default='./yolov5s.pt', help='weights path')  # from yolov5/models/
    parser.add_argument('--img-size', nargs='+', type=int, default=[640, 640], help='image size')  # height, width
    parser.add_argument('--batch-size', type=int, default=1, help='batch size')
    parser.add_argument('--dynamic', action='store_true', help='dynamic ONNX axes')
    parser.add_argument('--grid', action='store_true', help='export Detect() layer grid')
    parser.add_argument('--device', default='cpu', help='cuda device, i.e. 0 or 0,1,2,3 or cpu')
    opt = parser.parse_args()
    opt.img_size *= 2 if len(opt.img_size) == 1 else 1  # expand
    print(opt)
    set_logging()
    t = time.time()

    # Load PyTorch model
    device = select_device(opt.device)
    model = attempt_load(opt.weights, map_location=device)  # load FP32 model
    labels = model.names

    # Checks
    gs = int(max(model.stride))  # grid size (max stride)
    opt.img_size = [check_img_size(x, gs) for x in opt.img_size]  # verify img_size are gs-multiples

    # Input
    img = torch.zeros(opt.batch_size, 3, *opt.img_size).to(device)  # image size(1,3,320,192) iDetection

    # Update model
    for k, m in model.named_modules():
        m._non_persistent_buffers_set = set()  # pytorch 1.6.0 compatibility
        if isinstance(m, models.common.Conv):  # assign export-friendly activations
            if isinstance(m.act, nn.Hardswish):
                m.act = Hardswish()
            elif isinstance(m.act, nn.SiLU):
                m.act = SiLU()
        # elif isinstance(m, models.yolo.Detect):
        #     m.forward = m.forward_export  # assign forward (optional)
    #model.model[-1].export = not opt.grid  # set Detect() layer grid export
    model.model[-1].export = False
    y = model(img)  # dry run

    # TorchScript export
    try:
        print('\nStarting TorchScript export with torch %s...' % torch.__version__)
        f = opt.weights.replace('.pt', '.torchscript.pt')  # filename
        ts = torch.jit.trace(model, img)
        ts.save(f)
        print('TorchScript export success, saved as %s' % f)
    except Exception as e:
        print('TorchScript export failure: %s' % e)
# 以下代码省略，无需求改
......

接着在conda环境激活yolov5.4的虚拟环境，执行下面脚本：

python models/export.py --weights ./weights/yolov5s.pt --img 640 --batch 1

错误解决：1、bash窗口可能提示 not module utils；这是因为没有将源码根目录添加进环境变量，linux下执行以下命令就行

export PYTHONPATH="$PWD"

win下，我建议直接用pycharm打开yolov5.4工程，在ide中去执行export.py就行，如果你没有下载好yolovs.pt，他会自动下载，下载链接会打印在控制台，如下，如果下不动，可以尝试复制链接到迅雷。

Downloading https://github.com/ultralytics/yolov5/releases/download/v4.0/yolov5s.pt to yolov5s.pt...

执行export.py后出现如下警告：

D:\yolov5-0327\models\yolo.py:50: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if self.grid[i].shape[2:4] != x[i].shape[2:4]:
D:\Program Files\Anaconda\envs\yolov5\lib\site-packages\torch\jit\_trace.py:934: TracerWarning: Encountering a list at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.
  module._c._create_method_from_trace(
TorchScript export success, saved as ./yolov5s.torchscript.pt
ONNX export failure: No module named 'onnx'
CoreML export failure: No module named 'coremltools'

Export complete (10.94s). Visualize with https://github.com/lutzroeder/netron.

警告内容以后分析，不影响部署

三、C++版本yolov5.4实现

libtorch在vs环境中配置(在项目属性中设置下面加粗项目)：

include：

D:\libtorch-win-shared-with-deps-debug-1.8.1+cpu\libtorch\include

D:\libtorch-win-shared-with-deps-debug-1.8.1+cpu\libtorch\include\torch\csrc\api\include

lib：

D:\libtorch-win-shared-with-deps-debug-1.8.1+cpu\libtorch\lib

依赖库（你可能用的更新的libtorch，所以具体lib目录下所有.lib文件都要自己贴到连接器-附加依赖中）：

asmjit.lib
c10.lib
c10d.lib
c10_cuda.lib
caffe2_detectron_ops_gpu.lib
caffe2_module_test_dynamic.lib
caffe2_nvrtc.lib
clog.lib
cpuinfo.lib
dnnl.lib
fbgemm.lib
fbjni.lib
gloo.lib
gloo_cuda.lib
libprotobuf-lite.lib
libprotobuf.lib
libprotoc.lib
mkldnn.lib
pthreadpool.lib
pytorch_jni.lib
torch.lib
torch_cpu.lib
torch_cuda.lib
XNNPACK.lib

环境变量（需要重启）：

D:\libtorch-win-shared-with-deps-debug-1.8.1+cpu\libtorch\lib

配置好之后，vs2017 设置为debug X64模式，下面是yolov5.4版本c++代码

输入是：

上述转好的.torchscript.pt格式的模型文件
coco.names
一张图

完整libtorch c++源码：

#include <torch/script.h>
#include <torch/torch.h>
#include<opencv2/opencv.hpp>
#include <iostream>


std::vector<std::string> LoadNames(const std::string& path)
{
    // load class names
    std::vector<std::string> class_names;
    std::ifstream infile(path);
    if (infile.is_open()) {
        std::string line;
        while (std::getline(infile, line)) {
            class_names.emplace_back(line);
        }
        infile.close();
    }
    else {
        std::cerr << "Error loading the class names!\n";
    }

    return class_names;
}

std::vector<float> LetterboxImage(const cv::Mat& src, cv::Mat& dst, const cv::Size& out_size)
{
    auto in_h = static_cast<float>(src.rows);
    auto in_w = static_cast<float>(src.cols);
    float out_h = out_size.height;
    float out_w = out_size.width;

    float scale = std::min(out_w / in_w, out_h / in_h);

    int mid_h = static_cast<int>(in_h * scale);
    int mid_w = static_cast<int>(in_w * scale);

    cv::resize(src, dst, cv::Size(mid_w, mid_h));

    int top = (static_cast<int>(out_h) - mid_h) / 2;
    int down = (static_cast<int>(out_h) - mid_h + 1) / 2;
    int left = (static_cast<int>(out_w) - mid_w) / 2;
    int right = (static_cast<int>(out_w) - mid_w + 1) / 2;

    cv::copyMakeBorder(dst, dst, top, down, left, right, cv::BORDER_CONSTANT, cv::Scalar(114, 114, 114));

    std::vector<float> pad_info{ static_cast<float>(left), static_cast<float>(top), scale };
    return pad_info;
}

enum Det
{
    tl_x = 0,
    tl_y = 1,
    br_x = 2,
    br_y = 3,
    score = 4,
    class_idx = 5
};

struct Detection
{
    cv::Rect bbox;
    float score;
    int class_idx;
};

void Tensor2Detection(const at::TensorAccessor<float, 2>& offset_boxes,
    const at::TensorAccessor<float, 2>& det,
    std::vector<cv::Rect>& offset_box_vec,
    std::vector<float>& score_vec)
{

    for (int i = 0; i < offset_boxes.size(0); i++) {
        offset_box_vec.emplace_back(
            cv::Rect(cv::Point(offset_boxes[i][Det::tl_x], offset_boxes[i][Det::tl_y]),
                cv::Point(offset_boxes[i][Det::br_x], offset_boxes[i][Det::br_y]))
        );
        score_vec.emplace_back(det[i][Det::score]);
    }
}

void ScaleCoordinates(std::vector<Detection>& data, float pad_w, float pad_h,
    float scale, const cv::Size& img_shape)
{
    auto clip = [](float n, float lower, float upper)
    {
        return std::max(lower, std::min(n, upper));
    };

    std::vector<Detection> detections;
    for (auto & i : data) {
        float x1 = (i.bbox.tl().x - pad_w) / scale;  // x padding
        float y1 = (i.bbox.tl().y - pad_h) / scale;  // y padding
        float x2 = (i.bbox.br().x - pad_w) / scale;  // x padding
        float y2 = (i.bbox.br().y - pad_h) / scale;  // y padding

        x1 = clip(x1, 0, img_shape.width);
        y1 = clip(y1, 0, img_shape.height);
        x2 = clip(x2, 0, img_shape.width);
        y2 = clip(y2, 0, img_shape.height);

        i.bbox = cv::Rect(cv::Point(x1, y1), cv::Point(x2, y2));
    }
}


torch::Tensor xywh2xyxy(const torch::Tensor& x)
{
    auto y = torch::zeros_like(x);
    // convert bounding box format from (center x, center y, width, height) to (x1, y1, x2, y2)
    y.select(1, Det::tl_x) = x.select(1, 0) - x.select(1, 2).div(2);
    y.select(1, Det::tl_y) = x.select(1, 1) - x.select(1, 3).div(2);
    y.select(1, Det::br_x) = x.select(1, 0) + x.select(1, 2).div(2);
    y.select(1, Det::br_y) = x.select(1, 1) + x.select(1, 3).div(2);
    return y;
}

std::vector<std::vector<Detection>> PostProcessing(const torch::Tensor& detections,
    float pad_w, float pad_h, float scale, const cv::Size& img_shape,
    float conf_thres, float iou_thres)
{
    /***
     * 结果纬度为batch index(0), top-left x/y (1,2), bottom-right x/y (3,4), score(5), class id(6)
     * 13*13*3*(1+4)*80
     */
    constexpr int item_attr_size = 5;
    int batch_size = detections.size(0);
    // number of classes, e.g. 80 for coco dataset
    auto num_classes = detections.size(2) - item_attr_size;

    // get candidates which object confidence > threshold
    auto conf_mask = detections.select(2, 4).ge(conf_thres).unsqueeze(2);

    std::vector<std::vector<Detection>> output;
    output.reserve(batch_size);

    // iterating all images in the batch
    for (int batch_i = 0; batch_i < batch_size; batch_i++) {
        // apply constrains to get filtered detections for current image
        auto det = torch::masked_select(detections[batch_i], conf_mask[batch_i]).view({ -1, num_classes + item_attr_size });

        // if none detections remain then skip and start to process next image
        if (0 == det.size(0)) {
            continue;
        }

        // compute overall score = obj_conf * cls_conf, similar to x[:, 5:] *= x[:, 4:5]
        det.slice(1, item_attr_size, item_attr_size + num_classes) *= det.select(1, 4).unsqueeze(1);

        // box (center x, center y, width, height) to (x1, y1, x2, y2)
        torch::Tensor box = xywh2xyxy(det.slice(1, 0, 4));

        // [best class only] get the max classes score at each result (e.g. elements 5-84)
        std::tuple<torch::Tensor, torch::Tensor> max_classes = torch::max(det.slice(1, item_attr_size, item_attr_size + num_classes), 1);

        // class score
        auto max_conf_score = std::get<0>(max_classes);
        // index
        auto max_conf_index = std::get<1>(max_classes);

        max_conf_score = max_conf_score.to(torch::kFloat).unsqueeze(1);
        max_conf_index = max_conf_index.to(torch::kFloat).unsqueeze(1);

        // shape: n * 6, top-left x/y (0,1), bottom-right x/y (2,3), score(4), class index(5)
        det = torch::cat({ box.slice(1, 0, 4), max_conf_score, max_conf_index }, 1);

        // for batched NMS
        constexpr int max_wh = 4096;
        auto c = det.slice(1, item_attr_size, item_attr_size + 1) * max_wh;
        auto offset_box = det.slice(1, 0, 4) + c;

        std::vector<cv::Rect> offset_box_vec;
        std::vector<float> score_vec;

        // copy data back to cpu
        auto offset_boxes_cpu = offset_box.cpu();
        auto det_cpu = det.cpu();
        const auto& det_cpu_array = det_cpu.accessor<float, 2>();

        // use accessor to access tensor elements efficiently
        Tensor2Detection(offset_boxes_cpu.accessor<float, 2>(), det_cpu_array, offset_box_vec, score_vec);

        // run NMS
        std::vector<int> nms_indices;
        cv::dnn::NMSBoxes(offset_box_vec, score_vec, conf_thres, iou_thres, nms_indices);

        std::vector<Detection> det_vec;
        for (int index : nms_indices) {
            Detection t;
            const auto& b = det_cpu_array[index];
            t.bbox =
                cv::Rect(cv::Point(b[Det::tl_x], b[Det::tl_y]),
                    cv::Point(b[Det::br_x], b[Det::br_y]));
            t.score = det_cpu_array[index][Det::score];
            t.class_idx = det_cpu_array[index][Det::class_idx];
            det_vec.emplace_back(t);
        }

        ScaleCoordinates(det_vec, pad_w, pad_h, scale, img_shape);

        // save final detection for the current image
        output.emplace_back(det_vec);
    } // end of batch iterating

    return output;
}

void Demo(cv::Mat& img,
    const std::vector<std::vector<Detection>>& detections,
    const std::vector<std::string>& class_names,
    bool label = true)
{
    if (!detections.empty()) {
        for (const auto& detection : detections[0]) {
            const auto& box = detection.bbox;
            float score = detection.score;
            int class_idx = detection.class_idx;

            cv::rectangle(img, box, cv::Scalar(0, 0, 255), 2);

            if (label) {
                std::stringstream ss;
                ss << std::fixed << std::setprecision(2) << score;
                std::string s = class_names[class_idx] + " " + ss.str();

                auto font_face = cv::FONT_HERSHEY_DUPLEX;
                auto font_scale = 1.0;
                int thickness = 1;
                int baseline = 0;
                auto s_size = cv::getTextSize(s, font_face, font_scale, thickness, &baseline);
                cv::rectangle(img,
                    cv::Point(box.tl().x, box.tl().y - s_size.height - 5),
                    cv::Point(box.tl().x + s_size.width, box.tl().y),
                    cv::Scalar(0, 0, 255), -1);
                cv::putText(img, s, cv::Point(box.tl().x, box.tl().y - 5),
                    font_face, font_scale, cv::Scalar(255, 255, 255), thickness);
            }
        }
    }

    cv::namedWindow("Result", cv::WINDOW_NORMAL);
    cv::imshow("Result", img);

}

int main()
{
    // yolov5Ns.torchscript.pt 报错，所以仅能读取yolov5.4模型
    torch::jit::script::Module module = torch::jit::load("yolov5sxxx.torchscript.pt");
    torch::DeviceType device_type = torch::kCPU;
    module.to(device_type);
    /*module.to(torch::kHalf);*/
    module.eval();

    // img 必须读取3-channels图片
    cv::Mat img = cv::imread("zidane.jpg", -1);
    // 读取类别
    std::vector<std::string> class_names = LoadNames("coco.names");
    if (class_names.empty()) {
        return -1;
    }

    // set up threshold
    float conf_thres = 0.4;
    float iou_thres = 0.5;

    //inference
    torch::NoGradGuard no_grad;
    cv::Mat img_input = img.clone();
    std::vector<float> pad_info = LetterboxImage(img_input, img_input, cv::Size(640, 640));
    const float pad_w = pad_info[0];
    const float pad_h = pad_info[1];
    const float scale = pad_info[2];
    cv::cvtColor(img_input, img_input, cv::COLOR_BGR2RGB);  // BGR -> RGB
    //归一化需要是浮点类型
    img_input.convertTo(img_input, CV_32FC3, 1.0f / 255.0f);  // normalization 1/255
    // 加载图像到设备
    auto tensor_img = torch::from_blob(img_input.data, { 1, img_input.rows, img_input.cols, img_input.channels() }).to(device_type);
    // BHWC -> BCHW
    tensor_img = tensor_img.permute({ 0, 3, 1, 2 }).contiguous();  // BHWC -> BCHW (Batch, Channel, Height, Width)

    std::vector<torch::jit::IValue> inputs;
    // 在容器尾部添加一个元素，这个元素原地构造，不需要触发拷贝构造和转移构造
    inputs.emplace_back(tensor_img);

    torch::jit::IValue output = module.forward(inputs);

    // 解析结果
    auto detections = output.toTuple()->elements()[0].toTensor();
    auto result = PostProcessing(detections, pad_w, pad_h, scale, img.size(), conf_thres, iou_thres);
    // visualize detections
    if (true) {
        Demo(img, result, class_names);
        cv::waitKey(0);
    }
    return 1;
}

四、问题记录

我参考的是链接[1][2]代码，非常坑，[1][2]代码是一样的，也不知道谁抄谁的，代码中没有说明yolov5具体版本，而且有很多问题，不过还是感谢给了参考。

4.1、注释 detector.h中，注释如下头文件

//#include <c10/cuda/CUDAStream.h>
#//include <ATen/cuda/CUDAEvent.h>

4.2、错误： “std”: 不明确的符号

解决办法1：项目->属性->c/c++->语言->符合模式->选择否

(看清楚vs项目属性窗口对应的到底是Debug还是Release，血的教训！）

解决办法2：还有有个老哥给出的方法是，在std报错的地方改为："::std"，不推荐！

4.3、建议常被debug版本libtorch

libtorch中，执行到加载模型那一行代码，跳进libtorch库中的Assert，提示错误：AT_ASSERT(isTuple(), "Expected Tuple but got ", tagKind());（咱们是libtorch debug版本，还能跳到这一行，要是release，你都不知道错在哪里，所以常备debug版本，很有必要）

可能是你转模型的yolov5版本不是5.4，而是5.3、5.3.1、5.3、5.1；还有可能是你export.py脚本中没有按照上面设置。

参考：https://blog.csdn.net/weixin_42398658/article/details/111954760