<br>计算机爱好者<br>云计算<br>云存储<br>网络技术<br>非洲酋长<br>可爱的胖子<br>十年攻城狮

    <好记性不如烂笔头> usbredir-qemu架构usb重定向 SW64系统移植 过程记录

    [busuanzi]
    字数统计: 8.2k字   |   阅读时长≈ 44分

    目的

    把usb和uvc的优化支持移植到sw架构下

    关于sw

    https://baike.baidu.com/item/%E7%94%B3%E5%A8%81%E5%A4%84%E7%90%86%E5%99%A8/9468374?fr=aladdin

    • 指令集 ALPHA - RISC ?

    关于国产CPU指令集
    http://news.sohu.com/a/527655245_121290918

    第一个是申威CPU用使用的SW64。申威在早期获得了Alpha指令集的授权，使用的是Alpha指令集，但后来申威推出了100%自研的SW64指令集，放弃了原有的Alpha指令集。

    第二个是龙芯，龙芯最早期是获得了MIPS的授权，后来又在MIPS的基础上扩展形成了LoongISA，但到最近发布的3A5000后，龙芯放弃了MIPS指令集和LoongISA指令集，采用了100%自研的LoongArch指令集。

    我们将要移植的就是SW64的架构

    关于SW qemu

    内网 git@192.5.1.31:module_virtual/qemu-kvm-ksvd.git
    分支 8.1.8-1-sw

    调试服务器

    10.90.6.134 (后更换为 10.60.8.52)
    root/123123
    Linux node1 4.19.180-sw_64.ky3.kb2.sw_64 #1 SMP Thu Apr 28 09:31:24 CST 2022 sw_64 sw_64 sw_64 GNU/Linux

    经观察，usbredir包为0.7.1版本

    编译环境

    初始环境选择

    1 办公环境docker

    caotao已经进行过一个sw环境搭建，并且上传docker image到44
    尝试直接用docker在我本机启动

    1
    2
    docker pull 192.168.120.44/dev_builders/uos_sw_64:v1
    docker run -itd --network host --user $(id -u ygh):$(id -g ygh) --privileged=true -v /home/ygh/code/kylin:/home/uos/kylin -w /home/uos/kylin --name sw 192.168.120.44/dev_builders/uos_sw_64:v1 /bin/bash
    • 失败，原因：架构不同，我的pc是x86的，显然不是同一个CPU架构

    2 ssh到办公环境的SW架构系统

    尝试直接连接到caotao搭建的环境
    10.20.3.78
    uos/ksvd@KylinOs

    • 失败，原因：这个是客户端的，用来编译sw uos客户端一体化包的，系统是uos，不是我们需要的kylin系统

    3 ssh到编译环境已有的SW架构kylin系统

    尝试在编译环境里连接已经有的一套SW架构kylin系统
    192.5.1.205
    jenkins/kylinksvdypj

    可以编译，但是使用困难，代码的内外互通性差，编译环境卡，205本身资源紧缺等多方面原因

    4 尝试把编译环境已有的SW架构kylin系统 做成docker image

    打包命令记录

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    tar --numeric-owner -czvpf /tmp/system.tar \
    --directory=/  \
    --exclude=proc --exclude=sys --exclude=dev  \
    --exclude=run --exclude=boot --exclude=mnt  \
    --exclude=home/* --exclude=var/log  \
    --exclude=var/cache --exclude=var/lib/samba  \
    --exclude=usr/lib/ksvd --exclude=usr/lib/jvm  \
    --exclude=tmp --exclude=.readahead --exclude=root/*  \
    --exclude=1 --exclude=usr/local  \
    --exclude=usr/lib64/libwebkit2*  \
    --exclude=usr/lib64/webkit2*   \
    --exclude=usr/libexec/webkit2gtk-4.0   \
    --exclude=usr/share \
    .

    docker run -itd --network host --user $(id -u uos):$(id -g uos) --privileged=true -v /home/uos/ygh/:/home/jenkins/code -w /home/jenkins/code --name swtest1 test_sw /bin/bash

    docker run -it --network host -v /home/uos/ygh/:/root/code -w /root/code --name  test_sw /bin/bash
    • 因删减过多导致失败，办公/编译环境传输困难，不得不进行大量删减导致失败。

    5 搭建本地虚拟机环境搭建，安装SW架构kylin系统，然后导出做docker

    镜像

    首先尝试
    http://192.168.120.103/iso_download/?name=KylinSec-3.3-9A-2205-201636-sw_64.iso
    后改成
    由liqing提供的黄金镜像

    • 192.168.120.103 账号密码 zhujian/123123

    本地ubuntu环境 尝试 qemu+libvirt+virt-manager搭建

    1
    2
    3
    4
    5
    sudo apt install qemu
    sudo apt install qemu-system
    sudo apt install qemu-kvm
    sudo apt install libvirt-daemon libvirt-daemon-system
    sudo apt install virt-manager
    安装镜像

    sudo vim /etc/libvirt/libvirtd.conf 修改写权限
    重启libvirtd,

    • 失败 原因: RISC和ALPHA架构，没有对应的qemu，且后来查阅资料，当前的SW采用的是自研的SW64指令集，网上更不可能有对应的qemu了

    本地ubuntu环境 尝试 virt-manager 远程连接虚拟机

    峰回路转，发现我们已有的一台sw uos上，apt list中是有sw64的qemu-system的，可以尝试virtmanger远程连接进行尝试

    1
    2
    sudo apt-get install libvirt-daemon libvirt-daemon-system
    sudo apt-get install qemu-*
    • 把我自己主机的ssh pub导入到uos上（~/.ssh/authorized_keys）
    • 失败 原因: virt-manager连接后提示sw_64不可安装，依然只有导入功能

    采用公司sw架构的服务器上安装

    • 连接 10.90.6.134（没错，就是前面记录的测试服务器）

    • 导入指定ISO并尝试安装

    • 失败，原因：黄金镜像卡死（后经确认，系统要经过虚拟化兼容性移植，才可以用作虚拟机）

    • 尝试旧版本(之前已经做过虚拟化兼容性移植的系统)安装，通过

    导出docker image，并导入到办公环境的SW架构系统

    • 打包

      1
      tar --numeric-owner -czvpf /tmp/system.tar --directory=/ --exclude=proc --exclude=sys --exclude=dev --exclude=run --exclude=boot --exclude=mnt  --exclude=tmp .

    • 导出后把system.tar scp到10.20.3.78这台机器上（uos(sw)）

    • 导入

      1
      docker import ygh/sw_init.tar kylin_sw64:init

    • 启动

      1
      docker run -it --network host -v /home/uos/ygh/:/root/code -w /root/code --name ksw kylin_sw64:init bash

    • 测试系统正常, 保存最初版本, 上传为192.168.120.44/dev_builders/kylin_sw_64:init

    • 在此版本上增加软件，用来搭建编译环境，也可以在此版本基础上，卸载软件（如浏览器，图形界面，等编译不需要的软件），精简镜像

    通过docker启动导入的镜像
    1
    2
    3
    4
    docker run -itd --network host -v /home/uos/ygh/:/root/code -w /root/code --name ksw kylin_sw64:init bash
    #报错 docker: Error response from daemon: cannot share the host's network namespace when user namespaces are enabled.
    #原因是同事在/etc/docker/daemon.json中加了user_remap的配置，解决方法如下
    docker run -itd --userns=host --network host -v /home/uos/ygh/:/root/code -w /root/code --name ksw kylin_sw64:init bash
    • 问题记录
      docker每天重启，都找不到昨天的images
      后来发现，是因为userns_remap 指定了用户，
      导致docker默认目录(Docker Root Dir)改变
      /var/lib/docker ==> /var/lib/docker/$uid.$gid下面了
      /etc/subuid又修改了用户的uid导致 docker找不到指定目录
      https://man7.org/linux/man-pages/man5/subuid.5.html

    至此有了一个初步的环境。

    编译环境搭建(usbredir)

    修改源

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    [root@uos-PC code]# cat /etc/yum.repos.d/Kylin-Base.repo
    # Kylin-Base.repo
    #

    [base]
    name=KylinSecOS-$releasever - Base
    gpgcheck=0
    #baseurl=file:///run/media/$user/KylinSec/
    #baseurl=http://yumrepo.unikylin.com.cn/repos/$osversion/$basearch/base/latest/
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kylin-release
    enabled=0

    [addon]
    name=KylinSecOS-$releasever - Addon
    gpgcheck=0
    #baseurl=file:///run/media/$user/KylinSec-Addon/
    #baseurl=http://yumrepo.unikylin.com.cn/repos/$osversion/$basearch/base/latest/
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-kylin-release
    enabled=0

    [kylin]
    name=Kylin
    gpgcheck=0
    baseurl=http://192.168.120.17/kojifiles/repos/KY3.3-9-SW64-build/latest/sw_64/
    enabled=1
    1
    2
    yum clean all
    yum makecache

    安装依赖

    1
    2
    3
    4
    # 下载rpm
    yum install rpm-build
    # 下载libusbx
    yum install libusbx-devel

    编译

    1
    2
    3
    4
    # 需要一个tmp目录
    mkdir /tmp
    # 编译
    bash mkrpm.sh

    • 报错

      1
      2
      3
      4
      5
      checking whether we are cross compiling... configure: error: in `/root/rpmbuild/BUILD/usbredir-0.7.1':
      configure: error: cannot run C++ compiled programs.
      If you meant to cross compile, use `--host'.
      See `config.log' for more details
      错误：/var/tmp/rpm-tmp.srPjar (%build) 退出状态不好

    • 查看发现sw环境下虽然没有源码，但是有编译过

      1
      2
      3
      4
      5
      6
      7
      8
      9
      [root@uos-PC code]# yumdownloader --source usbredir
      已加载插件：langpacks
      No source RPM found for usbredir-0.7.1-3.ky3.sw_64
      Nothing to download
      [root@uos-PC code]# rpm -qa | grep usbredir
      [root@uos-PC code]# yum list | grep usbredir
      usbredir.sw_64                         0.7.1-3.ky3                 kylin        
      usbredir-devel.sw_64                   0.7.1-3.ky3                 kylin        
      usbredir-server.sw_64                  0.7.1-3.ky3                 kylin

    • 下载编译过的版本进行对比
      http://192.168.120.17/koji/buildinfo?buildID=61887
      下载的源码，进行编译
      rpm -ivh
      然后到/root/rpmbuild/SPECS目录下 rpmbuild -bb usbredir.spec

    也会报一样的错误，所以问题应该出在环境上

    • 查看config.log

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      configure:3289: checking whether we are cross compiling
      configure:3297: sw_64sw6a-sunway-linux-gnu-gcc -o conftest    conftest.c  >&5
      In file included from /usr/include/stdio.h:27:0,
                       from conftest.c:11:
      /usr/include/features.h:368:12: fatal error: sys/cdefs.h: No such file or directory
       #  include <sys/cdefs.h>
                  ^~~~~~~~~~~~~
      compilation terminated.
      configure:3301: $? = 1
      configure:3308: ./conftest
      ./configure: line 3310: ./conftest: No such file or directory
      configure:3312: $? = 127
      configure:3319: error: in `/root/rpmbuild/BUILD/usbredir-0.7.1':
      configure:3321: error: cannot run C compiled programs.
      If you meant to cross compile, use `--host'.
      See `config.log' for more details

    • 解决方法，下载glibc

      1
      yum install glibc-headers

    • 继续编译，还有明显报错没有x264

      1
      yum install x264-devel

    • 至此编译通过，但是我们注意到一个问题

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      checking for native Win32... host is sw_64sw6a-sunway-linux-gnu
      linux
      checking for code... checking for x264_encoder_encode in -lx264... yes
      x264 enable
      checking for mpp_init in -lrockchip_mpp... no
      code is x264
      x264
      checking for srvdecode... checking for avcodec_find_decoder in -lavcodec... no
      decode is 

      checking for sw_64sw6a-sunway-linux-gnu-pkg-config... no

    • 这里服务端decode显示是空,表示服务端没有开启解码功能的，查看configure.ac

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      AC_MSG_CHECKING([for srvdecode])
      AC_CHECK_LIB(avcodec, avcodec_find_decoder, [
              AC_CHECK_LIB(avformat, av_register_all, [
                      AC_CHECK_FILE([/usr/include/ffmpeg], [
                      echo "avcodec enable"
                      decode=enable
                      ])
              ])
      ])
      echo "decode is $decode"
      AC_MSG_RESULT([$decode])
      AM_CONDITIONAL([DECODE_AV],[test "$decode" = "enable"])

    • x86环境下，看下库是哪个包

      1
      2
      [System not activated][root@node2 ~]# rpm -qf /usr/lib64/libavcodec.so
      ffmpeg-devel-3.3.2-1.ky3.kb2.x86_64

    • 解决方法

      1
      yum install ffmpeg-devel

    • 这次编译通过，应该没有什么问题了

    编译环境搭建(qemu)

    安装依赖

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    # mkrpm.sh中依赖
    yum install git

    # 安装依赖包(spec发现)
    yum install bison libseccomp-devel libcap-devel libattr-devel flex libxkbcommon-devel
    yum install spice-server-devel spice-protocol

    # 安装依赖包(configure发现)
    yum install gnutls-devel
    yum install libgcrypt-devel.sw_64

    编译

    • 问题:目录不存在

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      install -d -m 0755 "/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/qemu/keymaps"
      set -e; for x in da     en-gb  et  fr     fr-ch  is  lt  modifiers  no  pt-br  sv ar      de     en-us  fi  fr-be  hr     it  lv  nl         pl  ru     th common  de-ch  es     fo  fr-ca  hu     ja  mk  nl-be      pt  sl     tr bepo    cz; do \
      	install -c -m 0644 /root/code/qemu-kvm-kylin/RPMFILE/cut/build/qemu-sw/pc-bios/keymaps/$x "/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/qemu/keymaps"; \
      done
      install -c -m 0644 /root/code/qemu-kvm-kylin/RPMFILE/cut/build/qemu-sw/trace-events-all "/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/qemu/trace-events-all"
      for d in sw64-softmmu; do \
      make --no-print-directory --quiet BUILD_DIR=/root/code/qemu-kvm-kylin/RPMFILE/cut/build/qemu-sw TARGET_DIR=$d/ -C $d install || exit 1 ; \
              done
      + cp pc-bios/core3-hmcode pc-bios/core3-reset pc-bios/core4-hmcode /root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/qemu
      + install -m 0644 /root/code/qemu-kvm-kylin/RPMFILE/cut/root/uefi-bios-sw /root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/share/qemu/
      + /usr/lib/rpm/check-buildroot
      + /usr/lib/rpm/sunway/brp-compress
      + /usr/lib/rpm/sunway/brp-strip /usr/bin/strip
      /usr/bin/strip: Unable to recognise the format of the input file `/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/qemu/core4-hmcode'
      + /usr/lib/rpm/sunway/brp-strip-comment-note /usr/bin/strip /usr/bin/objdump
      + /usr/lib/rpm/sunway/brp-strip-static-archive /usr/bin/strip
      + /usr/lib/rpm/brp-python-bytecompile /usr/bin/python 1
      + /usr/lib/rpm/sunway/brp-python-hardlink
      + /usr/lib/rpm/sunway/brp-java-repack-jars
      处理文件：qemu-3.1.0-2.sw2109.ky3.kb3.sw_64
      Provides: qemu = 3.1.0-2.sw2109.ky3.kb3 qemu(sw_64-64) = 3.1.0-2.sw2109.ky3.kb3
      Requires(rpmlib): rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
      处理文件：qemu-common-3.1.0-2.sw2109.ky3.kb3.sw_64
      错误：没有找到目录：/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/locale


      RPM 构建错误：
          没有找到目录：/root/code/qemu-kvm-kylin/RPMFILE/cut/ksvd/usr/local/share/locale

    • 查找发现目录是在make po的时候创建的

      1
      2
      cut/build/qemu-sw/po/Makefile:40:	    $(INSTALL) -d $(DESTDIR)$(prefix)/share/locale/$$base/LC_MESSAGES; \
      cut/build/qemu-sw/po/Makefile:41:	    $(INSTALL) -m644 $$obj $(DESTDIR)$(prefix)/share/locale/$$base/LC_MESSAGES/qemu.mo; \

    • 和jenkins环境对比，我这里没有执行

      1
      make -C po install

    • 查看Makefile文件

      1
      2
      3
      4
      # qemu-sw/Makefile
      ifdef CONFIG_GTK
              $(MAKE) -C po $@
      endif

    • 查看configure文件

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      # qemu-sw/configure
      gtk=""
      #...
        ;;
        --enable-gtk) gtk="yes"
      #...
      if test "$gtk" = "yes" ; then
        echo "CONFIG_GTK=m" >> $config_host_mak
        echo "GTK_CFLAGS=$gtk_cflags" >> $config_host_mak
        echo "GTK_LIBS=$gtk_libs" >> $config_host_mak
        if test "$gtk_gl" = "yes" ; then
          echo "CONFIG_GTK_GL=y" >> $config_host_mak
        fi
      fi

    都是使用run_kvm_config.sh
    我的configure结果

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    target list       sw64-softmmu
    gprof enabled     no
    sparse enabled    no
    strip binaries    no
    profiler          no
    static build      no
    SDL support       no 
    GTK support       no 
    GTK GL support    no
    VTE support       no

    jenkins的configure结果

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    target list       sw64-softmmu
    gprof enabled     no
    sparse enabled    no
    strip binaries    no
    profiler          no
    static build      no
    SDL support       yes (1.2.15)
    GTK support       yes (3.22.30)
    GTK GL support    no
    VTE support       no

    执行 yum install gtk3-devel 解决这个问题（SDL先且不管他）

    • 编译通过

    合代码

    合入usb重定向改动，来支持摄像头

    • 由于819改动较多，穿插其他人改动，难以直接merge，先进行尝试

    先在819代码下执行

    1
    tar -cvf 819.file redirect.* ksvd_redirect.c hook_* scsi.* fs_fat.h stat.h

    打包usb改动主要相关文件

    • 记录下Makefile.obj的改动

      1
      common-obj-$(CONFIG_USB_REDIR) += ksvd_redirect.o hook_scsi.o hook_uvc.o hook_net.o hook_udev.o hook_stream.o quirks.o scsi.o

    • 记录下configure的改动

      1
      2
      3
      4
      5
      6
      7
      8
      # check for usbredirparser for usb network redirection support
      if test "$usb_redir" != "no" ; then
          if $pkg_config --atleast-version=0.6 libusbredirparser-0.5; then
              usb_redir="yes"
              usb_redir_cflags=$($pkg_config --cflags libusbredirparser-0.5)
              usb_redir_libs=$($pkg_config --libs libusbredirparser-0.5)
              usb_redir_libs+=$($pkg_config --libs libavcodec libavdevice libavfilter libavformat libavutil libswresample libpostproc libswscale)
              usb_redir_libs+=" -luvccli -lusbparse"

    • 切换到sw分支，解压819.file

      1
      tar -xvf 819.file

    • 对比唯一的改动文件,发现改动不大

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      ygh@ygh:~/code/kylin/qemu-kvm-kylin/hw/usb$ git diff redirect.c
      diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
      index 18a42d19..8064a94f 100644
      --- a/hw/usb/redirect.c
      +++ b/hw/usb/redirect.c
      @@ -26,7 +26,6 @@
        */
       
       #include "qemu/osdep.h"
      -#include "qemu/units.h"
       #include "qapi/error.h"
       #include "qemu-common.h"
       #include "qemu/timer.h"
      @@ -1299,7 +1298,7 @@ static int usbredir_chardev_can_read(void *opaque)
           }
       
           /* usbredir_parser_do_read will consume *all* data we give it */
      -    return 1 * MiB;
      +    return 1024 * 1024;
       }
       
       static void usbredir_chardev_read(void *opaque, const uint8_t *buf, int size)

    • 分析后，对其进行回退，这两处改动，都是不需要的

    • 修改Makefile.obj后，尝试编译

    • 解决缺少的文件，从对应版本下载并加入的到该分支（最终更新前文记录的tar参数中，这里就不再记录了）

    • 编译通过

    修改uniqb

    • 记录改动
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    # da852f0c31a7a627b4b33b875bfaa1a8ffde9d1d
    diff --git a/KSVD/src/uniqb-runtime/main.cpp b/KSVD/src/uniqb-runtime/main.cpp
    index 86f7432..17aa388 100644
    --- a/KSVD/src/uniqb-runtime/main.cpp
    +++ b/KSVD/src/uniqb-runtime/main.cpp
    @@ -3901,7 +3901,7 @@ void set_kvm_args(Launcher &kvm)
                                    if (cfg.usb_degrade) {
                                            kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d,bus=usb.0,port=%d", i, cfg.usb_filter_degrade_for_qemu, cfg.usb
                                    } else {
    -                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d,bus=usb.0,port=%d", i, cfg.usb_filter_json_path, cfg.usb_d
    +                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d,bus=usb.0,port=%d", i, cfg.usb_filter_json_path, cfg.usb_
                                    }
                            }
                    } else {
    @@ -3913,7 +3913,7 @@ void set_kvm_args(Launcher &kvm)
                                    if (cfg.usb_degrade) {
                                            kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d", i, cfg.usb_filter_degrade_for_qemu, cfg.usb_degrade, i);
                                    } else {
    -                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d", i, cfg.usb_filter_json_path, cfg.usb_degrade, i);
    +                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d", i, cfg.usb_filter_json_path, cfg.usb_degrade, i);
                                    }
                            }
                    }
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    # 6eca69a13773f25af89cb9e83b9f3152c06946de
    diff --git a/KSVD/src/uniqb-runtime/main.cpp b/KSVD/src/uniqb-runtime/main.cpp
    index 17aa388..121994e 100644
    --- a/KSVD/src/uniqb-runtime/main.cpp
    +++ b/KSVD/src/uniqb-runtime/main.cpp
    @@ -3126,10 +3126,10 @@ void set_kvm_args(Launcher &kvm)
                            kvm.push("-device", "virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0");
                            kvm.push("-chardev", "spicevmc,name=vdagent,id=vdagent");
                            kvm.push("-device", "virtserialport,bus=virtio-serial0.0,nr=1,chardev=vdagent,name=com.redhat.spice.0");
    -                       kvm.push("-device", "nec-usb-xhci,id=usb,bus=pci.7,addr=0x0");
    -                       kvm.push("-device", "usb-tablet,id=input0,bus=usb.0,port=1");
    -                       kvm.push("-device", "usb-mouse,id=input1,bus=usb.0,port=2");
    -                       kvm.push("-device", "usb-kbd,id=input2,bus=usb.0,port=3");
    +                       kvm.push("-device", "nec-usb-xhci,id=usb1,bus=pci.7,addr=0x0");
    +                       kvm.push("-device", "usb-tablet,id=input0,bus=usb1.0,port=1");
    +                       kvm.push("-device", "usb-mouse,id=input1,bus=usb1.0,port=2");
    +                       kvm.push("-device", "usb-kbd,id=input2,bus=usb1.0,port=3");
                    }
            }
     
    @@ -3864,11 +3864,13 @@ void set_kvm_args(Launcher &kvm)
     //             }
     
                    if (3 == usb_mode) {
    -                       if (m_arch_type != ARCH_TYPE_MIPS64) {
    -                               kvm.push("-device", "nec-usb-xhci,id=xhci");
    -                       } else {
    -                               kvm.push("-device", "nec-usb-xhci,id=xhci,bus=pci,addr=0x6");
    +                       if (m_arch_type == ARCH_TYPE_MIPS64) {
    +                               kvm.push("-device", "nec-usb-xhci,id=redirhci,bus=pci,addr=0x6");
                                    kvm.push("-device", "usb-host,id=hostdev0,bus=xhci.0");
    +                       } else if (m_arch_type == ARCH_TYPE_SW_64){
    +                               kvm.push("-device", "nec-usb-xhci,id=redirhci,bus=pci.8");
    +                       }else {
    +                               kvm.push("-device", "nec-usb-xhci,id=redirhci");
                            }
                            if (m_arch_type == ARCH_TYPE_X86_64) {
                                    if (cfg.os_type == UNIQB_MACHTYPE_OS_WIN10  &&  runmode_installing) {
    @@ -3879,46 +3881,47 @@ void set_kvm_args(Launcher &kvm)
                            }
                    } else {
                            if (m_arch_type == ARCH_TYPE_MIPS64) {
    -                               kvm.push("-device", "ich9-usb-ehci1,id=usb,bus=pci,addr=0x4.0x7");
    +                               kvm.push("-device", "ich9-usb-ehci1,id=redirhci,bus=pci,addr=0x4.0x7");
                                    kvm.push("-device", "ich9-usb-uhci1,masterbus=usb.0,firstport=0,multifunction=on,addr=0x4");
                                    kvm.push("-device", "ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci,addr=0x4.0x1");
                                    kvm.push("-device", "ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci,addr=0x4.0x2");
                                    kvm.push("-device", "usb-tablet,id=input0,bus=usb.0,port=1");
                                    kvm.push("-device", "usb-kbd,id=input1,bus=usb.0,port=2");
    -                       } else {
    -                               kvm.push("-device", "ich9-usb-ehci1,id=usb");
    +                       } else if (m_arch_type == ARCH_TYPE_SW_64) {
    +                               kvm.push("-device", "ich9-usb-ehci1,id=redirhci,bus=pci.8");
    +                               kvm.push("-device", "ich9-usb-uhci1,masterbus=usb.0,firstport=0,multifunction=on");
    +                               kvm.push("-device", "ich9-usb-uhci2,masterbus=usb.0,firstport=2");
    +                               kvm.push("-device", "ich9-usb-uhci3,masterbus=usb.0,firstport=4");
    +                       }else {
    +                               kvm.push("-device", "ich9-usb-ehci1,id=redirhci");
                                    kvm.push("-device", "ich9-usb-uhci1,masterbus=usb.0,firstport=0,multifunction=on");
                                    kvm.push("-device", "ich9-usb-uhci2,masterbus=usb.0,firstport=2");
                                    kvm.push("-device", "ich9-usb-uhci3,masterbus=usb.0,firstport=4");
                            }
                    }
    -               INFO_("KVM USB filter: " << cfg.usb_filter_json_path);
    +               INFO_("KVM USB filter: " << cfg.usb_filter_json_path << " degrade:" << cfg.usb_degrade);
                    if (m_arch_type == ARCH_TYPE_MIPS64 && usb_mode  == 2) {
                            kvm.push("-chardev", "spicevmc,name=usbredir,id=usbredirchardev0");
                            // usb controler can only support 6 devices, usb-tablet and usb-kbd occupy 2, only leave 4 for usb devices
                            for (int i = 1; i <= 4; i++) {
                                    kvm.push("-chardev", "spicevmc,name=usbredir,id=usbredirchardev%d", i);
                                    if (cfg.usb_degrade) {
    -                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d,bus=usb.0,port=%d", i, cfg.usb_filter_degrade_for_qemu, cfg.usb
    +                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d,bus=redirhci.0,port=%d", i, cfg.usb_filter_degrade_for_qemu, cf
                                    } else {
    -                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d,bus=usb.0,port=%d", i, cfg.usb_filter_json_path, cfg.usb_
    +                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d,bus=redirhci.0,port=%d", i, cfg.usb_filter_json_path, cfg
                                    }
                            }
                    } else {
                            for (int i = 1; i <= usb_num; i++) {
                                    kvm.push("-chardev", "spicevmc,name=usbredir,id=usbredirchardev%d", i);
    -                               if (m_arch_type == ARCH_TYPE_SW_64) {
    -                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,id=usbredirdev%d", i, cfg.usb_filter_degrade_for_qemu, i);
    -                               } else {
                                    if (cfg.usb_degrade) {
    -                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d", i, cfg.usb_filter_degrade_for_qemu, cfg.usb_degrade, i);
    +                                       kvm.push("-device", "usb-redir,chardev=usbredirchardev%d,filter=%s,usb_degrade=%d,id=usbredirdev%d,bus=redirhci.0", i, cfg.usb_filter_degrade_for_qemu, cfg.usb_de
                                    } else {
    -                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d", i, cfg.usb_filter_json_path, cfg.usb_degrade, i);
    +                                       kvm.push("-device", "ksvd_redir,chardev=usbredirchardev%d,filter-path=%s,usb_degrade=%d,id=usbredirdev%d,bus=redirhci.0", i, cfg.usb_filter_json_path, cfg.usb_deg
                                    }
                            }
                    }
            }
    -       }
            if (cfg.kvm210 && ret){
                    set_kvm_uvcs_args(kvm);
            }

    ycmd编译

    • 配置
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      "uniqb-runtime_sw_b":{
          "ref":"view/申威KSVD编译/job/uniqb_back_module_shenwei/build?delay=0sec",
          "data":{
              "parameter":[
              {"name": "USER_NAME", "value": "ygh"}, 
              {"name": "BRANCH", "value": "8.1.8-1-sw"}, 
              {"name": "COMMON_USER_NAME", "value": "kycd_dev"}, 
              {"name": "COMMON_SUBMODULE_BRANCH", "value": "8.1.8-sw"}, 
              {"name": "INIT_WORKSPACE","value":true}, 
              {"name": "DEBUG"}, 
              {"name": "TARGET", "value": "uniqb-runtime"}
              ]
          }
      },
    • 名令
      1
      ycmd.py host.hostByName[\'jenkins_30\'].web getPageData uniqb-runtime_sw_b

    镜像精简

    • 初始镜像过大，对于后续保存，复用都不方便，所以对其进行精简
    • 卸载大部分不用的包
    • 保存精简后的镜像为 192.168.120.44/dev_builders/uos_sw_64:simple
    • 为了编译，又重新安装了一些包需要加装的包
    • 保存安装编译需求后的包为 192.168.120.44/dev_builders/uos_sw_64:v3

    调试

    由于之前的服务器被拆除
    这里更换了测试服务器为如下

    • 服务器 SW_S_TEST 10.60.8.50 后更换为 10.60.8.52

    编译qemu并安装

    编译环境下编译

    1
    2
    3
    4
    cd qemu-kvm-kylin/RPMFILE/
    rm /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* -f
    bash mkrpm.sh
    scp /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* root@10.60.8.52:/home/ygh/debug/

    测试环境下安装(该方法不影响别人使用，可以同步调试，后面有介绍)

    1
    2
    cd /home/ygh/debug/
    rpm2cpio qemu-system-sw64-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm | cpio -div

    编译usbreidr并安装

    编译环境下编译

    1
    2
    3
    4
    cd usbredir/RPMFILE/
    rm /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* -f
    bash mkrpm.sh
    scp /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* root@10.60.8.52:/home/ygh/debug/

    测试环境下安装(该方法不影响别人使用，可以同步调试，后面有介绍)

    1
    2
    cd /home/ygh/debug/
    rpm2cpio usbredir-0.7.1-79.ky3.kb7.sw_64.rpm | cpio -div

    快速编译脚本(usbredir+qemu)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    cd usbredir/RPMFILE/
    rm /root/rpmbuild/RPMS/sw_64/* -f
    bash mkrpm.sh
    cd -
    echo "mv to dst"
    mv /root/rpmbuild/RPMS/sw_64/* dst/


    cd qemu-kvm-kylin/RPMFILE/
    rm /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* -f
    bash mkrpm.sh
    cd -
    echo "mv to dst"
    mv /root/code/qemu-kvm-kylin/RPMFILE/sw_64/* dst/

    编译uniqbruntime

    • 这里采用april辅助工具（自研<工欲善其事，必先利其器> april）避免频繁网页选择操作
      1
      2
      3
      4
      5
      6
      7
      8
      9
      # 编译
      ycmd.py host.hostByName[\'jenkins_30\'].web getPageData uniqb-runtime_sw_b

      # 在网页上确认编译完成
      # 获取
      ycmd.py host.hostByName[\'jenkins_30\'].web getPageData uniqb-runtime_sw_g ./uniqb-runtime

      # 放在服务器
      ycmd.py host.hostByName[\"SW_S_TEST\"].ssh putFile ./uniqb-runtime /home/ygh/debug/usr/lib/ksvd/bin/

    多人复用调试

    • home
      • ygh
        • debug
          • usr/lib/ksvd/bin/uniqb-runtime
          • usr/local/bin/qemu-system-sw64
          • usr/lib/libusbredir*.so*
        • start.sh (管理单元)
        • hook.sh (hook脚本)
    • 查看影响的文件
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      [Service not authorized][root@node2 ygh]# bash start.sh -l
      kylindebug12581 is exist
      already hookdir
      /usr/local/bin
      /usr/local/share/qemu
      /usr/lib/ksvd/bin
      src:
      /home/kylindebug12581/bindir.txt[unknown  ASCII text]
      /home/kylindebug12581/libdir.txt[unknown  ASCII text]
      /home/kylindebug12581/usr/lib/ksvd/bin/uniqb-runtime[executable]
      /home/kylindebug12581/usr/local/bin/qemu-system-sw64[executable]
      /home/kylindebug12581/usr/local/share/qemu/palcode-clipper[executable]
      user:
      /home/ygh/debug/usbredir-server-0.7.1-79.ky3.kb7.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 usbredir-server-0.7.1-79.ky3.kb7]
      /home/ygh/debug/usbredir-debuginfo-0.7.1-79.ky3.kb7.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 usbredir-debuginfo-0.7.1-79.ky3.kb7]
      /home/ygh/debug/qemu-user-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-user-3.1.0-6.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-3.1.0-6.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-user-3.1.0-7.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-user-3.1.0-7.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-system-sw64-3.1.0-10.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-system-sw64-3.1.0-10.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-3.1.0-7.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-3.1.0-7.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-system-sw64-3.1.0-7.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-system-sw64-3.1.0-7.sw2109.ky3.kb3]
      /home/ygh/debug/usbredir-0.7.1-79.ky3.kb7.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 usbredir-0.7.1-79.ky3.kb7]
      /home/ygh/debug/qemu-common-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-common-3.1.0-6.sw2109.ky3.kb3]
      /home/ygh/debug/qemu-system-sw64-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-system-sw64-3.1.0-6.sw2109.ky3.kb3]
      /home/ygh/debug/usbredir-0.7.1-82.ky3.kb7.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 usbredir-0.7.1-82.ky3.kb7]
      /home/ygh/debug/usbredir-devel-0.7.1-79.ky3.kb7.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 usbredir-devel-0.7.1-79.ky3.kb7]
      /home/ygh/debug/qemu-common-3.1.0-7.sw2109.ky3.kb3.sw_64.rpm[unknown  RPM v3.0 bin Alpha/Sparc64 qemu-common-3.1.0-7.sw2109.ky3.kb3]
      /home/ygh/debug/usr/lib/libusbredirparser.so.1.0.0[sharedobj]
      /home/ygh/debug/usr/lib/libusbparse.so.0.0.0[sharedobj]
      /home/ygh/debug/usr/lib/ksvd/bin/uniqb-runtime[executable]
      /home/ygh/debug/usr/lib/libuvcsrv.so.0.0.0[sharedobj]
      /home/ygh/debug/usr/lib/libusbredirhost.so.1.0.0[sharedobj]
      /home/ygh/debug/usr/lib/libuvcproxy.so.0.0.0[sharedobj]
      /home/ygh/debug/usr/lib/libuvccli.so.0.0.0[sharedobj]
      /home/ygh/debug/usr/local/bin/qemu-system-sw64[executable]
      /home/ygh/debug/usr/local/share/qemu/palcode-clipper[executable]
      /home/ygh/debug/usr/share/doc/usbredir-0.7.1/README[unknown  ASCII text]
      /home/ygh/debug/usr/share/doc/usbredir-0.7.1/COPYING.LIB[unknown  ASCII text]
      /home/ygh/debug/usr/share/doc/usbredir-0.7.1/TODO[unknown  ASCII text]
      /home/ygh/debug/usr/share/doc/usbredir-0.7.1/ChangeLog[unknown  ASCII text]

    文件快速传输废弃思路

    • 尝试把本机id_rsa.pub 给服务器 .ssh/
    • scp -r uos@10.20.3.78:~/ygh/dst/ ./
    • scp -r root@10.60.8.52:/home/ygh/debug/
    • 废弃原因 服务器authorized_keys一直在变化，只能放弃了

    文件快速传输替换思路

    • 使用april工具进行 跨主机cp
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    # init copy hook工具
    ycmd.py host.hostByName[\"SW_S_TEST\"].ssh putFile /home/ygh/code/crazyboy/shtool/hook/start.sh /home/ygh/start.sh
    ycmd.py host.hostByName[\"SW_S_TEST\"].ssh putFile /home/ygh/code/crazyboy/shtool/hook/hook.sh /home/ygh/hook.sh
    # debug 每次修改，copy usbredir和qemu包，78的authorized_keys不会改变，直接使用scp
    scp -r uos@10.20.3.78:~/ygh/dst/ ./ 
    ycmd.py host.hostByName[\"SW_S_TEST\"].ssh putFile /home/ygh/dst/ /home/ygh/debug/
    # SW_S_TEST上执行 
    # 把工具放到ygh目录下
    cd /home/ygh
    mv hook* ./
    # 把要调试的两个包，解压
    rpm2cpio usbredir-0.7.1-79.ky3.kb7.sw_64.rpm | cpio -div
    rpm2cpio qemu-system-sw64-3.1.0-6.sw2109.ky3.kb3.sw_64.rpm | cpio -div
    # 开始hook
    ./start.sh s
    # 检查hook文件
    ./start.sh l

    • 为了方便调试，首先建立资源，让虚拟机确认运行在 SW_S_TEST 服务器上

    问题记录

    libavdevice.so.57 问题

    1
    error while loading shared libraries: libavdevice.so.57: cannot open shared object file: No such file or directory
    • 安装 ffmpeg 包

    驱动支持问题

    usb设备qemu显示attach。但是虚拟机里面没有设备插入信息
    查看发现，应该链HCD都没进去
    怀疑和SW服务器的PCI排列有关
    怀疑SW 64 不支持EHCI/UHCI
    怀疑SW 64 kylin系统，不支持uvc驱动
    已经确认，确实是不支持导致

    摄像头无法工作问题

    确认时内核没有包含uvc模块
    需要重新编译内核

    编译内核

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    [Service not authorized][root@node2 SPECS]# rpmbuild -bp kernel-alt.spec 
    error: Failed build dependencies:
    	redhat-rpm-config >= 1.0.0 is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	xmlto is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	asciidoc is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	bison is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	xz-lzma-compat is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	hmaccalc is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	python-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	newt-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	perl(ExtUtils::Embed) is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	elfutils-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	binutils-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	audit-libs-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	numactl-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	pciutils-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64
    	ncurses-devel is needed by kernel-alt-4.19.90-2003.4.0.0036.ky3.kb27.sw_64

    • yum install redhat-rpm-config 失败

    • 解决

      1
      2
      3
      yum install xmlto asciidoc bison xz-lzma-compat hmaccalc python-devel newt-devel perl perl-ExtUtils-Embed elfutils-devel binutils-devel audit-libs-devel numactl-devel pciutils-devel ncurses-devel

      rpmbuild -bp kernel-alt.spec  --nodeps

    • 配置
      Device Drivers —>
      <> Multimedia support —>
      [      ] Cameras/video grabbers support
      [] Analog TV support
      [ ] Digital TV support (NEW)
      [      ] AM/FM radio receivers/transmitters support
      [] Software defined radio support
      [      ] USB support —>
      <> USB Gadget Support —>
      <*> USB Gadget functions configurable through configfs
      [      ] USB Webcam function

    • load configs下面的sw的config | load arm/sw/configs下面的 config | cp arch/sw_64/configs/guest_defconfig ./.config -f

    • 按照依赖，配置完成 uvc

    • 生成的.confg cp到configs目录下对应的sw

    • 在头部加上 # sw_64

    • 修改spec文件，条过prep步骤（避免重新解压覆盖修改）

    • rpmbuild -bb xx.spec完成打包

    1
    rpmbuild -bb kernel-alt.spec --nodeps
    • undefined reference to `ima_load_digest_blobs’

    尝试打开
    [] Digital signature verification using multiple keyrings
    [    ] Integrity Measurement Architecture(IMA)
    [] Enable multiple writes to the IMA policy
    [    ] Enable reading back the current IMA policy

    • error: implicit declaration of function ‘ima_digest_upload’ [-Werror=implicit-function-declaration]
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      #ifdef CONFIG_IMA_APPRAISE
      int ima_appraise_measurement(enum ima_hooks func,
                                   struct integrity_iint_cache *iint,
                                   struct file *file, const unsigned char *filename,
                                   struct evm_ima_xattr_data *xattr_value,
                                   int xattr_len, const struct modsig *modsig);
      int ima_must_appraise(struct inode *inode, int mask, enum ima_hooks func);
      void ima_update_xattr(struct integrity_iint_cache *iint, struct file *file);
      enum integrity_status ima_get_cache_status(struct integrity_iint_cache *iint,
                                                 enum ima_hooks func);
      enum hash_algo ima_get_hash_algo(struct evm_ima_xattr_data *xattr_value,
                                       int xattr_len);
      int ima_read_xattr(struct dentry *dentry,
                         struct evm_ima_xattr_data **xattr_value);

      int ima_digest_upload(char *data, size_t datalen);
      int ima_parse_compact_list(u8 *buf, size_t len, bool insert);
      struct ima_digest *ima_lookup_digest(u8 *digest);
      void ima_init_digest(void);
      #else
      [] Integrity Measurement Architecture(IMA)
      [      ] Appraise integrity measurements
      [*] ima_appraise boot parameter (NEW)
    总结

    依照menuconfig依赖关系，以此打开下列三个配置

    • CONFIG_USB_VIDEO_CLASS
    • IMA
    • IMA_PARSE
    • security/integrity/ima/ima_digest.c文件
      引入了vmalloc 头文件

    ioerror

    1
    libv4l2 error turning on stream 输入/输出错误

    echo 0xFFFF > /sys/module/uvcvideo/parameters/trace 开启uvc日志

    dmesg 发现关键日志

    1
    2
    Device requested 2400 B/frame bandwidth
    No fast enough alt setting for requested bandwidth

    对应代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    for (i = 0; i < intf->num_altsetting; ++i) {
            struct usb_host_interface *alts;
            unsigned int psize;

            alts = &intf->altsetting[i];
            ep = uvc_find_endpoint(alts,
                    stream->header.bEndpointAddress);
            if (ep == NULL)
                    continue;

            /* Check if the bandwidth is high enough. */
            psize = uvc_endpoint_max_bpi(stream->dev->udev, ep);
            if (psize >= bandwidth && psize <= best_psize) {
                    altsetting = alts->desc.bAlternateSetting;
                    best_psize = psize;
                    best_ep = ep;
            }
    }

    if (best_ep == NULL) {
            uvc_trace(UVC_TRACE_VIDEO, "No fast enough alt setting "
                    "for requested bandwidth.\n");
            return -EIO;
    }

    检查发现设备为 HIGH_SPEED（2.0）
    HIGH_SPEED bandwidth 算法为 usb_endpoint_maxp * usb_endpoint_maxp_mult
    如下

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    #define USB_ENDPOINT_MAXP_MASK  0x07ff
    #define USB_EP_MAXP_MULT_SHIFT  11 
    #define USB_EP_MAXP_MULT_MASK   (3 << USB_EP_MAXP_MULT_SHIFT) //0x3 << 11 = 0x1800
    #define USB_EP_MAXP_MULT(m) \
            (((m) & USB_EP_MAXP_MULT_MASK) >> USB_EP_MAXP_MULT_SHIFT)

    //...
    static inline int usb_endpoint_maxp(const struct usb_endpoint_descriptor *epd)
    {
            return __le16_to_cpu(epd->wMaxPacketSize) & USB_ENDPOINT_MAXP_MASK;
    }
    //...
    static inline int
    usb_endpoint_maxp_mult(const struct usb_endpoint_descriptor *epd)
    {
            int maxp = __le16_to_cpu(epd->wMaxPacketSize);

            return USB_EP_MAXP_MULT(maxp) + 1;
    }
    //...
    static unsigned int uvc_endpoint_max_bpi(struct usb_device *dev,
                                             struct usb_host_endpoint *ep)
    {
            u16 psize;
            u16 mult;

            switch (dev->speed) {
            case USB_SPEED_SUPER:
            case USB_SPEED_SUPER_PLUS:
                    return le16_to_cpu(ep->ss_ep_comp.wBytesPerInterval);
            case USB_SPEED_HIGH:
                    psize = usb_endpoint_maxp(&ep->desc);
                    mult = usb_endpoint_maxp_mult(&ep->desc);
                    return psize * mult;
            case USB_SPEED_WIRELESS:
                    psize = usb_endpoint_maxp(&ep->desc);
                    return psize;
            default:
                    psize = usb_endpoint_maxp(&ep->desc);
                    return psize;
            }
    }

    查看设备，最大Endpoint的MxPS = 1023 = 0x3FF
    即，最大计算结果为 1023,确实无法满足日志中的2400

    那么进一步查看2400哪里来的

    1
    2
    //uvc_video.c
                    bandwidth = stream->ctrl.dwMaxPayloadTransferSize;

    2400控制请求所带，没问题
    在qemu中查看，却发现,最大MxPS和系统里显示的1023 不同为1400
    系统显示为
    128 256 800 800 800 1023

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    UVC_VS_Interface[0x1238a55a0][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 1|bNumEndpoints: 1
     UVC_VS_Endpoint[0x1238a56a0][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize:  80|bInterval: 1
    UVC_VS_Interface[0x1238a58e0][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 2|bNumEndpoints: 1
     UVC_VS_Endpoint[0x122d00940][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize: 100|bInterval: 1
    UVC_VS_Interface[0x1238a5ba0][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 3|bNumEndpoints: 1
     UVC_VS_Endpoint[0x1223601e0][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize: 320|bInterval: 1
    UVC_VS_Interface[0x122261ce0][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 4|bNumEndpoints: 1
     UVC_VS_Endpoint[0x1238a5e90][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize: b20|bInterval: 1
    UVC_VS_Interface[0x1238a6070][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 5|bNumEndpoints: 1
     UVC_VS_Endpoint[0x12234f500][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize:1320|bInterval: 1
    UVC_VS_Interface[0x1238a62d0][0x00000009]: | bLength:0x09 | bDescriptorType:0x04:|bInterfaceNumber: 1|bAlternateSetting: 6|bNumEndpoints: 1
     UVC_VS_Endpoint[0x1238a6380][0x00000007]: | bLength:0x07 | bDescriptorType:0x05:|bEndpointAddress:81|bmAttributes: 5|wMaxPacketSize:1400|bInterval: 1

    对比qemu显示，应该是已经计算过的，只显示了maxp 没有显示maxp_mult
    实际应为128 256 800 b20(5800) 1320(10 * 800) 1400(10 1024)
    足以满足
    查看显示代码，是否没有显示maxp_mult,结果发现显示了

    回过头再查看配置，发现设备加载速度竟然不是HIGH_SPEED的480M 而是12M
    原因是设备并非挂载在480M的 HIGH_SPEED root hub下面，而是在一个12M子hub下
    检查qemu参数, 筛选usb相关的

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    -device nec-usb-xhci,id=usb1,bus=pci.7,addr=0x0
    -device usb-tablet,id=input0,bus=usb1.0,port=1
    -device usb-mouse,id=input1,bus=usb1.0,port=2
    -device usb-kbd,id=input2,bus=usb1.0,port=3
    ...
    -device nec-usb-xhci,id=xhci
    -chardev spicevmc,name=usbredir,id=usbredirchardev1
    -device ksvd_redir,chardev=usbredirchardev1,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev1
    -chardev spicevmc,name=usbredir,id=usbredirchardev2
    -device ksvd_redir,chardev=usbredirchardev2,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev2
    -chardev spicevmc,name=usbredir,id=usbredirchardev3
    -device ksvd_redir,chardev=usbredirchardev3,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev3
    -chardev spicevmc,name=usbredir,id=usbredirchardev4
    -device ksvd_redir,chardev=usbredirchardev4,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev4
    -chardev spicevmc,name=usbredir,id=usbredirchardev5
    -device ksvd_redir,chardev=usbredirchardev5,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev5
    -chardev spicevmc,name=usbredir,id=usbredirchardev6
    -device ksvd_redir,chardev=usbredirchardev6,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev6
    -chardev spicevmc,name=usbredir,id=usbredirchardev7
    -device ksvd_redir,chardev=usbredirchardev7,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev7
    -chardev spicevmc,name=usbredir,id=usbredirchardev8
    -device ksvd_redir,chardev=usbredirchardev8,filter-path=/home/kylin-ksvd/.ksvd/usbSettings.82524a06-a33c-469f-ab82-7358bff5514f,usb_degrade=0,id=usbredirdev8

    和系统下的体现，差距较大
    系统下有6个总线，和我们定义的2个，不太相同
    1.3.5为2.0
    2.4.6为3.0
    都是xhci
    并且1下两个480M(2.0)hub
    2下1个12M的hub
    4下3个480M（2.0）hub

    去查看qemu源代码，未见问题。尝试uniqb中都做绑定，能看的更清晰

    qemu中

    1
    qemu-system-sw64: info: [28 09:49:39][0000:0000][12792]usbredirhost: attaching high speed device 1b17:0210 version 0.0 class ef

    虚拟机dmesg中

    1
    new full-speed USB device

    high speed 被识别成full-speed，原因无疑是因为位置挂在了哪个full-speed的hub下
    经过结构分析，是根hub只有4个口，超过4个的通道，会产生一个二级hub,类似之前非sw的一个已知问题

    1
    2
    DEFINE_PROP_UINT32("p2",    XHCIState, numports_2, 4),
    DEFINE_PROP_UINT32("p3",    XHCIState, numports_3, 4),

    修改为

    1
    2
    DEFINE_PROP_UINT32("p2",    XHCIState, numports_2, 15),
    DEFINE_PROP_UINT32("p3",    XHCIState, numports_3, 15),

    参考

    http://10.0.0.6/issues/35514

    <工欲善其事，必先利其器> hook_debug

    产出文档

    SW架构usbredir/qemu编译环境搭建操作指南 USB优化，qemu移植