perf: add scripts to build docker image and run perfetto trace analysis

Most of the work of this MR was done by Tomeu.

Changes:
  - update the Dockerfile to use the new virglrenderer tracing option
  - use command line parsing to pass the parameters when running a trace
  - move some files and rename scripts

v2: - unify gfx-pps build (Tomeu) and use to suggested branch (Fahien)
    - use upstream tagged version for igt-gpu-tools (Tomeu)
    - add parameters to set benchmark loop count (Louis-Francis)
    - add parameter to set perfetto loop count or loopless run

v3: - compile virglrenderer against minigbm (Rohan)
    - compile minigbm with driver i915 (Rohan)

v4: Don't build mesa-gbm (instead of deleting the library and
    header later)

v5: add option to run script whether to wait after each frame

v6: correct gfx-pps version and add gdb

v7: fix tag for gfx-pps

v8: - when building the container use the user ID and group
      id of the user who runs the container build
    - rework passing command line parameters
    - add a command line parameter to record per-frame
      images
    - Use debugoptimized build for gfx-pps (Tomeu)

v9: disable buffer storage, there seems to be a bug in
    context handling triggered by this

v10: move to upstream kernel 5.9.12

v11: - remove writing txt version of host trace
     - add debug flagto enable tracing EGL
     - don't force-remove libgbm1 it is no longer a problem to
       have it

v12: update kernel to 5.20.5

v13: refactor trace merge (Rohan)

v14: - Fix refactoring (Rohan)
     - Add license infro to perfetto merge script

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Reviewed-By: Rohan Garg <rohan.garg@collabora.com>
macos/master
Gert Wollny 4 years ago committed by Gert Wollny
parent d5cced1e82
commit e7f47fd3db
  1. 308
      perf-testing/Docker/Dockerfile
  2. 18
      perf-testing/Docker/init.sh
  3. 154
      perf-testing/Docker/merge_traces.py
  4. 49
      perf-testing/Docker/perfetto-guest.cfg
  5. 55
      perf-testing/Docker/perfetto-host.cfg
  6. 227
      perf-testing/Docker/run.sh
  7. 13
      perf-testing/Docker/run_perfetto_ui.sh
  8. 119
      perf-testing/Docker/run_traces.sh
  9. 37
      perf-testing/Docker/x86_64.config
  10. 70
      perf-testing/README.md
  11. 18
      perf-testing/build-dockerimage.sh
  12. 25
      perf-testing/perfetto-ui.sh
  13. 200
      perf-testing/run-trace-in-container.sh

@ -0,0 +1,308 @@
# Copyright 2018 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
FROM debian:buster
LABEL description="Test crosvm using a command like the following: \
docker run --privileged -v /dev/log:/dev/log -v <path to crosvm>:/platform/crosvm:ro <crosvm base image>"
# should be set to the ID/GROUP_ID of the user running the docker image
ARG USER_ID
ARG GROUP_ID
RUN apt-get update && \
apt-get install -y --no-install-recommends ca-certificates gnupg wget && \
echo 'deb https://deb.debian.org/debian buster-backports main' >> /etc/apt/sources.list && \
echo 'deb-src https://deb.debian.org/debian buster main' >> /etc/apt/sources.list && \
echo 'deb https://apt.llvm.org/buster/ llvm-toolchain-buster-9 main' >> /etc/apt/sources.list && \
cat /etc/apt/sources.list && \
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && \
apt-get update && \
apt-get build-dep -y mesa && \
apt-get install -y --no-install-recommends ca-certificates \
python3-setuptools \
llvm-9-dev \
libxcb-shm0-dev \
libelf-dev \
cmake \
bc \
flex \
bison \
debootstrap \
cpio \
xz-utils \
libegl1-mesa-dev \
autoconf \
automake \
curl \
g++ \
gcc \
gdb \
git \
kmod \
libcap-dev \
libdbus-1-dev \
libegl1-mesa-dev \
libfdt-dev \
libgl1-mesa-dev \
libgles2-mesa-dev \
libpciaccess-dev \
libssl-dev \
libtool \
libusb-1.0-0-dev \
libwayland-dev \
make \
nasm \
ninja-build \
pkg-config \
protobuf-compiler \
python \
libtinfo5 \
g++-7 \
python3-protobuf \
clang \
iptables \
libunwind-dev \
libprotobuf-dev \
protobuf-compiler \
libprotoc-dev \
libdw-dev \
libprotobuf-dev \
libdocopt-dev \
&& \
apt-get -y build-dep intel-gpu-tools && \
apt-get remove -y llvm-7-dev llvm-7 llvm-7-runtime
ENV RUSTUP_HOME=/usr/local/rustup \
CARGO_HOME=/usr/local/cargo \
PATH=/usr/local/cargo/bin:$PATH \
RUST_VERSION=1.45.2 \
RUSTFLAGS='--cfg hermetic'
# Debian usually has an old rust version in the repository. Instead of using that, we use rustup to
# pull in a toolchain versions of our choosing.
RUN curl -LO "https://static.rust-lang.org/rustup/archive/1.22.1/x86_64-unknown-linux-gnu/rustup-init" \
&& echo "49c96f3f74be82f4752b8bffcf81961dea5e6e94ce1ccba94435f12e871c3bdb *rustup-init" | sha256sum -c - \
&& chmod +x rustup-init \
&& ./rustup-init -y --no-modify-path --default-toolchain $RUST_VERSION \
&& rm rustup-init \
&& chmod -R a+w $RUSTUP_HOME $CARGO_HOME \
&& rustup --version \
&& cargo --version \
&& rustc --version
# Set the default toolchain to 'stable' to match the one that bin/smoke_test
# uses. This allows kokoro runs to avoid re-downloading the toolchain as long
# as the version matches RUST_VERSION.
RUN rustup default stable
# Warms up the cargo registry cache for future cargo runs. Cargo will still update the cache using a
# git pull, but it only needs to download files that were changed since this image was built.
RUN cargo install thisiznotarealpackage -q || true
# Used /scratch for building dependencies which are too new or don't exist on Debian stretch.
WORKDIR /scratch
# Suppress warnings about detached HEAD, which will happen a lot and is meaningless in this context.
RUN git config --global advice.detachedHead false
# New libepoxy and libdrm-dev requires newer meson than is in Debian stretch.
ARG MESON_COMMIT=master
RUN git clone https://github.com/mesonbuild/meson /meson \
&& cd /meson \
&& git checkout $MESON_COMMIT \
&& rm -f /usr/bin/meson \
&& ln -s $PWD/meson.py /usr/bin/meson
# The libdrm-dev in distro can be too old to build minigbm,
# so we build it from upstream.
ARG DRM_COMMIT=master
RUN git clone https://gitlab.freedesktop.org/mesa/drm.git/ \
&& cd drm \
&& git checkout $DRM_COMMIT \
&& meson build -Dlibdir=lib \
&& ninja -C build/ install
# The gbm used by upstream linux distros is not compatible with crosvm, which must use Chrome OS's
# minigbm.
RUN git clone https://chromium.googlesource.com/chromiumos/platform/minigbm \
&& cd minigbm \
&& sed 's/-Wall/-Wno-maybe-uninitialized/g' -i Makefile \
&& make CPPFLAGS="-DDRV_I915" DRV_I915=1 install -j$(nproc)
# New libepoxy has EGL_KHR_DEBUG entry points needed by crosvm.
ARG LIBEPOXY_COMMIT=master
RUN git clone https://github.com/anholt/libepoxy.git \
&& cd libepoxy \
&& git checkout $LIBEPOXY_COMMIT \
&& mkdir build \
&& meson build -Dtests=false -Dlibdir=lib \
&& ninja -C build/ install
RUN git clone https://gitlab.freedesktop.org/virgl/virglrenderer.git \
&& cd virglrenderer \
&& mkdir -p build \
&& meson build/ -Dprefix=/usr/local -Dlibdir=lib \
&& ninja -C build/ install
# Install libtpm2 so that tpm2-sys/build.rs does not try to build it in place in
# the read-only source directory.
ARG TPM2_COMMIT=073dc25aa4dda42475a7a5a140399fc5db61b20f
RUN git clone https://chromium.googlesource.com/chromiumos/third_party/tpm2 \
&& cd tpm2 \
&& git checkout $TPM2_COMMIT \
&& make -j$(nproc) \
&& cp build/libtpm2.a /lib
# PUll down platform2 repositroy and install librendernodehost.
# Note that we clone the repository outside of /scratch not to be removed
# because crosvm depends on libvda.
ENV PLATFORM2_ROOT=/platform2
ARG PLATFORM2_COMMIT=2dce812fc9091e41a33094929610199468ee322b
RUN git clone https://chromium.googlesource.com/chromiumos/platform2 $PLATFORM2_ROOT \
&& cd $PLATFORM2_ROOT \
&& git checkout $PLATFORM2_COMMIT
# Set up sysroot from which system_api proto files are built.
ENV SYSROOT=/sysroot
RUN mkdir -p $SYSROOT/usr/include/chromeos/dbus/trunks \
&& cp $PLATFORM2_ROOT/trunks/interface.proto \
$SYSROOT/usr/include/chromeos/dbus/trunks
# Copy it under rustc's sysroot as well for cargo clippy.
RUN export RUST_SYSROOT=$(rustc --print sysroot); echo $RUST_SYSROOT
RUN mkdir -p $RUST_SYSROOT/usr/include/chromeos/dbus/trunks \
&& cp $PLATFORM2_ROOT/trunks/interface.proto \
$RUST_SYSROOT/usr/include/chromeos/dbus/trunks
# Reduces image size and prevents accidentally using /scratch files
RUN rm -r /scratch
WORKDIR /
# The manual installation of shared objects requires an ld.so.cache refresh.
RUN ldconfig
# Pull down repositories that crosvm depends on to cros checkout-like locations.
ENV CROS_ROOT=/
ENV THIRD_PARTY_ROOT=$CROS_ROOT/third_party
RUN mkdir -p $THIRD_PARTY_ROOT
ENV PLATFORM_ROOT=$CROS_ROOT/platform
RUN mkdir -p $PLATFORM_ROOT
ENV AOSP_EXTERNAL_ROOT=$CROS_ROOT/aosp/external
RUN mkdir -p $AOSP_EXTERNAL_ROOT
# minijail does not exist in upstream linux distros.
ARG MINIJAIL_COMMIT=5f9e3001c61626d2863dad91248ba8496c3ef511
RUN git clone https://android.googlesource.com/platform/external/minijail $AOSP_EXTERNAL_ROOT/minijail \
&& cd $AOSP_EXTERNAL_ROOT/minijail \
&& git checkout $MINIJAIL_COMMIT \
&& make -j$(nproc) \
&& cp libminijail.so /usr/lib/x86_64-linux-gnu/
# Pull the cras library for audio access.
ARG ADHD_COMMIT=5068bdd18b51de8f2d5bcff754cdecda80de8f44
RUN git clone https://chromium.googlesource.com/chromiumos/third_party/adhd $THIRD_PARTY_ROOT/adhd \
&& cd $THIRD_PARTY_ROOT/adhd \
&& git checkout $ADHD_COMMIT
ARG VPERFETTO_COMMIT=3ce4813ae114e5f2e6e0b3f29517a88246c00363
RUN git clone https://github.com/741g/vperfetto.git && \
cd vperfetto && \
git checkout $VPERFETTO_COMMIT && \
cmake -G Ninja -B_build -DOPTION_BUILD_TESTS=FALSE && \
ninja -C _build install
ARG CROSVM_COMMIT=3f9373f474a295df0f8a38592472ae59adc98e29
RUN mkdir -p /platform/ \
&& cd /platform \
&& git clone --single-branch -b perfetto https://gitlab.freedesktop.org/tomeu/crosvm.git \
&& cd crosvm \
&& cargo install --debug --features 'default-no-sandbox wl-dmabuf gpu x virtio-gpu-next' --path . --root /usr/local
RUN export uid=$USER_ID gid=$GROUP_ID && \
mkdir -p /home/chronos && \
echo "chronos:x:${uid}:${gid}:Developer,,,:/home/chronos:/bin/bash" >> /etc/passwd && \
echo "chronos:x:${uid}:" >> /etc/group && \
chown ${uid}:${gid} -R /home/chronos
ENV EXTRA_PACKAGES="sudo,strace,libxcb-dri2-0,libxcb-dri3-0,libx11-xcb1,libxcb-xfixes0,libxcb-present0,libxcb-sync1,libxshmfence1,libx11-6,gdb,sysvinit-core,libwayland-client0,libwayland-server0,time,chrony,inetutils-ping,dnsutils,libpng16-16,libprocps7,gdb,valgrind"
RUN ulimit -n 1024 && \
debootstrap --variant=minbase --components main,contrib,non-free --include=$EXTRA_PACKAGES buster /rootfs http://deb.debian.org/debian && \
chroot /rootfs /bin/bash -c "echo 'deb http://deb.debian.org/debian buster-backports main' >>/etc/apt/sources.list && apt-get update && apt-get install -y --no-install-recommends libsensors5 python3 wget gnupg ca-certificates" && \
chroot /rootfs /bin/bash -c "echo 'deb https://apt.llvm.org/buster/ llvm-toolchain-buster-9 main' >/etc/apt/sources.list.d/llvm9.list && wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && apt-get update && apt-get install -y libllvm9" && \
chroot /rootfs /bin/bash -c "dpkg-query -Wf '\${Installed-Size}\t\${Package}\n' | sort -n " && \
chroot /rootfs /bin/bash -c "useradd -u 1001 -r -d / -s /sbin/nologin -c 'crossvm image user' perfetto"
COPY perf-testing/Docker/init.sh /rootfs/.
RUN cd /rootfs && \
find -H | cpio -H newc -o | xz --check=crc32 -T4 - > /rootfs.cpio.gz
COPY perf-testing/Docker/x86_64.config /tmp/.
RUN mkdir -p kernel && \
wget -O- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.5.tar.xz | tar -xJ --strip-components=1 -C kernel && \
cd kernel && \
./scripts/kconfig/merge_config.sh arch/x86/configs/x86_64_defconfig /tmp/x86_64.config && \
make -j12 vmlinux && \
cp vmlinux /. && \
cd .. && \
rm -rf kernel
# Need an unreleased version of Waffle for surfaceless support in apitrace
# Replace this build with the Debian package once that's possible
ENV WAFFLE_VERSION="e3c995d9a2693b687501715b6550619922346089"
RUN git clone https://gitlab.freedesktop.org/mesa/waffle.git --single-branch --no-checkout /waffle && \
cd /waffle && \
git checkout "$WAFFLE_VERSION" && \
cmake -B_build -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Debug -Dwaffle_has_surfaceless_egl=1 . && \
make -j12 -C _build install && \
mkdir -p build/lib build/bin && \
cp _build/lib/libwaffle-1.so build/lib/libwaffle-1.so.0 && \
cp _build/bin/wflinfo build/bin/wflinfo
ENV APITRACE_VERSION="perfetto"
RUN git clone https://gitlab.freedesktop.org/tomeu/apitrace.git --single-branch -b perfetto --no-checkout /apitrace && \
cd /apitrace && \
git checkout "$APITRACE_VERSION" && \
cmake -G Ninja -B_build -H. -DCMAKE_BUILD_TYPE=Debug -DENABLE_GUI=False -DENABLE_WAFFLE=on -DWaffle_DIR=/usr/local/lib/cmake/Waffle/ && \
ninja -C _build && \
mkdir build && \
cp _build/apitrace build && \
cp _build/eglretrace build
ENV GN_ARGS="is_debug=false use_custom_libcxx=false"
ENV CFG=linux_trusty-gcc7-x86_64-release
RUN git clone --single-branch -b virgl https://gitlab.freedesktop.org/tomeu/perfetto.git && \
cd perfetto && \
python3 tools/install-build-deps && \
python3 tools/install-build-deps --ui && \
tools/gn gen out/dist --args="${GN_ARGS}" --check && \
tools/ninja -C out/dist traced traced_probes perfetto trace_to_text ui trace_processor_shell && \
mkdir -p /usr/local/lib/python3.7/site-packages && \
protoc --python_out=/usr/local/lib/python3.7/site-packages protos/perfetto/trace/perfetto_trace.proto && \
tools/gen_amalgamated --gn_args 'target_os="linux" is_debug=false'
RUN mkdir -p /traces-db && chown chronos:chronos /traces-db && mkdir -p /wd && chown -R chronos:chronos /wd
ENV IGT_GPU_TOOLS_VERSION="igt-gpu-tools-1.25"
RUN git clone --single-branch -b master https://gitlab.freedesktop.org/drm/igt-gpu-tools.git && \
cd igt-gpu-tools && \
git checkout "$IGT_GPU_TOOLS_VERSION" && \
meson build -Doverlay=disabled -Dchamelium=disabled -Dvalgrind=disabled -Dman=disabled -Ddocs=disabled -Dtests=disabled -Drunner=disabled && \
ninja -C build install
ENV GFX_PPS_VERSION="v0.2.0"
RUN git clone --single-branch -b master https://gitlab.freedesktop.org/Fahien/gfx-pps.git && \
cd gfx-pps && \
git checkout "$GFX_PPS_VERSION" && \
meson build -Dweston=false -Dtest=false -Dbuildtype=debugoptimized && \
ninja -C build
COPY perf-testing/Docker/run_traces.sh /usr/local/.
COPY perf-testing/Docker/run_perfetto_ui.sh /usr/local/.
COPY perf-testing/Docker/run.sh /usr/local/.
COPY perf-testing/Docker/perfetto-guest.cfg /usr/local/.
COPY perf-testing/Docker/perfetto-host.cfg /usr/local/.
COPY perf-testing/Docker/merge_traces.py /usr/local/.
ENTRYPOINT ["/usr/local/run.sh"]

@ -0,0 +1,18 @@
#!/bin/sh
set +xe
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev || echo possibly already mounted
mkdir -p /dev/pts
mount -t devpts devpts /dev/pts
mount -t virtiofs local /usr/local
mount -t debugfs none /sys/kernel/debug
echo "nameserver 8.8.8.8" > /etc/resolv.conf
#for i in 1 2 3; do sntp -sS pool.ntp.org && break || sleep 2; done
bash /usr/local/run_traces.sh
sync
sleep 1

@ -0,0 +1,154 @@
#!/usr/bin/python3
#
# Copyright (C) 2020 Collabora Ltd
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included
# in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
from google import protobuf
import protos.perfetto.trace.perfetto_trace_pb2
from protos.perfetto.trace.perfetto_trace_pb2 import BUILTIN_CLOCK_BOOTTIME
from protos.perfetto.trace.perfetto_trace_pb2 import BUILTIN_CLOCK_REALTIME
import math
import sys
import operator
import time
def add_ftrace_event(out_message, in_packet, in_event, max_host_sequence_id = 0):
out_packet = out_message.packet.add()
out_packet.ftrace_events.cpu = in_packet.ftrace_events.cpu
out_packet.trusted_uid = in_packet.trusted_uid
out_packet.trusted_packet_sequence_id += max_host_sequence_id
out_packet.ftrace_events.event.add().CopyFrom(in_event)
virtio_gpu_pids = set()
print('%d Loading host trace' % time.time())
in_message = protos.perfetto.trace.perfetto_trace_pb2.Trace()
in_message.ParseFromString(open(sys.argv[1], 'rb').read())
print('%d Copying host trace' % time.time())
out_message = protos.perfetto.trace.perfetto_trace_pb2.Trace()
max_host_sequence_id = 0
first_host_virtio_gpu_cmd = math.inf
host_boot_ts = -1
for in_packet in in_message.packet:
max_host_sequence_id = max(max_host_sequence_id,
in_packet.trusted_packet_sequence_id)
if in_packet.HasField('ftrace_events'):
for event in in_packet.ftrace_events.event:
if event.HasField('sched_switch'):
if 'virtio_gpu' == event.sched_switch.prev_comm:
virtio_gpu_pids.add(event.sched_switch.prev_pid)
if 'virtio_gpu' == event.sched_switch.next_comm:
virtio_gpu_pids.add(event.sched_switch.next_pid)
if event.sched_switch.prev_pid in virtio_gpu_pids or \
event.sched_switch.next_pid in virtio_gpu_pids:
add_ftrace_event(out_message, in_packet, event)
elif event.HasField('sched_wakeup'):
if 'virtio_gpu' == event.sched_wakeup.comm:
virtio_gpu_pids.add(event.sched_wakeup.pid)
if event.sched_wakeup.pid in virtio_gpu_pids:
add_ftrace_event(out_message, in_packet, event)
elif event.HasField('print'):
event_type, guest_pid, label, cookie = event.print.buf.split('|')
# Replace host PID with the guest PID
event.pid = int(guest_pid)
add_ftrace_event(out_message, in_packet, event)
else:
if in_packet.HasField('track_descriptor'):
if in_packet.track_descriptor.HasField('name'):
in_packet.track_descriptor.name += ' (Host)'
elif in_packet.HasField('track_event'):
if in_packet.track_event.type == in_packet.track_event.TYPE_SLICE_BEGIN and \
in_packet.track_event.name == 'GetCapset':
first_host_virtio_gpu_cmd = min(first_host_virtio_gpu_cmd, in_packet.timestamp)
elif host_boot_ts == -1 and in_packet.HasField('clock_snapshot'):
for clock in in_packet.clock_snapshot.clocks:
if clock.clock_id == BUILTIN_CLOCK_BOOTTIME:
host_boottime = clock.timestamp
elif clock.clock_id == BUILTIN_CLOCK_REALTIME:
host_realtime = clock.timestamp
host_boot_ts = host_realtime - host_boottime
out_packet = out_message.packet.add()
out_packet.CopyFrom(in_packet)
print('%d Loading guest trace' % time.time())
in_message.ParseFromString(open(sys.argv[2], 'rb').read())
#print('%d Writing guest trace txt' % time.time())
#open('../traces-db/perfetto-guest.txt', 'w').write(str(in_message))
first_guest_virtio_gpu_cmd = math.inf
guest_boot_ts = -1
for in_packet in in_message.packet:
if guest_boot_ts == -1 and in_packet.HasField('clock_snapshot'):
for clock in in_packet.clock_snapshot.clocks:
if clock.clock_id == BUILTIN_CLOCK_BOOTTIME:
guest_boottime = clock.timestamp
elif clock.clock_id == BUILTIN_CLOCK_REALTIME:
guest_realtime = clock.timestamp
guest_boot_ts = guest_realtime - guest_boottime
elif in_packet.HasField('track_event'):
if in_packet.track_event.type == in_packet.track_event.TYPE_SLICE_BEGIN and \
in_packet.track_event.name == 'DRM_IOCTL_VIRTGPU_GET_CAPS':
first_guest_virtio_gpu_cmd = min(first_guest_virtio_gpu_cmd, in_packet.timestamp)
delta = guest_boot_ts - host_boot_ts
cmd_delta = first_host_virtio_gpu_cmd - first_guest_virtio_gpu_cmd - delta
print("boottime delta %ds." % (delta / 1000 / 1000 / 1000))
print("cmd delta %dus." % (cmd_delta / 1000))
for in_packet in in_message.packet:
if in_packet.HasField('process_tree') or \
in_packet.HasField('service_event') or \
in_packet.HasField('track_event') or \
in_packet.HasField('trace_packet_defaults') or \
in_packet.HasField('track_descriptor'):
out_packet = out_message.packet.add()
out_packet.CopyFrom(in_packet)
out_packet.trusted_packet_sequence_id += max_host_sequence_id
out_packet.timestamp += delta
if out_packet.HasField('track_descriptor'):
if out_packet.track_descriptor.HasField('name'):
out_packet.track_descriptor.name += ' (Guest)'
elif in_packet.HasField('ftrace_events'):
for event in in_packet.ftrace_events.event:
event.timestamp += delta
add_ftrace_event(out_message, in_packet, event, max_host_sequence_id)
def get_timestamp(packet):
if packet.HasField('timestamp'):
return packet.timestamp
elif packet.HasField('ftrace_events') and \
packet.ftrace_events.event:
return packet.ftrace_events.event[0].timestamp
return 0
out_message.packet.sort(key=get_timestamp)
print('%d Writing merged trace' % time.time())
open(sys.argv[3], 'wb').write(out_message.SerializeToString())
#print('%d Writing merged trace txt' % time.time())
#open('../traces-db/perfetto.txt', 'w').write(str(out_message))

@ -0,0 +1,49 @@
buffers {
size_kb: 655360
fill_policy: RING_BUFFER
}
data_sources {
config {
name: "linux.ftrace"
target_buffer: 0
ftrace_config {
ftrace_events: "virtio_gpu/virtio_gpu_cmd_queue"
ftrace_events: "virtio_gpu/virtio_gpu_cmd_response"
ftrace_events: "sched_switch"
ftrace_events: "sched_wakeup"
atrace_apps: "*"
compact_sched {
enabled: true
}
}
}
}
data_sources {
config {
name: "track_event"
track_event_config {
disabled_categories: "*"
enabled_categories: "Driver"
# enabled_categories: "EGL"
enabled_categories: "Gallium"
enabled_categories: "OpenGL"
}
}
}
# Resolve process commandlines and parent/child relationships, to better
# interpret the ftrace events, which are in terms of pids.
data_sources {
config {
name: "linux.process_stats"
target_buffer: 0
}
}
write_into_file: true
flush_period_ms: 10000

@ -0,0 +1,55 @@
buffers {
size_kb: 655360
fill_policy: RING_BUFFER
}
data_sources {
config {
name: "linux.ftrace"
target_buffer: 0
ftrace_config {
ftrace_events: "sched_switch"
ftrace_events: "sched_wakeup"
atrace_apps: "*"
compact_sched {
enabled: true
}
}
}
}
data_sources {
config {
name: "track_event"
track_event_config {
disabled_categories: "*"
enabled_categories: "Driver"
enabled_categories: "EGL"
enabled_categories: "OpenGL"
enabled_categories: "VMM"
}
}
}
# Resolve process commandlines and parent/child relationships, to better
# interpret the ftrace events, which are in terms of pids.
data_sources {
config {
name: "linux.process_stats"
target_buffer: 0
}
}
data_sources {
config {
name: "gpu.metrics"
gpu_counter_config {
counter_period_ns: 1000000
}
}
}
write_into_file: true
flush_period_ms: 250

@ -0,0 +1,227 @@
#!/bin/bash
# This script is to be run on the KVM host, inside the container
set -ex
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
export PYTHONPATH=/usr/local/lib/python3.7/site-packages
benchmark_loops=0
perfetto_loops=10
wait_after_frame=
debug=no
trace=
command=""
prep_snapshot=
while [ -n "$1" ] ; do
case "$1" in
--trace|-t)
trace="$2"
shift
;;
--benchmark|-b)
command="$command benchmark=$2"
benchmark_loops=$2
shift
;;
--perfetto|-p)
command="$command perfetto=$2"
perfetto_loops=$2
shift
;;
--wait-after-frame|-w)
command="$command wait-after-frame=1"
wait_after_frame="--wait-after-frame"
;;
--snapshot|-s)
command="$command record-frame=1"
prep_snapshot=yes
;;
--debug)
debug=yes
;;
*)
echo "Unknown option '$1' given, run with option --help to see supported options"
exit
;;
esac
shift
done
if [ "x$trace" = "x" ]; then
echo "No trace given in run script, you must pass is as free parameter to the docker call"
exit 1
fi
pushd /mesa
mkdir -p build
if [ ! -f build/build.ninja ]; then
meson build/ \
-Dprefix=/usr/local \
-Ddri-drivers=i965 \
-Dgallium-drivers=swrast,virgl,radeonsi,iris \
-Dbuildtype=debugoptimized \
-Dllvm=true \
-Dglx=dri \
-Degl=true \
-Dgbm=false \
-Dgallium-vdpau=false \
-Dgallium-va=false \
-Dvulkan-drivers=[] \
-Dvalgrind=false \
-Dlibdir=lib
else
pushd build
meson configure \
-Dprefix=/usr/local \
-Ddri-drivers=i965 \
-Dgallium-drivers=swrast,virgl,radeonsi,iris \
-Dbuildtype=debugoptimized \
-Dllvm=true \
-Dglx=dri \
-Degl=true \
-Dgbm=false \
-Dgallium-vdpau=false \
-Dgallium-va=false \
-Dvulkan-drivers=[] \
-Dvalgrind=false \
-Dlibdir=lib
popd
fi
ninja -C build/ install
popd
pushd /virglrenderer
mkdir -p build
if [ ! -f build/build.ninja ]; then
meson build/ \
-Dprefix=/usr/local \
-Dlibdir=lib \
-Dplatforms=glx,egl \
-Dminigbm_allocation=true \
-Dtracing=perfetto
else
pushd build
meson configure \
-Dprefix=/usr/local \
-Dlibdir=lib \
-Dplatforms=glx,egl \
-Dminigbm_allocation=true \
-Dtracing=perfetto
popd
fi
ninja -C build/ install
popd
# Crosvm needs to link with minigbm, due to incompatible ABI
export LD_PRELOAD=/usr/lib/libminigbm.so.1.0.0
export PATH="/apitrace/build:$PATH"
export PATH="/waffle/build/bin:$PATH"
export LD_LIBRARY_PATH="/waffle/build/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/local/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH"
trace_name=$(basename $trace)
trace_base=${trace_name%.*}
datadir="/traces-db/${trace_base}-out"
echo "Host:"
wflinfo --platform surfaceless_egl --api gles2
export EGL_PLATFORM="surfaceless"
export WAFFLE_PLATFORM="surfaceless_egl"
export DISPLAY=
if [ "x$benchmark_loops" != "x0" ]; then
echo "Measuring rendering times:"
eglretrace --benchmark --loop=$benchmark_loops --headless "/traces-db/${trace}"
fi
# To keep Perfetto happy
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
/perfetto/out/dist/traced &
/perfetto/out/dist/traced_probes &
sleep 1
/gfx-pps/build/src/gpu/producer-gpu &
sleep 1
/perfetto/out/dist/perfetto --txt -c /usr/local/perfetto-host.cfg -o /tmp/perfetto-host.trace --detach=mykey
sleep 1
if [ "x$perfetto_loops" != "x" ] ; then
echo "perfetto_loops parameter not given"
fi
echo "Replaying for Perfetto:"
LOOP=
if [ "x$perfetto_loops" != "x0" ]; then
LOOP="--loop=$perfetto_loops"
fi
eglretrace --benchmark --singlethread $LOOP $wait_after_frame --headless "/traces-db/${trace}"
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward
# store name of trace to be replayed so the guest can obtain the name
echo $trace_base > /traces-db/current_trace
echo $command > /traces-db/command
guest_perf="$datadir/${trace_base}-guest.perfetto"
host_perf="$datadir/${trace_base}-host.perfetto"
summary_perf="$datadir/${trace_base}-summary.perfetto"
mkdir -p $datadir
# work around Crosvm crashing because of errors in context
# handling, could be a problem with the kernel and/or with virglrenderer
export MESA_EXTENSION_OVERRIDE="-GL_ARB_buffer_storage -GL_EXT_buffer_storage"
if [ "x$debug" = "xyes" ]; then
export EGL_DEBUG=debug
fi
crosvm run \
--gpu gles=false\
-m 4096 \
-c 4 \
-i /rootfs.cpio.gz \
--shared-dir "/usr/local:local:type=fs" \
--shared-dir "/waffle:waffle-tag:type=fs" \
--shared-dir "/apitrace:apitrace-tag:type=fs" \
--shared-dir "/traces-db:traces-db-tag:type=fs" \
--shared-dir "/perfetto:perfetto-tag:type=fs" \
--host_ip 192.168.0.1 --netmask 255.255.255.0 --mac AA:BB:CC:00:00:12 \
-p "root=/dev/ram0 rdinit=/init.sh ip=192.168.0.2::192.168.0.1:255.255.255.0:crosvm:eth0 nohz=off clocksource=kvm-clock" \
/vmlinux
rm -f /traces-db/current_trace
rm -f /traces-db/command
/perfetto/out/dist/perfetto --attach=mykey --stop
mv /tmp/perfetto-host.trace "$host_perf"
chmod a+rw "$host_perf"
# sometimes one of these processes seems to crash or exit before, so
# check whether it is still
kill `pidof producer-gpu` || echo "producer-gpu was not running (anymore)"
kill `pidof traced_probes` || echo "traced_probes was not running (anymore)"
kill `pidof traced` || echo "traced was not running (anymore="
/usr/local/merge_traces.py "$host_perf" "$guest_perf" "$summary_perf"
sleep 1

@ -0,0 +1,13 @@
#!/bin/bash
root=""
if [ "x$1" != "x" ]; then
root="$1"
fi
cd "$root/perfetto"
GN_ARGS="is_debug=false use_custom_libcxx=false"
tools/install-build-deps --ui
tools/gn gen out/dist --args="${GN_ARGS}" --check
tools/ninja -C out/dist traced traced_probes perfetto trace_to_text ui trace_processor_shell
ui/run-dev-server out/dist/

@ -0,0 +1,119 @@
# This script is to be run on the KVM guest
set -ex
mkdir /waffle
mount -t virtiofs waffle-tag /waffle
mkdir /apitrace
mount -t virtiofs apitrace-tag /apitrace
mkdir /traces-db
mount -t virtiofs traces-db-tag /traces-db
mkdir /perfetto
mount -t virtiofs perfetto-tag /perfetto
echo 3 > /proc/sys/kernel/printk
export PATH="/apitrace/build:$PATH"
export PATH="/waffle/build/bin:$PATH"
export LD_LIBRARY_PATH="/waffle/build/lib:$LD_LIBRARY_PATH"
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export EGL_PLATFORM="surfaceless"
export WAFFLE_PLATFORM="surfaceless_egl"
export MESA_GL_VERSION_OVERRIDE="4.5"
export DISPLAY=
# Comment out any other sources, so it only syncs to the host via PTP
sed -i '/pool/s/^/#/' /etc/chrony/chrony.conf
echo refclock PHC /dev/ptp0 poll 1 dpoll -2 offset 0 >> /etc/chrony/chrony.conf
echo cmdport 0 >> /etc/chrony/chrony.conf
echo bindcmdaddress / >> /etc/chrony/chrony.conf
time chronyd -q # Initial synchronization, will take some time
chronyd # Keep clocks in sync
# Get trace cached
trace_base=$(cat /traces-db/current_trace)
if [ "x$trace_base" = "x" ]; then
echo "No trace given, bailing out"
exit 1
fi
command=$(cat /traces-db/command)
echo command=$command
WAIT=
RECORD=
benchmark_loops=0
perfetto_loops=10
for c in $command; do
val=(${c//=/ })
case "${val[0]}" in
benchmark)
benchmark_loops=${val[1]}
;;
perfetto)
perfetto_loops=${val[1]}
;;
wait-after-frame)
WAIT="--wait-after-frame"
;;
record-frame)
RECORD="--snapshot"
;;
esac
done
if [ -e /traces-db/wait_after_frame ]; then
WAIT=-wait-after-frame
fi
trace="/traces-db/${trace_base}.trace"
datadir="/traces-db/${trace_base}-out"
guest_perf="$datadir/${trace_base}-guest.perfetto"
cat "$trace" > /dev/null
# To keep Perfetto happy
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracer
echo "Guest:"
wflinfo --platform surfaceless_egl --api gles2 -v
/perfetto/out/dist/traced &
/perfetto/out/dist/traced_probes &
sleep 1
/perfetto/out/dist/perfetto --txt -c /usr/local/perfetto-guest.cfg -o "$guest_perf" --detach=mykey
sleep 1
# The first virtio-gpu event has to be captured in the guest, so we correlate correctly to the host event
echo "Replaying for Perfetto:"
LOOP=
if [ "x$perfetto_loops" != "x" -a "x$perfetto_loops" != "x0" ]; then
LOOP="--loop=$perfetto_loops"
fi
eglretrace --benchmark --singlethread $LOOP $WAIT --headless "$trace"
sleep 1
/perfetto/out/dist/perfetto --attach=mykey --stop
chmod a+rw "$guest_perf"
if [ "x$benchmark_loops" != "x0" ]; then
echo "Measuring rendering times:"
eglretrace --benchmark --loop=$benchmark_loops --headless "$trace"
fi
if [ "x$RECORD" != "x" ]; then
eglretrace --snapshot frame --snapshot-prefix=${datadir}/ --headless "$trace"
fi

@ -0,0 +1,37 @@
CONFIG_DRM=y
CONFIG_MMU=y
CONFIG_FUSE_FS=y
CONFIG_FUNCTION_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE=y
CONFIG_TRACEPOINTS=y
CONFIG_DMA_FENCE_TRACE=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_VIRTUALIZATION=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_FS=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_BLK_SCSI=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_INPUT=y
CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
CONFIG_DRM_VIRTIO_GPU=y
CONFIG_CRYPTO_DEV_VIRTIO=y
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_PTP_1588_CLOCK=y
CONFIG_PTP_1588_CLOCK_KVM=y
CONFIG_KVM=y
CONFIG_KVM_GUEST=y
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y

@ -0,0 +1,70 @@
The files in this directory help with testing Virgl on the virtio-gpu winsys
by means of Crosvm.
A whole environment will be built in a Docker image, then Mesa and Virglrenderer
will be built from local directories to be used both in the host and the guest.
The container image builds on top of other images built by scripts in the crosvm repository.
Instructions for building base images:
```console
$ git clone https://chromium.googlesource.com/chromiumos/platform/crosvm
$ pushd crosvm
$ sh docker/build_crosvm_base.sh
$ sh docker/build_crosvm.sh
```
Instructions for building target image:
```console
$ cd virglrenderer
$ sh perf-testing/build-dockerimage.sh
```
Instructions for running the container:
```console
$ cd virglrenderer
$ bash perf-testing/run_trace-in-container.sh \
--root $PATH_THAT_CONTAINS_MESA_CHECKOUT_VIRGLRENDERER_AND_TRACES_DB_CHECKOUT \
--trace $API_TRACE_TO_RUN
```
There are also options for run_trace-in-container.sh that allow specifying the
path to mesa, virglrenderer, and the traces db. These override the root path.
In addition, the root path defaults to the current working directory.
As a conveniance for shell autocompletion users running the script from the default
root that contains the traces db as subdirectory the the trace file name can be
also given with this traces db sudirectory name, i.e. if the traces db is located
in '$workdir/traces-db', root=$workdir, and the trace is calles 'sometrace.trace',
then both commands
```
perf-testing/run_trace-in-container.sh -r $rootdir -t traces-db/sometrace.trace
```
and
```
perf-testing/run_trace-in-container.sh -r $rootdir -t sometrace.trace
```
will work equally.
At the moment of writing, the branch perfetto-tracing is needed for mesa,
and the for virglrenderer at least commit 7db2faa354 is needed,
so these projects emit the required traces.
The perfetto traces will be saved to the a subdirectory of the traces-db checkout
directory with a name based on the api trace passed in with the --trace parameter.
Once the run_trace-in-container.sh script finishes, 3 Perfetto trace files will be written:
$(API_TRACE_TO_RUN%.*}-host.perfetto, $(API_TRACE_TO_RUN%.*}-guest.perfetto
and $(API_TRACE_TO_RUN%.*}-summary.perfetto. The last one is the fusion of the two first.
In order to visualize the traces, the Perfetto UI needs to be running in a local
service which can be started as follows:
```console
$ perf-testing/perfetto-ui.sh
```
The Perfetto UI can be loaded then on Chromium on the http://localhost:10000 address.

@ -0,0 +1,18 @@
#!/bin/bash
# Copyright 2019 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
set -ex
cd "${0%/*}"
export USER_ID=$(id -u)
export GROUP_ID=$(id -g)
src_root="$(realpath ..)"
docker build -t mesa \
-f Docker/Dockerfile \
--build-arg USER_ID=${USER_ID} \
--build-arg GROUP_ID=${GROUP_ID} \
"$@" \
"${src_root}"

@ -0,0 +1,25 @@
#!/bin/bash
# Copyright 2019 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# This script is to be run on the KVM host, outside the container
set -ex
# grab the pwd before changing it to this script's directory
pwd="${PWD}"
cd "${0%/*}"
exec docker run -it --rm \
--privileged \
--ipc=host \
-v /dev/log:/dev/log \
-v /dev/vhost-net:/dev/vhost-net \
-v /sys/kernel/debug:/sys/kernel/debug \
--volume "$pwd":/wd \
--workdir /wd \
-p 127.0.0.1:10000:10000/tcp \
--entrypoint /usr/local/run_perfetto_ui.sh \
mesa

@ -0,0 +1,200 @@
#!/bin/bash
# Copyright 2019 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
# This script is to be run on the KVM host, outside the container
#set -ex
# grab the pwd before changing it to this script's directory
pwd="${PWD}"
root_dir="$(pwd)"
cd "${0%/*}"
mesa_src=""
virgl_src=""
traces_db=""
kernel_src=""
benchmark_loops=0
perfetto_loops=10
wait_after_frame=
print_help() {
echo "Run GL trace with perfetto"
echo "Usage mesa_wrapper.sh [options]"
echo ""
echo " --root, -r Path to a root directory that contains the sources for mesa, virglrenderer, and the trace-db"
echo " --mesa, -m Path to Mesa source code (overrides path generated from root)"
echo " --virgl, -v Path to virglrenderer source code (overrides path generated from root)"
echo " --kernel, -k Path to Linux kernel source code"
echo " --traces-db, -d Path to the directory containing the traces (overrides path generated from root)"
echo " --trace, -t Trace to be run (path relative to traces-db) (required)"
echo " --benchmark, -b Number of times the last frame should be run for benchmarking (default 0=disabled)"
echo " --perfetto, -p Number of times the last frame should be loop for perfetto (default 10; 0=run trace normally)"
echo " --snapshot, -s Make per-frame snapshots"
echo " --debug Enable extra logging"
echo ""
echo " --help, -h Print this help"
}
command=""
while [ -n "$1" ] ; do
case "$1" in
--root|-r)
root_dir="$2"
shift
;;
--mesa|-m)
mesa_src="$2"
shift
;;
--virgl|-v)
virgl_src="$2"
shift
;;
--traces-db|-d)
traces_db="$2"
shift
;;
--kernel|-k)
kernel_src="$2"
shift
;;
--help|-h)
print_help
exit
;;
--trace|-t)
trace="$2"
shift
;;
--benchmark|-b)
command="$command -b $2"
shift
;;
--perfetto|-p)
command="$command -p $2"
shift
;;
--wait-after-frame|-w)
command="$command -w"
;;
--snapshot|-s)
command="$command -s"
;;
--debug)
command="$command --debug"
;;
*)
echo "Unknown option '$1' given, run with option --help to see supported options"
exit
;;
esac
shift
done
if [ "x$mesa_src" = "x" ] ; then
mesa_src="$root_dir/mesa"
fi
if [ "x$virgl_src" = "x" ] ; then
virgl_src="$root_dir/virglrenderer"
fi
if [ "x$traces_db" = "x" ] ; then
traces_db="$root_dir/traces-db"
fi
can_run=1
if [ "x$trace" = "x" ]; then
echo "No trace given" >&2;
can_run=0
fi
if [ "x$mesa_src" = "x" ]; then
echo "no mesa src dir given" >&2;
can_run=0
fi
if [ ! -d "$mesa_src/src/mesa" ]; then
echo "mesa src dir '$mesa_src' is not a mesa source tree" >&2;
can_run=0
fi
if [ "x$virgl_src" = "x" ]; then
echo "no virglrenderer src dir given" >&2;
can_run=0
fi
if [ ! -d "$virgl_src/vtest" ]; then
echo "virglrenderer src dir '$virgl_src' is not a virglrenderer source tree" >&2;
can_run=0
fi
if [ "x$traces_db" = "x" ]; then
echo "no traces_db dir given" >&2;
can_run=0
fi
if [ ! -f "$traces_db/$trace" ]; then
echo "Given trace file '$trace' doesn't exist in traces db dir '$traces_db'" >&2;
# check whether the trace has been given with a path relative to the root dir
# that can be removed
trace=${trace#*/}
echo "Trying $traces_db/$trace" >&2;
if [ ! -f "$traces_db/$trace" ]; then
echo "Given trace file '$trace' not found " >&2;
can_run=0
fi
fi
if [ "x$can_run" = "x0" ]; then
echo "Missing or command line options with errors were given" >&2;
exit 1
fi
re='^[0-9]+$'
if ! [[ 1$benchmark_loops =~ $re ]] ; then
echo "error: benchmark_loops '" $benchmark_loops "'is not a number" >&2;
exit 1
fi
if ! [[ 1$perfetto_loops =~ $re ]] ; then
echo "error: perfetto_loops '" $perfetto_loops "' is not a number" >&2;
exit 1
fi
echo "command=$command"
docker run -it --rm \
--privileged \
--ipc=host \
-v /dev/log:/dev/log \
-v /dev/vhost-net:/dev/vhost-net \
-v /sys/kernel/debug:/sys/kernel/debug \
-v "$mesa_src":/mesa \
-v "$virgl_src":/virglrenderer \
-v "$traces_db":/traces-db \
-v "$kernel_src":/kernel \
--volume "$pwd":/wd \
--workdir /wd \
mesa \
-t "$trace" \
$command
Loading…
Cancel
Save