Demonstrations of kvmexit-x86_64, for kylin server sp1/2/3.


Considering virtual machines' frequent exits can cause performance problems,
this tool aims to locate the frequent exited reasons and then find solutions
to reduce or even avoid the exit, by displaying the detail exit reasons and
the counts of each vm exit for all vms running on one physical machine.

Add new item @EXIT_TIME_AVG for Outputing, its mean the average of all interval
from kvm_exit to the next kvm_entry. unit:ns


Features of this tool
=====================

- Although there is a patch: [KVM: x86: add full vm-exit reason debug entries]
  (https://patchwork.kernel.org/project/kvm/patch/1555939499-30854-1-git-send-email-pizhenwei@bytedance.com/)
  trying to fill more vm-exit reason debug entries, just as the comments said,
  the code allocates lots of memory that may never be consumed, misses some
  arch-specific kvm causes, and can not do kernel aggregation. Instead bcc, as
  a user space tool, can implement all these functions more easily and flexibly.
- The bcc python logic could provide nice kernel aggregation and custom output,
  like collpasing all tids for one pid (e.i. one vm's qemu process id) with exit
  reasons sorted in descending order. For more information, see the following
  #USAGE message.
- The bpf in-kernel percpu_array and percpu_cache further improves performance.
  For more information, see the following #Help to understand.


Requirements:
1. bcc
2. python
3. chmod +x kvmexit-x86_64.py
4. only use for Intel and AMD


Example:
# ./kvmexit-x86_64.py
Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end.
^C
PID      TID      KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
9352     9385     EXTERNAL_INTERRUPT                  4        6600
9352     9385     HLT                                 697      67104726
9352     9385     MSR_READ                            47       1519
9352     9385     MSR_WRITE                           2295     1616
9352     9385     EPT_VIOLATION                       1        30603
9352     9385     EPT_MISCONFIG                       1        72435
9352     9385     PREEMPTION_TIMER                    7        2382

# ./kvmexit-x86_64.py 5
Display kvm exit reasons and statistics for all threads after sleeping 5 secs.
PID      TID      KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
9352     9386     HLT                                 22       218686732
9352     9386     MSR_READ                            2        1645
9352     9386     MSR_WRITE                           47       2100
9352     9384     HLT                                 20       209676451
9352     9384     MSR_READ                            3        1458

# ./kvmexit-x86_64.py -p 9352
Display kvm exit reasons and statistics for PID 9352... Hit Ctrl-C to end.
^C
KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
MSR_WRITE                           1964
HLT                                 922
EPT_VIOLATION                       177
CPUID                               159
EXTERNAL_INTERRUPT                  61
PREEMPTION_TIMER                    57
MSR_READ                            34
EPT_MISCONFIG                       33
PAUSE_INSTRUCTION                   1
EXCEPTION_NMI                       1

# ./kvmexit-x86_64.py -p 9352 -a
Display kvm exit reasons and statistics for PID 9352 and its all threads... Hit Ctrl-C to end.
^C
TID      KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
9383     EXTERNAL_INTERRUPT                  2        987
9383     HLT                                 48       187967635
9383     MSR_READ                            8        1431
9383     MSR_WRITE                           119      1941
9382     EXTERNAL_INTERRUPT                  1        2218
9382     HLT                                 13       718107462
9382     MSR_READ                            7        1332
9382     MSR_WRITE                           42       1588
9386     EXTERNAL_INTERRUPT                  1        1131
9386     HLT                                 24       377772380
9386     MSR_READ                            8        1372

# ./kvmexit-x86_64.py -p 9352 -v 0
Display kvm exit reasons and statistics for PID 9352 VCPU 0... Hit Ctrl-C to end.
^C
KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
MSR_WRITE                           197
IO_INSTRUCTION                      160
HLT                                 79
MSR_READ                            7
EPT_MISCONFIG                       4
PREEMPTION_TIMER                    2
EOI_INDUCED                         1
PAUSE_INSTRUCTION                   1

# ./kvmexit-x86_64.py -T '9387,9381'
Display kvm exit reasons and statistics for TIDS ['9387', '9381']... Hit Ctrl-C to end.
^C
TIDS     KVM_EXIT_REASON                     COUNT EXIT_TIME_AVG
9387     HLT                                 108      61142617
9387     MSR_READ                            7        1408
9387     MSR_WRITE                           175      2383
9381     HLT                                 23       267240960
9381     MSR_READ                            5        1363
9381     MSR_WRITE                           76       1801


Parameter Explanation:
TID: the user space's thread of each vcpu of that virtual machine.
TGID: thread group id, means qmeu's pid in user space.
COMM: the name of vcpu thread.
KVM_EXIT_REASON: the reason why the vm exits on aarch64.
COUNT: the counts of the @KVM_EXIT_REASONS.
AVG_TIME: the average of all time interval from kvm_exit to the next kvm_entry. unit:ns


Limited:
In view of the hardware-assisted virtualization technology of
different architectures, currently we only adapt on Intel and AMD.


USAGE message:

# ./kvmexit-x86_64.py -h
usage: kvmexit-x86_64.py [-h] [-p PID [-v VCPU | -a] ] [-t TID | -T 'TID1,TID2'] [duration]

Display kvm_exit_reason and its statistics at a timed interval

optional arguments:
  -h, --help            show this help message and exit
  -p PID, --pid PID     display process with this PID only, collpase all tids with exit reasons sorted in descending order
  -v VCPU, --v VCPU     display this VCPU only for this PID
  -a, --alltids         display all TIDS for this PID
  -t TID, --tid TID     display thread with this TID only with exit reasons sorted in descending order
  -T 'TID1,TID2', --tids 'TID1,TID2'
                        display threads for a union like {395490, 395491}
  duration              duration of display, after sleeping several seconds

examples:
    ./kvmexit-x86_64                         # Display kvm_exit_reason and its statistics in real-time until Ctrl-C
    ./kvmexit-x86_64 5                       # Display in real-time after sleeping 5s
    ./kvmexit-x86_64 -p 3195281              # Collpase all tids for pid 3195281 with exit reasons sorted in descending order
    ./kvmexit-x86_64 -p 3195281 20           # Collpase all tids for pid 3195281 with exit reasons sorted in descending order, and display after sleeping 20s
    ./kvmexit-x86_64 -p 3195281 -v 0         # Display only vcpu0 for pid 3195281, descending sort by default
    ./kvmexit-x86_64 -p 3195281 -a           # Display all tids for pid 3195281
    ./kvmexit-x86_64 -t 395490               # Display only for tid 395490 with exit reasons sorted in descending order
    ./kvmexit-x86_64 -t 395490 20            # Display only for tid 395490 with exit reasons sorted in descending order after sleeping 20s
    ./kvmexit-x86_64 -T '395490,395491'      # Display for a union like {395490, 395491}
