Hi Rik,

Do you have time to look at this? Thanks.

On 5/19/2021 4:22 PM, kernel test robot wrote:

Greeting,

FYI, we noticed a -29.1% regression of tbench.throughput-MB/sec due to commit:


commit: c722f35b513f807629603bbf24640b1a48be21b5 ("sched/fair: Bring back select_idle_smt(), but differently")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master


in testcase: tbench
on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 128G memory
with following parameters:

	nr_threads: 100%
	cluster: cs-localhost
	cpufreq_governor: performance
	ucode: 0x5003006




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@intel.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml  # generate the yaml file for lkp run
        bin/lkp run                    generated-yaml-file

=========================================================================================
cluster/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase/ucode:
  cs-localhost/gcc-9/performance/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp8/tbench/0x5003006

commit: 
  6db12ee045 ("psi: allow unprivileged users with CAP_SYS_RESOURCE to write psi files")
  c722f35b51 ("sched/fair: Bring back select_idle_smt(), but differently")

6db12ee0456d0e36 c722f35b513f807629603bbf246 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     15027           -29.1%      10651   5%  tbench.throughput-MB/sec
 1.893e+09   2%     -87.5%   2.36e+08  73%  tbench.time.involuntary_context_switches
     51693  21%     +65.7%      85645   7%  tbench.time.minor_page_faults
      5200           -45.9%       2815   9%  tbench.time.percent_of_cpu_this_job_got
     32467           -48.9%      16595  10%  tbench.time.system_time
      4990           -26.1%       3686   6%  tbench.time.user_time
 1.387e+08  25%    +760.7%  1.194e+09   8%  tbench.time.voluntary_context_switches
      3954   8%    +307.9%      16131   8%  uptime.idle
  93556034   4%     -24.5%   70597092   5%  numa-numastat.node0.local_node
  93594173   4%     -24.5%   70643519   5%  numa-numastat.node0.numa_hit
 1.011e+08   2%     -29.7%   71069877   6%  numa-numastat.node1.local_node
 1.012e+08   2%     -29.7%   71110181   6%  numa-numastat.node1.numa_hit
  47900004   3%     -21.4%   37635451   6%  numa-vmstat.node0.numa_hit
  47817433   3%     -21.5%   37543255   6%  numa-vmstat.node0.numa_local
  51712415   2%     -26.3%   38110581   8%  numa-vmstat.node1.numa_hit
  51561304   2%     -26.4%   37969236   8%  numa-vmstat.node1.numa_local
      2.17  31%    +861.5%      20.83   9%  vmstat.cpu.id
     88.67           -19.5%      71.33   2%  vmstat.cpu.sy
    132.50           -16.2%     111.00        vmstat.procs.r
    197679            +6.4%     210333        vmstat.system.in
 1.445e+09  24%    +838.9%  1.357e+10  10%  cpuidle.C1.time
 1.585e+08  24%    +803.8%  1.433e+09   8%  cpuidle.C1.usage
    238575  17%    +101.0%     479497   9%  cpuidle.C1E.usage
  16003682  25%    +636.0%  1.178e+08   7%  cpuidle.POLL.time
   1428997  25%    +572.1%    9604232   8%  cpuidle.POLL.usage
     22.00           +14.7       36.65   4%  mpstat.cpu.all.idle%
      0.81            +0.1        0.90        mpstat.cpu.all.irq%
     15.57            +4.6       20.20   2%  mpstat.cpu.all.soft%
     54.87           -18.2       36.68   5%  mpstat.cpu.all.sys%
      6.75            -1.2        5.57   3%  mpstat.cpu.all.usr%
 1.948e+08           -27.3%  1.417e+08   4%  proc-vmstat.numa_hit
 1.947e+08           -27.3%  1.416e+08   4%  proc-vmstat.numa_local
 1.516e+09           -28.0%  1.092e+09   5%  proc-vmstat.pgalloc_normal
   2867654   2%      +5.3%    3018753        proc-vmstat.pgfault
 1.512e+09           -28.1%  1.087e+09   5%  proc-vmstat.pgfree
      2663           -15.7%       2244        turbostat.Avg_MHz
     97.59           -15.8       81.80   2%  turbostat.Busy%
 1.585e+08  24%    +803.8%  1.433e+09   8%  turbostat.C1
      2.09  24%     +17.5       19.58  10%  turbostat.C1%
    237770  17%    +101.5%     479071   9%  turbostat.C1E
      2.38  17%    +663.9%      18.18  10%  turbostat.CPU%c1
     74.39            -6.2%      69.77        turbostat.RAMWatt
      0.01  59%    +886.2%       0.10  48%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.00  62%   +2215.4%       0.05  40%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.tcp_sendmsg
      0.01  24%    +378.9%       0.05  22%  perf-sched.sch_delay.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
      0.05  24%     +47.3%       0.07  12%  perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open.isra
      2347  20%     +28.2%       3010        perf-sched.sch_delay.max.ms.pipe_read.new_sync_read.vfs_read.ksys_read
      1.41  25%     -34.8%       0.92  19%  perf-sched.sch_delay.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
    790.43  10%     +30.2%       1029        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
    215752  29%     -94.4%      12122  46%  perf-sched.wait_and_delay.count.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      2004  12%     -37.7%       1248   6%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
      0.06  23%    +762.8%       0.53  42%  perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe.[unknown]
      0.02   8%    +421.8%       0.10  32%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.__release_sock.release_sock.tcp_sendmsg
      0.03  12%    +191.3%       0.09  18%  perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb
    685.22   7%     +17.4%     804.65  12%  perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
    785.50  10%     +30.3%       1023        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
      1.18  35%     -33.6%       0.78  18%  perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      1.58  15%     -34.9%       1.03  14%  perf-sched.wait_time.max.ms.preempt_schedule_common.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg
     51178  93%   +1451.5%     794049   9%  sched_debug.cfs_rq:/.MIN_vruntime.avg
   2407970  59%    +466.9%   13650279   4%  sched_debug.cfs_rq:/.MIN_vruntime.max
    330425  71%    +821.3%    3044331   6%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      6186   5%     -91.9%     499.13 165%  sched_debug.cfs_rq:/.load.min
      4.01   7%     -39.0%       2.45  15%  sched_debug.cfs_rq:/.load_avg.min
     51178  93%   +1451.5%     794049   9%  sched_debug.cfs_rq:/.max_vruntime.avg
   2407973  59%    +466.9%   13650279   4%  sched_debug.cfs_rq:/.max_vruntime.max
    330425  71%    +821.3%    3044331   6%  sched_debug.cfs_rq:/.max_vruntime.stddev
  23445823           -41.9%   13622977  11%  sched_debug.cfs_rq:/.min_vruntime.avg
  25062917   2%     -32.7%   16866064   8%  sched_debug.cfs_rq:/.min_vruntime.max
  22902926   2%     -42.8%   13102629  11%  sched_debug.cfs_rq:/.min_vruntime.min
      0.92           -19.0%       0.75   3%  sched_debug.cfs_rq:/.nr_running.avg
      0.74  12%     -89.7%       0.08 115%  sched_debug.cfs_rq:/.nr_running.min
      0.07  32%    +432.3%       0.38  10%  sched_debug.cfs_rq:/.nr_running.stddev
      1096   2%     -18.1%     898.07        sched_debug.cfs_rq:/.runnable_avg.avg
    203.00   4%      +8.1%     219.36   2%  sched_debug.cfs_rq:/.runnable_avg.stddev
    996603  48%    +225.1%    3240056  25%  sched_debug.cfs_rq:/.spread0.max
    855.14           -35.7%     549.90   6%  sched_debug.cfs_rq:/.util_avg.avg
      1230   6%     -21.1%     971.21   6%  sched_debug.cfs_rq:/.util_avg.max
    142.21   3%     -38.9%      86.89  17%  sched_debug.cfs_rq:/.util_avg.stddev
    630.23   5%     -86.6%      84.51  65%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      1148   5%     -57.8%     484.29  17%  sched_debug.cfs_rq:/.util_est_enqueued.max
    162.67  21%     -92.9%      11.56 138%  sched_debug.cfs_rq:/.util_est_enqueued.min
    189.55   5%     -60.4%      75.05  31%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     13.27   7%     -56.2%       5.81  16%  sched_debug.cpu.clock.stddev
      2627           -22.2%       2042   3%  sched_debug.cpu.curr->pid.avg
      1954  10%     -92.0%     157.14 137%  sched_debug.cpu.curr->pid.min
    963.90   6%     +58.2%       1525   4%  sched_debug.cpu.curr->pid.stddev
      0.00   7%     -47.7%       0.00  15%  sched_debug.cpu.next_balance.stddev
      1.30           -14.2%       1.12   2%  sched_debug.cpu.nr_running.avg
      2.50   6%     +79.0%       4.47   6%  sched_debug.cpu.nr_running.max
      0.76  10%     -89.8%       0.08 115%  sched_debug.cpu.nr_running.min
      0.51   3%     +73.6%       0.89   6%  sched_debug.cpu.nr_running.stddev
      8.53   3%    +154.8%      21.73   6%  perf-stat.i.MPKI
 2.178e+10           -13.8%  1.878e+10   3%  perf-stat.i.branch-instructions
      1.69            +0.2        1.90        perf-stat.i.branch-miss-rate%
 3.675e+08            -3.5%  3.545e+08        perf-stat.i.branch-misses
     18.30   7%     -13.5        4.80  36%  perf-stat.i.cache-miss-rate%
 1.546e+08   9%     -48.7%   79255656  18%  perf-stat.i.cache-misses
 9.224e+08   2%    +113.6%  1.971e+09   4%  perf-stat.i.cache-references
 2.553e+11           -14.5%  2.183e+11        perf-stat.i.cpu-cycles
    240694  23%    +737.9%    2016725   7%  perf-stat.i.cpu-migrations
      1700  10%     +78.2%       3029  11%  perf-stat.i.cycles-between-cache-misses
      0.02  19%      +0.2        0.18   7%  perf-stat.i.dTLB-load-miss-rate%
   6572698  19%    +582.9%   44886023   7%  perf-stat.i.dTLB-load-misses
 3.024e+10           -14.7%   2.58e+10   3%  perf-stat.i.dTLB-loads
      0.01  16%      +0.1        0.06   8%  perf-stat.i.dTLB-store-miss-rate%
   1539288  16%    +464.4%    8687418   8%  perf-stat.i.dTLB-store-misses
 1.709e+10           -14.7%  1.457e+10   3%  perf-stat.i.dTLB-stores
     73.36           -12.9       60.46   2%  perf-stat.i.iTLB-load-miss-rate%
 1.847e+08   2%     -10.9%  1.645e+08   2%  perf-stat.i.iTLB-load-misses
  67816199   2%     +59.3%   1.08e+08   3%  perf-stat.i.iTLB-loads
 1.073e+11           -13.7%  9.264e+10   3%  perf-stat.i.instructions
      2.66           -14.5%       2.27        perf-stat.i.metric.GHz
      1188   4%     -48.1%     616.51  13%  perf-stat.i.metric.K/sec
    729.51           -12.6%     637.79   2%  perf-stat.i.metric.M/sec
      3886   2%      +5.4%       4097        perf-stat.i.minor-faults
  31516592   8%     -29.4%   22240687  13%  perf-stat.i.node-load-misses
   5444441   5%     -16.3%    4559306   4%  perf-stat.i.node-loads
     96.59            -4.9       91.73        perf-stat.i.node-store-miss-rate%
    432854   9%     +96.5%     850464  17%  perf-stat.i.node-stores
      3887   2%      +5.4%       4098        perf-stat.i.page-faults
      8.60   3%    +147.7%      21.30   6%  perf-stat.overall.MPKI
      1.69            +0.2        1.89        perf-stat.overall.branch-miss-rate%
     16.76   9%     -12.7        4.07  23%  perf-stat.overall.cache-miss-rate%
      1666   9%     +69.9%       2830  14%  perf-stat.overall.cycles-between-cache-misses
      0.02  18%      +0.2        0.17   8%  perf-stat.overall.dTLB-load-miss-rate%
      0.01  16%      +0.1        0.06   9%  perf-stat.overall.dTLB-store-miss-rate%
     73.13           -12.8       60.37   2%  perf-stat.overall.iTLB-load-miss-rate%
     95.70            -4.9       90.84        perf-stat.overall.node-store-miss-rate%
 2.175e+10           -13.7%  1.876e+10   3%  perf-stat.ps.branch-instructions
  3.67e+08            -3.5%   3.54e+08        perf-stat.ps.branch-misses
 1.543e+08   9%     -48.7%   79211813  18%  perf-stat.ps.cache-misses
 9.214e+08   2%    +113.5%  1.967e+09   4%  perf-stat.ps.cache-references
 2.549e+11           -14.5%   2.18e+11        perf-stat.ps.cpu-cycles
    240897  23%    +735.5%    2012779   7%  perf-stat.ps.cpu-migrations
   6575045  19%    +581.3%   44797852   7%  perf-stat.ps.dTLB-load-misses
  3.02e+10           -14.7%  2.577e+10   3%  perf-stat.ps.dTLB-loads
   1539214  16%    +463.3%    8670426   8%  perf-stat.ps.dTLB-store-misses
 1.707e+10           -14.7%  1.456e+10   3%  perf-stat.ps.dTLB-stores
 1.844e+08   2%     -10.9%  1.643e+08   2%  perf-stat.ps.iTLB-load-misses
  67743849   2%     +59.2%  1.079e+08   3%  perf-stat.ps.iTLB-loads
 1.072e+11           -13.7%  9.253e+10   3%  perf-stat.ps.instructions
      3886   2%      +5.5%       4098        perf-stat.ps.minor-faults
  31470314   8%     -29.4%   22222434  13%  perf-stat.ps.node-load-misses
   5442735   5%     -16.2%    4560730   3%  perf-stat.ps.node-loads
    432399   9%     +96.4%     849120  17%  perf-stat.ps.node-stores
      3886   2%      +5.5%       4098        perf-stat.ps.page-faults
  7.73e+13           -13.7%  6.672e+13   3%  perf-stat.total.instructions
      0.00       +4.1e+103%      40.83 117%  interrupts.102:PCI-MSI.31981635-edge.i40e-eth0-TxRx-66
    163522   5%    +659.4%    1241719  40%  interrupts.CAL:Function_call_interrupts
      2717  28%    +275.9%      10215  52%  interrupts.CPU0.CAL:Function_call_interrupts
     30618  41%    +290.0%     119405   9%  interrupts.CPU0.RES:Rescheduling_interrupts
      1686  25%    +483.4%       9836  44%  interrupts.CPU1.CAL:Function_call_interrupts
     26633  50%    +342.3%     117798  10%  interrupts.CPU1.RES:Rescheduling_interrupts
      1633  25%    +563.1%      10833  50%  interrupts.CPU10.CAL:Function_call_interrupts
     27398  51%    +319.3%     114895  10%  interrupts.CPU10.RES:Rescheduling_interrupts
      1857  21%    +446.8%      10154  50%  interrupts.CPU11.CAL:Function_call_interrupts
     26206  53%    +312.9%     108207   8%  interrupts.CPU11.RES:Rescheduling_interrupts
      1423  15%    +614.7%      10176  52%  interrupts.CPU12.CAL:Function_call_interrupts
     26880  58%    +322.7%     113610   8%  interrupts.CPU12.RES:Rescheduling_interrupts
      1457  18%    +585.6%       9988  52%  interrupts.CPU13.CAL:Function_call_interrupts
     26741  57%    +330.3%     115064  10%  interrupts.CPU13.RES:Rescheduling_interrupts
      1440   8%    +626.7%      10466  52%  interrupts.CPU14.CAL:Function_call_interrupts
     27280  58%    +321.3%     114920  10%  interrupts.CPU14.RES:Rescheduling_interrupts
      1547  13%    +550.6%      10065  51%  interrupts.CPU15.CAL:Function_call_interrupts
     24652  60%    +366.8%     115077  10%  interrupts.CPU15.RES:Rescheduling_interrupts
      1613  14%    +518.7%       9981  52%  interrupts.CPU16.CAL:Function_call_interrupts
     26360  60%    +338.3%     115544   9%  interrupts.CPU16.RES:Rescheduling_interrupts
      1783  22%    +476.1%      10273  50%  interrupts.CPU17.CAL:Function_call_interrupts
     29465  57%    +290.9%     115190  10%  interrupts.CPU17.RES:Rescheduling_interrupts
      1692  15%    +539.1%      10816  49%  interrupts.CPU18.CAL:Function_call_interrupts
     25242  56%    +355.0%     114843   8%  interrupts.CPU18.RES:Rescheduling_interrupts
      1546  14%    +576.6%      10463  48%  interrupts.CPU19.CAL:Function_call_interrupts
     27541  53%    +318.0%     115135  10%  interrupts.CPU19.RES:Rescheduling_interrupts
      1539  20%    +597.8%      10744  50%  interrupts.CPU2.CAL:Function_call_interrupts
     26770  52%    +333.8%     116123  10%  interrupts.CPU2.RES:Rescheduling_interrupts
      1904  25%    +432.4%      10138  52%  interrupts.CPU20.CAL:Function_call_interrupts
     25460  61%    +353.1%     115361   8%  interrupts.CPU20.RES:Rescheduling_interrupts
      1762  23%    +469.0%      10028  47%  interrupts.CPU21.CAL:Function_call_interrupts
     26522  54%    +331.6%     114472   8%  interrupts.CPU21.RES:Rescheduling_interrupts
      1684  19%    +506.5%      10214  49%  interrupts.CPU22.CAL:Function_call_interrupts
     26964  52%    +320.4%     113356   9%  interrupts.CPU22.RES:Rescheduling_interrupts
      1664  16%    +552.7%      10863  47%  interrupts.CPU23.CAL:Function_call_interrupts
     25881  53%    +331.9%     111768   9%  interrupts.CPU23.RES:Rescheduling_interrupts
      2767  70%    +456.6%      15402  58%  interrupts.CPU24.CAL:Function_call_interrupts
     34637  23%    +245.6%     119696   7%  interrupts.CPU24.RES:Rescheduling_interrupts
      1644  19%    +790.7%      14649  62%  interrupts.CPU25.CAL:Function_call_interrupts
     31698  34%    +271.3%     117692   9%  interrupts.CPU25.RES:Rescheduling_interrupts
      1600  20%    +885.1%      15766  63%  interrupts.CPU26.CAL:Function_call_interrupts
     32053  35%    +263.6%     116533   8%  interrupts.CPU26.RES:Rescheduling_interrupts
      1906  40%    +773.1%      16646  65%  interrupts.CPU27.CAL:Function_call_interrupts
     32311  37%    +257.4%     115489   8%  interrupts.CPU27.RES:Rescheduling_interrupts
      1847  15%    +729.0%      15312  64%  interrupts.CPU28.CAL:Function_call_interrupts
     30973  38%    +271.9%     115189   8%  interrupts.CPU28.RES:Rescheduling_interrupts
      1637  24%    +825.0%      15146  64%  interrupts.CPU29.CAL:Function_call_interrupts
     32830  34%    +250.8%     115184   6%  interrupts.CPU29.RES:Rescheduling_interrupts
      1433  14%    +603.8%      10090  50%  interrupts.CPU3.CAL:Function_call_interrupts
     26983  49%    +320.2%     113392  10%  interrupts.CPU3.RES:Rescheduling_interrupts
      1597  14%    +863.9%      15400  62%  interrupts.CPU30.CAL:Function_call_interrupts
     31376  38%    +260.7%     113167   8%  interrupts.CPU30.RES:Rescheduling_interrupts
      1648  16%    +828.7%      15304  62%  interrupts.CPU31.CAL:Function_call_interrupts
     32435  33%    +258.4%     116241   8%  interrupts.CPU31.RES:Rescheduling_interrupts
      2081  28%    +635.8%      15314  64%  interrupts.CPU32.CAL:Function_call_interrupts
     30678  34%    +276.2%     115409   8%  interrupts.CPU32.RES:Rescheduling_interrupts
      1508  19%    +951.6%      15860  65%  interrupts.CPU33.CAL:Function_call_interrupts
     32920  39%    +251.5%     115730   8%  interrupts.CPU33.RES:Rescheduling_interrupts
      1529  22%    +936.7%      15857  65%  interrupts.CPU34.CAL:Function_call_interrupts
     31214  27%    +266.4%     114381   8%  interrupts.CPU34.RES:Rescheduling_interrupts
      1543  19%    +933.2%      15950  60%  interrupts.CPU35.CAL:Function_call_interrupts
     30887  33%    +269.9%     114244   7%  interrupts.CPU35.RES:Rescheduling_interrupts
      1566  17%    +890.1%      15505  64%  interrupts.CPU36.CAL:Function_call_interrupts
     31545  41%    +254.5%     111828   9%  interrupts.CPU36.RES:Rescheduling_interrupts
      1518  17%    +889.4%      15021  61%  interrupts.CPU37.CAL:Function_call_interrupts
     31749  38%    +266.0%     116188   9%  interrupts.CPU37.RES:Rescheduling_interrupts
      1507  18%    +946.1%      15770  62%  interrupts.CPU38.CAL:Function_call_interrupts
     32670  36%    +257.7%     116863   7%  interrupts.CPU38.RES:Rescheduling_interrupts
      1662  22%    +818.7%      15277  63%  interrupts.CPU39.CAL:Function_call_interrupts
     32667  37%    +255.3%     116081   9%  interrupts.CPU39.RES:Rescheduling_interrupts
      1908  49%    +447.9%      10454  49%  interrupts.CPU4.CAL:Function_call_interrupts
     26121  56%    +341.3%     115279   7%  interrupts.CPU4.RES:Rescheduling_interrupts
      1599  30%    +883.5%      15734  66%  interrupts.CPU40.CAL:Function_call_interrupts
     31759  37%    +259.8%     114285   8%  interrupts.CPU40.RES:Rescheduling_interrupts
      1535  19%    +889.5%      15194  62%  interrupts.CPU41.CAL:Function_call_interrupts
     31350  37%    +263.1%     113826   9%  interrupts.CPU41.RES:Rescheduling_interrupts
      1565  19%    +959.8%      16588  63%  interrupts.CPU42.CAL:Function_call_interrupts
     30781  33%    +269.8%     113842   7%  interrupts.CPU42.RES:Rescheduling_interrupts
      1581  11%    +929.0%      16272  63%  interrupts.CPU43.CAL:Function_call_interrupts
     31679  33%    +268.0%     116568   7%  interrupts.CPU43.RES:Rescheduling_interrupts
      1520  22%    +914.5%      15422  65%  interrupts.CPU44.CAL:Function_call_interrupts
     31967  41%    +264.6%     116553   9%  interrupts.CPU44.RES:Rescheduling_interrupts
      1503  16%    +931.3%      15503  63%  interrupts.CPU45.CAL:Function_call_interrupts
     30829  32%    +272.3%     114770   8%  interrupts.CPU45.RES:Rescheduling_interrupts
      1492  15%    +948.3%      15643  64%  interrupts.CPU46.CAL:Function_call_interrupts
     32136  37%    +256.1%     114426   8%  interrupts.CPU46.RES:Rescheduling_interrupts
      6530   4%    +186.1%      18685  49%  interrupts.CPU47.CAL:Function_call_interrupts
     32843  36%    +239.2%     111397   8%  interrupts.CPU47.RES:Rescheduling_interrupts
      1501  21%    +552.3%       9795  49%  interrupts.CPU48.CAL:Function_call_interrupts
     27770  49%    +306.1%     112779  10%  interrupts.CPU48.RES:Rescheduling_interrupts
      1716  12%    +479.8%       9951  47%  interrupts.CPU49.CAL:Function_call_interrupts
     27441  50%    +306.8%     111620  12%  interrupts.CPU49.RES:Rescheduling_interrupts
      1536  20%    +561.4%      10164  45%  interrupts.CPU5.CAL:Function_call_interrupts
     26043  58%    +336.2%     113601   8%  interrupts.CPU5.RES:Rescheduling_interrupts
      1466  15%    +652.1%      11028  52%  interrupts.CPU50.CAL:Function_call_interrupts
     25157  58%    +346.9%     112432  11%  interrupts.CPU50.RES:Rescheduling_interrupts
      1480  14%    +607.5%      10472  49%  interrupts.CPU51.CAL:Function_call_interrupts
     25280  53%    +326.7%     107877  10%  interrupts.CPU51.RES:Rescheduling_interrupts
      1718  28%    +513.8%      10546  49%  interrupts.CPU52.CAL:Function_call_interrupts
     25402  61%    +343.7%     112698   9%  interrupts.CPU52.RES:Rescheduling_interrupts
      1471  18%    +584.4%      10067  51%  interrupts.CPU53.CAL:Function_call_interrupts
     26502  51%    +317.7%     110702   9%  interrupts.CPU53.RES:Rescheduling_interrupts
      1520  10%    +577.5%      10301  49%  interrupts.CPU54.CAL:Function_call_interrupts
     26868  53%    +311.4%     110540  11%  interrupts.CPU54.RES:Rescheduling_interrupts
      1378  15%    +629.6%      10059  48%  interrupts.CPU55.CAL:Function_call_interrupts
     26538  55%    +328.1%     113615   9%  interrupts.CPU55.RES:Rescheduling_interrupts
      1432  12%    +613.7%      10226  50%  interrupts.CPU56.CAL:Function_call_interrupts
     27378  54%    +318.3%     114536   9%  interrupts.CPU56.RES:Rescheduling_interrupts
      1578  23%    +551.4%      10283  51%  interrupts.CPU57.CAL:Function_call_interrupts
     25737  62%    +337.5%     112612  10%  interrupts.CPU57.RES:Rescheduling_interrupts
      1483  22%    +630.0%      10829  52%  interrupts.CPU58.CAL:Function_call_interrupts
     27343  53%    +315.8%     113705   9%  interrupts.CPU58.RES:Rescheduling_interrupts
      1460  18%    +602.5%      10261  51%  interrupts.CPU59.CAL:Function_call_interrupts
     25911  58%    +298.8%     103334   7%  interrupts.CPU59.RES:Rescheduling_interrupts
      1569   8%    +530.0%       9889  50%  interrupts.CPU6.CAL:Function_call_interrupts
     27162  52%    +310.3%     111462  10%  interrupts.CPU6.RES:Rescheduling_interrupts
      1587  31%    +530.4%      10005  51%  interrupts.CPU60.CAL:Function_call_interrupts
     26060  56%    +330.3%     112129   9%  interrupts.CPU60.RES:Rescheduling_interrupts
      1414  16%    +608.1%      10012  49%  interrupts.CPU61.CAL:Function_call_interrupts
     25916  59%    +345.2%     115375  10%  interrupts.CPU61.RES:Rescheduling_interrupts
      1501  18%    +601.8%      10534  53%  interrupts.CPU62.CAL:Function_call_interrupts
     27209  60%    +317.1%     113499   9%  interrupts.CPU62.RES:Rescheduling_interrupts
      1411  18%    +612.3%      10052  51%  interrupts.CPU63.CAL:Function_call_interrupts
     25584  59%    +345.4%     113944   9%  interrupts.CPU63.RES:Rescheduling_interrupts
      1890  17%    +437.8%      10169  50%  interrupts.CPU64.CAL:Function_call_interrupts
     26884  60%    +322.0%     113444  10%  interrupts.CPU64.RES:Rescheduling_interrupts
      1729  25%    +475.6%       9955  48%  interrupts.CPU65.CAL:Function_call_interrupts
     26952  54%    +321.5%     113608  10%  interrupts.CPU65.RES:Rescheduling_interrupts
      1897  16%    +481.0%      11020  46%  interrupts.CPU66.CAL:Function_call_interrupts
     25947  58%    +327.7%     110971   9%  interrupts.CPU66.RES:Rescheduling_interrupts
      1761  18%    +467.0%       9987  50%  interrupts.CPU67.CAL:Function_call_interrupts
     27578  52%    +307.5%     112380  10%  interrupts.CPU67.RES:Rescheduling_interrupts
      1784  18%    +451.1%       9835  49%  interrupts.CPU68.CAL:Function_call_interrupts
     25578  60%    +340.7%     112722  10%  interrupts.CPU68.RES:Rescheduling_interrupts
      1837  17%    +436.5%       9857  49%  interrupts.CPU69.CAL:Function_call_interrupts
     26266  59%    +329.5%     112809   9%  interrupts.CPU69.RES:Rescheduling_interrupts
      1465  15%    +583.8%      10020  49%  interrupts.CPU7.CAL:Function_call_interrupts
     25720  57%    +351.1%     116026   8%  interrupts.CPU7.RES:Rescheduling_interrupts
      1810  21%    +425.7%       9515  44%  interrupts.CPU70.CAL:Function_call_interrupts
     27634  51%    +304.4%     111767  10%  interrupts.CPU70.RES:Rescheduling_interrupts
      1891  12%    +415.1%       9740  48%  interrupts.CPU71.CAL:Function_call_interrupts
     26205  55%    +323.3%     110934   9%  interrupts.CPU71.RES:Rescheduling_interrupts
      2946  93%    +429.1%      15592  63%  interrupts.CPU72.CAL:Function_call_interrupts
     33476  29%    +241.2%     114214   8%  interrupts.CPU72.RES:Rescheduling_interrupts
      1797  28%    +730.7%      14931  64%  interrupts.CPU73.CAL:Function_call_interrupts
     31188  31%    +269.5%     115229   9%  interrupts.CPU73.RES:Rescheduling_interrupts
      1649  23%    +918.0%      16794  64%  interrupts.CPU74.CAL:Function_call_interrupts
     30456  37%    +274.1%     113927   9%  interrupts.CPU74.RES:Rescheduling_interrupts
      1875  31%    +785.5%      16610  62%  interrupts.CPU75.CAL:Function_call_interrupts
     31503  38%    +259.7%     113310   8%  interrupts.CPU75.RES:Rescheduling_interrupts
      1833  17%    +766.5%      15885  64%  interrupts.CPU76.CAL:Function_call_interrupts
     29718  38%    +278.7%     112554   9%  interrupts.CPU76.RES:Rescheduling_interrupts
      1650  15%    +857.9%      15811  66%  interrupts.CPU77.CAL:Function_call_interrupts
     31625  29%    +252.2%     111391   7%  interrupts.CPU77.RES:Rescheduling_interrupts
      1740  16%    +798.0%      15632  62%  interrupts.CPU78.CAL:Function_call_interrupts
     32549  34%    +239.6%     110549   9%  interrupts.CPU78.RES:Rescheduling_interrupts
      1675  19%    +830.7%      15595  64%  interrupts.CPU79.CAL:Function_call_interrupts
     31016  36%    +266.4%     113656   8%  interrupts.CPU79.RES:Rescheduling_interrupts
      1437  11%    +635.4%      10570  49%  interrupts.CPU8.CAL:Function_call_interrupts
     27602  60%    +322.8%     116699  10%  interrupts.CPU8.RES:Rescheduling_interrupts
      1683  27%    +802.2%      15187  65%  interrupts.CPU80.CAL:Function_call_interrupts
     31149  43%    +268.5%     114795   8%  interrupts.CPU80.RES:Rescheduling_interrupts
      1583  24%    +911.9%      16020  66%  interrupts.CPU81.CAL:Function_call_interrupts
     31699  38%    +260.9%     114418   8%  interrupts.CPU81.RES:Rescheduling_interrupts
      1752  36%    +827.9%      16262  66%  interrupts.CPU82.CAL:Function_call_interrupts
     29360  36%    +288.9%     114167   8%  interrupts.CPU82.RES:Rescheduling_interrupts
      1558  13%    +932.4%      16090  64%  interrupts.CPU83.CAL:Function_call_interrupts
     30230  36%    +270.2%     111904   8%  interrupts.CPU83.RES:Rescheduling_interrupts
      1455  17%    +958.3%      15407  64%  interrupts.CPU84.CAL:Function_call_interrupts
     30911  34%    +254.7%     109645  10%  interrupts.CPU84.RES:Rescheduling_interrupts
      1603  10%    +827.8%      14881  62%  interrupts.CPU85.CAL:Function_call_interrupts
     31085  37%    +265.4%     113578   8%  interrupts.CPU85.RES:Rescheduling_interrupts
      1575  17%    +904.7%      15827  64%  interrupts.CPU86.CAL:Function_call_interrupts
     33065  38%    +248.7%     115301   8%  interrupts.CPU86.RES:Rescheduling_interrupts
      1498  18%    +920.9%      15295  64%  interrupts.CPU87.CAL:Function_call_interrupts
     30477  42%    +274.6%     114181   9%  interrupts.CPU87.RES:Rescheduling_interrupts
      1418  17%    +960.6%      15044  64%  interrupts.CPU88.CAL:Function_call_interrupts
     31735  32%    +255.3%     112741   9%  interrupts.CPU88.RES:Rescheduling_interrupts
      1459  21%    +984.3%      15826  65%  interrupts.CPU89.CAL:Function_call_interrupts
     31037  41%    +260.3%     111826   9%  interrupts.CPU89.RES:Rescheduling_interrupts
      1505  18%    +562.6%       9975  47%  interrupts.CPU9.CAL:Function_call_interrupts
     27015  64%    +327.1%     115392   9%  interrupts.CPU9.RES:Rescheduling_interrupts
      1592  24%    +935.1%      16487  66%  interrupts.CPU90.CAL:Function_call_interrupts
     31256  37%    +257.9%     111877   8%  interrupts.CPU90.RES:Rescheduling_interrupts
      1576  19%    +882.7%      15489  66%  interrupts.CPU91.CAL:Function_call_interrupts
     31276  37%    +266.3%     114573   8%  interrupts.CPU91.RES:Rescheduling_interrupts
      1706  18%    +786.3%      15121  66%  interrupts.CPU92.CAL:Function_call_interrupts
     31400  43%    +260.4%     113170   8%  interrupts.CPU92.RES:Rescheduling_interrupts
      1347  18%   +1012.1%      14981  66%  interrupts.CPU93.CAL:Function_call_interrupts
     29869  35%    +277.4%     112727   8%  interrupts.CPU93.RES:Rescheduling_interrupts
      1372  12%    +937.5%      14243  67%  interrupts.CPU94.CAL:Function_call_interrupts
     30286  40%    +271.0%     112366   7%  interrupts.CPU94.RES:Rescheduling_interrupts
      1739  27%    +735.8%      14538  64%  interrupts.CPU95.CAL:Function_call_interrupts
     30315  39%    +266.8%     111198   8%  interrupts.CPU95.RES:Rescheduling_interrupts
   2791109  14%    +291.3%   10921566   8%  interrupts.RES:Rescheduling_interrupts
      1.73  27%      -1.5        0.26 100%  perf-profile.calltrace.cycles-pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.22  22%      -1.2        0.97        perf-profile.calltrace.cycles-pp.kill
      2.08  23%      -1.2        0.84        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill
      2.03  24%      -1.2        0.80        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
      1.99  24%      -1.2        0.76        perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
      1.85  26%      -1.2        0.62        perf-profile.calltrace.cycles-pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
     45.78            -0.9       44.84        perf-profile.calltrace.cycles-pp.__send
     45.45            -0.9       44.56        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send
     44.92            -0.6       44.37        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     44.85            -0.5       44.31        perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     44.78            -0.5       44.25        perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
     44.54            -0.5       44.03        perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.47   4%      -0.2        1.29   2%  perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
      2.48            -0.1        2.33        perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
      1.66            -0.1        1.57        perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
      0.74   3%      -0.1        0.64   2%  perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      0.62   5%      -0.1        0.54   2%  perf-profile.calltrace.cycles-pp.skb_release_all.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      0.61   4%      -0.1        0.53   2%  perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.__kfree_skb.tcp_recvmsg_locked.tcp_recvmsg
      0.88   3%      -0.1        0.79        perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit
      0.80   4%      -0.1        0.72        perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output
      1.27   3%      -0.1        1.20        perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.14   3%      -0.1        1.08        perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.96   2%      -0.1        0.90        perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
      1.59            -0.1        1.53        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
      1.57            -0.1        1.51        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
      1.06   3%      -0.1        1.01        perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
      0.90   2%      -0.1        0.85        perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
      0.95            -0.0        0.91        perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg
      1.76   3%      +0.1        1.82        perf-profile.calltrace.cycles-pp.perf_swevent_overflow.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle
      1.81   3%      +0.1        1.88        perf-profile.calltrace.cycles-pp.perf_tp_event.perf_trace_sched_switch.__schedule.schedule_idle.do_idle
      0.61   2%      +0.1        0.68        perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      1.94   3%      +0.1        2.01        perf-profile.calltrace.cycles-pp.perf_trace_sched_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.58   2%      +0.1        0.67   2%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.89   6%      +0.2        1.07   5%  perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up
      0.92   3%      +0.2        1.10   5%  perf-profile.calltrace.cycles-pp.update_load_avg.dequeue_entity.dequeue_task_fair.__schedule.schedule
      2.55            +0.2        2.78        perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      6.25   3%      +0.2        6.48        perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_task_fair.__schedule.schedule
      2.10   3%      +0.2        2.34        perf-profile.calltrace.cycles-pp._raw_spin_lock.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      1.96   3%      +0.2        2.20        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      3.94   2%      +0.2        4.19        perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common
      4.83   2%      +0.3        5.11        perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
      4.86   2%      +0.3        5.15        perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      5.42   2%      +0.3        5.72        perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
      3.86   2%      +0.5        4.31        perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      3.90   2%      +0.5        4.36        perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.18 141%      +0.5        0.66   5%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__schedule.schedule_idle.do_idle.cpu_startup_entry
      0.00            +0.5        0.53   2%  perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__schedule.schedule_idle.do_idle
      7.74   2%      +0.6        8.32        perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.schedule_timeout
     33.75            +0.6       34.34        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
      0.00            +0.6        0.59   6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__schedule.schedule_idle.do_idle
     23.16            +0.6       23.75        perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
     23.06            +0.6       23.66        perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
      8.59   2%      +0.6        9.21        perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_timeout.wait_woken
     22.39            +0.6       23.03        perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
     33.98            +0.7       34.64        perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
     33.89            +0.7       34.56        perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
     16.58   2%      +0.7       17.32        perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data
     31.96            +0.7       32.70        perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
     16.76   2%      +0.8       17.51        perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked
     16.91   2%      +0.8       17.66        perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg
     17.24   2%      +0.8       18.00        perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
     18.29            +0.8       19.07        perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
     31.40            +0.9       32.29        perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll
     31.33            +0.9       32.22        perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
     31.29            +0.9       32.18        perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
     31.15            +0.9       32.05        perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
     29.63            +1.0       30.61        perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
     29.48            +1.0       30.49        perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
      0.00            +1.1        1.05        perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up
     24.86   2%      +1.2       26.07        perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
     24.61   2%      +1.2       25.82        perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
     25.65   2%      +1.2       26.87        perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
     25.34   2%      +1.2       26.58        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
     14.20   2%      +1.6       15.78        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     14.62   2%      +1.6       16.26        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     14.64   2%      +1.6       16.28        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
      0.00            +2.0        2.03        perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up_common
     19.87   2%      +2.3       22.16        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     19.89   2%      +2.3       22.19        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     19.89   2%      +2.3       22.19        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     20.12   2%      +2.3       22.45        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
      0.00            +2.4        2.36        perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock
     74.35            -2.1       72.23        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     73.04            -1.8       71.27        perf-profile.children.cycles-pp.do_syscall_64
      2.25  22%      -1.3        0.99        perf-profile.children.cycles-pp.kill
      1.99  24%      -1.2        0.76        perf-profile.children.cycles-pp.__x64_sys_kill
      1.85  26%      -1.2        0.62        perf-profile.children.cycles-pp.kill_something_info
      1.73  27%      -1.2        0.50        perf-profile.children.cycles-pp.kill_pid_info
      1.71  28%      -1.2        0.48        perf-profile.children.cycles-pp.group_send_sig_info
      1.59  30%      -1.2        0.38        perf-profile.children.cycles-pp.security_task_kill
      1.58  30%      -1.2        0.37        perf-profile.children.cycles-pp.apparmor_task_kill
     45.86            -1.0       44.90        perf-profile.children.cycles-pp.__send
      1.28  21%      -0.9        0.39   2%  perf-profile.children.cycles-pp.aa_sk_perm
      0.85  30%      -0.6        0.24   3%  perf-profile.children.cycles-pp.aa_get_task_label
      0.91  19%      -0.6        0.32   3%  perf-profile.children.cycles-pp.security_socket_recvmsg
      0.93  19%      -0.6        0.34   3%  perf-profile.children.cycles-pp.sock_recvmsg
     44.86            -0.5       44.31        perf-profile.children.cycles-pp.__x64_sys_sendto
     44.79            -0.5       44.25        perf-profile.children.cycles-pp.__sys_sendto
     44.54            -0.5       44.03        perf-profile.children.cycles-pp.sock_sendmsg
      1.09   8%      -0.3        0.75        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
      0.91   9%      -0.3        0.59        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
      0.44  22%      -0.3        0.14   2%  perf-profile.children.cycles-pp.security_socket_sendmsg
      1.53   4%      -0.2        1.35   2%  perf-profile.children.cycles-pp.__dev_queue_xmit
      2.59            -0.1        2.44        perf-profile.children.cycles-pp.tcp_ack
      1.46   2%      -0.1        1.34        perf-profile.children.cycles-pp.__kfree_skb
      1.74   2%      -0.1        1.64        perf-profile.children.cycles-pp.tcp_clean_rtx_queue
      0.72   4%      -0.1        0.62   2%  perf-profile.children.cycles-pp.skb_release_all
      0.70   4%      -0.1        0.61   2%  perf-profile.children.cycles-pp.skb_release_head_state
      0.82   4%      -0.1        0.74        perf-profile.children.cycles-pp.loopback_xmit
      0.90   4%      -0.1        0.82        perf-profile.children.cycles-pp.dev_hard_start_xmit
      1.28   3%      -0.1        1.20        perf-profile.children.cycles-pp._copy_from_iter_full
      0.69   4%      -0.1        0.62   2%  perf-profile.children.cycles-pp._raw_spin_lock_bh
      0.54   6%      -0.1        0.46   2%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.53   4%      -0.1        0.45   2%  perf-profile.children.cycles-pp.dst_release
      1.58            -0.1        1.51        perf-profile.children.cycles-pp.__skb_datagram_iter
      0.96   2%      -0.1        0.90        perf-profile.children.cycles-pp.sk_stream_alloc_skb
      1.60            -0.1        1.53        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      0.40   3%      -0.1        0.34   4%  perf-profile.children.cycles-pp.tcp_send_mss
      0.36   4%      -0.1        0.31   3%  perf-profile.children.cycles-pp.tcp_current_mss
      0.91   2%      -0.1        0.86        perf-profile.children.cycles-pp.__alloc_skb
      0.23   5%      -0.0        0.18   4%  perf-profile.children.cycles-pp.irqtime_account_irq
      0.27   5%      -0.0        0.23        perf-profile.children.cycles-pp.load_new_mm_cr3
      0.17   2%      -0.0        0.13   4%  perf-profile.children.cycles-pp.tcp_ack_update_rtt
      0.53            -0.0        0.49   2%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.61   4%      -0.0        0.58   3%  perf-profile.children.cycles-pp.release_sock
      0.33   8%      -0.0        0.29   2%  perf-profile.children.cycles-pp.next_token
      0.95            -0.0        0.92        perf-profile.children.cycles-pp._copy_to_iter
      0.49   2%      -0.0        0.46   2%  perf-profile.children.cycles-pp.switch_fpu_return
      0.41   3%      -0.0        0.38   3%  perf-profile.children.cycles-pp.tcp_event_new_data_sent
      0.39   3%      -0.0        0.36   2%  perf-profile.children.cycles-pp.netif_rx
      0.31   2%      -0.0        0.28        perf-profile.children.cycles-pp.reweight_entity
      0.21   4%      -0.0        0.18   4%  perf-profile.children.cycles-pp.kmem_cache_alloc_node
      0.33   2%      -0.0        0.30   3%  perf-profile.children.cycles-pp.__entry_text_start
      0.20   4%      -0.0        0.17   3%  perf-profile.children.cycles-pp.enqueue_to_backlog
      0.38            -0.0        0.35   2%  perf-profile.children.cycles-pp.__check_object_size
      0.38   3%      -0.0        0.35        perf-profile.children.cycles-pp.netif_rx_internal
      0.32   3%      -0.0        0.29        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.16   4%      -0.0        0.13   5%  perf-profile.children.cycles-pp.ip_local_out
      0.28   3%      -0.0        0.25   2%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.13   5%      -0.0        0.11   6%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.13   3%      -0.0        0.11   6%  perf-profile.children.cycles-pp.nf_hook_slow
      0.31            -0.0        0.29   2%  perf-profile.children.cycles-pp.simple_copy_to_iter
      0.11   6%      -0.0        0.09   5%  perf-profile.children.cycles-pp.import_single_range
      0.11   9%      -0.0        0.09   4%  perf-profile.children.cycles-pp.sock_put
      0.26   2%      -0.0        0.24   3%  perf-profile.children.cycles-pp.check_preempt_curr
      0.09   5%      -0.0        0.07   7%  perf-profile.children.cycles-pp.__might_fault
      0.13   5%      -0.0        0.11   4%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
      0.17   2%      -0.0        0.15   3%  perf-profile.children.cycles-pp.tcp_rearm_rto
      0.13   6%      -0.0        0.11   5%  perf-profile.children.cycles-pp.__cond_resched
      0.10   7%      -0.0        0.09   4%  perf-profile.children.cycles-pp.tcp_event_data_recv
      0.11   6%      -0.0        0.09   5%  perf-profile.children.cycles-pp.check_kill_permission
      0.23   3%      -0.0        0.22        perf-profile.children.cycles-pp.__ksize
      0.10   4%      -0.0        0.08   5%  perf-profile.children.cycles-pp.tcp_update_pacing_rate
      0.07   7%      -0.0        0.05   8%  perf-profile.children.cycles-pp.server
      0.08   4%      +0.0        0.09   4%  perf-profile.children.cycles-pp.attach_entity_load_avg
      0.07   5%      +0.0        0.09   5%  perf-profile.children.cycles-pp.is_module_text_address
      0.10   8%      +0.0        0.12   5%  perf-profile.children.cycles-pp.set_next_task_idle
      0.09   5%      +0.0        0.11   4%  perf-profile.children.cycles-pp.__update_idle_core
      0.29            +0.0        0.32   2%  perf-profile.children.cycles-pp.tcp_cleanup_rbuf
      0.30            +0.0        0.33        perf-profile.children.cycles-pp.kmem_cache_free
      0.13   5%      +0.0        0.15   3%  perf-profile.children.cycles-pp.perf_instruction_pointer
      0.08   7%      +0.0        0.10   6%  perf-profile.children.cycles-pp.cpus_share_cache
      0.10   4%      +0.0        0.13   5%  perf-profile.children.cycles-pp.pick_next_task_idle
      0.20   3%      +0.0        0.23   2%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.55   2%      +0.0        0.58        perf-profile.children.cycles-pp.update_rq_clock
      0.32   4%      +0.0        0.34   2%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.23   2%      +0.0        0.26   2%  perf-profile.children.cycles-pp._find_next_bit
      0.19   5%      +0.0        0.22   3%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.49   2%      +0.0        0.52   2%  perf-profile.children.cycles-pp.finish_task_switch
      0.33   3%      +0.0        0.37        perf-profile.children.cycles-pp.migrate_task_rq_fair
      0.22   5%      +0.0        0.26   4%  perf-profile.children.cycles-pp.start_kernel
      0.19   2%      +0.0        0.23   2%  perf-profile.children.cycles-pp.cpumask_next
      0.29   4%      +0.0        0.34   2%  perf-profile.children.cycles-pp.__slab_free
      0.60            +0.0        0.65   2%  perf-profile.children.cycles-pp.set_next_entity
      0.90            +0.0        0.94        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.62   2%      +0.1        0.69        perf-profile.children.cycles-pp.menu_select
      2.01            +0.1        2.10        perf-profile.children.cycles-pp.select_idle_cpu
      1.16            +0.1        1.27        perf-profile.children.cycles-pp.available_idle_cpu
      2.62            +0.2        2.86        perf-profile.children.cycles-pp.select_task_rq_fair
      4.07   2%      +0.2        4.32        perf-profile.children.cycles-pp.enqueue_entity
      5.00   2%      +0.3        5.28        perf-profile.children.cycles-pp.ttwu_do_activate
      4.97   2%      +0.3        5.26        perf-profile.children.cycles-pp.enqueue_task_fair
      5.60   3%      +0.3        5.90        perf-profile.children.cycles-pp.set_task_cpu
      2.54   4%      +0.4        2.91   4%  perf-profile.children.cycles-pp.update_load_avg
      3.37   2%      +0.4        3.76        perf-profile.children.cycles-pp._raw_spin_lock
      2.61   3%      +0.4        3.02   2%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.46   8%      +0.4        1.88   8%  perf-profile.children.cycles-pp.update_cfs_group
      3.95   2%      +0.5        4.42        perf-profile.children.cycles-pp.schedule_idle
     17.05            +0.5       17.54        perf-profile.children.cycles-pp.schedule
      7.76   2%      +0.6        8.34        perf-profile.children.cycles-pp.dequeue_entity
     23.16            +0.6       23.76        perf-profile.children.cycles-pp.inet_recvmsg
     23.07            +0.6       23.67        perf-profile.children.cycles-pp.tcp_recvmsg
      8.61   2%      +0.6        9.23        perf-profile.children.cycles-pp.dequeue_task_fair
     22.41            +0.6       23.05        perf-profile.children.cycles-pp.tcp_recvmsg_locked
     34.18            +0.7       34.86        perf-profile.children.cycles-pp.__local_bh_enable_ip
     33.99            +0.7       34.68        perf-profile.children.cycles-pp.do_softirq
     34.00            +0.7       34.69        perf-profile.children.cycles-pp.__softirqentry_text_start
     33.40            +0.7       34.14        perf-profile.children.cycles-pp.net_rx_action
     33.18            +0.8       33.94        perf-profile.children.cycles-pp.__napi_poll
     16.91   2%      +0.8       17.66        perf-profile.children.cycles-pp.schedule_timeout
     17.24   2%      +0.8       18.00        perf-profile.children.cycles-pp.wait_woken
     33.09            +0.8       33.86        perf-profile.children.cycles-pp.process_backlog
     18.30            +0.8       19.07        perf-profile.children.cycles-pp.sk_wait_data
     32.81            +0.8       33.61        perf-profile.children.cycles-pp.__netif_receive_skb_one_core
     32.47            +0.8       33.30        perf-profile.children.cycles-pp.ip_rcv
     31.91            +0.9       32.81        perf-profile.children.cycles-pp.ip_local_deliver_finish
     31.97            +0.9       32.87        perf-profile.children.cycles-pp.ip_local_deliver
     31.86            +0.9       32.77        perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
     31.72            +0.9       32.63        perf-profile.children.cycles-pp.tcp_v4_rcv
     20.81            +0.9       21.74        perf-profile.children.cycles-pp.__schedule
     30.18            +1.0       31.16        perf-profile.children.cycles-pp.tcp_v4_do_rcv
     30.03            +1.0       31.04        perf-profile.children.cycles-pp.tcp_rcv_established
     25.54   2%      +1.2       26.76        perf-profile.children.cycles-pp.__wake_up_common
     25.29   2%      +1.2       26.52        perf-profile.children.cycles-pp.try_to_wake_up
     26.34   2%      +1.2       27.58        perf-profile.children.cycles-pp.sock_def_readable
     26.04   2%      +1.2       27.29        perf-profile.children.cycles-pp.__wake_up_common_lock
     14.36   2%      +1.6       15.96        perf-profile.children.cycles-pp.intel_idle
     14.80   2%      +1.7       16.46        perf-profile.children.cycles-pp.cpuidle_enter_state
     14.80   2%      +1.7       16.47        perf-profile.children.cycles-pp.cpuidle_enter
     19.89   2%      +2.3       22.19        perf-profile.children.cycles-pp.start_secondary
     20.10   2%      +2.3       22.43        perf-profile.children.cycles-pp.do_idle
     20.12   2%      +2.3       22.45        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     20.12   2%      +2.3       22.45        perf-profile.children.cycles-pp.cpu_startup_entry
      0.00            +2.4        2.43        perf-profile.children.cycles-pp.select_idle_sibling
      1.18  22%      -0.9        0.30   2%  perf-profile.self.cycles-pp.aa_sk_perm
      0.84  30%      -0.6        0.23   4%  perf-profile.self.cycles-pp.aa_get_task_label
      0.66  31%      -0.6        0.10   4%  perf-profile.self.cycles-pp.apparmor_task_kill
      0.61   5%      -0.1        0.55   3%  perf-profile.self.cycles-pp._raw_spin_lock_bh
      0.52   4%      -0.1        0.45   2%  perf-profile.self.cycles-pp.dst_release
      0.33   2%      -0.1        0.26   2%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.43   6%      -0.1        0.38   2%  perf-profile.self.cycles-pp.__tcp_transmit_skb
      0.70   3%      -0.1        0.65   2%  perf-profile.self.cycles-pp.tcp_sendmsg_locked
      0.27   5%      -0.0        0.23        perf-profile.self.cycles-pp.load_new_mm_cr3
      0.49   2%      -0.0        0.45   2%  perf-profile.self.cycles-pp.switch_fpu_return
      0.16   4%      -0.0        0.12   3%  perf-profile.self.cycles-pp.tcp_ack_update_rtt
      0.33   3%      -0.0        0.30   3%  perf-profile.self.cycles-pp.tcp_clean_rtx_queue
      0.18   6%      -0.0        0.15   2%  perf-profile.self.cycles-pp.__ip_queue_xmit
      0.34   2%      -0.0        0.30   3%  perf-profile.self.cycles-pp.tcp_write_xmit
      0.25   7%      -0.0        0.22   6%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.13   7%      -0.0        0.10  12%  perf-profile.self.cycles-pp.tcp_current_mss
      0.25   5%      -0.0        0.22   3%  perf-profile.self.cycles-pp.ip_finish_output2
      0.10   7%      -0.0        0.07   8%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
      0.13   5%      -0.0        0.10   3%  perf-profile.self.cycles-pp.do_user_addr_fault
      0.13   5%      -0.0        0.11   6%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.21   6%      -0.0        0.18   3%  perf-profile.self.cycles-pp.tcp_wfree
      0.32   3%      -0.0        0.29   2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.23   4%      -0.0        0.21   2%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.11   8%      -0.0        0.09        perf-profile.self.cycles-pp.import_single_range
      0.15   3%      -0.0        0.14   3%  perf-profile.self.cycles-pp.tcp_rearm_rto
      0.09   8%      -0.0        0.07   5%  perf-profile.self.cycles-pp.sched_clock_cpu
      0.22            -0.0        0.20   3%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.11   5%      -0.0        0.09   5%  perf-profile.self.cycles-pp.__netif_receive_skb_one_core
      0.07   8%      -0.0        0.05   8%  perf-profile.self.cycles-pp.tcp_v4_do_rcv
      0.10   4%      -0.0        0.08   7%  perf-profile.self.cycles-pp.perf_callchain
      0.13   5%      -0.0        0.11   3%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
      0.13   5%      -0.0        0.11   4%  perf-profile.self.cycles-pp.kmem_cache_alloc_node
      0.09            -0.0        0.08   6%  perf-profile.self.cycles-pp.__napi_poll
      0.08   6%      -0.0        0.06   6%  perf-profile.self.cycles-pp.__x64_sys_sendto
      0.12   3%      -0.0        0.11   3%  perf-profile.self.cycles-pp.__sys_sendto
      0.11   6%      -0.0        0.09   5%  perf-profile.self.cycles-pp.__x64_sys_recvfrom
      0.14   3%      -0.0        0.12   3%  perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      0.07   7%      -0.0        0.05   8%  perf-profile.self.cycles-pp.ftrace_ops_trampoline
      0.09   5%      -0.0        0.08   5%  perf-profile.self.cycles-pp.tcp_update_pacing_rate
      0.06   8%      +0.0        0.07        perf-profile.self.cycles-pp.cpuidle_enter_state
      0.06            +0.0        0.07   6%  perf-profile.self.cycles-pp.rcu_nocb_flush_deferred_wakeup
      0.08   7%      +0.0        0.10   4%  perf-profile.self.cycles-pp.cpus_share_cache
      0.05   8%      +0.0        0.07   5%  perf-profile.self.cycles-pp.perf_instruction_pointer
      0.28            +0.0        0.30   3%  perf-profile.self.cycles-pp.tcp_cleanup_rbuf
      0.18   5%      +0.0        0.20        perf-profile.self.cycles-pp.migrate_task_rq_fair
      0.24   3%      +0.0        0.27   2%  perf-profile.self.cycles-pp.menu_select
      0.41   3%      +0.0        0.44   2%  perf-profile.self.cycles-pp.update_rq_clock
      0.22   4%      +0.0        0.26   2%  perf-profile.self.cycles-pp._find_next_bit
      0.16   2%      +0.0        0.20   3%  perf-profile.self.cycles-pp.do_idle
      0.29   3%      +0.0        0.34   2%  perf-profile.self.cycles-pp.__slab_free
      0.87            +0.0        0.92        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.24            +0.0        1.29   2%  perf-profile.self.cycles-pp.__schedule
      1.15            +0.1        1.26        perf-profile.self.cycles-pp.available_idle_cpu
      0.00            +0.1        0.14   3%  perf-profile.self.cycles-pp.select_idle_sibling
      0.43   7%      +0.2        0.61   8%  perf-profile.self.cycles-pp.set_task_cpu
      1.78   4%      +0.3        2.12   5%  perf-profile.self.cycles-pp.update_load_avg
      2.61   3%      +0.4        3.01   2%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      1.46   8%      +0.4        1.88   8%  perf-profile.self.cycles-pp.update_cfs_group
     14.36   2%      +1.6       15.96        perf-profile.self.cycles-pp.intel_idle
  20074513   7%     -24.9%   15075828   7%  softirqs.CPU0.NET_RX
     38385  66%    +476.2%     221177   9%  softirqs.CPU0.SCHED
     32352  12%     -78.4%       6999  45%  softirqs.CPU0.TIMER
  20359880   5%     -26.9%   14879840   7%  softirqs.CPU1.NET_RX
     36748  67%    +492.7%     217810  10%  softirqs.CPU1.SCHED
     31252  13%     -85.5%       4524  59%  softirqs.CPU1.TIMER
  20405448   5%     -25.5%   15210284   7%  softirqs.CPU10.NET_RX
     36443  69%    +499.5%     218467   9%  softirqs.CPU10.SCHED
     30609  11%     -84.5%       4741  63%  softirqs.CPU10.TIMER
  20394573   4%     -29.0%   14484845  10%  softirqs.CPU11.NET_RX
     36636  70%    +464.1%     206683   7%  softirqs.CPU11.SCHED
     30678  11%     -84.8%       4653  67%  softirqs.CPU11.TIMER
  20405829   4%     -26.1%   15089688   7%  softirqs.CPU12.NET_RX
     37061  70%    +494.1%     220185   8%  softirqs.CPU12.SCHED
     31218  10%     -84.2%       4919  66%  softirqs.CPU12.TIMER
  20444043   5%     -26.2%   15095203   7%  softirqs.CPU13.NET_RX
     36503  70%    +497.6%     218132   9%  softirqs.CPU13.SCHED
     30651  11%     -84.6%       4710  62%  softirqs.CPU13.TIMER
  20419827   5%     -25.6%   15186252   7%  softirqs.CPU14.NET_RX
     36704  70%    +494.3%     218149   9%  softirqs.CPU14.SCHED
     30681  11%     -84.3%       4806  65%  softirqs.CPU14.TIMER
  20510872   5%     -25.7%   15232936   7%  softirqs.CPU15.NET_RX
     36871  69%    +493.3%     218761   9%  softirqs.CPU15.SCHED
     31009  12%     -84.5%       4793  63%  softirqs.CPU15.TIMER
  20418436   5%     -25.4%   15224178   7%  softirqs.CPU16.NET_RX
     36854  69%    +495.1%     219327   9%  softirqs.CPU16.SCHED
     31013  11%     -84.7%       4758  64%  softirqs.CPU16.TIMER
  20372676   4%     -25.9%   15095272   7%  softirqs.CPU17.NET_RX
     36811  70%    +494.0%     218669   9%  softirqs.CPU17.SCHED
     31390  13%     -77.9%       6929  67%  softirqs.CPU17.TIMER
  20416587   5%     -26.0%   15118161   7%  softirqs.CPU18.NET_RX
     36394  70%    +502.6%     219293   9%  softirqs.CPU18.SCHED
     30912  11%     -84.5%       4784  63%  softirqs.CPU18.TIMER
  20432227   4%     -25.8%   15168258   7%  softirqs.CPU19.NET_RX
     36602  70%    +497.8%     218796   9%  softirqs.CPU19.SCHED
     30843  11%     -82.9%       5259  56%  softirqs.CPU19.TIMER
  20304573   6%     -26.0%   15034216   7%  softirqs.CPU2.NET_RX
     36968  69%    +495.1%     220012   9%  softirqs.CPU2.SCHED
     30526  12%     -84.4%       4764  62%  softirqs.CPU2.TIMER
  20444514   5%     -25.5%   15240754   7%  softirqs.CPU20.NET_RX
     36759  70%    +495.9%     219056   9%  softirqs.CPU20.SCHED
     30846  11%     -84.3%       4831  61%  softirqs.CPU20.TIMER
  20253081   6%     -25.1%   15174237   7%  softirqs.CPU21.NET_RX
     36699  70%    +495.2%     218452   8%  softirqs.CPU21.SCHED
     30719  12%     -83.5%       5060  65%  softirqs.CPU21.TIMER
  20460332   4%     -25.8%   15186744   7%  softirqs.CPU22.NET_RX
     36652  70%    +497.1%     218854   9%  softirqs.CPU22.SCHED
     30899  11%     -84.4%       4811  63%  softirqs.CPU22.TIMER
  20389322   5%     -26.5%   14979439   8%  softirqs.CPU23.NET_RX
     36717  70%    +492.5%     217548   8%  softirqs.CPU23.SCHED
     30864  11%     -83.1%       5216  60%  softirqs.CPU23.TIMER
  21350836   3%     -30.0%   14945967   6%  softirqs.CPU24.NET_RX
     47117  46%    +359.7%     216593   8%  softirqs.CPU24.SCHED
     31954  20%     -84.0%       5123  66%  softirqs.CPU24.TIMER
  21494795   3%     -28.6%   15347973   6%  softirqs.CPU25.NET_RX
     46755  49%    +362.6%     216302   8%  softirqs.CPU25.SCHED
     30392   9%     -82.7%       5245  47%  softirqs.CPU25.TIMER
  21529925   3%     -28.8%   15338064   6%  softirqs.CPU26.NET_RX
     46989  48%    +360.6%     216437   8%  softirqs.CPU26.SCHED
     30126   9%     -84.3%       4715  60%  softirqs.CPU26.TIMER
  21609177   3%     -28.9%   15374089   6%  softirqs.CPU27.NET_RX
     47211  48%    +358.1%     216275   8%  softirqs.CPU27.SCHED
     30806   9%     -84.4%       4806  59%  softirqs.CPU27.TIMER
  21434518   3%     -28.0%   15426523   6%  softirqs.CPU28.NET_RX
     46072  49%    +369.3%     216211   8%  softirqs.CPU28.SCHED
     29926  10%     -83.7%       4871  60%  softirqs.CPU28.TIMER
  21436007   3%     -28.3%   15360927   6%  softirqs.CPU29.NET_RX
     46751  48%    +363.2%     216561   8%  softirqs.CPU29.SCHED
     29983   9%     -83.7%       4876  58%  softirqs.CPU29.TIMER
  20556029   5%     -27.1%   14991636   9%  softirqs.CPU3.NET_RX
     36617  69%    +485.7%     214485   9%  softirqs.CPU3.SCHED
     30777  11%     -82.7%       5311  54%  softirqs.CPU3.TIMER
  21514774   3%     -29.0%   15275932   6%  softirqs.CPU30.NET_RX
     46934  48%    +361.6%     216667   8%  softirqs.CPU30.SCHED
     30153   9%     -84.1%       4799  58%  softirqs.CPU30.TIMER
  21317864   4%     -28.1%   15318620   6%  softirqs.CPU31.NET_RX
     46511  49%    +365.4%     216450   8%  softirqs.CPU31.SCHED
     29862  10%     -83.9%       4819  60%  softirqs.CPU31.TIMER
  21191794   5%     -27.4%   15375714   6%  softirqs.CPU32.NET_RX
     45002  47%    +380.8%     216377   8%  softirqs.CPU32.SCHED
     30188  13%     -84.0%       4825  60%  softirqs.CPU32.TIMER
  21594693   3%     -28.5%   15432125   6%  softirqs.CPU33.NET_RX
     46875  48%    +361.2%     216187   8%  softirqs.CPU33.SCHED
     30026  10%     -84.0%       4803  60%  softirqs.CPU33.TIMER
  21459214   3%     -28.2%   15402703   6%  softirqs.CPU34.NET_RX
     45836  47%    +371.2%     215962   8%  softirqs.CPU34.SCHED
     29899  10%     -84.1%       4764  58%  softirqs.CPU34.TIMER
  21334265   4%     -28.2%   15311475   6%  softirqs.CPU35.NET_RX
     46008  47%    +369.7%     216108   8%  softirqs.CPU35.SCHED
     30033   9%     -83.8%       4872  61%  softirqs.CPU35.TIMER
  21434454   3%     -29.8%   15051817   6%  softirqs.CPU36.NET_RX
     47102  48%    +353.6%     213647   8%  softirqs.CPU36.SCHED
     30159   9%     -84.3%       4724  59%  softirqs.CPU36.TIMER
  21446302   3%     -28.5%   15324037   6%  softirqs.CPU37.NET_RX
     46540  48%    +365.2%     216506   8%  softirqs.CPU37.SCHED
     29955   9%     -84.3%       4688  60%  softirqs.CPU37.TIMER
  21408080   4%     -28.1%   15398099   6%  softirqs.CPU38.NET_RX
     46514  47%    +364.5%     216055   8%  softirqs.CPU38.SCHED
     30224  10%     -84.2%       4763  59%  softirqs.CPU38.TIMER
  21488127   3%     -28.2%   15418075   6%  softirqs.CPU39.NET_RX
     46334  49%    +366.6%     216186   8%  softirqs.CPU39.SCHED
     29912  10%     -84.1%       4757  59%  softirqs.CPU39.TIMER
  20494991   5%     -25.8%   15203629   7%  softirqs.CPU4.NET_RX
     36606  70%    +500.9%     219950   8%  softirqs.CPU4.SCHED
     30735  11%     -83.9%       4963  64%  softirqs.CPU4.TIMER
  21463890   3%     -28.3%   15380723   6%  softirqs.CPU40.NET_RX
     46863  48%    +361.7%     216368   8%  softirqs.CPU40.SCHED
     30014   9%     -84.1%       4763  59%  softirqs.CPU40.TIMER
  21482678   3%     -28.7%   15323146   6%  softirqs.CPU41.NET_RX
     46944  48%    +361.1%     216451   8%  softirqs.CPU41.SCHED
     30167   9%     -83.9%       4864  59%  softirqs.CPU41.TIMER
  21294399   4%     -28.3%   15278181   6%  softirqs.CPU42.NET_RX
     45452  47%    +376.3%     216501   8%  softirqs.CPU42.SCHED
     29866  10%     -82.6%       5185  54%  softirqs.CPU42.TIMER
  21365880   3%     -28.1%   15358507   6%  softirqs.CPU43.NET_RX
     46603  49%    +364.4%     216428   8%  softirqs.CPU43.SCHED
     30073   9%     -84.0%       4809  59%  softirqs.CPU43.TIMER
  21520046   3%     -28.2%   15449623   6%  softirqs.CPU44.NET_RX
     46912  48%    +361.0%     216249   8%  softirqs.CPU44.SCHED
     29983  10%     -83.8%       4865  58%  softirqs.CPU44.TIMER
  21519719   3%     -28.4%   15408388   6%  softirqs.CPU45.NET_RX
     46866  48%    +361.2%     216163   8%  softirqs.CPU45.SCHED
     29935   9%     -83.0%       5082  55%  softirqs.CPU45.TIMER
  21419897   3%     -28.3%   15362156   6%  softirqs.CPU46.NET_RX
     46939  48%    +361.6%     216649   8%  softirqs.CPU46.SCHED
     29931  10%     -83.4%       4981  56%  softirqs.CPU46.TIMER
  21420535   3%     -28.7%   15271884   6%  softirqs.CPU47.NET_RX
     45846  47%    +370.1%     215519   8%  softirqs.CPU47.SCHED
     29858  10%     -84.1%       4740  60%  softirqs.CPU47.TIMER
  20312914   5%     -25.7%   15093130   7%  softirqs.CPU48.NET_RX
     36150  69%    +505.3%     218836   9%  softirqs.CPU48.SCHED
     30533  12%     -84.7%       4657  65%  softirqs.CPU48.TIMER
  20470745   4%     -28.8%   14574011   8%  softirqs.CPU49.NET_RX
     36483  70%    +488.7%     214762  11%  softirqs.CPU49.SCHED
     30893  11%     -85.0%       4630  54%  softirqs.CPU49.TIMER
  20156700   7%     -24.7%   15172321   7%  softirqs.CPU5.NET_RX
     35518  68%    +517.8%     219449   9%  softirqs.CPU5.SCHED
     30265  13%     -83.0%       5157  53%  softirqs.CPU5.TIMER
  20423756   5%     -26.0%   15104088   7%  softirqs.CPU50.NET_RX
     36375  70%    +499.0%     217904   9%  softirqs.CPU50.SCHED
     30692  11%     -83.8%       4974  56%  softirqs.CPU50.TIMER
  20528259   5%     -27.9%   14802498  11%  softirqs.CPU51.NET_RX
     36471  71%    +481.1%     211940   9%  softirqs.CPU51.SCHED
     30830  11%     -84.8%       4699  64%  softirqs.CPU51.TIMER
  20406992   4%     -25.5%   15212005   7%  softirqs.CPU52.NET_RX
     36584  70%    +498.1%     218812   9%  softirqs.CPU52.SCHED
     30640  11%     -83.2%       5158  60%  softirqs.CPU52.TIMER
  20215189   6%     -25.1%   15148729   7%  softirqs.CPU53.NET_RX
     35623  68%    +515.2%     219165   9%  softirqs.CPU53.SCHED
     30368  13%     -82.7%       5247  54%  softirqs.CPU53.TIMER
  20363531   5%     -25.9%   15088299   7%  softirqs.CPU54.NET_RX
     36829  70%    +495.7%     219407   9%  softirqs.CPU54.SCHED
     30604  12%     -84.6%       4718  63%  softirqs.CPU54.TIMER
  20446122   5%     -26.0%   15123333   7%  softirqs.CPU55.NET_RX
     36648  70%    +496.4%     218572   9%  softirqs.CPU55.SCHED
     30692  11%     -84.3%       4807  61%  softirqs.CPU55.TIMER
  20460809   5%     -26.2%   15103440   6%  softirqs.CPU56.NET_RX
     36546  71%    +499.2%     218984   9%  softirqs.CPU56.SCHED
     30796  11%     -85.0%       4612  60%  softirqs.CPU56.TIMER
  20579289   5%     -25.7%   15290133   7%  softirqs.CPU57.NET_RX
     36635  70%    +497.9%     219057   9%  softirqs.CPU57.SCHED
     30824  11%     -84.1%       4887  61%  softirqs.CPU57.TIMER
  20340209   5%     -25.1%   15226056   7%  softirqs.CPU58.NET_RX
     36661  70%    +497.4%     219034   9%  softirqs.CPU58.SCHED
     30643  12%     -83.8%       4958  67%  softirqs.CPU58.TIMER
  20407456   5%     -31.2%   14032788  14%  softirqs.CPU59.NET_RX
     36581  70%    +445.7%     199631   7%  softirqs.CPU59.SCHED
     30705  11%     -84.8%       4662  66%  softirqs.CPU59.TIMER
  20391632   5%     -26.0%   15081271   7%  softirqs.CPU6.NET_RX
     36990  70%    +492.1%     219031   9%  softirqs.CPU6.SCHED
     31051  10%     -84.9%       4685  66%  softirqs.CPU6.TIMER
  20313212   5%     -25.8%   15082404   7%  softirqs.CPU60.NET_RX
     36711  70%    +497.9%     219487   9%  softirqs.CPU60.SCHED
     30611  12%     -77.9%       6758  62%  softirqs.CPU60.TIMER
  20370636   5%     -25.5%   15182632   7%  softirqs.CPU61.NET_RX
     36456  69%    +499.3%     218484   9%  softirqs.CPU61.SCHED
     30655  11%     -83.6%       5031  56%  softirqs.CPU61.TIMER
  20447745   5%     -25.8%   15175825   7%  softirqs.CPU62.NET_RX
     36878  70%    +491.8%     218266   9%  softirqs.CPU62.SCHED
     30914  12%     -83.8%       5002  63%  softirqs.CPU62.TIMER
  20585734   4%     -26.3%   15163240   7%  softirqs.CPU63.NET_RX
     36766  69%    +493.6%     218247   9%  softirqs.CPU63.SCHED
     30783  11%     -84.3%       4848  63%  softirqs.CPU63.TIMER
  20529909   5%     -26.0%   15201926   7%  softirqs.CPU64.NET_RX
     36996  69%    +492.3%     219138   9%  softirqs.CPU64.SCHED
     31589  12%     -84.3%       4944  59%  softirqs.CPU64.TIMER
  20469049   5%     -26.2%   15105347   7%  softirqs.CPU65.NET_RX
     36816  70%    +495.3%     219168   9%  softirqs.CPU65.SCHED
     31163  11%     -84.5%       4843  62%  softirqs.CPU65.TIMER
  20451275   5%     -26.1%   15117422   7%  softirqs.CPU66.NET_RX
     36621  70%    +498.4%     219158   9%  softirqs.CPU66.SCHED
     31143  11%     -84.5%       4812  63%  softirqs.CPU66.TIMER
  20535600   4%     -26.1%   15183175   7%  softirqs.CPU67.NET_RX
     36716  70%    +496.0%     218827   9%  softirqs.CPU67.SCHED
     31123  11%     -84.5%       4811  62%  softirqs.CPU67.TIMER
  20606503   4%     -25.9%   15263709   7%  softirqs.CPU68.NET_RX
     36843  71%    +493.6%     218696   9%  softirqs.CPU68.SCHED
     31375  10%     -84.5%       4850  61%  softirqs.CPU68.TIMER
  20565997   4%     -26.1%   15207162   7%  softirqs.CPU69.NET_RX
     36903  70%    +493.4%     218980   9%  softirqs.CPU69.SCHED
     31377  10%     -84.6%       4840  63%  softirqs.CPU69.TIMER
  20398774   5%     -26.3%   15029635   7%  softirqs.CPU7.NET_RX
     36601  69%    +495.4%     217915   9%  softirqs.CPU7.SCHED
     30791  11%     -84.9%       4640  66%  softirqs.CPU7.TIMER
  20433298   4%     -26.2%   15079571   7%  softirqs.CPU70.NET_RX
     36471  70%    +499.8%     218755   9%  softirqs.CPU70.SCHED
     31125  11%     -84.6%       4791  61%  softirqs.CPU70.TIMER
  20514519   4%     -26.6%   15064099   7%  softirqs.CPU71.NET_RX
     36787  70%    +493.4%     218283   9%  softirqs.CPU71.SCHED
     31230  11%     -84.3%       4914  64%  softirqs.CPU71.TIMER
  21343935   3%     -29.8%   14976558   6%  softirqs.CPU72.NET_RX
     47439  47%    +356.6%     216628   8%  softirqs.CPU72.SCHED
     33558  25%     -86.3%       4611  62%  softirqs.CPU72.TIMER
  21417052   3%     -28.3%   15349903   6%  softirqs.CPU73.NET_RX
     46794  48%    +362.8%     216549   8%  softirqs.CPU73.SCHED
     30149  10%     -83.8%       4893  57%  softirqs.CPU73.TIMER
  21398667   3%     -28.4%   15321598   6%  softirqs.CPU74.NET_RX
    117019  29%     -36.5%      74286  24%  softirqs.CPU74.RCU
     46710  47%    +363.0%     216256   8%  softirqs.CPU74.SCHED
     30250  10%     -84.3%       4756  61%  softirqs.CPU74.TIMER
  21158934   6%     -27.3%   15387291   6%  softirqs.CPU75.NET_RX
    116793  29%     -36.3%      74379  24%  softirqs.CPU75.RCU
     46862  48%    +361.5%     216265   8%  softirqs.CPU75.SCHED
     29925  11%     -83.8%       4847  57%  softirqs.CPU75.TIMER
  21327908   4%     -27.8%   15394469   6%  softirqs.CPU76.NET_RX
     45803  47%    +372.3%     216338   8%  softirqs.CPU76.SCHED
     30109  10%     -83.7%       4920  57%  softirqs.CPU76.TIMER
  21477994   4%     -28.6%   15330910   6%  softirqs.CPU77.NET_RX
     46487  47%    +365.6%     216466   8%  softirqs.CPU77.SCHED
     30216  10%     -84.4%       4726  59%  softirqs.CPU77.TIMER
  21460057   4%     -29.1%   15225183   6%  softirqs.CPU78.NET_RX
     46884  48%    +362.1%     216676   8%  softirqs.CPU78.SCHED
     30421  10%     -84.4%       4730  60%  softirqs.CPU78.TIMER
  21541321   3%     -28.9%   15319139   6%  softirqs.CPU79.NET_RX
     46875  48%    +362.6%     216857   8%  softirqs.CPU79.SCHED
     30230   9%     -82.5%       5291  76%  softirqs.CPU79.TIMER
  20493021   4%     -26.0%   15160469   7%  softirqs.CPU8.NET_RX
     36579  70%    +498.1%     218781   9%  softirqs.CPU8.SCHED
     30682  11%     -84.4%       4790  59%  softirqs.CPU8.TIMER
  21172364   7%     -27.3%   15402317   6%  softirqs.CPU80.NET_RX
     46254  47%    +368.0%     216468   8%  softirqs.CPU80.SCHED
     29465  12%     -83.2%       4953  56%  softirqs.CPU80.TIMER
  21567300   3%     -28.5%   15430307   6%  softirqs.CPU81.NET_RX
     46695  48%    +362.7%     216072   8%  softirqs.CPU81.SCHED
     30093   9%     -84.0%       4801  60%  softirqs.CPU81.TIMER
  21491215   3%     -28.3%   15417714   6%  softirqs.CPU82.NET_RX
     46133  47%    +368.7%     216210   8%  softirqs.CPU82.SCHED
     29921  10%     -76.8%       6934  67%  softirqs.CPU82.TIMER
  21460124   3%     -28.5%   15348172   6%  softirqs.CPU83.NET_RX
     46491  48%    +365.6%     216476   8%  softirqs.CPU83.SCHED
     29970  10%     -83.7%       4888  56%  softirqs.CPU83.TIMER
  21445385   4%     -30.8%   14839411   9%  softirqs.CPU84.NET_RX
     46941  48%    +350.7%     211545   9%  softirqs.CPU84.SCHED
     30037  10%     -84.3%       4718  60%  softirqs.CPU84.TIMER
  21551635   3%     -28.9%   15333781   6%  softirqs.CPU85.NET_RX
     46814  48%    +362.1%     216323   8%  softirqs.CPU85.SCHED
     29996  10%     -83.9%       4817  59%  softirqs.CPU85.TIMER
  21556404   3%     -28.6%   15382397   6%  softirqs.CPU86.NET_RX
     46934  48%    +360.7%     216231   8%  softirqs.CPU86.SCHED
     30053   9%     -83.7%       4900  55%  softirqs.CPU86.TIMER
  21549266   3%     -28.5%   15417462   6%  softirqs.CPU87.NET_RX
     46429  49%    +366.0%     216351   8%  softirqs.CPU87.SCHED
     29937  10%     -83.9%       4806  58%  softirqs.CPU87.TIMER
  21537823   3%     -28.7%   15361608   6%  softirqs.CPU88.NET_RX
     46887  48%    +360.1%     215706   8%  softirqs.CPU88.SCHED
     29981   9%     -84.1%       4781  59%  softirqs.CPU88.TIMER
  21559950   3%     -29.0%   15300783   6%  softirqs.CPU89.NET_RX
     46878  48%    +361.6%     216403   8%  softirqs.CPU89.SCHED
     30017   9%     -84.1%       4783  59%  softirqs.CPU89.TIMER
  20571529   5%     -25.7%   15286193   7%  softirqs.CPU9.NET_RX
     36585  70%    +498.6%     219007   9%  softirqs.CPU9.SCHED
     30724  11%     -83.7%       5009  58%  softirqs.CPU9.TIMER
  21116816   5%     -27.5%   15303522   6%  softirqs.CPU90.NET_RX
     46158  48%    +368.6%     216275   8%  softirqs.CPU90.SCHED
     29583  11%     -83.6%       4842  58%  softirqs.CPU90.TIMER
  21247319   4%     -27.6%   15382623   6%  softirqs.CPU91.NET_RX
     46803  48%    +362.0%     216238   8%  softirqs.CPU91.SCHED
     29690  10%     -83.6%       4856  58%  softirqs.CPU91.TIMER
  21660224   3%     -28.6%   15460344   6%  softirqs.CPU92.NET_RX
     46935  48%    +359.9%     215852   8%  softirqs.CPU92.SCHED
     30143   9%     -84.0%       4814  59%  softirqs.CPU92.TIMER
  21653985   3%     -28.9%   15405532   6%  softirqs.CPU93.NET_RX
     46972  48%    +359.5%     215821   8%  softirqs.CPU93.SCHED
     30034  10%     -83.5%       4964  59%  softirqs.CPU93.TIMER
  21602751   3%     -28.8%   15385232   6%  softirqs.CPU94.NET_RX
     46892  48%    +362.2%     216728   8%  softirqs.CPU94.SCHED
     29959  10%     -83.6%       4921  58%  softirqs.CPU94.TIMER
  21277877   4%     -28.4%   15224899   6%  softirqs.CPU95.NET_RX
     45607  48%    +373.5%     215952   8%  softirqs.CPU95.SCHED
     29711  10%     -83.0%       5065  55%  softirqs.CPU95.TIMER
 2.009e+09           -27.4%   1.46e+09   5%  softirqs.NET_RX
   3996567  14%    +421.3%   20835185   8%  softirqs.SCHED
   2929735   3%     -83.8%     474988  53%  softirqs.TIMER


                                                                                
                              tbench.throughput-MB_sec                          
                                                                                
  16000 +-------------------------------------------------------------------+   
        |                  .+.          .+.                                 |   
  15000 |.+.+..+. .+.+.+..+   +.+.+..+.+   +. .+..+. .+.+.    .+.+.+.+..+.+.|   
        |        +                           +      +     +..+              |   
  14000 |-+                                                                 |   
        |                                                                   |   
  13000 |-+                                                                 |   
        |                                                                   |   
  12000 |-+                                                                 |   
        | O                     O O    O O                           O      |   
  11000 |-+            O    O O                     O          O        O   |   
        |      O O O O    O          O     O O O  O   O O O      O        O |   
  10000 |-+ O                                                O     O        |   
        |                                                                   |   
   9000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               tbench.time.user_time                            
                                                                                
  5200 +--------------------------------------------------------------------+   
  5000 |-+.+..          .+.+   +..   .+.+..     + +    + :      .+. .+..   .|   
       |.+    +.+. .+..+    + +   +.+      +. .+   + .+  :    .+   +    +.+ |   
  4800 |-+        +          +               +      +     +.+.              |   
  4600 |-+                                                                  |   
       |                                                                    |   
  4400 |-+                                                                  |   
  4200 |-+                                                                  |   
  4000 |-+                     O  O   O O                            O      |   
       | O                   O                      O          O        O   |   
  3800 |-+             O   O                   O O                          |   
  3600 |-+    O O   O    O                   O        O O                 O |   
       |   O      O                                       O O    O          |   
  3400 |-+                          O      O                       O        |   
  3200 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                               tbench.time.system_time                          
                                                                                
  34000 +-------------------------------------------------------------------+   
  32000 |.+     .+  :            .+.. .+.+. .+.    .+.        .+   +.+..+.+.|   
        | +.+..+    : .+..+.+.+.+    +     +   +..+   +.+.+..+              |   
  30000 |-+          +                                                      |   
  28000 |-+                                                                 |   
        |                                                                   |   
  26000 |-+                                                                 |   
  24000 |-+                                                                 |   
  22000 |-+                                                                 |   
        |                                                                   |   
  20000 |-+                       O      O                           O      |   
  18000 |-O                 O   O      O            O          O            |   
        |              O      O      O     O O O  O     O               O O |   
  16000 |-+ O  O O O O    O                           O   O  O   O          |   
  14000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                      tbench.time.percent_of_cpu_this_job_got                   
                                                                                
  5500 +--------------------------------------------------------------------+   
       |.       +.+              .+. .+.+.. .+     .+         .+.+.    .+.+.|   
  5000 |-+.    +   +  .+.+.+.+.+.   +      +  + .+.  + .+.+.+.     +.+.     |   
       |   +..+     +.                         +      +                     |   
       |                                                                    |   
  4500 |-+                                                                  |   
       |                                                                    |   
  4000 |-+                                                                  |   
       |                                                                    |   
  3500 |-+                                                                  |   
       |                                                             O      |   
       |                       O  O   O O                      O            |   
  3000 |-O                 O O                 O O  O                   O   |   
       |      O O O O  O O          O      O O        O O O O    O        O |   
  2500 +--------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                        tbench.time.voluntary_context_switches                  
                                                                                
  1.4e+09 +-----------------------------------------------------------------+   
          |   O O    O     O                                 O     O        |   
  1.2e+09 |-O      O   O O     O      O     O O O O    O O O     O      O O |   
          |                  O   O      O            O         O            |   
    1e+09 |-+                       O     O                           O     |   
          |                                                                 |   
    8e+08 |-+                                                               |   
          |                                                                 |   
    6e+08 |-+                                                               |   
          |                                                                 |   
    4e+08 |-+                                                               |   
          |  .+.+..    +.                       +.     +.                   |   
    2e+08 |.+         +  +.+.+.+.+.. .+.   .+. +  +.. +  +.+.+.   .+..+.   .|   
          |        +.+              +   +.+   +      +         +.+      +.+ |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                       tbench.time.involuntary_context_switches                 
                                                                                
    2e+09 +-----------------------------------------------------------------+   
  1.8e+09 |.+.    .+  + .+.+.+.+.+..+.+.+   +.+. .+..+. .+.   .+   +..+.+.+.|   
          |   +.+.     +                        +      +   +.+              |   
  1.6e+09 |-+                                                               |   
  1.4e+09 |-+                                                               |   
          |                                                                 |   
  1.2e+09 |-+                                                               |   
    1e+09 |-+                                                               |   
    8e+08 |-+                                                               |   
          |                                                                 |   
    6e+08 |-+                       O     O                           O     |   
    4e+08 |-O                O   O      O            O         O            |   
          |              O     O      O     O   O O      O              O   |   
    2e+08 |-+ O O  O O O   O                  O        O   O O   O        O |   
        0 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


---
0DAY/LKP+ Test Infrastructure                   Open Source Technology Center
https://lists.01.org/hyperkitty/list/lkp@lists.01.org       Intel Corporation

Thanks,
Oliver Sang


_______________________________________________
LKP mailing list -- lkp@lists.01.org
To unsubscribe send an email to lkp-leave@lists.01.org

-- 
Zhengjun Xing