[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression
by kernel test robot
Greeting,
FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit:
commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: fio-basic
on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
with following parameters:
disk: 2pmem
fs: btrfs
runtime: 200s
nr_task: 50%
time_based: tb
rw: randwrite
bs: 4k
ioengine: libaio
test_size: 100G
cpufreq_governor: performance
ucode: 0x3d
test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode:
4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d
commit:
ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check")
8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio")
ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 25% 1:4 dmesg.BUG:unable_to_handle_kernel
:4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt
:4 25% 1:4 dmesg.Oops:#[##]
:4 25% 1:4 dmesg.RIP:cpuidle_enter_state
:4 25% 1:4 dmesg.RIP:native_write_msr
:4 25% 1:4 dmesg.RIP:perf_prepare_sample
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms%
4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms%
8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms%
1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms%
66.81 -5.9 60.90 ± 2% fio.latency_4ms%
0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us%
1.017e+08 -28.5% 72701525 fio.time.file_system_outputs
109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got
197.97 -26.4% 145.79 ± 2% fio.time.system_time
38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time
12388015 -28.5% 8858878 fio.time.voluntary_context_switches
12711059 -28.5% 9083954 fio.workload
247.91 -28.6% 176.96 fio.write_bw_MBps
309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us
17574 +40.0% 24609 fio.write_clat_mean_us
57989 ± 4% +82.2% 105661 fio.write_clat_stddev
63463 -28.6% 45302 fio.write_iops
564.61 +40.2% 791.54 fio.write_slat_mean_us
9375 ± 7% +96.3% 18406 fio.write_slat_stddev
21.25 +8.5% 23.06 ± 9% boot-time.dhcp
93.32 +1.4% 94.58 iostat.cpu.idle
6.43 -18.6% 5.23 ± 2% iostat.cpu.system
6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys%
0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr%
93.00 +1.1% 94.00 vmstat.cpu.id
591200 -30.0% 413921 vmstat.io.bo
31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache
11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free
5.00 -26.7% 3.67 ± 12% vmstat.procs.r
635910 ± 2% -27.6% 460384 vmstat.system.cs
7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node
7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit
224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss
242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node
7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node
224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign
7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit
9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time
53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage
1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time
16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage
5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time
16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage
9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage
19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time
10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active
10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file)
31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached
115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree
6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M
534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty
11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree
37604344 ± 2% -9.4% 34060483 meminfo.Memused
806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback
370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs
370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs
15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs
17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs
12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs
7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs
8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs
2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs
2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs
263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz
9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy%
53502248 ± 6% -31.8% 36503898 turbostat.C1
5.86 ± 8% -2.0 3.82 turbostat.C1%
16540396 ± 13% -45.0% 9102733 turbostat.C1E
6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E%
16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3
33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3%
9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6
18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3
53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp
4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2
56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp
11.84 -4.8% 11.28 turbostat.RAMWatt
7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active
7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file)
308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty
8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive
8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file)
572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback
15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages
11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive
105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon)
11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file)
5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack
6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree
18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed
4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables
857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim
109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem
986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab
575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max
100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev
830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max
29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg
591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max
107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev
2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev
2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev
24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg
541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max
93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev
519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max
87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev
480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max
80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev
408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max
71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev
381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max
64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev
106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly
2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file
33315360 -30.4% 23192301 proc-vmstat.nr_dirtied
133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty
7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages
28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma
2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages
5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file
430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable
197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback
33243480 -30.4% 23143756 proc-vmstat.nr_written
2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file
5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file
134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending
15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit
15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local
2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate
16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal
17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree
1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout
5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd
5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd
32760 -7.6% 30285 proc-vmstat.slabs_scanned
1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file
10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied
76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty
2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file
142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback
10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written
1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file
2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file
77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending
5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit
5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local
68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss
119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other
30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes
6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied
3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages
28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma
1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages
26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon
2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file
5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack
29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped
1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages
27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem
214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable
75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback
6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written
26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon
2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file
68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign
5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit
4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local
18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI
2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions
1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate%
37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses
28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses
2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references
645248 ± 2% -27.4% 468301 perf-stat.i.context-switches
1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi
1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles
1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate%
31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses
3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads
0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate%
2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses
2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores
13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads
1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions
8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss
0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc
16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses
8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads
2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses
1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores
17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI
1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate%
1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi
689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses
0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate%
0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate%
10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate%
7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss
0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc
202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length
2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions
36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses
28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses
2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references
642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches
1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles
31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses
3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads
2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses
2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores
13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads
1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions
15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses
8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads
2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses
1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores
2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions
36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU
37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU
37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU
39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU
38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU
37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU
37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU
33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU
34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU
34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU
44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU
43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU
36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU
67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER
39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU
42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU
40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU
42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU
42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU
41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU
39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU
40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU
40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU
41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU
36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU
39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU
39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU
38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU
39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU
39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU
39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU
40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU
39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU
41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU
38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU
36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU
40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU
40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU
40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU
41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU
38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU
39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU
42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU
40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU
38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU
37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU
39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU
40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU
40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU
39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU
40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU
41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU
38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU
39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU
39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU
41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU
36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU
37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU
36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU
35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU
35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU
35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU
36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU
37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU
36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU
35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU
35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU
37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU
36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU
36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU
37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU
37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU
2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU
249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6
154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7
150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49
10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts
12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts
945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts
945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts
13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts
11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts
13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts
13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts
7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts
7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts
1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts
1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts
2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts
194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts
194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts
2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts
11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts
2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts
2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts
10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts
2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts
12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts
2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts
9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts
980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts
980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts
11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts
11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts
728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts
728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts
10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts
2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts
8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts
2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts
2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts
10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts
2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts
10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts
1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts
161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts
161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts
1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts
1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts
1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts
12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts
2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts
2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts
2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts
2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts
150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49
11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts
1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts
1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts
11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts
2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts
2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts
11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts
2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts
10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts
2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts
9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts
1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts
1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts
11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts
601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts
601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts
8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts
9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts
249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6
12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts
7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts
971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts
971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts
9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts
2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts
2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts
8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts
7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts
8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts
154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7
9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts
8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts
911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts
911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts
12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts
1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts
1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts
60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts
60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts
749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts
15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread
5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit
5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit
5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit
5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback
6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work
5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64
6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode
5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit
5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work
1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io
1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper
2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper
0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper
0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io
0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums
1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread
0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread
0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io
1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter
0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write
0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper
0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space
1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one
1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write
1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range
1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc
1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage
0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write
0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages
0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook
2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook
4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range
0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents
0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space
0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread
7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work
49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64
1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback
0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes
61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode
55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page
0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb
0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages
0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages
15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper
12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io
7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot
7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages
7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages
7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage
6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64
6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit
5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit
5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one
5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written
4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums
5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write
4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks
5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter
5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write
5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc
5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range
5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow
2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block
2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block
2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping
1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search
0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk
0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit
2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items
3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock
1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit
2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule
1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits
1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit
1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search
2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert
2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs
2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs
1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule
1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up
1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em
0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock
0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block
1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages
1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio
1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search
1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer
0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum
1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range
1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range
1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common
0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit
1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right
0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item
1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function
1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache
0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path
0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full
0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32
1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page
0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page
0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc
1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io
1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes
0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done
1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle
0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc
0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages
0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit
0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent
0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path
1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache
1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit
0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free
0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb
0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair
0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref
0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket
0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup
0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load
0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent
0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items
0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity
0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate
0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook
0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg
0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head
0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit
0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio
0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath
0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc
0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending
0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode
0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback
0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range
0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist
0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer
0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty
0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range
0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user
0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry
0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent
0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc
0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state
0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin
0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode
0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state
0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc
0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state
0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer
0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty
0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up
0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block
0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref
0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free
0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right
0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback
0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent
0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark
0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer
0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map
0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent
0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic
0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref
0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head
0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached
0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start
0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock
0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio
0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase
0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start
0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock
0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space
0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map
0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used
0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref
0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist
0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se
0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback
0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark
0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep
0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock
0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent
0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node
0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed
0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid
0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid
0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64
0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref
0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling
0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents
0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents
0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages
0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io
0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search
0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map
0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate
0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked
0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking
0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent
0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node
0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable
0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to
0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached
0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev
0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u
0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu
0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock
0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep
0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock
0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit
0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid
0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events
0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event
0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile
0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle
0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events
0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping
0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect
0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter
0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret
0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys
0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance
0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter
0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline
0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains
1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer
2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt
0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8
2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents
0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction
0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64
1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32
0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode
8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn
50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle
61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64
61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry
8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages
56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state
61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary
1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page
1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages
1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb
1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio
0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook
0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf
0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio
2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping
1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search
2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search
0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page
0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc
0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32
0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule
0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup
0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load
0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer
0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head
0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up
0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath
0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free
0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free
0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer
0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot
0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc
0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head
0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state
0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se
0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase
0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock
0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc
0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath
0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map
0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg
0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item
0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping
0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark
0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep
0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid
0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair
0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid
0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking
0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed
0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair
0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up
0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked
0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node
0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock
0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio
0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity
0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete
0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock
0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u
0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock
0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock
0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to
0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit
0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev
0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag
0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu
0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid
0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep
0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer
0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event
0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right
0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty
0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit
0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt
0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance
0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline
0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit
0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret
0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state
0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8
0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer
0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf
0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64
1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32
0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer
49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle
fio.write_bw_MBps
300 +-+-------------------------------------------------------------------+
| |
250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ |
| : : +. +..+ + +..+ +. + |
| : : : : : : |
200 +-+ : : : : : : O O |
O O O: O :O O O O O O O O O O: O:O O: O: O O O O O
150 +-+ : : : : : : |
| : : : : : : |
100 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
50 +-+ :: :: :: |
| : : : |
0 +-+---------------------------------------------O-------------O-------+
fio.write_iops
70000 +-+-----------------------------------------------------------------+
|..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ |
60000 +-+ : : +. : : + : |
| : : : : : : |
50000 +-+ : : : : : : |
O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O
40000 +-+ : O: : : : : |
| : : : : : : |
30000 +-+ : : : : : : |
| : : : : : : |
20000 +-+ : : : : : : |
| :: :: :: |
10000 +-+ : : : |
| : : : |
0 +-+--------------------------------------------O------------O-------+
fio.write_clat_mean_us
30000 +-+-----------------------------------------------------------------+
| |
25000 O-+O O O O O O O O O O O O O O O O O O O |
| O O O O O
| |
20000 +-+ |
|..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ |
15000 +-+ : : : : : : |
| : : : : : : |
10000 +-+ : : : : : : |
| : : : : : : |
| : : : : : : |
5000 +-+ : : : : :: |
| : : : |
0 +-+--------------------------------------------O------------O-------+
fio.write_clat_stddev
160000 +-+----------------------------------------------------------------+
| |
140000 O-+O O O O O O O O O O O O O O |
120000 +-+ O O |
| |
100000 +-+ O O O O O O O O
| |
80000 +-+ |
| .+.. |
60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ |
40000 +-+ : : : : : : |
| : : : : : : |
20000 +-+ : : : : : : |
| :: :: :: |
0 +-+-------------------------------------------O------------O-------+
fio.write_slat_mean_us
900 +-+-------------------------------------------------------------------+
O O O O O O O O O O O O O O O O O O |
800 +-+O O O O O O O
700 +-+ |
| |
600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ |
500 +-+ : : : : : : |
| : : : : : : |
400 +-+ : : : : : : |
300 +-+ : : : : : : |
| : : : : : : |
200 +-+ : : : : : : |
100 +-+ : : :: :: |
| : : : |
0 +-+---------------------------------------------O-------------O-------+
fio.write_slat_stddev
25000 +-+-----O---------O-------------------------------------------------+
O O O O O O O O O O O O O O O |
| |
20000 +-+ |
| O O O O O O O O
| |
15000 +-+ |
| |
10000 +-++. .+.. .+.. .+..+ .+ .+..+ |
|. + +..+..+ +..+ + : +. : +..+.+. |
| : : : : : : |
5000 +-+ : : : : : : |
| : : : : : : |
| :: : :: |
0 +-+--------------------------------------------O------------O-------+
fio.latency_1000ms_
0.4 +-+------------------------------------------------------------------+
| |
0.35 +-+ O |
0.3 +-+ |
| O O |
0.25 +-+ O O |
| O O O
0.2 O-+ O O O O O O O O O O |
| O O |
0.15 +-+O O O O |
0.1 +-+ |
| |
0.05 +-+ |
| .+. .+..+..+.+..+.. .+. .+.+..+.. |
0 +-+---------------------------------------------O------------O-------+
fio.workload
1.4e+07 +-+---------------------------------------------------------------+
|..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ |
1.2e+07 +-+ : : : : + : |
| : : : : : : |
1e+07 +-+ : : : : : : |
O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O
8e+06 +-+ : O: : : : : |
| : : : : : : |
6e+06 +-+ : : : : : : |
| : : : : : : |
4e+06 +-+ : : : : : : |
| :: :: :: |
2e+06 +-+ : : : |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
fio.time.voluntary_context_switches
1.4e+07 +-+---------------------------------------------------------------+
| |
1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ |
| : : : : : : |
1e+07 +-+ : : : : : : |
| O O: : O O O : : :O : O O O O O O O
8e+06 O-+ : O:O O O O O O O: O:O O: : |
| : : : : : : |
6e+06 +-+ : : : : : : |
| : : : : : : |
4e+06 +-+ : : : : : : |
| : : : : :: |
2e+06 +-+ : : :: |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
fio.time.file_system_outputs
1.2e+08 +-+---------------------------------------------------------------+
| |
1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ |
| : : : : : : |
| : : : : : : |
8e+07 +-+ : : : : : : O O O O O
O O O: O:O O O O O O O O O O: O:O O:O : O O |
6e+07 +-+ : : : : : : |
| : : : : : : |
4e+07 +-+ : : : : : : |
| : : : : : : |
| : : : : :: |
2e+07 +-+ : : :: |
| : : : |
0 +-+-------------------------------------------O-----------O-------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
10 months, 2 weeks
Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression
by Xing Zhengjun
On 5/31/2019 3:10 AM, Trond Myklebust wrote:
> On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote:
>>
>> On 5/30/2019 10:00 AM, Trond Myklebust wrote:
>>> Hi Xing,
>>>
>>> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
>>>> Hi Trond,
>>>>
>>>> On 5/20/2019 1:54 PM, kernel test robot wrote:
>>>>> Greeting,
>>>>>
>>>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due
>>>>> to
>>>>> commit:
>>>>>
>>>>>
>>>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC:
>>>>> Convert
>>>>> socket page send code to use iov_iter()")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>>>> master
>>>>>
>>>>> in testcase: fsmark
>>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
>>>>> 3.00GHz with 384G memory
>>>>> with following parameters:
>>>>>
>>>>> iterations: 1x
>>>>> nr_threads: 64t
>>>>> disk: 1BRD_48G
>>>>> fs: xfs
>>>>> fs2: nfsv4
>>>>> filesize: 4M
>>>>> test_size: 40G
>>>>> sync_method: fsyncBeforeClose
>>>>> cpufreq_governor: performance
>>>>>
>>>>> test-description: The fsmark is a file system benchmark to test
>>>>> synchronous write workloads, for example, mail servers
>>>>> workload.
>>>>> test-url: https://sourceforge.net/projects/fsmark/
>>>>>
>>>>>
>>>>>
>>>>> Details are as below:
>>>>> -------------------------------------------------------------
>>>>> ----
>>>>> --------------------------------->
>>>>>
>>>>>
>>>>> To reproduce:
>>>>>
>>>>> git clone https://github.com/intel/lkp-tests.git
>>>>> cd lkp-tests
>>>>> bin/lkp install job.yaml # job file is attached in
>>>>> this
>>>>> email
>>>>> bin/lkp run job.yaml
>>>>>
>>>>> ===============================================================
>>>>> ====
>>>>> ======================
>>>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf
>>>>> ig/n
>>>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>>>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
>>>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
>>>>> ep01/40G/fsmark
>>>>>
>>>>> commit:
>>>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
>>>>> iov_iter_kvec()")
>>>>> 0472e47660 ("SUNRPC: Convert socket page send code to use
>>>>> iov_iter()")
>>>>>
>>>>> e791f8e9380d945e 0472e476604998c127f3c80d291
>>>>> ---------------- ---------------------------
>>>>> fail:runs %reproduction fail:runs
>>>>> | | |
>>>>> :4 50% 2:4 dmesg.WARNING:a
>>>>> t#for
>>>>> _ip_interrupt_entry/0x
>>>>> %stddev %change %stddev
>>>>> \ | \
>>>>> 15118573
>>>>> ± 2% +16.0% 17538083 fsmark.app_overhead
>>>>> 510.93 -
>>>>> 22.7% 395.12 fsmark.files_per_sec
>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>> psed_
>>>>> time
>>>>> 24.90 +22.8% 30.57 fsmark.time.ela
>>>>> psed_
>>>>> time.max
>>>>> 288.00 ± 2% -
>>>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
>>>>> 70.03 ± 2% -
>>>>> 11.3% 62.14 fsmark.time.system_time
>>>>>
>>>>
>>>> Do you have time to take a look at this regression?
>>>
>>> From your stats, it looks to me as if the problem is increased
>>> NUMA
>>> overhead. Pretty much everything else appears to be the same or
>>> actually performing better than previously. Am I interpreting that
>>> correctly?
>> The real regression is the throughput(fsmark.files_per_sec) is
>> decreased
>> by 22.7%.
>
> Understood, but I'm trying to make sense of why. I'm not able to
> reproduce this, so I have to rely on your performance stats to
> understand where the 22.7% regression is coming from. As far as I can
> see, the only numbers in the stats you published that are showing a
> performance regression (other than the fsmark number itself), are the
> NUMA numbers. Is that a correct interpretation?
>
We re-test the case yesterday, the test result almost is the same.
we will do more test and also check the test case itself, if you need
more information, please let me know, thanks.
>>> If my interpretation above is correct, then I'm not seeing where
>>> this
>>> patch would be introducing new NUMA regressions. It is just
>>> converting
>>> from using one method of doing socket I/O to another. Could it
>>> perhaps
>>> be a memory artefact due to your running the NFS client and server
>>> on
>>> the same machine?
>>>
>>> Apologies for pushing back a little, but I just don't have the
>>> hardware available to test NUMA configurations, so I'm relying on
>>> external testing for the above kind of scenario.
>>>
>> Thanks for looking at this. If you need more information, please let
>> me
>> know.
>>> Thanks
>>> Trond
>>>
--
Zhengjun Xing
1 year, 7 months
[btrfs] c8eaeac7b7: aim7.jobs-per-min -11.7% regression
by kernel test robot
Greeting,
FYI, we noticed a -11.7% regression of aim7.jobs-per-min due to commit:
commit: c8eaeac7b734347c3afba7008b7af62f37b9c140 ("btrfs: reserve delalloc metadata differently")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:
disk: 4BRD_12G
md: RAID0
fs: btrfs
test: disk_rr
load: 1500
cpufreq_governor: performance
test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/1500/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_rr/aim7
commit:
4297ff84dc ("btrfs: track DIO bytes in flight")
c8eaeac7b7 ("btrfs: reserve delalloc metadata differently")
4297ff84dc24d120 c8eaeac7b734347c3afba7008b7
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:4 75% 3:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
:4 75% 3:4 dmesg.WARNING:stack_recursion
%stddev %change %stddev
\ | \
43654 -11.7% 38526 aim7.jobs-per-min
206.46 +13.3% 233.89 aim7.time.elapsed_time
206.46 +13.3% 233.89 aim7.time.elapsed_time.max
611427 ± 3% +36.6% 834981 ± 2% aim7.time.involuntary_context_switches
103580 +1.7% 105361 aim7.time.minor_page_faults
7636 +14.8% 8763 aim7.time.system_time
2731877 +7.6% 2940550 aim7.time.voluntary_context_switches
0.83 ± 2% -0.1 0.73 ± 2% mpstat.cpu.all.usr%
94.44 +7.0% 101.08 iostat.md0.w/s
16028 +7.2% 17182 iostat.md0.wkB/s
2040 +24.8% 2547 ± 18% meminfo.Inactive(file)
17154 -10.2% 15396 meminfo.max_used_kB
2.4e+08 ± 45% -87.4% 30268527 ±145% cpuidle.C3.time
731742 ± 18% -88.7% 82339 ±109% cpuidle.C3.usage
4.447e+08 ± 12% +35.0% 6.005e+08 ± 19% cpuidle.C6.time
10907 +6.9% 11662 vmstat.io.bo
55.33 ± 4% +13.9% 63.00 ± 2% vmstat.procs.r
24049 -6.9% 22394 vmstat.system.cs
10588 ± 30% -44.7% 5859 ± 25% numa-meminfo.node0.Inactive
9856 ± 36% -47.8% 5145 ± 34% numa-meminfo.node0.Inactive(anon)
4691 ± 9% +243.5% 16115 ± 69% numa-meminfo.node0.KernelStack
2202 ± 29% +1132.8% 27151 ± 92% numa-meminfo.node0.PageTables
11332 ± 29% +46.9% 16649 ± 8% numa-meminfo.node1.Inactive
10023 ± 36% +47.8% 14817 ± 12% numa-meminfo.node1.Inactive(anon)
1185 +15.5% 1369 ± 7% slabinfo.btrfs_ordered_extent.active_objs
1185 +15.5% 1369 ± 7% slabinfo.btrfs_ordered_extent.num_objs
2748 ± 3% +6.5% 2927 ± 3% slabinfo.kmalloc-rcl-64.active_objs
2748 ± 3% +6.5% 2927 ± 3% slabinfo.kmalloc-rcl-64.num_objs
367.33 ± 3% +29.2% 474.75 ± 10% slabinfo.skbuff_fclone_cache.active_objs
367.33 ± 3% +29.2% 474.75 ± 10% slabinfo.skbuff_fclone_cache.num_objs
1545 ± 11% +14.2% 1764 ± 13% slabinfo.xfrm_dst_cache.num_objs
2464 ± 36% -47.8% 1285 ± 34% numa-vmstat.node0.nr_inactive_anon
4690 ± 9% +243.7% 16120 ± 69% numa-vmstat.node0.nr_kernel_stack
550.00 ± 29% +1134.5% 6790 ± 92% numa-vmstat.node0.nr_page_table_pages
152291 ± 3% +19.0% 181153 ± 5% numa-vmstat.node0.nr_written
2464 ± 36% -47.8% 1285 ± 34% numa-vmstat.node0.nr_zone_inactive_anon
2505 ± 36% +47.9% 3705 ± 12% numa-vmstat.node1.nr_inactive_anon
151216 ± 3% +18.3% 178821 ± 4% numa-vmstat.node1.nr_written
2505 ± 36% +47.9% 3705 ± 12% numa-vmstat.node1.nr_zone_inactive_anon
493640 +1.0% 498686 proc-vmstat.nr_file_pages
510.00 +24.8% 636.50 ± 18% proc-vmstat.nr_inactive_file
584507 ± 2% +20.8% 706353 ± 2% proc-vmstat.nr_written
510.00 +24.8% 636.50 ± 18% proc-vmstat.nr_zone_inactive_file
29772 ± 5% +4.0% 30949 ± 5% proc-vmstat.numa_hint_faults
15567 ± 6% +5.4% 16416 ± 6% proc-vmstat.numa_hint_faults_local
13816 -1.7% 13583 proc-vmstat.numa_other
582286 +10.0% 640790 proc-vmstat.pgfault
2337841 ± 2% +20.9% 2825298 ± 2% proc-vmstat.pgpgout
2730 +5.2% 2872 turbostat.Avg_MHz
731158 ± 18% -88.8% 81753 ±110% turbostat.C3
2.80 ± 44% -2.5 0.32 ±146% turbostat.C3%
4.98 ± 25% -37.4% 3.12 ± 17% turbostat.CPU%c1
1.14 ± 78% -79.4% 0.24 ±160% turbostat.CPU%c3
135.74 +10.2% 149.62 turbostat.CorWatt
18209025 +13.7% 20695723 turbostat.IRQ
162.95 +8.5% 176.76 turbostat.PkgWatt
9586 ± 3% +8.6% 10410 ± 2% turbostat.SMI
24351 -6.9% 22675 perf-stat.i.context-switches
1.095e+11 +5.2% 1.151e+11 perf-stat.i.cpu-cycles
2942 -8.8% 2682 perf-stat.i.cpu-migrations
4026 ± 2% +5.5% 4248 perf-stat.i.cycles-between-cache-misses
0.11 ± 7% +0.0 0.15 ± 27% perf-stat.i.dTLB-store-miss-rate%
1.788e+09 -10.4% 1.602e+09 perf-stat.i.dTLB-stores
4679972 ± 2% -36.2% 2983497 ± 47% perf-stat.i.iTLB-loads
0.24 -3.5% 0.23 perf-stat.i.ipc
2646 -2.6% 2578 perf-stat.i.minor-faults
40.94 +0.4 41.38 perf-stat.i.node-store-miss-rate%
2646 -2.6% 2578 perf-stat.i.page-faults
4.07 +4.1% 4.24 perf-stat.overall.cpi
4196 +5.1% 4411 perf-stat.overall.cycles-between-cache-misses
0.11 ± 8% +0.0 0.15 ± 29% perf-stat.overall.dTLB-store-miss-rate%
0.25 -4.0% 0.24 perf-stat.overall.ipc
24233 -6.8% 22576 perf-stat.ps.context-switches
1.089e+11 +5.2% 1.146e+11 perf-stat.ps.cpu-cycles
2928 -8.8% 2670 perf-stat.ps.cpu-migrations
1.779e+09 -10.3% 1.595e+09 perf-stat.ps.dTLB-stores
4657162 ± 2% -36.2% 2970461 ± 47% perf-stat.ps.iTLB-loads
2635 -2.5% 2570 perf-stat.ps.minor-faults
2635 -2.5% 2570 perf-stat.ps.page-faults
5.677e+12 +14.2% 6.482e+12 perf-stat.total.instructions
23484 +27.9% 30030 ± 2% sched_debug.cfs_rq:/.load.avg
51446 ± 2% +492.7% 304901 ± 2% sched_debug.cfs_rq:/.load.max
11212 ± 5% +332.6% 48506 ± 2% sched_debug.cfs_rq:/.load.stddev
0.21 ± 14% -27.1% 0.15 ± 19% sched_debug.cfs_rq:/.nr_running.stddev
50.75 ± 10% +138.7% 121.12 ± 91% sched_debug.cfs_rq:/.runnable_load_avg.max
1.33 ± 93% +275.0% 5.00 ± 28% sched_debug.cfs_rq:/.runnable_load_avg.min
18594 ± 4% +39.0% 25853 ± 3% sched_debug.cfs_rq:/.runnable_weight.avg
41096 ± 7% +627.8% 299088 sched_debug.cfs_rq:/.runnable_weight.max
9009 ± 3% +431.5% 47888 ± 2% sched_debug.cfs_rq:/.runnable_weight.stddev
-4276 -670.0% 24374 ± 56% sched_debug.cfs_rq:/.spread0.avg
74100 ± 9% +24.2% 92022 ± 12% sched_debug.cfs_rq:/.spread0.max
-57421 -37.4% -35972 sched_debug.cfs_rq:/.spread0.min
463.62 +12.9% 523.36 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg
0.25 ± 81% +9225.0% 23.31 ±119% sched_debug.cfs_rq:/.util_est_enqueued.min
18.98 ± 4% +9.4% 20.76 ± 4% sched_debug.cpu.cpu_load[0].avg
47.67 ± 5% +20.8% 57.56 ± 8% sched_debug.cpu.cpu_load[0].max
1.92 ± 43% +154.3% 4.88 ± 40% sched_debug.cpu.cpu_load[0].min
44.75 +22.2% 54.69 ± 9% sched_debug.cpu.cpu_load[1].max
5.42 ± 15% +38.5% 7.50 ± 9% sched_debug.cpu.cpu_load[2].min
56.17 ± 5% +14.1% 64.06 ± 7% sched_debug.cpu.cpu_load[3].max
6.33 ± 9% +28.3% 8.12 ± 9% sched_debug.cpu.cpu_load[3].min
7.17 ± 4% +16.9% 8.38 ± 8% sched_debug.cpu.cpu_load[4].min
454.42 ± 86% +142.9% 1103 ± 21% sched_debug.cpu.curr->pid.min
23570 +20.9% 28493 ± 8% sched_debug.cpu.load.avg
51446 ± 2% +371.3% 242447 ± 45% sched_debug.cpu.load.max
4293 ± 80% +118.5% 9381 ± 21% sched_debug.cpu.load.min
2039 ± 7% +10.2% 2247 ± 2% sched_debug.cpu.nr_load_updates.stddev
1.06 ± 2% +22.2% 1.29 ± 5% sched_debug.cpu.nr_running.avg
0.25 ± 81% +150.0% 0.62 ± 20% sched_debug.cpu.nr_running.min
0.52 +13.5% 0.59 ± 10% sched_debug.cpu.nr_running.stddev
19645 ± 5% +9.4% 21489 ± 4% softirqs.CPU0.RCU
86145 +14.7% 98787 softirqs.CPU0.TIMER
87552 ± 7% +12.6% 98576 ± 4% softirqs.CPU1.TIMER
19349 ± 9% +13.9% 22048 ± 5% softirqs.CPU10.RCU
81218 ± 3% +29.7% 105305 ± 12% softirqs.CPU10.TIMER
19223 ± 2% +8.7% 20891 ± 7% softirqs.CPU11.RCU
83829 ± 2% +12.5% 94302 softirqs.CPU11.TIMER
84709 +16.0% 98222 ± 2% softirqs.CPU12.TIMER
85057 ± 2% +10.5% 93955 softirqs.CPU13.TIMER
20677 ± 7% +6.1% 21934 ± 7% softirqs.CPU14.RCU
84147 ± 2% +17.2% 98617 softirqs.CPU14.TIMER
19576 ± 4% +13.0% 22113 ± 7% softirqs.CPU16.RCU
84576 ± 2% +33.1% 112557 ± 14% softirqs.CPU16.TIMER
85248 ± 4% +14.5% 97595 ± 3% softirqs.CPU17.TIMER
19195 ± 6% +15.2% 22106 ± 11% softirqs.CPU18.RCU
85046 ± 2% +14.4% 97311 softirqs.CPU18.TIMER
85248 ± 3% +11.4% 94985 ± 2% softirqs.CPU19.TIMER
19415 ± 3% +11.4% 21619 ± 3% softirqs.CPU2.RCU
19095 ± 4% +10.6% 21115 ± 5% softirqs.CPU20.RCU
84097 ± 2% +15.0% 96686 softirqs.CPU20.TIMER
87892 ± 3% +13.1% 99386 ± 5% softirqs.CPU21.TIMER
19125 ± 3% +13.4% 21686 ± 5% softirqs.CPU22.RCU
14525 ± 5% -9.0% 13212 ± 4% softirqs.CPU22.SCHED
85902 ± 5% +15.4% 99114 softirqs.CPU22.TIMER
19616 ± 3% +12.4% 22042 ± 2% softirqs.CPU23.RCU
86349 ± 4% +12.0% 96741 softirqs.CPU23.TIMER
19747 ± 6% +13.5% 22407 ± 7% softirqs.CPU24.RCU
84333 +17.1% 98749 softirqs.CPU24.TIMER
19131 ± 2% +11.2% 21279 ± 5% softirqs.CPU25.RCU
19625 ± 5% +9.1% 21420 ± 5% softirqs.CPU26.RCU
84545 ± 3% +15.5% 97675 softirqs.CPU26.TIMER
20319 ± 7% +5.3% 21389 ± 6% softirqs.CPU27.RCU
86276 ± 4% +12.1% 96743 softirqs.CPU27.TIMER
18484 ± 4% +15.3% 21315 ± 7% softirqs.CPU28.RCU
83144 +16.5% 96862 softirqs.CPU28.TIMER
19786 ± 2% +12.0% 22157 ± 4% softirqs.CPU3.RCU
87487 ± 5% +12.2% 98117 softirqs.CPU3.TIMER
80763 ± 3% +20.0% 96896 ± 2% softirqs.CPU30.TIMER
83628 ± 2% +12.5% 94076 softirqs.CPU31.TIMER
85112 ± 4% +14.8% 97737 ± 3% softirqs.CPU32.TIMER
84626 ± 3% +10.9% 93812 softirqs.CPU33.TIMER
87944 ± 6% +12.1% 98575 softirqs.CPU34.TIMER
86859 +22.5% 106365 ± 13% softirqs.CPU36.TIMER
85336 ± 4% +16.8% 99655 ± 4% softirqs.CPU37.TIMER
85090 ± 2% +14.2% 97153 softirqs.CPU38.TIMER
19528 ± 4% +16.7% 22783 ± 8% softirqs.CPU4.RCU
84396 +18.0% 99614 softirqs.CPU4.TIMER
19042 ± 4% +8.8% 20714 ± 4% softirqs.CPU5.RCU
87389 ± 5% +12.7% 98473 ± 2% softirqs.CPU5.TIMER
20312 ± 3% +13.2% 22992 ± 6% softirqs.CPU6.RCU
84550 ± 3% +15.8% 97877 softirqs.CPU6.TIMER
18516 ± 7% +24.9% 23125 ± 5% softirqs.CPU8.RCU
82667 +28.2% 106005 ± 11% softirqs.CPU8.TIMER
764529 ± 2% +8.9% 832849 ± 5% softirqs.RCU
3442286 ± 2% +13.9% 3922206 softirqs.TIMER
275.00 ± 17% +64.0% 451.00 ± 28% interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
204.00 ± 56% +41.4% 288.50 ± 52% interrupts.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
103.67 +13.3% 117.50 interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
78055 +10.9% 86558 interrupts.CAL:Function_call_interrupts
425311 +13.1% 480959 interrupts.CPU0.LOC:Local_timer_interrupts
22301 ± 3% +25.5% 27997 interrupts.CPU0.RES:Rescheduling_interrupts
2033 ± 5% +12.1% 2278 ± 3% interrupts.CPU1.CAL:Function_call_interrupts
425286 +13.0% 480441 interrupts.CPU1.LOC:Local_timer_interrupts
22426 ± 3% +26.7% 28410 ± 2% interrupts.CPU1.RES:Rescheduling_interrupts
424918 +13.2% 480935 interrupts.CPU10.LOC:Local_timer_interrupts
22298 ± 2% +25.8% 28049 interrupts.CPU10.RES:Rescheduling_interrupts
1959 ± 5% +12.8% 2210 ± 6% interrupts.CPU11.CAL:Function_call_interrupts
424741 +13.1% 480460 interrupts.CPU11.LOC:Local_timer_interrupts
21914 ± 3% +27.7% 27983 interrupts.CPU11.RES:Rescheduling_interrupts
424836 +13.2% 480889 interrupts.CPU12.LOC:Local_timer_interrupts
21863 ± 3% +26.4% 27640 interrupts.CPU12.RES:Rescheduling_interrupts
1870 +21.7% 2276 ± 6% interrupts.CPU13.CAL:Function_call_interrupts
425017 +12.9% 479899 interrupts.CPU13.LOC:Local_timer_interrupts
1379 ±140% +232.7% 4590 ± 62% interrupts.CPU13.NMI:Non-maskable_interrupts
1379 ±140% +232.7% 4590 ± 62% interrupts.CPU13.PMI:Performance_monitoring_interrupts
21979 ± 2% +30.5% 28691 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts
424883 +13.2% 480884 interrupts.CPU14.LOC:Local_timer_interrupts
21978 +27.1% 27940 ± 2% interrupts.CPU14.RES:Rescheduling_interrupts
1919 ± 8% +18.7% 2278 ± 6% interrupts.CPU15.CAL:Function_call_interrupts
425309 +13.0% 480679 interrupts.CPU15.LOC:Local_timer_interrupts
22330 ± 3% +25.9% 28118 interrupts.CPU15.RES:Rescheduling_interrupts
1925 ± 6% +18.0% 2271 ± 4% interrupts.CPU16.CAL:Function_call_interrupts
424967 +13.1% 480548 interrupts.CPU16.LOC:Local_timer_interrupts
21993 ± 3% +28.3% 28215 interrupts.CPU16.RES:Rescheduling_interrupts
2102 +9.7% 2305 ± 5% interrupts.CPU17.CAL:Function_call_interrupts
424760 +13.0% 480175 interrupts.CPU17.LOC:Local_timer_interrupts
22036 ± 2% +28.5% 28307 ± 2% interrupts.CPU17.RES:Rescheduling_interrupts
424881 +13.1% 480700 interrupts.CPU18.LOC:Local_timer_interrupts
22029 ± 3% +26.9% 27950 interrupts.CPU18.RES:Rescheduling_interrupts
424963 +13.1% 480489 interrupts.CPU19.LOC:Local_timer_interrupts
23004 ± 3% +27.6% 29354 ± 2% interrupts.CPU19.RES:Rescheduling_interrupts
424990 +13.1% 480714 interrupts.CPU2.LOC:Local_timer_interrupts
22102 ± 3% +26.5% 27967 interrupts.CPU2.RES:Rescheduling_interrupts
1757 ± 6% +17.1% 2057 ± 6% interrupts.CPU20.CAL:Function_call_interrupts
425145 +13.1% 480765 interrupts.CPU20.LOC:Local_timer_interrupts
22303 ± 2% +25.8% 28066 interrupts.CPU20.RES:Rescheduling_interrupts
425267 +12.9% 480312 interrupts.CPU21.LOC:Local_timer_interrupts
21813 ± 2% +27.1% 27723 interrupts.CPU21.RES:Rescheduling_interrupts
424618 +13.2% 480680 interrupts.CPU22.LOC:Local_timer_interrupts
22015 ± 3% +26.0% 27734 interrupts.CPU22.RES:Rescheduling_interrupts
1912 ± 8% +18.9% 2273 ± 5% interrupts.CPU23.CAL:Function_call_interrupts
424735 +13.1% 480469 interrupts.CPU23.LOC:Local_timer_interrupts
22040 +27.5% 28098 ± 3% interrupts.CPU23.RES:Rescheduling_interrupts
1924 ± 5% +6.4% 2048 ± 4% interrupts.CPU24.CAL:Function_call_interrupts
424618 +13.2% 480598 interrupts.CPU24.LOC:Local_timer_interrupts
21873 ± 2% +28.1% 28009 interrupts.CPU24.RES:Rescheduling_interrupts
424698 +13.1% 480416 interrupts.CPU25.LOC:Local_timer_interrupts
22041 +28.3% 28287 ± 2% interrupts.CPU25.RES:Rescheduling_interrupts
275.00 ± 17% +64.0% 451.00 ± 28% interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
424902 +13.1% 480512 interrupts.CPU26.LOC:Local_timer_interrupts
22002 ± 3% +27.0% 27949 interrupts.CPU26.RES:Rescheduling_interrupts
424698 +13.2% 480806 interrupts.CPU27.LOC:Local_timer_interrupts
5859 ± 30% -81.3% 1093 ±173% interrupts.CPU27.NMI:Non-maskable_interrupts
5859 ± 30% -81.3% 1093 ±173% interrupts.CPU27.PMI:Performance_monitoring_interrupts
22290 +25.4% 27948 ± 3% interrupts.CPU27.RES:Rescheduling_interrupts
1904 ± 9% +12.6% 2144 ± 8% interrupts.CPU28.CAL:Function_call_interrupts
424963 +13.1% 480696 interrupts.CPU28.LOC:Local_timer_interrupts
21860 +28.7% 28145 interrupts.CPU28.RES:Rescheduling_interrupts
1933 ± 7% +18.4% 2288 ± 4% interrupts.CPU29.CAL:Function_call_interrupts
424503 +13.2% 480645 interrupts.CPU29.LOC:Local_timer_interrupts
22101 ± 4% +27.4% 28162 ± 3% interrupts.CPU29.RES:Rescheduling_interrupts
1994 ± 2% +11.0% 2214 ± 4% interrupts.CPU3.CAL:Function_call_interrupts
424920 +13.1% 480485 interrupts.CPU3.LOC:Local_timer_interrupts
21988 ± 2% +27.7% 28074 ± 2% interrupts.CPU3.RES:Rescheduling_interrupts
425052 +13.1% 480827 interrupts.CPU30.LOC:Local_timer_interrupts
22012 ± 2% +26.6% 27873 interrupts.CPU30.RES:Rescheduling_interrupts
1986 ± 6% +12.8% 2240 ± 5% interrupts.CPU31.CAL:Function_call_interrupts
424997 +13.1% 480581 interrupts.CPU31.LOC:Local_timer_interrupts
21947 ± 3% +27.3% 27935 ± 2% interrupts.CPU31.RES:Rescheduling_interrupts
204.00 ± 56% +41.4% 288.50 ± 52% interrupts.CPU32.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
424705 +13.1% 480486 interrupts.CPU32.LOC:Local_timer_interrupts
21887 +28.5% 28124 interrupts.CPU32.RES:Rescheduling_interrupts
425192 +13.0% 480284 interrupts.CPU33.LOC:Local_timer_interrupts
22206 ± 3% +25.4% 27853 ± 2% interrupts.CPU33.RES:Rescheduling_interrupts
1972 ± 6% +12.8% 2224 ± 6% interrupts.CPU34.CAL:Function_call_interrupts
424557 +13.2% 480684 interrupts.CPU34.LOC:Local_timer_interrupts
22117 ± 3% +25.6% 27777 interrupts.CPU34.RES:Rescheduling_interrupts
1894 +13.4% 2147 ± 4% interrupts.CPU35.CAL:Function_call_interrupts
424767 +13.1% 480496 interrupts.CPU35.LOC:Local_timer_interrupts
22043 ± 3% +28.0% 28225 interrupts.CPU35.RES:Rescheduling_interrupts
424967 +13.1% 480760 interrupts.CPU36.LOC:Local_timer_interrupts
21838 ± 3% +28.6% 28082 interrupts.CPU36.RES:Rescheduling_interrupts
1855 ± 3% +19.7% 2220 ± 7% interrupts.CPU37.CAL:Function_call_interrupts
424903 +13.0% 480330 interrupts.CPU37.LOC:Local_timer_interrupts
22221 ± 2% +27.8% 28392 ± 2% interrupts.CPU37.RES:Rescheduling_interrupts
1880 +15.3% 2167 ± 6% interrupts.CPU38.CAL:Function_call_interrupts
425042 +13.1% 480659 interrupts.CPU38.LOC:Local_timer_interrupts
21922 ± 3% +28.6% 28182 interrupts.CPU38.RES:Rescheduling_interrupts
1845 ± 2% +19.8% 2212 ± 6% interrupts.CPU39.CAL:Function_call_interrupts
425307 +13.0% 480459 interrupts.CPU39.LOC:Local_timer_interrupts
21737 ± 2% +30.3% 28324 ± 2% interrupts.CPU39.RES:Rescheduling_interrupts
1901 ± 8% +18.2% 2247 ± 4% interrupts.CPU4.CAL:Function_call_interrupts
424921 +13.2% 480871 interrupts.CPU4.LOC:Local_timer_interrupts
22205 ± 2% +26.1% 28002 interrupts.CPU4.RES:Rescheduling_interrupts
425096 +13.1% 480655 interrupts.CPU5.LOC:Local_timer_interrupts
22004 +28.7% 28319 ± 2% interrupts.CPU5.RES:Rescheduling_interrupts
103.67 +13.3% 117.50 interrupts.CPU6.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
425006 +13.1% 480843 interrupts.CPU6.LOC:Local_timer_interrupts
22177 +26.5% 28053 interrupts.CPU6.RES:Rescheduling_interrupts
1962 ± 5% +17.3% 2301 ± 7% interrupts.CPU7.CAL:Function_call_interrupts
424930 +13.1% 480702 interrupts.CPU7.LOC:Local_timer_interrupts
1402 ±138% +387.3% 6831 ± 49% interrupts.CPU7.NMI:Non-maskable_interrupts
1402 ±138% +387.3% 6831 ± 49% interrupts.CPU7.PMI:Performance_monitoring_interrupts
21891 ± 2% +28.1% 28048 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts
1765 ± 6% +15.8% 2044 ± 5% interrupts.CPU8.CAL:Function_call_interrupts
423965 +13.4% 480881 interrupts.CPU8.LOC:Local_timer_interrupts
22606 ± 4% +23.5% 27917 interrupts.CPU8.RES:Rescheduling_interrupts
424961 +13.1% 480710 interrupts.CPU9.LOC:Local_timer_interrupts
22153 +27.1% 28146 ± 2% interrupts.CPU9.RES:Rescheduling_interrupts
16996312 +13.1% 19224399 interrupts.LOC:Local_timer_interrupts
883558 ± 2% +27.2% 1124085 interrupts.RES:Rescheduling_interrupts
43.50 -10.6 32.91 perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
42.95 -10.6 32.38 perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
41.58 -10.4 31.23 perf-profile.calltrace.cycles-pp._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
41.28 -10.3 30.96 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
42.18 -10.3 31.90 perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter
42.48 -10.3 32.21 perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
42.65 -10.3 32.39 perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
42.08 -10.2 31.85 perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write
0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
0.88 ± 2% -0.5 0.40 ± 58% perf-profile.calltrace.cycles-pp.secondary_startup_64
2.01 -0.4 1.60 ± 3% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
1.55 ± 2% -0.3 1.22 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
1.78 -0.3 1.46 perf-profile.calltrace.cycles-pp.read
1.53 ± 2% -0.3 1.21 ± 3% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write
1.10 ± 2% -0.3 0.81 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages
1.05 ± 2% -0.3 0.77 ± 6% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent
1.31 -0.2 1.06 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.32 -0.2 1.09 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
1.22 -0.2 1.01 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.16 -0.2 0.95 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.43 ± 4% -0.2 1.23 ± 3% perf-profile.calltrace.cycles-pp.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
0.76 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
0.76 ± 2% -0.2 0.59 ± 3% perf-profile.calltrace.cycles-pp.close
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.74 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.btrfs_evict_inode.evict.__dentry_kill.dentry_kill.dput
0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run
0.74 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput
0.86 -0.1 0.73 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.72 ± 2% -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.lseek64
0.83 -0.1 0.70 perf-profile.calltrace.cycles-pp.generic_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
0.74 ± 4% -0.1 0.61 ± 5% perf-profile.calltrace.cycles-pp.pagecache_get_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
0.64 ± 5% -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
0.59 ± 4% -0.0 0.54 ± 2% perf-profile.calltrace.cycles-pp.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter
0.58 ± 5% -0.0 0.53 ± 2% perf-profile.calltrace.cycles-pp.__extent_read_full_page.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write
94.94 +1.2 96.10 perf-profile.calltrace.cycles-pp.write
94.38 +1.2 95.60 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
94.47 +1.2 95.69 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
94.49 +1.2 95.72 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
94.29 +1.2 95.53 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
93.96 +1.3 95.23 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
93.90 +1.3 95.19 perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
93.59 +1.3 94.92 perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
41.79 +12.8 54.57 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release
3.76 +22.7 26.46 perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
0.00 +23.2 23.16 perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent
0.00 +23.2 23.20 perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit
0.83 ± 3% +23.3 24.09 perf-profile.calltrace.cycles-pp.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
0.81 ± 3% +23.3 24.09 perf-profile.calltrace.cycles-pp.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
0.58 ± 2% +23.3 23.88 perf-profile.calltrace.cycles-pp.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages
0.60 ± 2% +23.3 23.90 perf-profile.calltrace.cycles-pp.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write
0.00 +23.4 23.36 perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit
0.00 +23.5 23.48 perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit
43.13 -10.6 32.50 perf-profile.children.cycles-pp.reserve_metadata_bytes
43.51 -10.6 32.92 perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
2.02 -0.4 1.60 ± 3% perf-profile.children.cycles-pp.btrfs_get_extent
0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.secondary_startup_64
0.87 ± 4% -0.4 0.50 ± 9% perf-profile.children.cycles-pp.start_secondary
0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.cpu_startup_entry
0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.do_idle
0.83 ± 3% -0.4 0.48 ± 9% perf-profile.children.cycles-pp.cpuidle_enter_state
0.82 ± 4% -0.3 0.47 ± 9% perf-profile.children.cycles-pp.intel_idle
1.64 ± 2% -0.3 1.30 ± 3% perf-profile.children.cycles-pp.btrfs_search_slot
1.55 ± 2% -0.3 1.22 ± 3% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
1.81 -0.3 1.50 perf-profile.children.cycles-pp.read
1.15 ± 3% -0.3 0.86 ± 4% perf-profile.children.cycles-pp.btrfs_read_lock_root_node
1.10 ± 4% -0.3 0.81 ± 6% perf-profile.children.cycles-pp.btrfs_tree_read_lock
1.01 ± 4% -0.3 0.74 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
1.23 -0.2 1.01 perf-profile.children.cycles-pp.ksys_read
1.43 ± 4% -0.2 1.24 ± 3% perf-profile.children.cycles-pp.prepare_pages
1.16 -0.2 0.96 perf-profile.children.cycles-pp.vfs_read
1.07 ± 3% -0.2 0.87 ± 3% perf-profile.children.cycles-pp.__set_extent_bit
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.task_work_run
0.65 ± 5% -0.2 0.48 ± 9% perf-profile.children.cycles-pp.prepare_to_wait_event
0.76 ± 2% -0.2 0.59 ± 3% perf-profile.children.cycles-pp.close
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.__fput
0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.dput
0.74 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.btrfs_evict_inode
0.75 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.dentry_kill
0.75 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.__dentry_kill
0.75 ± 2% -0.2 0.58 ± 4% perf-profile.children.cycles-pp.exit_to_usermode_loop
0.74 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.evict
0.92 ± 2% -0.2 0.76 ± 3% perf-profile.children.cycles-pp.pagecache_get_page
0.75 ± 2% -0.1 0.61 perf-profile.children.cycles-pp.lseek64
0.87 -0.1 0.73 perf-profile.children.cycles-pp.new_sync_read
0.84 -0.1 0.70 perf-profile.children.cycles-pp.generic_file_read_iter
0.68 ± 2% -0.1 0.55 perf-profile.children.cycles-pp.entry_SYSCALL_64
0.46 -0.1 0.34 ± 5% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
0.32 ± 6% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.creat
0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.do_sys_open
0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.do_filp_open
0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.path_openat
1.00 ± 2% -0.1 0.90 ± 4% perf-profile.children.cycles-pp.can_overcommit
0.62 -0.1 0.52 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.43 ± 5% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.lock_extent_bits
0.57 ± 4% -0.1 0.48 ± 3% perf-profile.children.cycles-pp.set_extent_bit
0.31 ± 5% -0.1 0.22 ± 9% perf-profile.children.cycles-pp.finish_wait
0.43 ± 2% -0.1 0.35 perf-profile.children.cycles-pp.alloc_extent_state
0.16 ± 17% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.call_rwsem_down_write_failed
0.16 ± 17% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.rwsem_down_write_failed
0.20 ± 12% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.down_write
0.23 ± 2% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.do_unlinkat
0.23 -0.1 0.17 ± 3% perf-profile.children.cycles-pp.unlink
0.31 ± 3% -0.1 0.25 ± 2% perf-profile.children.cycles-pp.truncate_inode_pages_range
0.64 ± 5% -0.1 0.58 ± 2% perf-profile.children.cycles-pp.prepare_uptodate_page
0.46 -0.1 0.40 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter
0.40 ± 5% -0.1 0.34 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc
0.48 ± 2% -0.1 0.42 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
0.46 -0.1 0.40 ± 2% perf-profile.children.cycles-pp.__do_readpage
0.38 ± 6% -0.1 0.32 ± 2% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
0.31 -0.1 0.25 ± 5% perf-profile.children.cycles-pp.security_file_permission
0.29 ± 4% -0.1 0.24 perf-profile.children.cycles-pp.find_get_entry
0.17 ± 4% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.btrfs_create
0.18 ± 9% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.btrfs_trans_release_metadata
0.59 ± 4% -0.0 0.54 ± 2% perf-profile.children.cycles-pp.extent_read_full_page
0.17 ± 4% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.btrfs_unlink
0.18 ± 9% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.__btrfs_end_transaction
0.17 ± 4% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.vfs_unlink
0.31 ± 5% -0.0 0.27 ± 3% perf-profile.children.cycles-pp.btrfs_alloc_data_chunk_ondemand
0.14 ± 11% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
0.33 ± 5% -0.0 0.29 ± 5% perf-profile.children.cycles-pp.btrfs_check_data_free_space
0.28 ± 7% -0.0 0.24 ± 5% perf-profile.children.cycles-pp.add_to_page_cache_lru
0.21 ± 8% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.memset_erms
0.11 ± 15% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.__btrfs_btree_balance_dirty
0.06 ± 7% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__btrfs_unlink_inode
0.19 ± 4% -0.0 0.15 ± 2% perf-profile.children.cycles-pp.___might_sleep
0.16 ± 5% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.xas_load
0.11 ± 19% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.btrfs_delayed_inode_release_metadata
0.24 ± 5% -0.0 0.20 ± 3% perf-profile.children.cycles-pp.btrfs_copy_from_user
0.07 ± 17% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.21 ± 3% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.selinux_file_permission
0.21 ± 6% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.set_state_bits
0.36 ± 2% -0.0 0.33 ± 3% perf-profile.children.cycles-pp.copyout
0.10 ± 8% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.start_transaction
0.09 ± 18% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode
0.18 ± 4% -0.0 0.15 ± 11% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.17 ± 5% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
0.12 ± 10% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.free_extent_state
0.07 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.btrfs_delalloc_release_metadata
0.09 ± 10% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.btrfs_block_rsv_add
0.10 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.truncate_cleanup_page
0.18 ± 7% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size
0.06 ± 7% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.btrfs_unlink_inode
0.14 ± 3% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.copyin
0.18 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
0.08 ± 10% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.kfree
0.12 ± 3% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.release_pages
0.09 ± 9% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.xas_store
0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.split_state
0.07 -0.0 0.05 ± 8% perf-profile.children.cycles-pp.current_time
0.12 ± 4% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.__pagevec_release
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.try_merge_map
0.10 -0.0 0.09 ± 4% perf-profile.children.cycles-pp.btrfs_add_extent_mapping
0.09 ± 24% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.uart_console_write
0.09 ± 24% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.serial8250_console_putchar
0.16 ± 7% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.btrfs_free_reserved_data_space_noquota
97.42 +0.6 98.04 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
97.35 +0.6 97.97 perf-profile.children.cycles-pp.do_syscall_64
94.98 +1.2 96.13 perf-profile.children.cycles-pp.write
94.38 +1.2 95.61 perf-profile.children.cycles-pp.ksys_write
94.30 +1.2 95.54 perf-profile.children.cycles-pp.vfs_write
93.96 +1.3 95.24 perf-profile.children.cycles-pp.new_sync_write
93.91 +1.3 95.19 perf-profile.children.cycles-pp.btrfs_file_write_iter
93.60 +1.3 94.93 perf-profile.children.cycles-pp.btrfs_buffered_write
85.03 +2.0 87.03 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
86.13 +2.3 88.43 perf-profile.children.cycles-pp._raw_spin_lock
42.75 +12.7 55.42 perf-profile.children.cycles-pp.space_info_add_old_bytes
43.08 +12.8 55.90 perf-profile.children.cycles-pp.block_rsv_release_bytes
42.99 +12.9 55.92 perf-profile.children.cycles-pp.btrfs_inode_rsv_release
3.76 +22.7 26.47 perf-profile.children.cycles-pp.btrfs_dirty_pages
1.36 ± 3% +23.2 24.57 perf-profile.children.cycles-pp.__clear_extent_bit
0.89 ± 2% +23.2 24.14 perf-profile.children.cycles-pp.clear_extent_bit
0.94 ± 2% +23.3 24.21 perf-profile.children.cycles-pp.clear_state_bit
0.67 +23.3 23.95 perf-profile.children.cycles-pp.btrfs_clear_delalloc_extent
0.82 ± 4% -0.3 0.47 ± 9% perf-profile.self.cycles-pp.intel_idle
0.60 ± 2% -0.1 0.49 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.62 -0.1 0.52 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.30 ± 13% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.reserve_metadata_bytes
0.48 -0.1 0.42 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
0.51 ± 5% -0.1 0.45 ± 3% perf-profile.self.cycles-pp.can_overcommit
0.23 ± 7% -0.0 0.18 ± 8% perf-profile.self.cycles-pp.__set_extent_bit
0.23 ± 8% -0.0 0.19 ± 6% perf-profile.self.cycles-pp.kmem_cache_alloc
0.08 ± 12% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.btrfs_get_extent
0.20 ± 6% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.memset_erms
0.18 ± 10% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.do_syscall_64
0.18 ± 6% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.___might_sleep
0.07 ± 17% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.14 ± 6% -0.0 0.11 perf-profile.self.cycles-pp.btrfs_calculate_inode_block_rsv_size
0.11 ± 8% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.free_extent_state
0.16 ± 6% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.find_get_entry
0.10 ± 12% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.xas_load
0.14 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.selinux_file_permission
0.14 -0.0 0.12 ± 8% perf-profile.self.cycles-pp.btrfs_file_write_iter
0.15 ± 5% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.btrfs_buffered_write
0.11 ± 4% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.generic_file_read_iter
0.10 ± 4% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.btrfs_dirty_pages
0.06 -0.0 0.05 perf-profile.self.cycles-pp.rb_next
0.16 ± 2% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.btrfs_drop_pages
0.05 ± 8% +0.0 0.09 ± 20% perf-profile.self.cycles-pp.btrfs_delalloc_reserve_metadata
0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.btrfs_free_reserved_data_space_noquota
84.44 +2.1 86.53 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
aim7.jobs-per-min
50000 +-+-----------------------------------------------------------------+
45000 +-+ .+..+..+..+..+..+..+..+..+..+.+.. .+.. .+ |
| +. +..+..+. +..+. : +..+..|
40000 O-+ O O O O O O O O O O O O O O O O O O |
35000 +-+O O : : |
| : : |
30000 +-+ : : |
25000 +-+ : : |
20000 +-+ : : |
| : : |
15000 +-+ : : |
10000 +-+ : : |
| :: |
5000 +-+ : |
0 +-+------------------------------O----------------O-----------------+
aim7.time.system_time
10000 +-+-----------------------------------------------------------------+
9000 O-+O O O O O |
| O O O O O O O O O O O O O O O |
8000 +-++.. .+..+.. .+..+.. +..+..|
7000 +-+ +..+..+..+..+..+..+..+..+..+.+..+. +. + : |
| : : |
6000 +-+ : : |
5000 +-+ : : |
4000 +-+ : : |
| : : |
3000 +-+ : : |
2000 +-+ : : |
| : : |
1000 +-+ : |
0 +-+------------------------------O----------------O-----------------+
aim7.time.elapsed_time
250 +-+------O------------------------------------------------------------+
O O O O O O O O O O O O O O O O O O O O |
| .+..+.. +..+..|
200 +-++..+..+..+..+..+..+..+..+..+..+...+..+. +..+..+..+ : |
| : : |
| : : |
150 +-+ : : |
| : : |
100 +-+ : : |
| : : |
| : : |
50 +-+ : : |
| : : |
| : |
0 +-+------------------------------O------------------O-----------------+
aim7.time.elapsed_time.max
250 +-+------O------------------------------------------------------------+
O O O O O O O O O O O O O O O O O O O O |
| .+..+.. +..+..|
200 +-++..+..+..+..+..+..+..+..+..+..+...+..+. +..+..+..+ : |
| : : |
| : : |
150 +-+ : : |
| : : |
100 +-+ : : |
| : : |
| : : |
50 +-+ : : |
| : : |
| : |
0 +-+------------------------------O------------------O-----------------+
aim7.time.involuntary_context_switches
1e+06 +-+----------------------------------------------------------------+
900000 O-+O O |
| O O O O O O O O O O O O O O |
800000 +-+ O O O O |
700000 +-+ |
| .+.. +.. |
600000 +-++..+..+..+..+.+..+..+..+..+..+..+..+. +..+..+.+..+ : +..|
500000 +-+ : : |
400000 +-+ : : |
| : : |
300000 +-+ : : |
200000 +-+ : : |
| : : |
100000 +-+ : |
0 +-+-----------------------------O-----------------O----------------+
[*] bisect-good sample
[O] bisect-bad sample
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
1 year, 8 months
[tcp] 8b27dae5a2: netperf.Throughput_Mbps -25.7% regression
by kernel test robot
Greeting,
FYI, we noticed a -25.7% regression of netperf.Throughput_Mbps due to commit:
commit: 8b27dae5a2e89a61c46c6dbc76c040c0e6d0ed4c ("tcp: add one skb cache for rx")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master
in testcase: netperf
on test machine: 104 threads Skylake with 192G memory
with following parameters:
ip: ipv4
runtime: 900s
nr_threads: 200%
cluster: cs-localhost
test: TCP_STREAM
cpufreq_governor: performance
test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance.
test-url: http://www.netperf.org/netperf/
In addition to that, the commit also has significant impact on the following tests:
+------------------+----------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_Mbps -25.6% regression |
| test machine | 104 threads Skylake with 192G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | ip=ipv4 |
| | nr_threads=200% |
| | runtime=900s |
| | test=TCP_MAERTS |
+------------------+----------------------------------------------------+
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/900s/lkp-skl-fpga01/TCP_STREAM/netperf
commit:
472c2e07ee ("tcp: add one skb cache for tx")
8b27dae5a2 ("tcp: add one skb cache for rx")
472c2e07eef04514 8b27dae5a2e89a61c46c6dbc76c
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
:4 25% 1:4 dmesg.WARNING:at_ip_ip_finish_output2/0x
%stddev %change %stddev
\ | \
3939 -25.7% 2925 netperf.Throughput_Mbps
819330 -25.7% 608425 netperf.Throughput_total_Mbps
3.725e+09 -25.8% 2.764e+09 netperf.time.involuntary_context_switches
7399 -1.8% 7266 netperf.time.percent_of_cpu_this_job_got
65271 -1.0% 64597 netperf.time.system_time
1529 -34.0% 1009 netperf.time.user_time
5.626e+09 -25.7% 4.178e+09 netperf.workload
12.36 ± 4% +6.5 18.84 mpstat.cpu.all.soft%
2.95 ± 2% -0.8 2.15 mpstat.cpu.all.usr%
9.557e+08 +13.2% 1.082e+09 numa-numastat.node0.local_node
9.557e+08 +13.2% 1.082e+09 numa-numastat.node0.numa_hit
4.845e+08 ± 2% +11.6% 5.407e+08 numa-vmstat.node0.numa_hit
4.845e+08 ± 2% +11.6% 5.406e+08 numa-vmstat.node0.numa_local
2662 ± 2% +4.2% 2774 turbostat.Avg_MHz
1.87 ± 86% -66.1% 0.63 ± 9% turbostat.CPU%c1
94.75 ± 2% +2.4% 97.00 vmstat.cpu.sy
8145843 ± 2% -24.7% 6134835 vmstat.system.cs
224329 -0.8% 222470 vmstat.system.in
1.884e+09 +10.9% 2.09e+09 proc-vmstat.numa_hit
1.884e+09 +10.9% 2.089e+09 proc-vmstat.numa_local
1.506e+10 +10.9% 1.671e+10 proc-vmstat.pgalloc_normal
1.506e+10 +10.9% 1.671e+10 proc-vmstat.pgfree
2104 ± 8% +86.4% 3922 ± 58% sched_debug.cpu.avg_idle.min
35810547 -25.7% 26600377 sched_debug.cpu.nr_switches.avg
37423347 -25.1% 28047296 sched_debug.cpu.nr_switches.max
32860210 -25.5% 24491190 sched_debug.cpu.nr_switches.min
968.00 ± 4% -18.2% 792.00 ± 7% slabinfo.kmalloc-rcl-128.active_objs
968.00 ± 4% -18.2% 792.00 ± 7% slabinfo.kmalloc-rcl-128.num_objs
16611 ± 2% +6.1% 17628 ± 4% slabinfo.skbuff_fclone_cache.active_objs
16827 ± 2% +5.7% 17794 ± 4% slabinfo.skbuff_fclone_cache.num_objs
18.78 ± 5% +28.9% 24.21 perf-stat.i.MPKI
1.942e+10 ± 2% -21.1% 1.531e+10 perf-stat.i.branch-instructions
2.08 ± 6% -0.1 1.97 perf-stat.i.branch-miss-rate%
3.892e+08 ± 2% -22.6% 3.014e+08 perf-stat.i.branch-misses
1.767e+09 ± 2% +4.3% 1.842e+09 perf-stat.i.cache-references
8171306 ± 2% -24.7% 6151961 perf-stat.i.context-switches
2.88 ± 2% +31.0% 3.78 perf-stat.i.cpi
2.761e+11 ± 2% +4.2% 2.876e+11 perf-stat.i.cpu-cycles
1.10 ± 3% -0.1 0.97 perf-stat.i.dTLB-load-miss-rate%
3.254e+08 ± 4% -32.1% 2.208e+08 perf-stat.i.dTLB-load-misses
2.893e+10 ± 2% -21.9% 2.26e+10 perf-stat.i.dTLB-loads
58990104 ± 5% -25.8% 43766302 perf-stat.i.dTLB-store-misses
1.658e+10 ± 2% -23.6% 1.266e+10 perf-stat.i.dTLB-stores
55.76 -3.4 52.34 perf-stat.i.iTLB-load-miss-rate%
63191122 ± 2% -29.1% 44815478 perf-stat.i.iTLB-load-misses
49413835 ± 2% -17.7% 40661018 perf-stat.i.iTLB-loads
9.688e+10 ± 2% -21.5% 7.605e+10 perf-stat.i.instructions
1607 ± 3% +8.1% 1737 perf-stat.i.instructions-per-iTLB-miss
0.35 -24.3% 0.27 perf-stat.i.ipc
86243 ± 29% +52.4% 131400 ± 18% perf-stat.i.node-stores
18.24 +32.8% 24.23 perf-stat.overall.MPKI
2.00 -0.0 1.97 perf-stat.overall.branch-miss-rate%
2.85 +32.7% 3.78 perf-stat.overall.cpi
1.11 ± 3% -0.1 0.97 perf-stat.overall.dTLB-load-miss-rate%
56.12 -3.7 52.43 perf-stat.overall.iTLB-load-miss-rate%
1533 ± 2% +10.7% 1697 perf-stat.overall.instructions-per-iTLB-miss
0.35 -24.6% 0.26 perf-stat.overall.ipc
15748 +4.3% 16430 perf-stat.overall.path-length
1.939e+10 ± 2% -21.1% 1.53e+10 perf-stat.ps.branch-instructions
3.887e+08 ± 2% -22.5% 3.011e+08 perf-stat.ps.branch-misses
1.764e+09 ± 2% +4.3% 1.84e+09 perf-stat.ps.cache-references
8161286 ± 2% -24.7% 6144845 perf-stat.ps.context-switches
2.757e+11 ± 2% +4.2% 2.872e+11 perf-stat.ps.cpu-cycles
3.25e+08 ± 4% -32.1% 2.206e+08 perf-stat.ps.dTLB-load-misses
2.889e+10 ± 2% -21.8% 2.258e+10 perf-stat.ps.dTLB-loads
58916703 ± 5% -25.8% 43714633 perf-stat.ps.dTLB-store-misses
1.656e+10 ± 2% -23.6% 1.265e+10 perf-stat.ps.dTLB-stores
63112575 ± 2% -29.1% 44762927 perf-stat.ps.iTLB-load-misses
49353553 ± 2% -17.7% 40614004 perf-stat.ps.iTLB-loads
9.676e+10 ± 2% -21.5% 7.596e+10 perf-stat.ps.instructions
86182 ± 29% +52.3% 131291 ± 18% perf-stat.ps.node-stores
8.86e+13 -22.5% 6.864e+13 perf-stat.total.instructions
136251 ± 8% -18.5% 110991 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts
128756 ± 3% -18.6% 104869 ± 6% interrupts.CPU10.RES:Rescheduling_interrupts
119826 ± 6% -12.1% 105329 ± 4% interrupts.CPU100.RES:Rescheduling_interrupts
121133 ± 10% -17.9% 99457 interrupts.CPU101.RES:Rescheduling_interrupts
9051 ± 3% +14.1% 10328 interrupts.CPU102.CAL:Function_call_interrupts
131446 ± 7% -17.8% 108104 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts
124627 ± 2% -16.6% 103971 ± 9% interrupts.CPU12.RES:Rescheduling_interrupts
131072 ± 11% -19.9% 105033 ± 4% interrupts.CPU13.RES:Rescheduling_interrupts
127845 ± 8% -17.0% 106112 ± 9% interrupts.CPU14.RES:Rescheduling_interrupts
126159 ± 10% -20.6% 100111 ± 11% interrupts.CPU15.RES:Rescheduling_interrupts
132232 ± 5% -17.9% 108535 ± 7% interrupts.CPU17.RES:Rescheduling_interrupts
127157 ± 4% -17.3% 105165 ± 11% interrupts.CPU18.RES:Rescheduling_interrupts
127948 ± 2% -15.5% 108165 ± 11% interrupts.CPU19.RES:Rescheduling_interrupts
122716 ± 6% -14.0% 105519 ± 11% interrupts.CPU23.RES:Rescheduling_interrupts
130754 ± 5% -20.3% 104222 ± 9% interrupts.CPU24.RES:Rescheduling_interrupts
126283 ± 4% -16.6% 105320 ± 6% interrupts.CPU27.RES:Rescheduling_interrupts
8997 ± 3% -12.4% 7883 ± 9% interrupts.CPU28.CAL:Function_call_interrupts
128467 ± 5% -20.9% 101674 ± 11% interrupts.CPU29.RES:Rescheduling_interrupts
133914 ± 6% -26.0% 99089 ± 10% interrupts.CPU3.RES:Rescheduling_interrupts
8987 ± 3% -11.9% 7922 ± 7% interrupts.CPU31.CAL:Function_call_interrupts
115389 ± 9% -17.7% 94962 ± 5% interrupts.CPU31.RES:Rescheduling_interrupts
8830 ± 5% -12.1% 7759 ± 4% interrupts.CPU32.CAL:Function_call_interrupts
134198 ± 6% -19.6% 107954 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts
8940 ± 3% -8.8% 8154 ± 4% interrupts.CPU34.CAL:Function_call_interrupts
123678 ± 12% -17.1% 102582 ± 2% interrupts.CPU34.RES:Rescheduling_interrupts
8969 ± 3% -9.2% 8143 ± 4% interrupts.CPU35.CAL:Function_call_interrupts
118372 ± 8% -10.9% 105507 ± 9% interrupts.CPU35.RES:Rescheduling_interrupts
8958 ± 3% -9.2% 8131 ± 4% interrupts.CPU36.CAL:Function_call_interrupts
132196 ± 3% -16.2% 110761 ± 7% interrupts.CPU4.RES:Rescheduling_interrupts
123005 ± 6% -17.2% 101838 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts
132802 ± 15% -23.6% 101521 ± 11% interrupts.CPU43.RES:Rescheduling_interrupts
131107 ± 4% -25.6% 97566 ± 3% interrupts.CPU44.RES:Rescheduling_interrupts
127673 ± 8% -24.0% 97028 ± 6% interrupts.CPU47.RES:Rescheduling_interrupts
123709 ± 6% -14.3% 106030 ± 12% interrupts.CPU48.RES:Rescheduling_interrupts
123709 ± 7% -12.4% 108355 ± 6% interrupts.CPU49.RES:Rescheduling_interrupts
135382 ± 5% -20.7% 107334 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts
130424 ± 4% -17.4% 107704 ± 6% interrupts.CPU52.RES:Rescheduling_interrupts
129234 ± 8% -18.6% 105171 ± 10% interrupts.CPU53.RES:Rescheduling_interrupts
131374 ± 6% -18.8% 106699 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts
126141 ± 10% -14.7% 107626 ± 11% interrupts.CPU57.RES:Rescheduling_interrupts
133750 ± 9% -19.9% 107102 interrupts.CPU6.RES:Rescheduling_interrupts
119663 ± 5% -16.7% 99633 ± 3% interrupts.CPU60.RES:Rescheduling_interrupts
121078 ± 7% -13.6% 104670 ± 9% interrupts.CPU61.RES:Rescheduling_interrupts
121662 ± 5% -15.3% 102992 ± 10% interrupts.CPU63.RES:Rescheduling_interrupts
118130 ± 2% -15.1% 100310 ± 5% interrupts.CPU65.RES:Rescheduling_interrupts
128075 ± 6% -18.4% 104495 ± 8% interrupts.CPU67.RES:Rescheduling_interrupts
124700 ± 6% -15.7% 105149 ± 4% interrupts.CPU68.RES:Rescheduling_interrupts
119607 ± 9% -12.7% 104432 ± 6% interrupts.CPU7.RES:Rescheduling_interrupts
119288 ± 4% -11.8% 105251 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts
125678 ± 7% -18.7% 102141 ± 8% interrupts.CPU72.RES:Rescheduling_interrupts
130174 ± 5% -19.8% 104394 ± 3% interrupts.CPU74.RES:Rescheduling_interrupts
9215 ± 2% +9.8% 10119 ± 4% interrupts.CPU76.CAL:Function_call_interrupts
113784 ± 8% -10.6% 101686 ± 7% interrupts.CPU78.RES:Rescheduling_interrupts
122103 ± 5% -16.2% 102305 ± 5% interrupts.CPU79.RES:Rescheduling_interrupts
122242 ± 3% -23.8% 93153 ± 8% interrupts.CPU81.RES:Rescheduling_interrupts
9154 ± 3% +10.3% 10092 ± 4% interrupts.CPU82.CAL:Function_call_interrupts
116414 ± 3% -18.8% 94476 ± 9% interrupts.CPU82.RES:Rescheduling_interrupts
128558 ± 12% -24.5% 97067 ± 2% interrupts.CPU83.RES:Rescheduling_interrupts
9147 ± 3% +10.2% 10078 ± 4% interrupts.CPU84.CAL:Function_call_interrupts
122842 ± 9% -19.8% 98521 ± 2% interrupts.CPU84.RES:Rescheduling_interrupts
9081 ± 3% +10.9% 10071 ± 4% interrupts.CPU85.CAL:Function_call_interrupts
9130 ± 2% +10.2% 10065 ± 4% interrupts.CPU86.CAL:Function_call_interrupts
9068 ± 3% +10.9% 10057 ± 5% interrupts.CPU87.CAL:Function_call_interrupts
9068 ± 3% +10.8% 10050 ± 5% interrupts.CPU88.CAL:Function_call_interrupts
122119 ± 5% -20.1% 97576 ± 3% interrupts.CPU88.RES:Rescheduling_interrupts
9065 ± 3% +12.8% 10223 ± 2% interrupts.CPU89.CAL:Function_call_interrupts
124799 ± 6% -20.4% 99390 ± 8% interrupts.CPU89.RES:Rescheduling_interrupts
130362 ± 2% -12.6% 113981 ± 5% interrupts.CPU9.RES:Rescheduling_interrupts
8902 +12.7% 10035 ± 5% interrupts.CPU90.CAL:Function_call_interrupts
9062 ± 3% +10.7% 10028 ± 5% interrupts.CPU91.CAL:Function_call_interrupts
119442 ± 8% -24.1% 90611 ± 2% interrupts.CPU91.RES:Rescheduling_interrupts
9058 ± 3% +10.7% 10023 ± 5% interrupts.CPU92.CAL:Function_call_interrupts
117081 ± 10% -21.2% 92294 ± 12% interrupts.CPU92.RES:Rescheduling_interrupts
9058 ± 3% +10.6% 10017 ± 5% interrupts.CPU93.CAL:Function_call_interrupts
130977 ± 17% -27.7% 94702 ± 6% interrupts.CPU94.RES:Rescheduling_interrupts
9055 ± 3% +10.5% 10003 ± 5% interrupts.CPU95.CAL:Function_call_interrupts
116952 ± 4% -12.4% 102488 ± 10% interrupts.CPU95.RES:Rescheduling_interrupts
9054 ± 3% +10.4% 9995 ± 5% interrupts.CPU96.CAL:Function_call_interrupts
9054 ± 3% +10.3% 9990 ± 5% interrupts.CPU97.CAL:Function_call_interrupts
118634 ± 5% -12.7% 103578 ± 9% interrupts.CPU98.RES:Rescheduling_interrupts
116356 ± 10% -16.5% 97115 ± 7% interrupts.CPU99.RES:Rescheduling_interrupts
12787758 ± 2% -14.8% 10896246 interrupts.RES:Rescheduling_interrupts
10.84 -10.8 0.00 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
10.56 -10.6 0.00 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
10.16 -10.2 0.00 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
10.07 -10.1 0.00 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
9.71 -9.7 0.00 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack
9.57 -9.6 0.00 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue
21.28 -6.8 14.49 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.19 -6.8 14.43 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.57 -6.6 14.00 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.46 -6.5 13.93 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
16.37 -3.0 13.34 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
16.18 -3.0 13.21 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
7.01 ± 4% -2.4 4.61 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
6.90 ± 5% -2.4 4.54 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
7.71 ± 2% -2.2 5.51 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
7.69 ± 2% -2.2 5.49 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
6.19 ± 5% -2.1 4.10 perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
6.04 ± 5% -2.0 4.00 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
5.89 ± 6% -2.0 3.90 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
7.03 ± 2% -2.0 5.04 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
5.71 ± 6% -1.9 3.78 perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
13.46 -1.9 11.54 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
6.86 ± 2% -1.9 4.93 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
6.72 ± 2% -1.9 4.83 ± 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
13.26 -1.8 11.41 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
5.49 -1.8 3.67 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
2.62 ± 14% -1.8 0.82 ± 5% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
5.29 -1.7 3.61 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
4.80 -1.6 3.17 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
4.59 -1.6 3.02 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg
4.50 -1.5 2.96 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg
4.41 -1.5 2.90 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data
4.81 -1.5 3.31 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
10.13 -1.5 8.63 perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
9.79 -1.4 8.36 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
2.14 -1.4 0.73 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
4.34 -1.4 2.96 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg
9.60 -1.4 8.23 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
2.10 ± 34% -1.4 0.73 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock
2.08 -1.4 0.71 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
3.88 ± 2% -1.2 2.71 perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
3.71 -1.2 2.55 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg
3.56 ± 2% -1.1 2.49 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
3.32 ± 2% -1.0 2.34 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
1.56 -0.9 0.61 ± 2% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
3.14 ± 2% -0.9 2.22 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
1.77 -0.6 1.16 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken
7.43 ± 2% -0.6 6.87 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
7.41 ± 2% -0.6 6.85 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.39 -0.5 0.87 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.34 -0.5 0.84 perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.30 -0.5 0.81 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.09 -0.4 0.72 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.20 -0.4 0.84 perf-profile.calltrace.cycles-pp.__switch_to
1.09 -0.3 0.76 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
6.67 ± 2% -0.3 6.38 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
0.83 -0.3 0.56 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg
0.84 -0.3 0.57 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout.wait_woken
0.95 -0.3 0.69 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
0.75 ± 2% -0.2 0.52 perf-profile.calltrace.cycles-pp._cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
0.73 ± 2% -0.2 0.51 perf-profile.calltrace.cycles-pp.preempt_schedule_common._cond_resched.__release_sock.release_sock.tcp_sendmsg
9.41 +1.1 10.47 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb
9.29 +1.1 10.39 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data
95.31 +1.4 96.67 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
95.15 +1.4 96.56 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.02 +7.2 18.20 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
12.28 ± 2% +7.2 19.51 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
12.21 ± 2% +7.3 19.47 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
12.00 ± 2% +7.3 19.34 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
11.98 ± 2% +7.3 19.32 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
11.74 ± 2% +7.4 19.14 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
11.43 ± 2% +7.4 18.87 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
11.37 ± 2% +7.5 18.83 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
9.15 +7.5 16.67 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim
9.26 +7.5 16.80 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb
9.43 +7.6 17.01 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked
9.79 +7.6 17.37 perf-profile.calltrace.cycles-pp.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
10.97 ± 2% +7.6 18.56 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
9.59 +7.6 17.22 perf-profile.calltrace.cycles-pp.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg
10.64 ± 3% +7.7 18.30 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
10.11 ± 3% +7.8 17.94 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
9.96 ± 3% +7.9 17.82 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
3.91 ± 2% +7.9 11.79 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
3.91 ± 2% +7.9 11.79 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
9.90 ± 3% +7.9 17.79 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
9.71 ± 3% +8.0 17.67 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
3.58 ± 2% +8.0 11.60 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
3.34 ± 2% +8.1 11.45 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
9.05 ± 2% +8.1 17.17 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
18.39 +8.7 27.12 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill
18.58 +8.7 27.31 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill
19.93 +8.7 28.67 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
19.92 +8.7 28.66 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
19.70 +8.7 28.45 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked
19.77 +8.8 28.54 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg
72.20 +8.8 81.00 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
72.09 +8.8 80.92 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.49 +9.0 80.48 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.15 +9.1 80.24 perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
0.00 +10.6 10.61 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv
0.00 +10.7 10.72 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu
0.00 +11.0 10.98 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.00 +11.1 11.11 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
54.40 +12.2 66.63 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
32407497 -26.7% 23748191 softirqs.CPU0.NET_RX
31936171 ± 3% -23.8% 24347695 softirqs.CPU1.NET_RX
31995720 ± 3% -23.5% 24474810 softirqs.CPU10.NET_RX
31851135 -28.4% 22807522 softirqs.CPU100.NET_RX
31165470 ± 2% -27.9% 22460444 ± 2% softirqs.CPU101.NET_RX
30427682 ± 3% -27.8% 21964060 ± 2% softirqs.CPU102.NET_RX
31822429 -30.3% 22190016 ± 2% softirqs.CPU103.NET_RX
31794681 ± 3% -24.0% 24170873 ± 2% softirqs.CPU11.NET_RX
31337588 ± 3% -25.0% 23487789 ± 2% softirqs.CPU12.NET_RX
30982384 ± 3% -21.6% 24285912 softirqs.CPU13.NET_RX
31985249 ± 2% -24.5% 24135878 ± 2% softirqs.CPU14.NET_RX
31569639 ± 3% -23.4% 24173977 ± 2% softirqs.CPU15.NET_RX
31672651 ± 2% -23.7% 24152941 ± 2% softirqs.CPU16.NET_RX
31845476 ± 2% -23.4% 24406051 softirqs.CPU17.NET_RX
32237888 -24.4% 24378972 softirqs.CPU18.NET_RX
32390163 -25.7% 24060765 softirqs.CPU19.NET_RX
32518154 -26.3% 23981309 ± 2% softirqs.CPU2.NET_RX
31625849 ± 2% -26.0% 23391632 ± 2% softirqs.CPU20.NET_RX
31800991 ± 2% -24.2% 24096267 ± 2% softirqs.CPU21.NET_RX
32165283 ± 2% -23.9% 24484882 softirqs.CPU22.NET_RX
32096347 ± 3% -24.6% 24198018 ± 2% softirqs.CPU23.NET_RX
32606868 -24.9% 24498919 softirqs.CPU24.NET_RX
31333059 ± 2% -23.5% 23972392 ± 2% softirqs.CPU25.NET_RX
29740104 ± 3% -23.9% 22620846 softirqs.CPU26.NET_RX
8933 ± 31% -24.6% 6736 ± 7% softirqs.CPU26.SCHED
31426927 -28.2% 22574373 ± 2% softirqs.CPU27.NET_RX
31494409 -27.6% 22811958 softirqs.CPU28.NET_RX
31749609 -29.8% 22274240 ± 2% softirqs.CPU29.NET_RX
31359195 ± 3% -23.3% 24058233 softirqs.CPU3.NET_RX
31891532 -28.5% 22801967 softirqs.CPU30.NET_RX
31022930 ± 3% -26.6% 22780850 softirqs.CPU31.NET_RX
31633941 -28.0% 22773366 softirqs.CPU32.NET_RX
31064582 ± 3% -28.2% 22291648 ± 2% softirqs.CPU33.NET_RX
30670424 ± 2% -26.0% 22706259 softirqs.CPU34.NET_RX
31155193 ± 3% -28.3% 22336447 ± 2% softirqs.CPU35.NET_RX
31767721 -29.3% 22462506 ± 2% softirqs.CPU36.NET_RX
31702765 -27.9% 22848963 softirqs.CPU37.NET_RX
31618979 -28.8% 22498908 ± 2% softirqs.CPU38.NET_RX
8157 ± 30% -22.3% 6342 softirqs.CPU38.SCHED
31178503 ± 2% -28.8% 22206116 ± 2% softirqs.CPU39.NET_RX
31543118 ± 3% -22.7% 24378208 softirqs.CPU4.NET_RX
29993167 ± 3% -26.6% 22006508 softirqs.CPU40.NET_RX
31389405 ± 2% -27.4% 22777491 softirqs.CPU41.NET_RX
31225889 ± 3% -28.1% 22466787 ± 2% softirqs.CPU42.NET_RX
31122849 ± 3% -27.5% 22548782 softirqs.CPU43.NET_RX
30449949 ± 3% -26.7% 22314973 ± 2% softirqs.CPU44.NET_RX
30708016 ± 3% -26.5% 22575709 softirqs.CPU45.NET_RX
30869514 ± 2% -27.1% 22517203 softirqs.CPU46.NET_RX
30349340 ± 3% -26.7% 22238899 ± 2% softirqs.CPU47.NET_RX
31690538 -27.9% 22844885 softirqs.CPU48.NET_RX
31442996 -27.6% 22774395 softirqs.CPU49.NET_RX
32539333 -24.8% 24475876 softirqs.CPU5.NET_RX
31777443 -28.1% 22839894 softirqs.CPU50.NET_RX
30319039 ± 4% -25.8% 22488386 softirqs.CPU51.NET_RX
32358234 -24.7% 24364666 softirqs.CPU52.NET_RX
31051127 ± 3% -23.5% 23764585 ± 2% softirqs.CPU53.NET_RX
31368731 ± 3% -22.3% 24384593 softirqs.CPU54.NET_RX
32739188 -26.5% 24063678 ± 2% softirqs.CPU55.NET_RX
32587893 -25.0% 24425586 softirqs.CPU56.NET_RX
31762403 ± 3% -24.6% 23947521 softirqs.CPU57.NET_RX
32342624 -25.5% 24095598 softirqs.CPU58.NET_RX
31850813 ± 3% -23.7% 24310413 softirqs.CPU59.NET_RX
32296552 -26.1% 23882250 ± 2% softirqs.CPU6.NET_RX
31208970 ± 3% -23.9% 23735529 ± 2% softirqs.CPU60.NET_RX
32655061 -25.7% 24259328 softirqs.CPU61.NET_RX
31834306 ± 3% -23.2% 24436288 softirqs.CPU62.NET_RX
32492263 -25.7% 24146807 ± 2% softirqs.CPU63.NET_RX
32160041 -24.6% 24237812 softirqs.CPU64.NET_RX
32146792 -25.1% 24083142 softirqs.CPU65.NET_RX
31886509 ± 3% -24.3% 24153287 ± 2% softirqs.CPU66.NET_RX
31124000 ± 2% -21.2% 24534615 softirqs.CPU67.NET_RX
31606731 ± 3% -23.7% 24115056 ± 2% softirqs.CPU68.NET_RX
31809649 ± 4% -27.3% 23140437 softirqs.CPU69.NET_RX
31832502 ± 3% -23.8% 24256886 softirqs.CPU7.NET_RX
31934664 ± 2% -23.5% 24445503 softirqs.CPU70.NET_RX
32283512 -24.5% 24374573 softirqs.CPU71.NET_RX
31559429 ± 3% -23.0% 24304361 softirqs.CPU72.NET_RX
32519181 -26.5% 23915099 ± 2% softirqs.CPU73.NET_RX
32064639 ± 3% -23.5% 24531236 softirqs.CPU74.NET_RX
32552635 -25.2% 24358616 softirqs.CPU75.NET_RX
32574031 -25.1% 24391309 softirqs.CPU76.NET_RX
29891561 -18.3% 24435768 softirqs.CPU77.NET_RX
30558797 ± 3% -27.1% 22290753 ± 2% softirqs.CPU78.NET_RX
30913524 ± 3% -27.1% 22542484 softirqs.CPU79.NET_RX
32338939 -25.2% 24184482 softirqs.CPU8.NET_RX
29930689 ± 3% -24.3% 22649621 softirqs.CPU80.NET_RX
31789053 -29.3% 22471154 ± 2% softirqs.CPU81.NET_RX
31080841 ± 3% -28.0% 22371709 ± 2% softirqs.CPU82.NET_RX
31004904 ± 3% -28.0% 22335427 softirqs.CPU83.NET_RX
30902184 ± 2% -26.4% 22746371 softirqs.CPU84.NET_RX
31508332 -28.4% 22567097 softirqs.CPU85.NET_RX
31437975 -29.0% 22331821 ± 2% softirqs.CPU86.NET_RX
31025912 ± 3% -26.1% 22913093 softirqs.CPU87.NET_RX
30270422 ± 2% -25.2% 22636977 ± 2% softirqs.CPU88.NET_RX
30385565 ± 4% -26.0% 22499521 ± 2% softirqs.CPU89.NET_RX
31577940 ± 3% -22.4% 24489124 softirqs.CPU9.NET_RX
30152844 ± 3% -24.7% 22717462 softirqs.CPU90.NET_RX
29900183 ± 2% -24.9% 22460973 softirqs.CPU91.NET_RX
31952707 -28.9% 22715130 softirqs.CPU92.NET_RX
30820437 ± 3% -26.6% 22619907 ± 2% softirqs.CPU93.NET_RX
30627580 ± 3% -25.6% 22790846 softirqs.CPU94.NET_RX
31367872 -27.7% 22670545 softirqs.CPU95.NET_RX
31045007 ± 3% -26.5% 22818368 softirqs.CPU96.NET_RX
30842779 ± 3% -28.2% 22143073 ± 2% softirqs.CPU97.NET_RX
30173663 ± 3% -24.8% 22686608 softirqs.CPU98.NET_RX
30750855 ± 3% -25.6% 22870339 softirqs.CPU99.NET_RX
3.273e+09 -25.8% 2.43e+09 softirqs.NET_RX
netperf.Throughput_Mbps
4000 +-+------------------------------------------------------------------+
| |
3500 +-+ |
3000 +-+ |
O O O OO OOO OO OO OOO OO OO OO OOO OO OO OO |
2500 +-+ |
| |
2000 +-+ |
| |
1500 +-+ |
1000 +-+ |
| |
500 +-+ |
| |
0 +O+O---O-------------------------------------------------------------+
netperf.Throughput_total_Mbps
900000 +-+----------------------------------------------------------------+
|+.++.+++.++.+++.++.+++.++.+++.++.+++.+++.++.+++.++.+++.++.+++.++.+|
800000 +-+ |
700000 +-+ |
| O |
600000 O-+ O O O OO OOO OO OOO OO OOO O OOO OOO OO O |
500000 +-+ |
| |
400000 +-+ |
300000 +-+ |
| |
200000 +-+ |
100000 +-+ |
| |
0 +O+O---O-----------------------------------------------------------+
netperf.workload
6e+09 +-+-----------------------------------------------------------------+
|+.++.++.+++.++.+++.++.++.+++.++.+++.++.++.+++.++.+++.++.++.+++.++.+|
5e+09 +-+ |
| |
O O O OOO OO OOO OO OO OOO OO OOO OO OO OOO |
4e+09 +-+ |
| |
3e+09 +-+ |
| |
2e+09 +-+ |
| |
| |
1e+09 +-+ |
| |
0 +O+O---O------------------------------------------------------------+
netperf.time.user_time
2000 +-+------------------------------------------------------------------+
1800 +-+ + + |
| : : + : |
1600 +-+ +. + .++.++. + : +.+ .++. .++.++. + .++ + .++.+|
1400 +-+++.++.+ + + + + + +++ + +.++.++ +.++ |
| |
1200 +-+ |
1000 O-+ O OO OOO OO OO OOO OO OO OO OOO OO OO OO |
800 +-+ |
| |
600 +-+ |
400 +-+ |
| |
200 +-+ |
0 +O+O--OO-------------------------------------------------------------+
netperf.time.percent_of_cpu_this_job_got
8000 +-+------------------------------------------------------------------+
O+.+O.++.OO.OOO.OO.OO.OOO.OO.OO.OO.OOO.OO.OO.OO+.++.++.++.+++.++.++.+|
7000 +-+ |
6000 +-+ |
| |
5000 +-+ |
| |
4000 +-+ |
| |
3000 +-+ |
2000 +-+ |
| |
1000 +-+ |
| |
0 +O+O--OO-------------------------------------------------------------+
netperf.time.involuntary_context_switches
4e+09 +-+---------------------------------------------------------------+
|+.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+|
3.5e+09 +-+ |
3e+09 +-+ |
O O O OOO OO OOO OOO OO OOO OOO OO OOO OO |
2.5e+09 +-+ |
| |
2e+09 +-+ |
| |
1.5e+09 +-+ |
1e+09 +-+ |
| |
5e+08 +-+ |
| |
0 +O+O--OO----------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-skl-fpga01: 104 threads Skylake with 192G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/900s/lkp-skl-fpga01/TCP_MAERTS/netperf
commit:
472c2e07ee ("tcp: add one skb cache for tx")
8b27dae5a2 ("tcp: add one skb cache for rx")
472c2e07eef04514 8b27dae5a2e89a61c46c6dbc76c
---------------- ---------------------------
%stddev %change %stddev
\ | \
3933 -25.6% 2926 netperf.Throughput_Mbps
818124 -25.6% 608758 netperf.Throughput_total_Mbps
57423 ± 19% -41.1% 33817 netperf.time.involuntary_context_switches
3010 +3.7% 3122 netperf.time.percent_of_cpu_this_job_got
25432 +3.9% 26424 netperf.time.system_time
3.735e+09 -25.6% 2.778e+09 netperf.time.voluntary_context_switches
5.618e+09 -25.6% 4.18e+09 netperf.workload
194721 ± 22% +41.3% 275069 ± 2% cpuidle.C6.usage
13.00 ± 7% +5.9 18.86 mpstat.cpu.all.soft%
3.39 ± 5% -1.2 2.23 mpstat.cpu.all.usr%
9.539e+08 +13.2% 1.08e+09 numa-numastat.node0.local_node
9.539e+08 +13.2% 1.08e+09 numa-numastat.node0.numa_hit
4.767e+08 +13.1% 5.389e+08 numa-vmstat.node0.numa_hit
4.767e+08 +13.1% 5.389e+08 numa-vmstat.node0.numa_local
95.75 +1.3% 97.00 vmstat.cpu.sy
8245666 -25.5% 6142829 vmstat.system.cs
2696 +2.9% 2775 turbostat.Avg_MHz
0.89 ± 5% -32.4% 0.60 turbostat.CPU%c1
108.00 -1.6% 106.23 turbostat.RAMWatt
1.881e+09 +11.2% 2.091e+09 proc-vmstat.numa_hit
1.881e+09 +11.2% 2.091e+09 proc-vmstat.numa_local
1.504e+10 +11.2% 1.672e+10 proc-vmstat.pgalloc_normal
1.504e+10 +11.2% 1.672e+10 proc-vmstat.pgfree
4053 ± 15% +19.6% 4848 ± 13% slabinfo.eventpoll_pwq.active_objs
4053 ± 15% +19.6% 4848 ± 13% slabinfo.eventpoll_pwq.num_objs
545.50 ± 11% -17.1% 452.00 ± 11% slabinfo.kernfs_iattrs_cache.active_objs
545.50 ± 11% -17.1% 452.00 ± 11% slabinfo.kernfs_iattrs_cache.num_objs
12.31 ± 12% +25.1% 15.40 sched_debug.cfs_rq:/.load_avg.stddev
1449447 ± 23% -29.2% 1026327 ± 22% sched_debug.cfs_rq:/.min_vruntime.stddev
0.78 ± 66% +155.6% 2.00 ± 11% sched_debug.cfs_rq:/.removed.load_avg.avg
5.98 ± 60% +84.9% 11.05 ± 4% sched_debug.cfs_rq:/.removed.load_avg.stddev
35.94 ± 66% +156.2% 92.09 ± 11% sched_debug.cfs_rq:/.removed.runnable_sum.avg
274.98 ± 60% +85.4% 509.82 ± 4% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
0.39 ± 66% +115.0% 0.84 ± 11% sched_debug.cfs_rq:/.removed.util_avg.avg
1.75 ± 14% +32.1% 2.31 ± 12% sched_debug.cfs_rq:/.runnable_load_avg.min
98724 ± 85% -66.3% 33282 ± 85% sched_debug.cfs_rq:/.runnable_weight.max
-3346961 -44.2% -1868641 sched_debug.cfs_rq:/.spread0.min
1449347 ± 23% -29.2% 1026220 ± 22% sched_debug.cfs_rq:/.spread0.stddev
156728 ± 15% +49.0% 233530 ± 14% sched_debug.cpu.avg_idle.avg
260031 ± 9% +33.3% 346707 ± 10% sched_debug.cpu.avg_idle.stddev
20.54 ± 5% -10.2% 18.44 ± 2% sched_debug.cpu.clock.stddev
20.54 ± 5% -10.2% 18.44 ± 2% sched_debug.cpu.clock_task.stddev
35742943 -25.6% 26602562 sched_debug.cpu.nr_switches.avg
37162589 -24.6% 28014480 sched_debug.cpu.nr_switches.max
32371186 -24.3% 24512619 sched_debug.cpu.nr_switches.min
18.74 +29.7% 24.31 perf-stat.i.MPKI
1.938e+10 -20.9% 1.534e+10 perf-stat.i.branch-instructions
2.05 -0.1 1.98 perf-stat.i.branch-miss-rate%
3.961e+08 -23.4% 3.033e+08 perf-stat.i.branch-misses
0.74 ± 31% -0.4 0.38 ± 18% perf-stat.i.cache-miss-rate%
12929011 ± 32% -49.1% 6575616 ± 19% perf-stat.i.cache-misses
1.813e+09 +2.3% 1.854e+09 perf-stat.i.cache-references
8267879 -25.5% 6157361 perf-stat.i.context-switches
2.89 +30.6% 3.77 perf-stat.i.cpi
2.794e+11 +2.9% 2.876e+11 perf-stat.i.cpu-cycles
23942 ± 24% +97.5% 47291 ± 21% perf-stat.i.cycles-between-cache-misses
1.13 -0.1 0.99 perf-stat.i.dTLB-load-miss-rate%
3.331e+08 -31.8% 2.274e+08 perf-stat.i.dTLB-load-misses
2.9e+10 -22.0% 2.263e+10 perf-stat.i.dTLB-loads
60002134 ± 2% -26.3% 44193691 ± 2% perf-stat.i.dTLB-store-misses
1.664e+10 -23.6% 1.271e+10 perf-stat.i.dTLB-stores
58.26 -4.7 53.57 perf-stat.i.iTLB-load-miss-rate%
70291090 -31.0% 48469793 perf-stat.i.iTLB-load-misses
50169394 -16.5% 41890643 perf-stat.i.iTLB-loads
9.677e+10 -21.2% 7.624e+10 perf-stat.i.instructions
1410 +13.6% 1602 perf-stat.i.instructions-per-iTLB-miss
0.35 -23.4% 0.27 perf-stat.i.ipc
2150902 ± 41% -42.7% 1233268 ± 27% perf-stat.i.node-load-misses
652857 ± 24% -39.8% 393122 ± 15% perf-stat.i.node-loads
77.08 ± 11% -20.6 56.48 ± 14% perf-stat.i.node-store-miss-rate%
386848 ± 40% -64.7% 136565 ± 27% perf-stat.i.node-store-misses
18.74 +29.8% 24.32 perf-stat.overall.MPKI
2.04 -0.1 1.98 perf-stat.overall.branch-miss-rate%
0.71 ± 32% -0.4 0.35 ± 20% perf-stat.overall.cache-miss-rate%
2.89 +30.6% 3.77 perf-stat.overall.cpi
23413 ± 23% +95.0% 45644 ± 21% perf-stat.overall.cycles-between-cache-misses
1.14 -0.1 0.99 perf-stat.overall.dTLB-load-miss-rate%
58.35 -4.7 53.64 perf-stat.overall.iTLB-load-miss-rate%
1376 +14.3% 1573 perf-stat.overall.instructions-per-iTLB-miss
0.35 -23.4% 0.27 perf-stat.overall.ipc
75.96 ± 13% -20.1 55.89 ± 14% perf-stat.overall.node-store-miss-rate%
15545 +5.9% 16457 perf-stat.overall.path-length
1.936e+10 -20.9% 1.532e+10 perf-stat.ps.branch-instructions
3.956e+08 -23.4% 3.03e+08 perf-stat.ps.branch-misses
12917167 ± 32% -49.1% 6568704 ± 19% perf-stat.ps.cache-misses
1.811e+09 +2.3% 1.852e+09 perf-stat.ps.cache-references
8258274 -25.5% 6150289 perf-stat.ps.context-switches
2.791e+11 +2.9% 2.872e+11 perf-stat.ps.cpu-cycles
3.328e+08 -31.8% 2.271e+08 perf-stat.ps.dTLB-load-misses
2.897e+10 -22.0% 2.26e+10 perf-stat.ps.dTLB-loads
59931265 ± 2% -26.3% 44142618 ± 2% perf-stat.ps.dTLB-store-misses
1.662e+10 -23.6% 1.269e+10 perf-stat.ps.dTLB-stores
70208114 -31.0% 48413951 perf-stat.ps.iTLB-load-misses
50111323 -16.5% 41842519 perf-stat.ps.iTLB-loads
9.666e+10 -21.2% 7.616e+10 perf-stat.ps.instructions
2148559 ± 41% -42.7% 1231893 ± 27% perf-stat.ps.node-load-misses
653248 ± 24% -39.9% 392846 ± 15% perf-stat.ps.node-loads
386418 ± 40% -64.7% 136417 ± 27% perf-stat.ps.node-store-misses
8.733e+13 -21.2% 6.879e+13 perf-stat.total.instructions
887.50 ± 21% +961.1% 9417 ±122% interrupts.39:PCI-MSI.67633154-edge.eth0-TxRx-1
906772 +4.4% 946334 interrupts.CAL:Function_call_interrupts
128451 ± 3% -20.1% 102684 ± 11% interrupts.CPU1.RES:Rescheduling_interrupts
122297 ± 12% -20.1% 97767 ± 5% interrupts.CPU100.RES:Rescheduling_interrupts
114996 ± 8% -12.8% 100315 ± 5% interrupts.CPU101.RES:Rescheduling_interrupts
8676 ± 2% +14.9% 9972 ± 3% interrupts.CPU102.CAL:Function_call_interrupts
145573 ± 15% -30.4% 101298 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts
8795 +13.2% 9956 ± 3% interrupts.CPU103.CAL:Function_call_interrupts
129834 ± 7% -15.4% 109820 ± 9% interrupts.CPU11.RES:Rescheduling_interrupts
122266 ± 5% -18.3% 99893 ± 4% interrupts.CPU13.RES:Rescheduling_interrupts
133695 ± 4% -23.7% 101984 ± 5% interrupts.CPU14.RES:Rescheduling_interrupts
115773 ± 7% -18.6% 94202 ± 8% interrupts.CPU18.RES:Rescheduling_interrupts
120005 ± 3% -15.7% 101149 ± 8% interrupts.CPU27.RES:Rescheduling_interrupts
119584 ± 3% -19.5% 96286 ± 6% interrupts.CPU29.RES:Rescheduling_interrupts
127064 ± 7% -15.1% 107858 ± 5% interrupts.CPU30.RES:Rescheduling_interrupts
887.50 ± 21% +961.1% 9417 ±122% interrupts.CPU31.39:PCI-MSI.67633154-edge.eth0-TxRx-1
118974 ± 11% -17.1% 98609 ± 7% interrupts.CPU33.RES:Rescheduling_interrupts
125463 ± 13% -23.9% 95477 ± 5% interrupts.CPU34.RES:Rescheduling_interrupts
125126 ± 16% -21.7% 97997 ± 7% interrupts.CPU35.RES:Rescheduling_interrupts
133035 ± 12% -28.2% 95517 ± 10% interrupts.CPU37.RES:Rescheduling_interrupts
120167 ± 11% -24.5% 90782 ± 8% interrupts.CPU38.RES:Rescheduling_interrupts
125040 ± 3% -19.8% 100223 ± 7% interrupts.CPU42.RES:Rescheduling_interrupts
119768 ± 15% -16.6% 99882 interrupts.CPU43.RES:Rescheduling_interrupts
125986 ± 7% -18.9% 102226 ± 8% interrupts.CPU45.RES:Rescheduling_interrupts
124516 ± 10% -15.7% 104961 ± 5% interrupts.CPU48.RES:Rescheduling_interrupts
125647 ± 5% -22.8% 96970 ± 3% interrupts.CPU49.RES:Rescheduling_interrupts
131345 ± 5% -17.2% 108796 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts
127858 ± 3% -18.9% 103683 ± 5% interrupts.CPU53.RES:Rescheduling_interrupts
125241 ± 15% -18.7% 101797 ± 5% interrupts.CPU54.RES:Rescheduling_interrupts
133398 ± 7% -22.1% 103961 ± 3% interrupts.CPU56.RES:Rescheduling_interrupts
129732 ± 10% -24.2% 98324 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts
132858 ± 12% -20.5% 105623 ± 9% interrupts.CPU60.RES:Rescheduling_interrupts
123442 ± 5% -20.7% 97934 ± 4% interrupts.CPU64.RES:Rescheduling_interrupts
132863 ± 3% -20.2% 106040 ± 4% interrupts.CPU65.RES:Rescheduling_interrupts
9002 +10.7% 9967 ± 2% interrupts.CPU66.CAL:Function_call_interrupts
8999 +10.7% 9958 ± 2% interrupts.CPU67.CAL:Function_call_interrupts
8774 ± 5% +13.4% 9951 ± 2% interrupts.CPU68.CAL:Function_call_interrupts
8991 +10.6% 9945 ± 2% interrupts.CPU69.CAL:Function_call_interrupts
125949 ± 9% -15.0% 107048 ± 4% interrupts.CPU69.RES:Rescheduling_interrupts
123443 ± 8% -15.7% 104008 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts
8987 +10.6% 9938 ± 2% interrupts.CPU70.CAL:Function_call_interrupts
8983 +10.5% 9927 ± 2% interrupts.CPU71.CAL:Function_call_interrupts
122925 ± 12% -18.0% 100786 ± 8% interrupts.CPU71.RES:Rescheduling_interrupts
8925 +11.0% 9910 ± 2% interrupts.CPU73.CAL:Function_call_interrupts
126258 ± 7% -19.4% 101773 ± 7% interrupts.CPU73.RES:Rescheduling_interrupts
129281 ± 6% -18.1% 105945 ± 5% interrupts.CPU74.RES:Rescheduling_interrupts
8969 +10.3% 9893 ± 2% interrupts.CPU75.CAL:Function_call_interrupts
8966 +10.3% 9887 ± 2% interrupts.CPU76.CAL:Function_call_interrupts
8965 +10.1% 9870 ± 2% interrupts.CPU77.CAL:Function_call_interrupts
129057 ± 9% -24.7% 97240 ± 3% interrupts.CPU77.RES:Rescheduling_interrupts
8683 ± 5% +13.7% 9874 ± 2% interrupts.CPU78.CAL:Function_call_interrupts
129114 ± 9% -27.6% 93483 ± 4% interrupts.CPU78.RES:Rescheduling_interrupts
8685 ± 4% +13.6% 9865 ± 2% interrupts.CPU79.CAL:Function_call_interrupts
121386 ± 7% -18.7% 98627 ± 5% interrupts.CPU8.RES:Rescheduling_interrupts
8956 +10.1% 9864 ± 2% interrupts.CPU80.CAL:Function_call_interrupts
125015 ± 11% -24.1% 94918 ± 5% interrupts.CPU80.RES:Rescheduling_interrupts
8633 ± 7% +14.2% 9856 ± 2% interrupts.CPU81.CAL:Function_call_interrupts
132492 ± 4% -20.7% 105014 ± 4% interrupts.CPU81.RES:Rescheduling_interrupts
8857 ± 2% +11.2% 9848 ± 2% interrupts.CPU82.CAL:Function_call_interrupts
111387 ± 6% -13.8% 96026 ± 8% interrupts.CPU82.RES:Rescheduling_interrupts
112445 ± 6% -9.6% 101699 ± 7% interrupts.CPU83.RES:Rescheduling_interrupts
121973 ± 7% -21.1% 96217 ± 6% interrupts.CPU84.RES:Rescheduling_interrupts
8883 +10.6% 9822 ± 2% interrupts.CPU85.CAL:Function_call_interrupts
115045 ± 6% -16.3% 96249 ± 4% interrupts.CPU85.RES:Rescheduling_interrupts
8668 ± 3% +12.6% 9762 interrupts.CPU86.CAL:Function_call_interrupts
8579 ± 3% +14.3% 9806 ± 2% interrupts.CPU87.CAL:Function_call_interrupts
8719 ± 2% +12.4% 9797 ± 2% interrupts.CPU88.CAL:Function_call_interrupts
8769 +11.6% 9788 ± 2% interrupts.CPU89.CAL:Function_call_interrupts
131248 ± 9% -27.5% 95130 ± 9% interrupts.CPU9.RES:Rescheduling_interrupts
8766 +11.6% 9780 ± 2% interrupts.CPU90.CAL:Function_call_interrupts
121573 ± 6% -14.4% 104044 ± 8% interrupts.CPU90.RES:Rescheduling_interrupts
8762 +11.5% 9768 ± 2% interrupts.CPU91.CAL:Function_call_interrupts
125741 ± 6% -21.2% 99116 ± 10% interrupts.CPU91.RES:Rescheduling_interrupts
8764 +11.3% 9754 ± 2% interrupts.CPU92.CAL:Function_call_interrupts
125340 ± 7% -21.1% 98921 ± 6% interrupts.CPU92.RES:Rescheduling_interrupts
8620 ± 4% +13.0% 9743 ± 2% interrupts.CPU93.CAL:Function_call_interrupts
129358 ± 10% -25.0% 97082 ± 6% interrupts.CPU93.RES:Rescheduling_interrupts
8753 +11.2% 9730 ± 2% interrupts.CPU94.CAL:Function_call_interrupts
8750 +11.1% 9719 ± 2% interrupts.CPU95.CAL:Function_call_interrupts
115441 ± 6% -15.6% 97484 ± 8% interrupts.CPU95.RES:Rescheduling_interrupts
121213 ± 5% -17.0% 100662 ± 12% interrupts.CPU99.RES:Rescheduling_interrupts
12650754 ± 3% -14.8% 10783909 interrupts.RES:Rescheduling_interrupts
176.50 ± 59% -79.7% 35.75 ± 47% interrupts.TLB:TLB_shootdowns
10.98 -11.0 0.00 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
10.70 -10.7 0.00 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
10.29 -10.3 0.00 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv
10.20 -10.2 0.00 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established
9.82 -9.8 0.00 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack
9.68 -9.7 0.00 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue
21.27 -6.8 14.52 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.18 -6.7 14.46 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.54 -6.5 14.04 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
20.43 -6.5 13.96 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
16.41 -3.1 13.34 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
16.22 -3.0 13.21 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto
7.69 -2.2 5.51 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
7.67 -2.2 5.50 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
6.75 ± 6% -2.2 4.60 ± 2% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
13.64 -2.1 11.53 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
13.43 -2.0 11.40 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
7.00 -2.0 5.04 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg
6.83 -1.9 4.93 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg
6.70 -1.9 4.83 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
6.39 ± 3% -1.9 4.53 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
5.43 -1.7 3.69 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
5.31 -1.7 3.59 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
5.68 ± 3% -1.6 4.11 ± 2% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu
4.76 -1.6 3.19 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
4.84 -1.5 3.30 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom
4.56 -1.5 3.04 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg
5.51 ± 3% -1.5 4.01 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
4.48 -1.5 2.99 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg
4.39 -1.5 2.92 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data
5.36 ± 3% -1.5 3.90 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv
5.19 ± 3% -1.4 3.78 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established
4.37 -1.4 2.96 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg
2.14 -1.4 0.73 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
10.04 -1.4 8.65 perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
2.08 -1.4 0.70 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock
9.73 -1.3 8.42 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
9.56 -1.3 8.29 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg
3.73 -1.2 2.55 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg
3.78 ± 2% -1.1 2.68 perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
1.86 ± 30% -1.0 0.84 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable
3.46 ± 2% -1.0 2.46 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.51 ± 6% -0.9 0.61 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
3.22 ± 2% -0.9 2.33 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg
3.02 ± 2% -0.8 2.21 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked
1.78 -0.6 1.16 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken
7.18 ± 2% -0.6 6.58 ± 2% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
7.16 ± 2% -0.6 6.56 ± 2% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.37 -0.5 0.88 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.33 -0.5 0.84 perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.29 -0.5 0.82 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.71 -0.5 0.25 ±100% perf-profile.calltrace.cycles-pp.__sched_text_start.preempt_schedule_common._cond_resched.__release_sock.release_sock
1.20 -0.4 0.82 perf-profile.calltrace.cycles-pp.__switch_to
1.09 -0.4 0.72 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
1.09 -0.3 0.77 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64
6.42 ± 2% -0.3 6.09 ± 2% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
0.83 -0.3 0.56 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg
0.84 -0.3 0.58 ± 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout.wait_woken
0.94 -0.3 0.68 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
0.74 -0.2 0.53 perf-profile.calltrace.cycles-pp._cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg
0.73 -0.2 0.52 perf-profile.calltrace.cycles-pp.preempt_schedule_common._cond_resched.__release_sock.release_sock.tcp_sendmsg
9.51 +0.7 10.17 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb
9.38 +0.7 10.08 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data
95.32 +1.4 96.69 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
95.16 +1.4 96.58 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.17 +7.3 18.47 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
11.81 ± 2% +7.4 19.18 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb
11.74 ± 2% +7.4 19.15 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit
11.52 ± 2% +7.5 18.98 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2
11.55 ± 2% +7.5 19.01 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output
11.27 ± 2% +7.5 18.81 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip
10.95 ± 2% +7.6 18.55 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq
9.28 +7.6 16.91 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim
9.39 +7.7 17.05 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb
10.83 ± 2% +7.7 18.52 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
9.57 +7.7 17.27 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked
9.94 +7.7 17.64 perf-profile.calltrace.cycles-pp.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
10.49 ± 2% +7.7 18.23 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack
9.73 +7.8 17.49 perf-profile.calltrace.cycles-pp.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg
10.16 ± 2% +7.8 17.98 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start
3.79 +8.0 11.77 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock
3.78 +8.0 11.77 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock
9.61 ± 2% +8.0 17.61 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action
9.45 ± 2% +8.1 17.50 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog
9.40 ± 2% +8.1 17.47 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core
9.21 ± 2% +8.1 17.35 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv
3.44 +8.1 11.58 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv
8.65 ± 2% +8.2 16.88 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
3.21 +8.2 11.44 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established
18.55 ± 2% +8.6 27.14 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill
20.09 ± 2% +8.6 28.68 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto
19.87 ± 2% +8.6 28.45 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked
20.08 ± 2% +8.6 28.67 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg
18.74 ± 2% +8.6 27.34 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill
19.94 ± 2% +8.6 28.53 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg
72.24 +8.7 80.98 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
72.13 +8.8 80.91 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.52 +9.0 80.48 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.18 +9.1 80.25 perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
0.00 +10.3 10.29 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv
0.00 +10.4 10.41 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu
0.00 +10.7 10.66 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
0.00 +10.8 10.81 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver
54.39 +12.2 66.64 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto
31487988 ± 3% -23.2% 24183132 softirqs.CPU0.NET_RX
32151929 -26.1% 23765617 ± 2% softirqs.CPU1.NET_RX
31962684 ± 3% -23.5% 24458409 softirqs.CPU10.NET_RX
31110228 ± 3% -26.2% 22964939 softirqs.CPU100.NET_RX
31626866 -27.6% 22906397 softirqs.CPU101.NET_RX
29779153 ± 3% -22.9% 22969929 softirqs.CPU102.NET_RX
31513116 -27.3% 22920216 softirqs.CPU103.NET_RX
31713243 ± 2% -24.9% 23826688 ± 2% softirqs.CPU11.NET_RX
32072474 -26.1% 23696606 ± 2% softirqs.CPU12.NET_RX
32279430 -24.8% 24260634 softirqs.CPU13.NET_RX
31697375 ± 3% -23.0% 24406003 softirqs.CPU14.NET_RX
32278026 -24.7% 24317293 softirqs.CPU15.NET_RX
31162122 ± 3% -21.7% 24396540 softirqs.CPU16.NET_RX
32209863 -24.2% 24418327 softirqs.CPU17.NET_RX
32329002 -25.6% 24054387 ± 2% softirqs.CPU18.NET_RX
32127509 -24.5% 24258223 softirqs.CPU19.NET_RX
32217120 -25.2% 24089001 softirqs.CPU2.NET_RX
31415502 ± 3% -22.5% 24349881 softirqs.CPU20.NET_RX
32186243 -25.0% 24137967 softirqs.CPU21.NET_RX
32271735 -26.9% 23586468 ± 2% softirqs.CPU22.NET_RX
31871656 ± 3% -23.8% 24292511 softirqs.CPU23.NET_RX
32321894 -25.2% 24177260 softirqs.CPU24.NET_RX
31969188 -24.7% 24057006 softirqs.CPU25.NET_RX
30938777 -27.7% 22360933 ± 2% softirqs.CPU26.NET_RX
30118051 ± 3% -25.9% 22321500 ± 2% softirqs.CPU27.NET_RX
31079253 -28.1% 22337317 ± 2% softirqs.CPU28.NET_RX
31429166 -27.6% 22746093 ± 2% softirqs.CPU29.NET_RX
30485045 ± 3% -20.5% 24221128 ± 2% softirqs.CPU3.NET_RX
30912223 ± 2% -26.8% 22622183 ± 2% softirqs.CPU30.NET_RX
31357027 -27.6% 22717601 softirqs.CPU31.NET_RX
31052178 -27.3% 22565776 ± 2% softirqs.CPU32.NET_RX
31101262 -28.2% 22323348 ± 2% softirqs.CPU33.NET_RX
31331660 -27.7% 22656208 softirqs.CPU34.NET_RX
30833278 ± 3% -25.4% 22998777 softirqs.CPU35.NET_RX
30738161 ± 3% -25.2% 22998898 softirqs.CPU36.NET_RX
30881460 ± 4% -26.6% 22671426 softirqs.CPU37.NET_RX
31140748 -27.4% 22621903 softirqs.CPU38.NET_RX
31218654 -27.3% 22684067 softirqs.CPU39.NET_RX
32583070 -24.8% 24513384 softirqs.CPU4.NET_RX
31462139 -28.4% 22517351 softirqs.CPU40.NET_RX
31566697 -27.9% 22768461 softirqs.CPU41.NET_RX
30975279 ± 2% -27.2% 22555119 softirqs.CPU42.NET_RX
30718280 ± 3% -26.2% 22661320 ± 2% softirqs.CPU43.NET_RX
31283332 -27.3% 22740697 softirqs.CPU44.NET_RX
30983467 -26.4% 22794814 softirqs.CPU45.NET_RX
30658730 ± 4% -28.4% 21946934 ± 2% softirqs.CPU46.NET_RX
31181856 -26.5% 22929226 softirqs.CPU47.NET_RX
31037938 ± 2% -26.9% 22691892 ± 2% softirqs.CPU48.NET_RX
30911500 ± 2% -26.3% 22777693 softirqs.CPU49.NET_RX
31802923 ± 3% -23.9% 24204592 softirqs.CPU5.NET_RX
31517206 -28.1% 22648510 softirqs.CPU50.NET_RX
30660453 ± 3% -27.2% 22327925 ± 2% softirqs.CPU51.NET_RX
31819962 ± 2% -23.9% 24206194 softirqs.CPU52.NET_RX
32142649 -26.4% 23668373 ± 2% softirqs.CPU53.NET_RX
32149559 -24.2% 24360340 softirqs.CPU54.NET_RX
32425853 -26.4% 23854082 ± 2% softirqs.CPU55.NET_RX
31817157 ± 2% -24.3% 24096785 softirqs.CPU56.NET_RX
31640490 ± 3% -24.4% 23920278 ± 2% softirqs.CPU57.NET_RX
31650729 ± 3% -23.1% 24327421 softirqs.CPU58.NET_RX
31269148 ± 3% -22.3% 24307789 softirqs.CPU59.NET_RX
31030416 ± 4% -22.0% 24203694 softirqs.CPU6.NET_RX
32255424 -25.2% 24141973 ± 2% softirqs.CPU60.NET_RX
32245715 -25.1% 24167151 ± 2% softirqs.CPU61.NET_RX
31763163 ± 2% -23.1% 24415760 softirqs.CPU62.NET_RX
31632095 ± 3% -22.7% 24451397 softirqs.CPU63.NET_RX
31490930 ± 2% -24.2% 23879322 softirqs.CPU64.NET_RX
31516754 ± 3% -23.5% 24108737 softirqs.CPU65.NET_RX
32359236 -24.5% 24440756 softirqs.CPU66.NET_RX
31784935 ± 3% -23.9% 24195033 ± 2% softirqs.CPU67.NET_RX
30967426 ± 4% -21.8% 24208434 softirqs.CPU68.NET_RX
32208978 -27.9% 23223363 softirqs.CPU69.NET_RX
31534273 ± 3% -23.6% 24083955 softirqs.CPU7.NET_RX
31800791 ± 2% -23.3% 24375677 softirqs.CPU70.NET_RX
31511220 ± 3% -23.1% 24225232 softirqs.CPU71.NET_RX
32055308 -25.2% 23985501 ± 2% softirqs.CPU72.NET_RX
32255683 -24.3% 24425926 softirqs.CPU73.NET_RX
31808226 ± 2% -23.7% 24280351 ± 2% softirqs.CPU74.NET_RX
31994746 ± 3% -25.5% 23828432 ± 2% softirqs.CPU75.NET_RX
31732779 ± 3% -24.0% 24121773 ± 2% softirqs.CPU76.NET_RX
31249558 ± 2% -23.8% 23816726 ± 2% softirqs.CPU77.NET_RX
29102709 ± 3% -24.1% 22087804 ± 2% softirqs.CPU78.NET_RX
31050911 -26.3% 22879670 softirqs.CPU79.NET_RX
32385972 -25.5% 24114907 softirqs.CPU8.NET_RX
31250746 -26.8% 22865137 softirqs.CPU80.NET_RX
30970512 ± 3% -27.5% 22460428 ± 2% softirqs.CPU81.NET_RX
30769035 ± 3% -26.4% 22640114 softirqs.CPU82.NET_RX
31163906 -28.0% 22426881 ± 2% softirqs.CPU83.NET_RX
30833151 ± 2% -26.9% 22535706 softirqs.CPU84.NET_RX
31213448 -27.8% 22530613 ± 2% softirqs.CPU85.NET_RX
31299014 -27.8% 22598945 ± 2% softirqs.CPU86.NET_RX
31380574 -26.8% 22966556 softirqs.CPU87.NET_RX
31419156 -26.9% 22966072 softirqs.CPU88.NET_RX
31485946 -29.0% 22360125 ± 2% softirqs.CPU89.NET_RX
32266157 -25.9% 23914440 ± 2% softirqs.CPU9.NET_RX
30336933 ± 3% -25.9% 22489963 softirqs.CPU90.NET_RX
30595348 ± 3% -26.1% 22614928 softirqs.CPU91.NET_RX
30924249 ± 2% -26.0% 22894665 softirqs.CPU92.NET_RX
30593902 ± 2% -24.8% 23001426 softirqs.CPU93.NET_RX
31556110 -29.2% 22330354 ± 2% softirqs.CPU94.NET_RX
30668287 ± 3% -29.0% 21759470 softirqs.CPU95.NET_RX
31245897 -26.9% 22847267 softirqs.CPU96.NET_RX
31136421 -28.7% 22213048 ± 2% softirqs.CPU97.NET_RX
30013471 ± 4% -23.9% 22844117 softirqs.CPU98.NET_RX
31291344 -28.5% 22378868 ± 2% softirqs.CPU99.NET_RX
3.269e+09 -25.6% 2.431e+09 softirqs.NET_RX
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
1 year, 8 months
CKI hackfest @Plumbers invite
by Veronika Kabatova
Hi,
as some of you have heard, CKI Project is planning hackfest CI meetings after
Plumbers conference this year (Sept. 12-13). We would like to invite everyone
who has interest in CI for kernel to come and join us.
The early agenda with summary is at the end of the email. If you think there's
something important missing let us know! Also let us know in case you'd want to
lead any of the sessions, we'd be happy to delegate out some work :)
Please send us an email as soon as you decide to come and feel free to invite
other people who should be present. We are not planning to cap the attendance
right now but need to solve the logistics based on the interest. The event is
free to attend, no additional registration except letting us know is needed.
Feel free to contact us if you have any questions,
Veronika
CKI Project
-----------------------------------------------------------
Here is an early agenda we put together:
- Introductions
- Common place for upstream results, result publishing in general
- The discussion on the mailing list is going strong so we might be able to
substitute this session for a different one in case everything is solved by
September.
- Test result interpretation and bug detection
- How to autodetect infrastructure failures, regressions/new bugs and test
bugs? How to handle continuous failures due to known bugs in both tests and
kernel? What's your solution? Can people always trust the results they
receive?
- Getting results to developers/maintainers
- Aimed at kernel developers and maintainers, share your feedback and
expectations.
- How much data should be sent in the initial communication vs. a click away
in a dashboard? Do you want incremental emails with new results as they come
in?
- What about adding checks to tested patches in Patchwork when patch series
are being tested?
- Providing enough data/script to reproduce the failure. What if special HW
is needed?
- Onboarding new kernel trees to test
- Aimed at kernel developers and maintainers.
- Which trees are most prone to bring in new problems? Which are the most
critical ones? Do you want them to be tested? Which tests do you feel are
most beneficial for specific trees or in general?
- Security when testing untrusted patches
- How do we merge, compile, and test patches that have untrusted code in them
and have not yet been reviewed? How do we avoid abuse of systems,
information theft, or other damage?
- Check out the original patch that sparked the discussion at
https://patchwork.ozlabs.org/patch/862123/
- Avoiding effort duplication
- Food for thought by GregKH
- X different CI systems running ${TEST} on latest stable kernel on x86_64
might look useless on the first look but is it? AMD/Intel CPUs, different
network cards, different graphic drivers, compilers, kernel configuration...
How do we distribute the workload to avoid doing the same thing all over
again while still running in enough different environments to get the most
coverage?
- Common hardware pools
- Is this something people are interested in? Would be helpful especially for
HW that's hard to access, eg. ppc64le or s390x systems. Companies could also
sing up to share their HW for testing to ensure kernel works with their
products.
1 year, 8 months
Re: [LKP] [rcu] kernel BUG at include/linux/pagemap.h:149!
by Frederic Weisbecker
On Fri, Sep 11, 2015 at 10:19:47AM +0800, Boqun Feng wrote:
> Subject: [PATCH 01/27] rcu: Don't disable preemption for Tiny and Tree RCU
> readers
>
> Because preempt_disable() maps to barrier() for non-debug builds,
> it forces the compiler to spill and reload registers. Because Tree
> RCU and Tiny RCU now only appear in CONFIG_PREEMPT=n builds, these
> barrier() instances generate needless extra code for each instance of
> rcu_read_lock() and rcu_read_unlock(). This extra code slows down Tree
> RCU and bloats Tiny RCU.
>
> This commit therefore removes the preempt_disable() and preempt_enable()
> from the non-preemptible implementations of __rcu_read_lock() and
> __rcu_read_unlock(), respectively.
>
> For debug purposes, preempt_disable() and preempt_enable() are still
> kept if CONFIG_PREEMPT_COUNT=y, which makes the detection of sleeping
> inside atomic sections still work in non-preemptible kernels.
>
> Signed-off-by: Boqun Feng <boqun.feng(a)gmail.com>
> Signed-off-by: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com>
> ---
> include/linux/rcupdate.h | 6 ++++--
> include/linux/rcutiny.h | 1 +
> kernel/rcu/tree.c | 9 +++++++++
> 3 files changed, 14 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index d63bb77..6c3cece 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -297,12 +297,14 @@ void synchronize_rcu(void);
>
> static inline void __rcu_read_lock(void)
> {
> - preempt_disable();
> + if (IS_ENABLED(CONFIG_PREEMPT_COUNT))
> + preempt_disable();
preempt_disable() is a no-op when !CONFIG_PREEMPT_COUNT, right?
Or rather it's a barrier(), which is anyway implied by rcu_read_lock().
So perhaps we can get rid of the IS_ENABLED() check?
1 year, 8 months
[sched/fair] cca2ae2c00: aim7.jobs-per-min -35.9% regression
by kernel test robot
Greeting,
FYI, we noticed a -35.9% regression of aim7.jobs-per-min due to commit:
commit: cca2ae2c00b5fc7893f1a184d822812ee3425b7e ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()")
git://linux-arm.org/linux-de.git upstream/rem_rq_load
in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:
disk: 1BRD_48G
fs: xfs
test: creat-clo
load: 1500
cpufreq_governor: performance
test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.6/1500/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/creat-clo/aim7
commit:
fe146f762b ("sched: Remove rq->cpu_load[] update code")
cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()")
fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x
%stddev %change %stddev
\ | \
68031 -35.9% 43605 ± 5% aim7.jobs-per-min
132.55 +56.4% 207.34 ± 5% aim7.time.elapsed_time
132.55 +56.4% 207.34 ± 5% aim7.time.elapsed_time.max
173584 +89.5% 328925 ± 3% aim7.time.involuntary_context_switches
92120 ± 2% +10.0% 101340 ± 4% aim7.time.minor_page_faults
4210 +36.0% 5725 ± 7% aim7.time.system_time
57.74 +101.6% 116.39 ± 5% aim7.time.user_time
2692874 ± 3% -42.8% 1540853 ± 7% aim7.time.voluntary_context_switches
22.16 +38.0% 30.58 ± 5% iostat.cpu.idle
76.74 -11.4% 67.98 ± 2% iostat.cpu.system
1.10 ± 2% +30.8% 1.44 ± 2% iostat.cpu.user
21.03 +8.9 29.93 ± 5% mpstat.cpu.all.idle%
77.85 -9.2 68.61 ± 2% mpstat.cpu.all.sys%
1.11 ± 2% +0.3 1.45 ± 2% mpstat.cpu.all.usr%
21.75 +37.9% 30.00 ± 5% vmstat.cpu.id
76.00 -11.2% 67.50 ± 2% vmstat.cpu.sy
41880 ± 3% -59.0% 17180 vmstat.system.cs
8.722e+08 ± 13% +131.5% 2.019e+09 ± 8% cpuidle.C6.time
1282389 ± 8% +136.9% 3038476 ± 17% cpuidle.C6.usage
1315243 ± 4% -93.1% 91226 ± 19% cpuidle.POLL.time
1772414 ± 4% -98.6% 24472 ± 17% cpuidle.POLL.usage
123678 +24.8% 154380 ± 2% meminfo.AnonHugePages
242624 ± 9% -18.6% 197568 ± 5% meminfo.DirectMap4k
40068 ± 8% -24.5% 30237 ± 2% meminfo.Shmem
16032 ± 2% -36.2% 10229 ± 3% meminfo.max_used_kB
1191648 ± 2% -21.3% 937552 numa-numastat.node0.local_node
1197443 ± 2% -21.5% 939550 numa-numastat.node0.numa_hit
1169975 ± 2% -26.9% 855025 ± 11% numa-numastat.node1.local_node
1172017 ± 2% -26.5% 860874 ± 11% numa-numastat.node1.numa_hit
933658 ± 6% -10.2% 838153 ± 3% numa-vmstat.node0.numa_hit
927596 ± 6% -24.4% 700855 ± 4% numa-vmstat.node0.numa_local
6061 ± 54% +2165.1% 137297 ± 2% numa-vmstat.node0.numa_other
906672 ± 5% -14.9% 771132 ± 7% numa-vmstat.node1.numa_hit
167353 -78.6% 35802 ± 7% numa-vmstat.node1.numa_other
1914 ± 4% +10.3% 2111 ± 3% slabinfo.eventpoll_pwq.active_objs
1914 ± 4% +10.3% 2111 ± 3% slabinfo.eventpoll_pwq.num_objs
68433 ± 4% -37.8% 42579 ± 3% slabinfo.filp.active_objs
2291 ± 4% -32.8% 1539 slabinfo.filp.active_slabs
73346 ± 4% -32.8% 49281 slabinfo.filp.num_objs
2291 ± 4% -32.8% 1539 slabinfo.filp.num_slabs
2867 ± 3% +9.2% 3130 ± 2% slabinfo.kmalloc-128.active_objs
2867 ± 3% +9.2% 3130 ± 2% slabinfo.kmalloc-128.num_objs
922.00 ± 9% +17.6% 1084 ± 4% slabinfo.task_group.active_objs
922.00 ± 9% +17.6% 1084 ± 4% slabinfo.task_group.num_objs
2334 -56.6% 1012 ± 3% turbostat.Avg_MHz
79.29 -8.5 70.79 ± 2% turbostat.Busy%
2943 -51.4% 1430 ± 3% turbostat.Bzy_MHz
1279422 ± 8% +137.3% 3035612 ± 17% turbostat.C6
15.91 ± 12% +8.0 23.87 ± 7% turbostat.C6%
12.74 ± 8% +46.4% 18.66 ± 9% turbostat.CPU%c1
127.15 -60.3% 50.54 ± 3% turbostat.CorWatt
68.75 -12.4% 60.25 turbostat.CoreTmp
11323915 +51.8% 17189663 ± 5% turbostat.IRQ
1.89 ± 34% -64.0% 0.68 ± 87% turbostat.Pkg%pc6
68.75 -12.4% 60.25 turbostat.PkgTmp
154.35 -50.2% 76.80 ± 2% turbostat.PkgWatt
36.98 -9.2% 33.59 turbostat.RAMWatt
10930 +54.8% 16920 ± 5% turbostat.SMI
79583 -2.1% 77935 proc-vmstat.nr_active_anon
275358 -1.3% 271685 proc-vmstat.nr_file_pages
5012 -2.0% 4914 proc-vmstat.nr_inactive_anon
30851 +1.5% 31301 proc-vmstat.nr_kernel_stack
6172 -2.0% 6046 proc-vmstat.nr_mapped
10015 ± 8% -24.5% 7558 ± 2% proc-vmstat.nr_shmem
32115 -4.7% 30603 proc-vmstat.nr_slab_unreclaimable
79583 -2.1% 77935 proc-vmstat.nr_zone_active_anon
5012 -2.0% 4914 proc-vmstat.nr_zone_inactive_anon
16174 +58.8% 25684 ± 13% proc-vmstat.numa_hint_faults
6212 ± 4% +117.6% 13517 ± 15% proc-vmstat.numa_hint_faults_local
2394858 -23.7% 1826339 ± 4% proc-vmstat.numa_hit
2387014 -23.8% 1818486 ± 4% proc-vmstat.numa_local
9960 +31.0% 13047 ± 10% proc-vmstat.numa_pages_migrated
17309 +188.4% 49918 ± 27% proc-vmstat.numa_pte_updates
5225 ± 16% -56.6% 2269 ± 9% proc-vmstat.pgactivate
4457838 -28.2% 3200155 ± 5% proc-vmstat.pgalloc_normal
413171 +41.1% 583119 ± 4% proc-vmstat.pgfault
4428619 -28.3% 3174333 ± 5% proc-vmstat.pgfree
9960 +31.0% 13047 ± 10% proc-vmstat.pgmigrate_success
4427 ± 4% +32.5% 5865 ± 3% proc-vmstat.pgpgout
932.00 ± 3% +3.4% 963.25 ± 4% proc-vmstat.unevictable_pgs_culled
23687 ± 91% +95.1% 46206 ± 72% sched_debug.cfs_rq:/.load.stddev
1212 ± 68% -70.6% 356.25 ± 11% sched_debug.cfs_rq:/.load_avg.max
13.08 ± 11% -49.8% 6.56 ± 30% sched_debug.cfs_rq:/.load_avg.min
0.35 ± 5% +19.6% 0.42 ± 4% sched_debug.cfs_rq:/.nr_running.stddev
15.34 +25.5% 19.25 ± 17% sched_debug.cfs_rq:/.runnable_load_avg.avg
8.52 ± 6% +290.6% 33.28 ± 67% sched_debug.cfs_rq:/.runnable_load_avg.stddev
22196 ± 96% +104.1% 45294 ± 75% sched_debug.cfs_rq:/.runnable_weight.stddev
23081 ± 26% -212.8% -26032 sched_debug.cfs_rq:/.spread0.avg
102531 ± 17% -46.3% 55090 ± 36% sched_debug.cfs_rq:/.spread0.max
-7190 +952.0% -75647 sched_debug.cfs_rq:/.spread0.min
439.42 ± 14% -23.7% 335.08 ± 15% sched_debug.cfs_rq:/.util_avg.min
173.42 ± 11% +32.0% 228.90 ± 6% sched_debug.cfs_rq:/.util_avg.stddev
284592 ± 6% +90.7% 542802 ± 4% sched_debug.cpu.avg_idle.avg
6069 ± 33% +1188.9% 78224 ± 32% sched_debug.cpu.avg_idle.min
228220 ± 5% +15.8% 264189 ± 3% sched_debug.cpu.avg_idle.stddev
3.07 ± 11% +97.8% 6.08 ± 10% sched_debug.cpu.clock.stddev
3.07 ± 11% +97.8% 6.08 ± 10% sched_debug.cpu.clock_task.stddev
4032 +17.3% 4729 ± 7% sched_debug.cpu.curr->pid.max
851.30 ± 2% +25.1% 1065 ± 3% sched_debug.cpu.curr->pid.stddev
0.00 ± 2% +9.0% 0.00 ± 2% sched_debug.cpu.next_balance.stddev
0.40 ± 4% +21.8% 0.49 ± 7% sched_debug.cpu.nr_running.stddev
61876 ± 2% -39.5% 37429 ± 15% sched_debug.cpu.nr_switches.avg
72542 ± 3% -32.2% 49204 ± 10% sched_debug.cpu.nr_switches.max
55154 ± 3% -39.5% 33393 ± 16% sched_debug.cpu.nr_switches.min
4093 ± 18% -21.0% 3234 ± 14% sched_debug.cpu.nr_switches.stddev
23.68 +13.1% 26.78 ± 4% sched_debug.cpu.nr_uninterruptible.avg
170.25 ± 11% +136.5% 402.67 ± 20% sched_debug.cpu.nr_uninterruptible.max
46.30 ± 10% +97.2% 91.30 ± 20% sched_debug.cpu.nr_uninterruptible.stddev
4118075 -50.0% 2059067 sched_debug.sysctl_sched.sysctl_sched_features
9.08e+09 -57.2% 3.887e+09 ± 2% perf-stat.i.branch-instructions
39772561 -30.2% 27751148 ± 4% perf-stat.i.branch-misses
30.87 -4.6 26.27 ± 2% perf-stat.i.cache-miss-rate%
34133998 -42.8% 19525491 ± 4% perf-stat.i.cache-misses
1.1e+08 ± 2% -31.0% 75953637 ± 3% perf-stat.i.cache-references
42802 ± 3% -59.5% 17355 perf-stat.i.context-switches
9.388e+10 -57.1% 4.024e+10 ± 3% perf-stat.i.cpu-cycles
1480 ± 2% +161.3% 3868 ± 3% perf-stat.i.cpu-migrations
2576 ± 2% -19.5% 2073 ± 4% perf-stat.i.cycles-between-cache-misses
1.461e+10 -57.9% 6.147e+09 ± 2% perf-stat.i.dTLB-loads
2.598e+09 -34.4% 1.704e+09 ± 5% perf-stat.i.dTLB-stores
3340458 ± 2% -27.2% 2431112 ± 4% perf-stat.i.iTLB-load-misses
4.759e+10 -57.5% 2.021e+10 ± 2% perf-stat.i.instructions
13273 ± 3% -39.0% 8098 ± 4% perf-stat.i.instructions-per-iTLB-miss
2917 -8.8% 2660 perf-stat.i.minor-faults
47.27 +1.2 48.42 perf-stat.i.node-load-miss-rate%
17915543 -42.3% 10339302 ± 5% perf-stat.i.node-load-misses
19859340 -45.4% 10837882 ± 5% perf-stat.i.node-loads
40.22 +1.5 41.74 perf-stat.i.node-store-miss-rate%
9587423 -42.5% 5512775 ± 5% perf-stat.i.node-store-misses
13693791 -45.2% 7500888 ± 5% perf-stat.i.node-stores
2917 -8.8% 2660 perf-stat.i.page-faults
2.31 ± 3% +62.6% 3.76 ± 3% perf-stat.overall.MPKI
0.44 ± 2% +0.3 0.71 ± 4% perf-stat.overall.branch-miss-rate%
31.03 -5.3 25.73 ± 4% perf-stat.overall.cache-miss-rate%
2751 ± 2% -25.1% 2062 perf-stat.overall.cycles-between-cache-misses
14261 ± 4% -41.6% 8325 ± 4% perf-stat.overall.instructions-per-iTLB-miss
47.43 +1.4 48.82 perf-stat.overall.node-load-miss-rate%
41.18 +1.2 42.36 perf-stat.overall.node-store-miss-rate%
9.011e+09 -57.1% 3.868e+09 ± 2% perf-stat.ps.branch-instructions
39471063 -30.0% 27616773 ± 4% perf-stat.ps.branch-misses
33874716 -42.6% 19430430 ± 4% perf-stat.ps.cache-misses
1.092e+08 ± 2% -30.8% 75583753 ± 3% perf-stat.ps.cache-references
42475 ± 3% -59.3% 17270 perf-stat.ps.context-switches
9.317e+10 -57.0% 4.004e+10 ± 3% perf-stat.ps.cpu-cycles
1469 ± 2% +162.0% 3849 ± 3% perf-stat.ps.cpu-migrations
1.45e+10 -57.8% 6.117e+09 ± 2% perf-stat.ps.dTLB-loads
2.578e+09 -34.2% 1.696e+09 ± 5% perf-stat.ps.dTLB-stores
3315089 ± 2% -27.0% 2419259 ± 4% perf-stat.ps.iTLB-load-misses
4.723e+10 -57.4% 2.011e+10 ± 2% perf-stat.ps.instructions
2898 -8.6% 2649 perf-stat.ps.minor-faults
17778569 -42.1% 10288528 ± 5% perf-stat.ps.node-load-misses
19707496 -45.3% 10784655 ± 5% perf-stat.ps.node-loads
9514153 -42.3% 5485718 ± 5% perf-stat.ps.node-store-misses
13589365 -45.1% 7464266 ± 5% perf-stat.ps.node-stores
2898 -8.6% 2649 perf-stat.ps.page-faults
6.414e+12 ± 2% -34.0% 4.231e+12 ± 4% perf-stat.total.instructions
13476 ± 3% +40.8% 18978 ± 2% softirqs.CPU0.SCHED
50649 ± 3% +55.9% 78958 ± 7% softirqs.CPU0.TIMER
10154 ± 4% +69.5% 17215 ± 4% softirqs.CPU1.SCHED
49566 ± 4% +96.4% 97347 ± 16% softirqs.CPU1.TIMER
10195 +57.5% 16054 ± 3% softirqs.CPU10.SCHED
50586 ± 2% +53.2% 77512 ± 8% softirqs.CPU10.TIMER
9932 +67.0% 16585 ± 3% softirqs.CPU11.SCHED
49178 ± 6% +72.3% 84755 ± 4% softirqs.CPU11.TIMER
9750 ± 2% +65.0% 16089 ± 4% softirqs.CPU12.SCHED
54719 ± 18% +42.4% 77900 ± 5% softirqs.CPU12.TIMER
154622 ± 4% -14.5% 132171 ± 7% softirqs.CPU13.RCU
10067 ± 2% +63.7% 16475 ± 3% softirqs.CPU13.SCHED
49427 ± 5% +67.0% 82545 ± 4% softirqs.CPU13.TIMER
10123 ± 3% +59.9% 16185 ± 3% softirqs.CPU14.SCHED
50112 ± 2% +58.2% 79263 ± 6% softirqs.CPU14.TIMER
153886 ± 3% -14.4% 131761 ± 8% softirqs.CPU15.RCU
9982 ± 2% +64.2% 16390 ± 2% softirqs.CPU15.SCHED
49303 ± 5% +69.2% 83407 ± 3% softirqs.CPU15.TIMER
10189 +60.1% 16315 ± 2% softirqs.CPU16.SCHED
50924 ± 2% +53.7% 78274 ± 7% softirqs.CPU16.TIMER
149727 ± 6% -12.1% 131573 ± 7% softirqs.CPU17.RCU
10052 ± 2% +64.1% 16493 ± 2% softirqs.CPU17.SCHED
49666 ± 4% +67.0% 82929 ± 4% softirqs.CPU17.TIMER
10146 ± 2% +61.4% 16375 ± 2% softirqs.CPU18.SCHED
57432 ± 20% +36.9% 78602 ± 7% softirqs.CPU18.TIMER
149562 ± 5% -11.6% 132252 ± 7% softirqs.CPU19.RCU
10032 ± 2% +63.9% 16446 ± 3% softirqs.CPU19.SCHED
49273 ± 4% +67.2% 82393 ± 4% softirqs.CPU19.TIMER
10206 ± 2% +57.5% 16075 ± 4% softirqs.CPU2.SCHED
51506 +51.9% 78234 ± 7% softirqs.CPU2.TIMER
9769 ± 2% +64.9% 16110 ± 3% softirqs.CPU20.SCHED
49181 ± 4% +59.6% 78481 ± 6% softirqs.CPU20.TIMER
152472 ± 4% -13.5% 131909 ± 6% softirqs.CPU21.RCU
11128 ± 9% +45.9% 16237 ± 2% softirqs.CPU21.SCHED
52047 ± 6% +76.1% 91663 ± 9% softirqs.CPU21.TIMER
9992 +61.5% 16142 ± 4% softirqs.CPU22.SCHED
49187 ± 3% +60.2% 78791 ± 8% softirqs.CPU22.TIMER
146898 ± 4% -11.3% 130257 ± 7% softirqs.CPU23.RCU
10528 ± 6% +56.5% 16472 ± 2% softirqs.CPU23.SCHED
50357 ± 3% +68.0% 84618 ± 6% softirqs.CPU23.TIMER
10130 +60.0% 16211 ± 3% softirqs.CPU24.SCHED
55993 ± 16% +40.1% 78467 ± 8% softirqs.CPU24.TIMER
151281 ± 5% -13.8% 130379 ± 6% softirqs.CPU25.RCU
10544 ± 8% +56.3% 16483 ± 2% softirqs.CPU25.SCHED
56974 ± 12% +46.4% 83428 ± 5% softirqs.CPU25.TIMER
9988 +62.1% 16192 ± 4% softirqs.CPU26.SCHED
49650 +82.2% 90457 ± 13% softirqs.CPU26.TIMER
149485 -11.9% 131759 ± 6% softirqs.CPU27.RCU
9873 +67.1% 16493 ± 3% softirqs.CPU27.SCHED
48333 ± 4% +71.3% 82778 ± 4% softirqs.CPU27.TIMER
9977 +60.9% 16053 ± 3% softirqs.CPU28.SCHED
49167 ± 3% +63.7% 80466 ± 5% softirqs.CPU28.TIMER
147449 ± 6% -11.7% 130131 ± 6% softirqs.CPU29.RCU
9834 +66.0% 16325 softirqs.CPU29.SCHED
48376 ± 4% +68.1% 81326 ± 5% softirqs.CPU29.TIMER
154504 ± 6% -12.7% 134808 ± 7% softirqs.CPU3.RCU
9968 +88.7% 18814 ± 4% softirqs.CPU3.SCHED
49472 ± 3% +79.9% 88977 ± 8% softirqs.CPU3.TIMER
9917 ± 2% +62.1% 16074 ± 5% softirqs.CPU30.SCHED
49682 ± 3% +54.7% 76867 ± 8% softirqs.CPU30.TIMER
152680 ± 3% -14.9% 129949 ± 7% softirqs.CPU31.RCU
9983 +64.8% 16453 ± 2% softirqs.CPU31.SCHED
48960 ± 6% +70.4% 83422 ± 5% softirqs.CPU31.TIMER
10167 +57.0% 15962 ± 3% softirqs.CPU32.SCHED
51076 ± 2% +51.2% 77243 ± 6% softirqs.CPU32.TIMER
151683 ± 5% -15.6% 128070 ± 6% softirqs.CPU33.RCU
10089 ± 3% +62.3% 16379 ± 2% softirqs.CPU33.SCHED
49549 ± 5% +66.7% 82575 ± 5% softirqs.CPU33.TIMER
10213 +58.4% 16174 ± 3% softirqs.CPU34.SCHED
52055 ± 5% +50.9% 78542 ± 6% softirqs.CPU34.TIMER
141065 ± 5% -10.0% 126977 ± 6% softirqs.CPU35.RCU
10022 +64.8% 16519 ± 3% softirqs.CPU35.SCHED
53261 ± 14% +57.0% 83625 ± 3% softirqs.CPU35.TIMER
10083 +61.2% 16257 ± 3% softirqs.CPU36.SCHED
51476 +51.1% 77803 ± 6% softirqs.CPU36.TIMER
148739 ± 2% -14.3% 127470 ± 7% softirqs.CPU37.RCU
9978 ± 2% +64.5% 16411 ± 2% softirqs.CPU37.SCHED
49715 ± 3% +66.3% 82678 ± 4% softirqs.CPU37.TIMER
10154 ± 2% +58.7% 16110 ± 2% softirqs.CPU38.SCHED
53442 ± 6% +59.1% 85001 ± 18% softirqs.CPU38.TIMER
150024 ± 4% -13.1% 130420 ± 8% softirqs.CPU39.RCU
10042 +59.0% 15967 softirqs.CPU39.SCHED
49248 ± 5% +66.3% 81906 ± 4% softirqs.CPU39.TIMER
10462 ± 5% +54.3% 16142 ± 3% softirqs.CPU4.SCHED
50777 ± 2% +53.9% 78169 ± 7% softirqs.CPU4.TIMER
150834 ± 2% -10.3% 135341 ± 7% softirqs.CPU5.RCU
10002 +67.0% 16704 ± 5% softirqs.CPU5.SCHED
49621 +69.4% 84052 ± 5% softirqs.CPU5.TIMER
10015 +61.4% 16164 ± 5% softirqs.CPU6.SCHED
49679 +63.2% 81070 ± 5% softirqs.CPU6.TIMER
150360 ± 5% -11.3% 133386 ± 6% softirqs.CPU7.RCU
9951 +65.1% 16432 ± 2% softirqs.CPU7.SCHED
53815 ± 20% +52.8% 82214 ± 5% softirqs.CPU7.TIMER
9958 +62.2% 16148 ± 3% softirqs.CPU8.SCHED
48872 ± 2% +61.7% 79048 ± 6% softirqs.CPU8.TIMER
152788 ± 3% -12.8% 133236 ± 7% softirqs.CPU9.RCU
9863 ± 3% +65.9% 16367 ± 2% softirqs.CPU9.SCHED
48422 ± 5% +68.2% 81429 ± 5% softirqs.CPU9.TIMER
406958 +61.6% 657494 ± 2% softirqs.SCHED
2030752 ± 2% +61.4% 3277241 ± 5% softirqs.TIMER
94.75 ± 13% +335.1% 412.25 ± 67% interrupts.41:IR-PCI-MSI.524291-edge.eth0-TxRx-2
90.75 ± 19% +64.5% 149.25 ± 18% interrupts.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
101.00 ± 23% +72.0% 173.75 ± 32% interrupts.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
66.25 +56.2% 103.50 ± 4% interrupts.96:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
53986 +51.2% 81621 ± 8% interrupts.CAL:Function_call_interrupts
1354 ± 6% +54.4% 2091 ± 4% interrupts.CPU0.CAL:Function_call_interrupts
267296 +58.0% 422236 ± 5% interrupts.CPU0.LOC:Local_timer_interrupts
6383 ± 8% -41.4% 3739 ± 9% interrupts.CPU0.RES:Rescheduling_interrupts
1358 ± 4% +46.0% 1982 ± 11% interrupts.CPU1.CAL:Function_call_interrupts
269685 ± 2% +56.6% 422237 ± 5% interrupts.CPU1.LOC:Local_timer_interrupts
7255 ± 7% -56.2% 3179 ± 9% interrupts.CPU1.RES:Rescheduling_interrupts
66.25 +56.2% 103.50 ± 4% interrupts.CPU10.96:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
1366 ± 5% +45.2% 1984 ± 14% interrupts.CPU10.CAL:Function_call_interrupts
267276 +58.0% 422206 ± 5% interrupts.CPU10.LOC:Local_timer_interrupts
6675 ± 10% -57.6% 2827 ± 8% interrupts.CPU10.RES:Rescheduling_interrupts
1309 ± 5% +53.9% 2015 ± 11% interrupts.CPU11.CAL:Function_call_interrupts
266188 +58.7% 422319 ± 5% interrupts.CPU11.LOC:Local_timer_interrupts
6719 ± 6% -56.8% 2906 ± 13% interrupts.CPU11.RES:Rescheduling_interrupts
1351 ± 4% +45.8% 1971 ± 8% interrupts.CPU12.CAL:Function_call_interrupts
269142 ± 2% +56.8% 422008 ± 5% interrupts.CPU12.LOC:Local_timer_interrupts
6698 ± 11% -57.6% 2839 ± 7% interrupts.CPU12.RES:Rescheduling_interrupts
1359 ± 4% +54.4% 2098 ± 12% interrupts.CPU13.CAL:Function_call_interrupts
266898 +58.2% 422104 ± 5% interrupts.CPU13.LOC:Local_timer_interrupts
6765 ± 7% -58.4% 2815 ± 17% interrupts.CPU13.RES:Rescheduling_interrupts
1340 ± 7% +50.0% 2010 ± 7% interrupts.CPU14.CAL:Function_call_interrupts
268735 ± 2% +57.1% 422054 ± 5% interrupts.CPU14.LOC:Local_timer_interrupts
6450 ± 7% -55.5% 2872 ± 8% interrupts.CPU14.RES:Rescheduling_interrupts
1376 ± 3% +48.0% 2037 ± 12% interrupts.CPU15.CAL:Function_call_interrupts
269196 ± 2% +56.8% 421995 ± 5% interrupts.CPU15.LOC:Local_timer_interrupts
6730 ± 5% -59.2% 2744 ± 13% interrupts.CPU15.RES:Rescheduling_interrupts
1286 ± 13% +59.2% 2046 ± 12% interrupts.CPU16.CAL:Function_call_interrupts
269168 ± 2% +56.7% 421841 ± 5% interrupts.CPU16.LOC:Local_timer_interrupts
6273 ± 10% -55.9% 2768 ± 9% interrupts.CPU16.RES:Rescheduling_interrupts
1361 ± 3% +48.3% 2019 ± 6% interrupts.CPU17.CAL:Function_call_interrupts
269146 ± 2% +56.9% 422160 ± 5% interrupts.CPU17.LOC:Local_timer_interrupts
6946 ± 3% -62.1% 2635 ± 10% interrupts.CPU17.RES:Rescheduling_interrupts
1326 ± 4% +51.3% 2006 ± 12% interrupts.CPU18.CAL:Function_call_interrupts
268729 ± 2% +57.0% 421848 ± 5% interrupts.CPU18.LOC:Local_timer_interrupts
6064 ± 12% -53.2% 2839 ± 9% interrupts.CPU18.RES:Rescheduling_interrupts
1275 +56.1% 1990 ± 4% interrupts.CPU19.CAL:Function_call_interrupts
269182 ± 2% +56.8% 422201 ± 5% interrupts.CPU19.LOC:Local_timer_interrupts
23.25 ±138% +6069.9% 1434 ±105% interrupts.CPU19.NMI:Non-maskable_interrupts
23.25 ±138% +6069.9% 1434 ±105% interrupts.CPU19.PMI:Performance_monitoring_interrupts
6504 ± 3% -57.0% 2798 ± 8% interrupts.CPU19.RES:Rescheduling_interrupts
90.75 ± 19% +64.5% 149.25 ± 18% interrupts.CPU2.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
1305 ± 9% +54.9% 2022 ± 14% interrupts.CPU2.CAL:Function_call_interrupts
266564 +58.3% 421875 ± 5% interrupts.CPU2.LOC:Local_timer_interrupts
6681 ± 17% -51.8% 3219 ± 14% interrupts.CPU2.RES:Rescheduling_interrupts
1379 ± 6% +40.5% 1937 ± 10% interrupts.CPU20.CAL:Function_call_interrupts
269083 ± 2% +56.8% 421789 ± 5% interrupts.CPU20.LOC:Local_timer_interrupts
6676 ± 12% -57.5% 2838 ± 9% interrupts.CPU20.RES:Rescheduling_interrupts
1376 ± 8% +51.3% 2083 ± 10% interrupts.CPU21.CAL:Function_call_interrupts
268308 +57.3% 421982 ± 5% interrupts.CPU21.LOC:Local_timer_interrupts
7344 ± 6% -63.1% 2711 ± 10% interrupts.CPU21.RES:Rescheduling_interrupts
1367 ± 5% +48.8% 2033 ± 5% interrupts.CPU22.CAL:Function_call_interrupts
267131 +57.9% 421846 ± 5% interrupts.CPU22.LOC:Local_timer_interrupts
6569 ± 12% -57.7% 2779 ± 7% interrupts.CPU22.RES:Rescheduling_interrupts
1347 ± 7% +51.1% 2036 ± 15% interrupts.CPU23.CAL:Function_call_interrupts
267612 +57.7% 422027 ± 5% interrupts.CPU23.LOC:Local_timer_interrupts
6903 ± 4% -59.9% 2771 ± 16% interrupts.CPU23.RES:Rescheduling_interrupts
1299 ± 7% +47.1% 1910 ± 6% interrupts.CPU24.CAL:Function_call_interrupts
266620 +58.2% 421923 ± 5% interrupts.CPU24.LOC:Local_timer_interrupts
3745 ± 66% -90.9% 340.00 ±173% interrupts.CPU24.NMI:Non-maskable_interrupts
3745 ± 66% -90.9% 340.00 ±173% interrupts.CPU24.PMI:Performance_monitoring_interrupts
6605 ± 9% -57.5% 2804 ± 5% interrupts.CPU24.RES:Rescheduling_interrupts
1392 ± 2% +50.3% 2092 ± 11% interrupts.CPU25.CAL:Function_call_interrupts
267144 +57.9% 421926 ± 5% interrupts.CPU25.LOC:Local_timer_interrupts
7062 ± 2% -60.9% 2761 ± 14% interrupts.CPU25.RES:Rescheduling_interrupts
1445 +32.0% 1907 ± 9% interrupts.CPU26.CAL:Function_call_interrupts
266423 +58.3% 421705 ± 5% interrupts.CPU26.LOC:Local_timer_interrupts
5617 ± 32% -99.8% 10.25 ±156% interrupts.CPU26.NMI:Non-maskable_interrupts
5617 ± 32% -99.8% 10.25 ±156% interrupts.CPU26.PMI:Performance_monitoring_interrupts
6698 ± 12% -58.1% 2803 ± 8% interrupts.CPU26.RES:Rescheduling_interrupts
1349 ± 7% +47.1% 1985 ± 10% interrupts.CPU27.CAL:Function_call_interrupts
267001 +58.1% 422055 ± 5% interrupts.CPU27.LOC:Local_timer_interrupts
3567 ± 9% -70.7% 1046 ±131% interrupts.CPU27.NMI:Non-maskable_interrupts
3567 ± 9% -70.7% 1046 ±131% interrupts.CPU27.PMI:Performance_monitoring_interrupts
6626 ± 4% -58.2% 2767 ± 18% interrupts.CPU27.RES:Rescheduling_interrupts
1343 ± 5% +53.3% 2058 ± 9% interrupts.CPU28.CAL:Function_call_interrupts
266397 +58.3% 421762 ± 5% interrupts.CPU28.LOC:Local_timer_interrupts
6308 ± 9% -55.8% 2789 ± 11% interrupts.CPU28.RES:Rescheduling_interrupts
1348 ± 5% +52.7% 2059 ± 9% interrupts.CPU29.CAL:Function_call_interrupts
269137 ± 2% +56.8% 422139 ± 5% interrupts.CPU29.LOC:Local_timer_interrupts
6912 ± 3% -60.0% 2764 ± 14% interrupts.CPU29.RES:Rescheduling_interrupts
1331 ± 4% +54.6% 2057 ± 5% interrupts.CPU3.CAL:Function_call_interrupts
269405 ± 2% +56.7% 422183 ± 5% interrupts.CPU3.LOC:Local_timer_interrupts
7171 ± 6% -61.2% 2785 ± 7% interrupts.CPU3.RES:Rescheduling_interrupts
1337 ± 6% +51.8% 2029 ± 5% interrupts.CPU30.CAL:Function_call_interrupts
266174 +58.6% 422198 ± 5% interrupts.CPU30.LOC:Local_timer_interrupts
6522 ± 16% -54.6% 2961 ± 12% interrupts.CPU30.RES:Rescheduling_interrupts
1397 ± 3% +51.0% 2110 ± 11% interrupts.CPU31.CAL:Function_call_interrupts
266913 +58.2% 422156 ± 5% interrupts.CPU31.LOC:Local_timer_interrupts
7094 ± 3% -60.9% 2771 ± 14% interrupts.CPU31.RES:Rescheduling_interrupts
1347 ± 7% +59.8% 2153 ± 8% interrupts.CPU32.CAL:Function_call_interrupts
266250 +58.1% 421070 ± 4% interrupts.CPU32.LOC:Local_timer_interrupts
6519 ± 14% -55.5% 2900 ± 9% interrupts.CPU32.RES:Rescheduling_interrupts
1351 ± 4% +47.0% 1985 ± 9% interrupts.CPU33.CAL:Function_call_interrupts
266078 +58.6% 421991 ± 5% interrupts.CPU33.LOC:Local_timer_interrupts
6204 ± 3% -58.2% 2590 ± 14% interrupts.CPU33.RES:Rescheduling_interrupts
94.75 ± 13% +335.1% 412.25 ± 67% interrupts.CPU34.41:IR-PCI-MSI.524291-edge.eth0-TxRx-2
1315 ± 8% +65.9% 2183 ± 8% interrupts.CPU34.CAL:Function_call_interrupts
266341 +58.3% 421683 ± 5% interrupts.CPU34.LOC:Local_timer_interrupts
6533 ± 15% -53.6% 3033 ± 14% interrupts.CPU34.RES:Rescheduling_interrupts
1373 ± 6% +55.2% 2132 ± 5% interrupts.CPU35.CAL:Function_call_interrupts
266140 +58.6% 422149 ± 5% interrupts.CPU35.LOC:Local_timer_interrupts
6663 ± 5% -58.7% 2753 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts
1381 ± 5% +49.1% 2059 ± 6% interrupts.CPU36.CAL:Function_call_interrupts
265961 +58.6% 421688 ± 5% interrupts.CPU36.LOC:Local_timer_interrupts
6267 ± 14% -54.7% 2840 ± 7% interrupts.CPU36.RES:Rescheduling_interrupts
1385 ± 4% +53.2% 2122 ± 11% interrupts.CPU37.CAL:Function_call_interrupts
266929 +58.2% 422152 ± 5% interrupts.CPU37.LOC:Local_timer_interrupts
7480 ± 34% -77.2% 1703 ± 40% interrupts.CPU37.NMI:Non-maskable_interrupts
7480 ± 34% -77.2% 1703 ± 40% interrupts.CPU37.PMI:Performance_monitoring_interrupts
7124 ± 2% -61.8% 2721 ± 17% interrupts.CPU37.RES:Rescheduling_interrupts
1405 ± 6% +40.8% 1979 ± 5% interrupts.CPU38.CAL:Function_call_interrupts
266168 +58.5% 421978 ± 5% interrupts.CPU38.LOC:Local_timer_interrupts
6570 ± 13% -56.6% 2854 ± 10% interrupts.CPU38.RES:Rescheduling_interrupts
36.50 ± 68% +575.3% 246.50 ± 65% interrupts.CPU38.TLB:TLB_shootdowns
1453 ± 2% +41.0% 2048 ± 12% interrupts.CPU39.CAL:Function_call_interrupts
266343 +58.1% 421194 ± 5% interrupts.CPU39.LOC:Local_timer_interrupts
6526 ± 24% -72.9% 1767 ±102% interrupts.CPU39.NMI:Non-maskable_interrupts
6526 ± 24% -72.9% 1767 ±102% interrupts.CPU39.PMI:Performance_monitoring_interrupts
6869 ± 3% -60.8% 2690 ± 14% interrupts.CPU39.RES:Rescheduling_interrupts
64.75 ± 46% +556.0% 424.75 ± 74% interrupts.CPU39.TLB:TLB_shootdowns
101.00 ± 23% +72.0% 173.75 ± 32% interrupts.CPU4.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
1350 ± 5% +60.3% 2165 ± 8% interrupts.CPU4.CAL:Function_call_interrupts
267121 +57.9% 421817 ± 5% interrupts.CPU4.LOC:Local_timer_interrupts
6429 ± 15% -52.2% 3072 ± 9% interrupts.CPU4.RES:Rescheduling_interrupts
1331 ± 4% +47.1% 1959 ± 15% interrupts.CPU5.CAL:Function_call_interrupts
269514 ± 2% +56.8% 422468 ± 5% interrupts.CPU5.LOC:Local_timer_interrupts
6580 ± 6% -57.6% 2790 ± 13% interrupts.CPU5.RES:Rescheduling_interrupts
1273 ± 3% +67.9% 2139 ± 9% interrupts.CPU6.CAL:Function_call_interrupts
269080 ± 2% +56.8% 421874 ± 5% interrupts.CPU6.LOC:Local_timer_interrupts
6094 ± 12% -49.5% 3079 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts
1281 ± 6% +64.3% 2106 ± 8% interrupts.CPU7.CAL:Function_call_interrupts
269338 ± 2% +56.8% 422190 ± 5% interrupts.CPU7.LOC:Local_timer_interrupts
6606 ± 5% -58.8% 2724 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts
1320 ± 4% +50.6% 1989 ± 12% interrupts.CPU8.CAL:Function_call_interrupts
269056 ± 2% +56.9% 422204 ± 5% interrupts.CPU8.LOC:Local_timer_interrupts
6206 ± 10% -54.1% 2850 ± 6% interrupts.CPU8.RES:Rescheduling_interrupts
1330 ± 6% +51.7% 2017 ± 8% interrupts.CPU9.CAL:Function_call_interrupts
266031 +58.7% 422174 ± 5% interrupts.CPU9.LOC:Local_timer_interrupts
6742 ± 7% -60.8% 2642 ± 7% interrupts.CPU9.RES:Rescheduling_interrupts
10704917 +57.7% 16879423 ± 5% interrupts.LOC:Local_timer_interrupts
149849 ± 17% -61.7% 57361 ± 11% interrupts.NMI:Non-maskable_interrupts
149849 ± 17% -61.7% 57361 ± 11% interrupts.PMI:Performance_monitoring_interrupts
266053 ± 5% -57.1% 114037 ± 10% interrupts.RES:Rescheduling_interrupts
1072 ± 9% +60.8% 1725 ± 20% interrupts.TLB:TLB_shootdowns
65.19 -11.9 53.32 ± 4% perf-profile.calltrace.cycles.osq_lock.rwsem_down_write_failed.down_write.path_openat.do_filp_open
67.76 -11.4 56.37 ± 4% perf-profile.calltrace.cycles.rwsem_down_write_failed.down_write.path_openat.do_filp_open.do_sys_open
68.21 -11.4 56.86 ± 4% perf-profile.calltrace.cycles.down_write.path_openat.do_filp_open.do_sys_open.do_syscall_64
31.80 ±120% -9.2 22.57 ±173% perf-profile.calltrace.cycles.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
33.22 ±120% -9.2 24.01 ±173% perf-profile.calltrace.cycles.do_syscall_64.entry_SYSCALL_64_after_hwframe
32.26 ±120% -9.2 23.06 ±173% perf-profile.calltrace.cycles.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
33.26 ±120% -9.2 24.06 ±173% perf-profile.calltrace.cycles.entry_SYSCALL_64_after_hwframe
93.10 -3.7 89.35 perf-profile.calltrace.cycles.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.45 ± 57% +0.2 0.69 ± 5% perf-profile.calltrace.cycles.osq_unlock.rwsem_down_write_failed.down_write.path_openat.do_filp_open
0.69 ± 15% +0.3 0.98 ± 12% perf-profile.calltrace.cycles.xlog_ungrant_log_space.xfs_log_done.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize
0.84 ± 18% +0.3 1.15 ± 3% perf-profile.calltrace.cycles.rwsem_spin_on_owner.rwsem_down_write_failed.down_write.path_openat.do_filp_open
0.43 ± 58% +0.3 0.75 ± 4% perf-profile.calltrace.cycles.complete_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.78 ± 16% +0.4 1.17 ± 6% perf-profile.calltrace.cycles.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_open
0.26 ±100% +0.4 0.66 ± 7% perf-profile.calltrace.cycles.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize
0.83 ± 16% +0.4 1.25 ± 6% perf-profile.calltrace.cycles.alloc_empty_file.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.40 ± 57% +0.4 0.82 ± 8% perf-profile.calltrace.cycles.xlog_grant_add_space.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize
0.47 ± 57% +0.4 0.89 ± 6% perf-profile.calltrace.cycles.selinux_inode_permission.security_inode_permission.link_path_walk.path_openat.do_filp_open
0.89 ± 17% +0.4 1.31 ± 10% perf-profile.calltrace.cycles.xfs_log_done.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_setattr_size
0.28 ±100% +0.4 0.72 ± 4% perf-profile.calltrace.cycles.unlazy_walk.complete_walk.path_openat.do_filp_open.do_sys_open
0.50 ± 57% +0.5 0.96 ± 5% perf-profile.calltrace.cycles.security_inode_permission.link_path_walk.path_openat.do_filp_open.do_sys_open
0.77 ± 18% +0.5 1.23 ± 3% perf-profile.calltrace.cycles.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.14 ±173% +0.5 0.61 ± 12% perf-profile.calltrace.cycles.dput.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.14 ±173% +0.5 0.65 ± 3% perf-profile.calltrace.cycles.legitimize_path.unlazy_walk.complete_walk.path_openat.do_filp_open
0.00 +0.6 0.55 ± 3% perf-profile.calltrace.cycles.strncpy_from_user.getname_flags.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.57 ± 5% perf-profile.calltrace.cycles.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.rwsem_down_write_failed.down_write
0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles.down_read.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_setattr_size
0.00 +0.6 0.61 ± 4% perf-profile.calltrace.cycles.apic_timer_interrupt.osq_lock.rwsem_down_write_failed.down_write.path_openat
0.93 ± 19% +0.6 1.54 ± 3% perf-profile.calltrace.cycles.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.14 ±173% +0.6 0.76 ± 9% perf-profile.calltrace.cycles.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_open
0.00 +0.6 0.63 ± 8% perf-profile.calltrace.cycles.may_open.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.00 +0.6 0.63 ± 9% perf-profile.calltrace.cycles.lookup_fast.walk_component.link_path_walk.path_openat.do_filp_open
1.59 +0.7 2.25 ± 6% perf-profile.calltrace.cycles.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size
1.64 +0.7 2.32 ± 5% perf-profile.calltrace.cycles.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr
1.75 ± 3% +0.8 2.54 ± 6% perf-profile.calltrace.cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64
2.35 +1.0 3.31 ± 6% perf-profile.calltrace.cycles.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr.notify_change
0.00 +1.0 0.98 ± 26% perf-profile.calltrace.cycles.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
0.00 +1.0 0.98 ± 26% perf-profile.calltrace.cycles.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.76 ± 19% +1.0 1.76 ± 7% perf-profile.calltrace.cycles._raw_spin_lock_irqsave.try_to_wake_up.wake_up_q.rwsem_wake.path_openat
0.45 ± 58% +1.0 1.47 ± 8% perf-profile.calltrace.cycles.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.try_to_wake_up.wake_up_q.rwsem_wake
0.94 ± 19% +1.1 2.06 ± 5% perf-profile.calltrace.cycles.try_to_wake_up.wake_up_q.rwsem_wake.path_openat.do_filp_open
1.02 ± 19% +1.2 2.18 ± 6% perf-profile.calltrace.cycles.wake_up_q.rwsem_wake.path_openat.do_filp_open.do_sys_open
0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.cpu_startup_entry.start_secondary.secondary_startup_64
0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.start_secondary.secondary_startup_64
1.99 ± 2% +1.2 3.22 ± 4% perf-profile.calltrace.cycles.rwsem_wake.path_openat.do_filp_open.do_sys_open.do_syscall_64
0.00 +1.2 1.25 ± 22% perf-profile.calltrace.cycles.secondary_startup_64
13.78 ± 9% +3.2 16.98 ± 9% perf-profile.calltrace.cycles.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr.notify_change.do_truncate
13.95 ± 9% +3.2 17.19 ± 9% perf-profile.calltrace.cycles.xfs_setattr_size.xfs_vn_setattr.notify_change.do_truncate.path_openat
14.60 ± 8% +3.5 18.06 ± 8% perf-profile.calltrace.cycles.xfs_vn_setattr.notify_change.do_truncate.path_openat.do_filp_open
15.35 ± 8% +3.7 19.04 ± 8% perf-profile.calltrace.cycles.notify_change.do_truncate.path_openat.do_filp_open.do_sys_open
15.55 ± 8% +3.8 19.31 ± 8% perf-profile.calltrace.cycles.do_truncate.path_openat.do_filp_open.do_sys_open.do_syscall_64
65.26 -11.9 53.37 ± 4% perf-profile.children.cycles.osq_lock
67.82 -11.4 56.42 ± 4% perf-profile.children.cycles.rwsem_down_write_failed
68.58 -11.3 57.31 ± 3% perf-profile.children.cycles.down_write
93.10 -3.7 89.36 perf-profile.children.cycles.path_openat
93.19 -3.7 89.52 perf-profile.children.cycles.do_filp_open
94.57 -3.0 91.53 perf-profile.children.cycles.do_sys_open
97.37 -1.9 95.45 perf-profile.children.cycles.do_syscall_64
97.49 -1.9 95.62 perf-profile.children.cycles.entry_SYSCALL_64_after_hwframe
0.10 ± 10% +0.0 0.12 ± 4% perf-profile.children.cycles.xfs_inode_item_size
0.06 ± 11% +0.0 0.08 ± 10% perf-profile.children.cycles.dax_layout_busy_page
0.05 ± 9% +0.0 0.08 ± 10% perf-profile.children.cycles.xfs_trans_ijoin
0.07 ± 5% +0.0 0.10 ± 15% perf-profile.children.cycles.xfs_da3_node_lookup_int
0.20 ± 6% +0.0 0.22 ± 4% perf-profile.children.cycles.run_ksoftirqd
0.07 ± 6% +0.0 0.10 ± 15% perf-profile.children.cycles.xfs_break_leased_layouts
0.07 ± 6% +0.0 0.10 ± 5% perf-profile.children.cycles.percpu_counter_add_batch
0.20 ± 3% +0.0 0.23 ± 3% perf-profile.children.cycles.__list_del_entry_valid
0.21 ± 5% +0.0 0.24 ± 3% perf-profile.children.cycles.smpboot_thread_fn
0.08 ± 5% +0.0 0.11 ± 9% perf-profile.children.cycles.__slab_free
0.07 ± 5% +0.0 0.10 ± 8% perf-profile.children.cycles.xfs_dir3_data_check
0.07 ± 5% +0.0 0.10 ± 8% perf-profile.children.cycles.__xfs_dir3_data_check
0.11 ± 4% +0.0 0.15 ± 7% perf-profile.children.cycles.xfs_dir_removename
0.22 ± 7% +0.0 0.25 perf-profile.children.cycles.terminate_walk
0.07 ± 12% +0.0 0.10 ± 18% perf-profile.children.cycles.selinux_task_getsecid
0.13 ± 3% +0.0 0.16 ± 12% perf-profile.children.cycles.xfs_create
0.11 ± 4% +0.0 0.15 ± 7% perf-profile.children.cycles.xfs_dir2_node_removename
0.06 ± 14% +0.0 0.10 ± 5% perf-profile.children.cycles.xfs_inode_item_format_data_fork
0.04 ± 58% +0.0 0.08 ± 8% perf-profile.children.cycles.__check_heap_object
0.11 ± 6% +0.0 0.15 ± 8% perf-profile.children.cycles.xfs_log_calc_unit_res
0.07 ± 12% +0.0 0.11 ± 12% perf-profile.children.cycles.xfs_inode_item_unlock
0.11 ± 7% +0.0 0.15 ± 5% perf-profile.children.cycles.__list_add_valid
0.11 ± 6% +0.0 0.15 ± 11% perf-profile.children.cycles.rcu_all_qs
0.13 ± 3% +0.0 0.17 ± 12% perf-profile.children.cycles.xfs_generic_create
0.13 ± 6% +0.0 0.17 ± 9% perf-profile.children.cycles.__call_rcu
0.09 ± 13% +0.0 0.14 ± 11% perf-profile.children.cycles.__legitimize_mnt
0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles.lockref_get
0.01 ±173% +0.0 0.06 ± 20% perf-profile.children.cycles.xfs_difree
0.11 ± 9% +0.0 0.16 ± 11% perf-profile.children.cycles.xfs_trans_del_item
0.25 ± 5% +0.0 0.30 ± 6% perf-profile.children.cycles.xfs_break_layouts
0.06 ± 14% +0.0 0.11 ± 24% perf-profile.children.cycles.io_serial_in
0.00 +0.1 0.05 perf-profile.children.cycles.__update_load_avg_se
0.10 ± 8% +0.1 0.16 ± 7% perf-profile.children.cycles.xfs_log_space_wake
0.01 ±173% +0.1 0.07 ± 17% perf-profile.children.cycles.update_curr
0.00 +0.1 0.05 ± 8% perf-profile.children.cycles.ktime_get_update_offsets_now
0.11 ± 7% +0.1 0.16 ± 18% perf-profile.children.cycles.__x86_indirect_thunk_rax
0.04 ± 57% +0.1 0.09 ± 14% perf-profile.children.cycles.setattr_prepare
0.01 ±173% +0.1 0.07 ± 6% perf-profile.children.cycles.__fd_install
0.01 ±173% +0.1 0.07 ± 10% perf-profile.children.cycles.rcu_segcblist_enqueue
0.17 ± 3% +0.1 0.22 ± 3% perf-profile.children.cycles.vfs_unlink
0.16 ± 4% +0.1 0.22 ± 7% perf-profile.children.cycles.xfs_remove
0.13 ± 3% +0.1 0.18 ± 9% perf-profile.children.cycles.schedule
0.00 +0.1 0.06 ± 14% perf-profile.children.cycles.inode_has_perm
0.00 +0.1 0.06 ± 14% perf-profile.children.cycles.xfs_log_ticket_put
0.01 ±173% +0.1 0.07 ± 17% perf-profile.children.cycles.xfs_file_release
0.09 ± 7% +0.1 0.15 ± 13% perf-profile.children.cycles.security_task_getsecid
0.18 ± 2% +0.1 0.24 ± 21% perf-profile.children.cycles.path_init
0.18 ± 5% +0.1 0.24 ± 5% perf-profile.children.cycles.__mnt_want_write
0.16 ± 4% +0.1 0.22 ± 5% perf-profile.children.cycles.xfs_vn_unlink
0.00 +0.1 0.06 ± 16% perf-profile.children.cycles.delay_tsc
0.32 ± 4% +0.1 0.38 ± 6% perf-profile.children.cycles.xlog_grant_push_ail
0.15 ± 5% +0.1 0.21 ± 11% perf-profile.children.cycles.xfs_iunlock
0.31 ± 3% +0.1 0.38 ± 5% perf-profile.children.cycles.up_write
0.16 ± 2% +0.1 0.23 ± 10% perf-profile.children.cycles.ima_file_check
0.15 ± 7% +0.1 0.22 ± 13% perf-profile.children.cycles.__fsnotify_parent
0.00 +0.1 0.07 ± 17% perf-profile.children.cycles.update_sd_lb_stats
0.00 +0.1 0.07 ± 23% perf-profile.children.cycles.read_tsc
0.09 ± 4% +0.1 0.16 ± 11% perf-profile.children.cycles._raw_spin_unlock_irqrestore
0.19 ± 14% +0.1 0.25 ± 7% perf-profile.children.cycles.__schedule
0.00 +0.1 0.07 ± 19% perf-profile.children.cycles.lapic_next_deadline
0.23 ± 8% +0.1 0.30 ± 8% perf-profile.children.cycles.ret_from_fork
0.11 ± 4% +0.1 0.18 ± 12% perf-profile.children.cycles.xfs_vn_setattr_size
0.42 ± 2% +0.1 0.49 ± 10% perf-profile.children.cycles.__x64_sys_close
0.00 +0.1 0.07 ± 14% perf-profile.children.cycles.find_busiest_group
0.00 +0.1 0.07 ± 14% perf-profile.children.cycles.xfs_can_free_eofblocks
0.23 ± 6% +0.1 0.30 ± 8% perf-profile.children.cycles.kthread
0.13 ± 14% +0.1 0.20 ± 12% perf-profile.children.cycles.avc_has_perm
0.10 ± 17% +0.1 0.17 ± 21% perf-profile.children.cycles.uart_console_write
0.09 ± 15% +0.1 0.17 ± 19% perf-profile.children.cycles.serial8250_console_putchar
0.09 ± 13% +0.1 0.16 ± 14% perf-profile.children.cycles.generic_permission
0.07 ± 21% +0.1 0.14 ± 13% perf-profile.children.cycles.xlog_grant_head_check
0.10 ± 15% +0.1 0.17 ± 19% perf-profile.children.cycles.serial8250_console_write
0.26 ± 5% +0.1 0.33 ± 7% perf-profile.children.cycles.up_read
0.15 ± 6% +0.1 0.22 ± 6% perf-profile.children.cycles.__virt_addr_valid
0.01 ±173% +0.1 0.09 ± 7% perf-profile.children.cycles.xlog_verify_grant_tail
0.33 +0.1 0.40 ± 7% perf-profile.children.cycles.do_unlinkat
0.20 ± 6% +0.1 0.28 ± 16% perf-profile.children.cycles.inode_security_rcu
0.10 ± 13% +0.1 0.17 ± 19% perf-profile.children.cycles.wait_for_xmitr
0.36 ± 4% +0.1 0.44 ± 8% perf-profile.children.cycles.lockref_get_not_dead
0.33 ± 2% +0.1 0.41 ± 8% perf-profile.children.cycles.xlog_space_left
0.26 ± 9% +0.1 0.34 ± 6% perf-profile.children.cycles.security_file_alloc
0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.irq_work_interrupt
0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.smp_irq_work_interrupt
0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.irq_work_run
0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.printk
0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.vprintk_emit
0.03 ±100% +0.1 0.11 ± 23% perf-profile.children.cycles.pick_next_task_fair
0.49 ± 2% +0.1 0.57 ± 3% perf-profile.children.cycles.down_read
0.00 +0.1 0.09 ± 20% perf-profile.children.cycles.native_write_msr
0.61 ± 2% +0.1 0.69 ± 5% perf-profile.children.cycles.osq_unlock
0.26 ± 7% +0.1 0.35 ± 10% perf-profile.children.cycles._cond_resched
0.11 ± 17% +0.1 0.20 ± 20% perf-profile.children.cycles.irq_work_run_list
0.29 ± 11% +0.1 0.39 ± 2% perf-profile.children.cycles.wake_q_add
0.27 ± 7% +0.1 0.36 ± 7% perf-profile.children.cycles.rcu_core
0.26 ± 5% +0.1 0.35 ± 11% perf-profile.children.cycles.xfs_ilock
0.17 +0.1 0.26 ± 4% perf-profile.children.cycles.__d_lookup_rcu
0.00 +0.1 0.09 ± 19% perf-profile.children.cycles.load_balance
0.10 ± 18% +0.1 0.20 ± 19% perf-profile.children.cycles.console_unlock
0.26 ± 5% +0.1 0.35 ± 11% perf-profile.children.cycles.__alloc_fd
0.03 ±100% +0.1 0.12 ± 18% perf-profile.children.cycles.clockevents_program_event
0.00 +0.1 0.10 ± 8% perf-profile.children.cycles.menu_select
0.23 ± 10% +0.1 0.33 ± 8% perf-profile.children.cycles.selinux_inode_setattr
0.15 ± 2% +0.1 0.26 ± 6% perf-profile.children.cycles.xfs_release
0.00 +0.1 0.11 ± 16% perf-profile.children.cycles.ktime_get
0.24 ± 7% +0.1 0.35 ± 14% perf-profile.children.cycles.xfs_trans_log_inode
0.00 +0.1 0.11 ± 17% perf-profile.children.cycles.perf_mux_hrtimer_handler
0.31 ± 10% +0.1 0.42 ± 5% perf-profile.children.cycles.__check_object_size
0.33 ± 11% +0.1 0.45 ± 3% perf-profile.children.cycles.__rwsem_mark_wake
0.19 ± 9% +0.1 0.31 ± 12% perf-profile.children.cycles.memset_erms
0.23 ± 10% +0.1 0.34 ± 5% perf-profile.children.cycles.fsnotify
0.26 ± 9% +0.1 0.38 ± 8% perf-profile.children.cycles.security_inode_setattr
0.26 ± 5% +0.1 0.39 ± 9% perf-profile.children.cycles.avc_has_perm_noaudit
0.28 ± 2% +0.1 0.41 ± 13% perf-profile.children.cycles.selinux_file_open
0.29 ± 5% +0.1 0.43 ± 8% perf-profile.children.cycles.xfs_trans_free_items
0.13 ± 11% +0.1 0.27 ± 14% perf-profile.children.cycles.task_tick_fair
0.52 ± 9% +0.1 0.65 ± 3% perf-profile.children.cycles.legitimize_path
0.41 ± 6% +0.1 0.56 ± 4% perf-profile.children.cycles.strncpy_from_user
0.28 ± 3% +0.1 0.42 ± 7% perf-profile.children.cycles.mnt_want_write
0.20 ± 6% +0.1 0.34 ± 9% perf-profile.children.cycles.__sb_start_write
0.30 ± 3% +0.1 0.45 ± 13% perf-profile.children.cycles.__inode_security_revalidate
0.51 ± 7% +0.2 0.67 ± 8% perf-profile.children.cycles.xlog_ticket_alloc
0.59 ± 9% +0.2 0.75 ± 4% perf-profile.children.cycles.complete_walk
0.55 ± 10% +0.2 0.72 ± 4% perf-profile.children.cycles.unlazy_walk
0.16 ± 9% +0.2 0.34 ± 9% perf-profile.children.cycles.scheduler_tick
0.16 ± 10% +0.2 0.34 ± 15% perf-profile.children.cycles.irq_exit
0.36 ± 9% +0.2 0.54 ± 5% perf-profile.children.cycles.xfs_inode_item_format
0.35 ± 3% +0.2 0.52 ± 10% perf-profile.children.cycles.security_file_open
0.35 ± 7% +0.2 0.52 ± 7% perf-profile.children.cycles.__softirqentry_text_start
0.39 ± 4% +0.2 0.57 ± 4% perf-profile.children.cycles.__might_sleep
0.45 ± 9% +0.2 0.64 ± 8% perf-profile.children.cycles.lookup_fast
0.40 ± 2% +0.2 0.59 ± 9% perf-profile.children.cycles.inode_permission
0.74 ± 8% +0.2 0.94 ± 10% perf-profile.children.cycles.dput
0.43 ± 7% +0.2 0.63 ± 8% perf-profile.children.cycles.may_open
0.39 ± 4% +0.2 0.60 ± 12% perf-profile.children.cycles.___might_sleep
0.93 ± 2% +0.2 1.15 ± 3% perf-profile.children.cycles.rwsem_spin_on_owner
0.76 ± 2% +0.2 0.99 ± 12% perf-profile.children.cycles.xlog_ungrant_log_space
0.52 ± 8% +0.2 0.76 ± 9% perf-profile.children.cycles.walk_component
0.41 ± 7% +0.3 0.68 ± 4% perf-profile.children.cycles.kmem_cache_free
0.24 ± 11% +0.3 0.51 ± 9% perf-profile.children.cycles.update_process_times
0.25 ± 10% +0.3 0.52 ± 9% perf-profile.children.cycles.tick_sched_handle
0.53 ± 4% +0.3 0.82 ± 8% perf-profile.children.cycles.xlog_grant_add_space
0.59 ± 6% +0.3 0.88 ± 7% perf-profile.children.cycles.kmem_zone_alloc
0.70 ± 4% +0.3 1.00 ± 4% perf-profile.children.cycles.getname_flags
0.27 ± 11% +0.3 0.57 ± 8% perf-profile.children.cycles.tick_sched_timer
0.86 ± 4% +0.3 1.18 ± 5% perf-profile.children.cycles.__alloc_file
0.67 ± 3% +0.3 1.00 ± 7% perf-profile.children.cycles.syscall_return_via_sysret
0.91 ± 4% +0.3 1.25 ± 6% perf-profile.children.cycles.alloc_empty_file
0.98 ± 2% +0.3 1.31 ± 10% perf-profile.children.cycles.xfs_log_done
0.84 ± 2% +0.4 1.23 ± 3% perf-profile.children.cycles.do_dentry_open
0.83 ± 3% +0.4 1.22 ± 7% perf-profile.children.cycles.entry_SYSCALL_64
0.89 ± 2% +0.4 1.28 ± 5% perf-profile.children.cycles.selinux_inode_permission
0.09 ± 9% +0.4 0.51 ± 34% perf-profile.children.cycles.intel_idle
0.97 ± 3% +0.4 1.39 ± 4% perf-profile.children.cycles.security_inode_permission
0.36 ± 8% +0.5 0.82 ± 9% perf-profile.children.cycles.__hrtimer_run_queues
1.02 ± 2% +0.5 1.54 ± 3% perf-profile.children.cycles.__fput
1.09 +0.6 1.67 ± 6% perf-profile.children.cycles.kmem_cache_alloc
0.46 ± 7% +0.6 1.08 ± 9% perf-profile.children.cycles.hrtimer_interrupt
1.41 +0.6 2.06 ± 4% perf-profile.children.cycles.task_work_run
1.60 +0.7 2.26 ± 6% perf-profile.children.cycles.xfs_log_reserve
1.64 +0.7 2.33 ± 5% perf-profile.children.cycles.xfs_trans_reserve
1.56 +0.7 2.26 ± 4% perf-profile.children.cycles.exit_to_usermode_loop
0.25 ± 5% +0.8 1.02 ± 26% perf-profile.children.cycles.cpuidle_enter
0.25 ± 5% +0.8 1.02 ± 26% perf-profile.children.cycles.cpuidle_enter_state
1.76 ± 3% +0.8 2.54 ± 6% perf-profile.children.cycles.link_path_walk
0.39 ± 6% +0.8 1.20 ± 22% perf-profile.children.cycles.start_secondary
0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.do_idle
0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.secondary_startup_64
0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.cpu_startup_entry
0.65 ± 6% +0.9 1.51 ± 11% perf-profile.children.cycles.smp_apic_timer_interrupt
0.69 ± 6% +0.9 1.62 ± 10% perf-profile.children.cycles.apic_timer_interrupt
0.86 ± 5% +0.9 1.80 ± 7% perf-profile.children.cycles._raw_spin_lock_irqsave
2.35 +1.0 3.33 ± 6% perf-profile.children.cycles.xfs_trans_alloc
1.04 ± 4% +1.0 2.08 ± 5% perf-profile.children.cycles.try_to_wake_up
1.13 ± 4% +1.1 2.18 ± 6% perf-profile.children.cycles.wake_up_q
1.99 ± 2% +1.2 3.23 ± 4% perf-profile.children.cycles.rwsem_wake
13.78 ± 9% +3.2 16.98 ± 9% perf-profile.children.cycles.xfs_setattr_nonsize
13.95 ± 9% +3.2 17.19 ± 9% perf-profile.children.cycles.xfs_setattr_size
14.60 ± 8% +3.5 18.06 ± 8% perf-profile.children.cycles.xfs_vn_setattr
15.34 ± 8% +3.7 19.04 ± 8% perf-profile.children.cycles.notify_change
15.55 ± 8% +3.8 19.31 ± 8% perf-profile.children.cycles.do_truncate
64.82 -12.2 52.65 ± 4% perf-profile.self.cycles.osq_lock
0.06 ± 11% +0.0 0.08 ± 12% perf-profile.self.cycles.dax_layout_busy_page
0.08 ± 5% +0.0 0.10 ± 10% perf-profile.self.cycles.__call_rcu
0.20 ± 3% +0.0 0.22 ± 3% perf-profile.self.cycles.__list_del_entry_valid
0.06 ± 6% +0.0 0.08 ± 13% perf-profile.self.cycles.xfs_trans_unreserve_and_mod_sb
0.07 ± 10% +0.0 0.10 ± 9% perf-profile.self.cycles.selinux_inode_setattr
0.07 ± 6% +0.0 0.09 ± 8% perf-profile.self.cycles.percpu_counter_add_batch
0.04 ± 58% +0.0 0.07 ± 12% perf-profile.self.cycles.xfs_trans_ijoin
0.08 ± 10% +0.0 0.11 ± 4% perf-profile.self.cycles.xfs_vn_setattr
0.08 ± 5% +0.0 0.11 ± 9% perf-profile.self.cycles.__slab_free
0.07 ± 7% +0.0 0.10 ± 15% perf-profile.self.cycles.xfs_break_leased_layouts
0.09 +0.0 0.12 ± 12% perf-profile.self.cycles.wake_up_q
0.06 ± 11% +0.0 0.09 ± 14% perf-profile.self.cycles.xfs_trans_free_items
0.04 ± 57% +0.0 0.07 ± 10% perf-profile.self.cycles.xfs_trans_reserve
0.07 ± 12% +0.0 0.10 ± 18% perf-profile.self.cycles.selinux_task_getsecid
0.12 ± 3% +0.0 0.16 ± 5% perf-profile.self.cycles.xfs_setattr_size
0.10 ± 9% +0.0 0.13 ± 10% perf-profile.self.cycles.lookup_fast
0.04 ± 58% +0.0 0.08 ± 5% perf-profile.self.cycles.__check_heap_object
0.12 ± 3% +0.0 0.15 ± 11% perf-profile.self.cycles.selinux_file_open
0.09 ± 8% +0.0 0.13 ± 5% perf-profile.self.cycles.exit_to_usermode_loop
0.04 ± 58% +0.0 0.08 ± 15% perf-profile.self.cycles.xfs_inode_item_format_data_fork
0.07 ± 7% +0.0 0.11 ± 10% perf-profile.self.cycles.__xfs_trans_commit
0.11 ± 4% +0.0 0.15 ± 5% perf-profile.self.cycles.xfs_log_calc_unit_res
0.08 ± 8% +0.0 0.12 ± 6% perf-profile.self.cycles.try_to_wake_up
0.11 ± 7% +0.0 0.15 ± 5% perf-profile.self.cycles.__list_add_valid
0.12 ± 9% +0.0 0.16 ± 6% perf-profile.self.cycles.task_work_run
0.09 ± 13% +0.0 0.13 ± 11% perf-profile.self.cycles.__legitimize_mnt
0.06 ± 13% +0.0 0.11 ± 8% perf-profile.self.cycles.do_filp_open
0.11 ± 6% +0.0 0.15 ± 11% perf-profile.self.cycles.rcu_all_qs
0.43 +0.0 0.48 ± 2% perf-profile.self.cycles.down_read
0.13 ± 6% +0.0 0.17 ± 9% perf-profile.self.cycles._cond_resched
0.11 ± 7% +0.0 0.15 ± 12% perf-profile.self.cycles.xfs_trans_del_item
0.08 ± 5% +0.0 0.13 ± 12% perf-profile.self.cycles._raw_spin_unlock_irqrestore
0.01 ±173% +0.0 0.06 ± 11% perf-profile.self.cycles.__rwsem_mark_wake
0.01 ±173% +0.0 0.06 ± 20% perf-profile.self.cycles.filp_close
0.07 ± 11% +0.0 0.12 ± 22% perf-profile.self.cycles.walk_component
0.06 ± 14% +0.0 0.11 ± 24% perf-profile.self.cycles.io_serial_in
0.01 ±173% +0.1 0.06 ± 13% perf-profile.self.cycles.do_truncate
0.11 ± 7% +0.1 0.16 ± 18% perf-profile.self.cycles.__alloc_fd
0.61 ± 2% +0.1 0.66 ± 4% perf-profile.self.cycles.osq_unlock
0.00 +0.1 0.05 ± 8% perf-profile.self.cycles.current_time
0.10 ± 7% +0.1 0.15 ± 9% perf-profile.self.cycles.xfs_log_space_wake
0.23 ± 3% +0.1 0.28 ± 5% perf-profile.self.cycles.rwsem_wake
0.11 ± 7% +0.1 0.16 ± 18% perf-profile.self.cycles.__x86_indirect_thunk_rax
0.00 +0.1 0.05 ± 9% perf-profile.self.cycles.inode_has_perm
0.01 ±173% +0.1 0.07 ± 6% perf-profile.self.cycles.__fd_install
0.01 ±173% +0.1 0.07 ± 10% perf-profile.self.cycles.rcu_segcblist_enqueue
0.01 ±173% +0.1 0.07 ± 16% perf-profile.self.cycles.xlog_grant_head_check
0.18 ± 7% +0.1 0.23 ± 4% perf-profile.self.cycles.__mnt_want_write
0.00 +0.1 0.06 ± 14% perf-profile.self.cycles.ktime_get
0.00 +0.1 0.06 ± 14% perf-profile.self.cycles.xfs_log_ticket_put
0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles.xfs_file_release
0.11 ± 9% +0.1 0.17 ± 13% perf-profile.self.cycles.entry_SYSCALL_64_after_hwframe
0.00 +0.1 0.06 ± 16% perf-profile.self.cycles.delay_tsc
0.31 ± 3% +0.1 0.37 ± 6% perf-profile.self.cycles.up_write
0.15 ± 7% +0.1 0.21 ± 12% perf-profile.self.cycles.__fsnotify_parent
0.00 +0.1 0.06 ± 13% perf-profile.self.cycles.menu_select
0.00 +0.1 0.07 ± 7% perf-profile.self.cycles.security_file_open
0.00 +0.1 0.07 ± 17% perf-profile.self.cycles.setattr_prepare
0.00 +0.1 0.07 ± 23% perf-profile.self.cycles.read_tsc
0.18 ± 6% +0.1 0.25 ± 13% perf-profile.self.cycles.do_dentry_open
0.09 ± 13% +0.1 0.16 ± 15% perf-profile.self.cycles.generic_permission
0.00 +0.1 0.07 ± 14% perf-profile.self.cycles.xfs_can_free_eofblocks
0.15 ± 6% +0.1 0.22 ± 8% perf-profile.self.cycles.__virt_addr_valid
0.13 ± 14% +0.1 0.20 ± 12% perf-profile.self.cycles.avc_has_perm
0.25 ± 5% +0.1 0.33 ± 7% perf-profile.self.cycles._raw_spin_lock_irqsave
0.01 ±173% +0.1 0.09 ± 9% perf-profile.self.cycles.xlog_verify_grant_tail
0.36 ± 4% +0.1 0.44 ± 8% perf-profile.self.cycles.lockref_get_not_dead
0.26 ± 6% +0.1 0.33 ± 6% perf-profile.self.cycles.up_read
0.20 ± 4% +0.1 0.27 ± 5% perf-profile.self.cycles.__fput
0.03 ±100% +0.1 0.10 ± 4% perf-profile.self.cycles.apic_timer_interrupt
0.32 +0.1 0.40 ± 7% perf-profile.self.cycles.xlog_space_left
0.15 ± 8% +0.1 0.23 ± 14% perf-profile.self.cycles.__inode_security_revalidate
0.08 ± 15% +0.1 0.16 ± 9% perf-profile.self.cycles.__sb_start_write
0.13 ± 13% +0.1 0.21 ± 12% perf-profile.self.cycles.xfs_setattr_nonsize
0.17 +0.1 0.26 ± 5% perf-profile.self.cycles.__d_lookup_rcu
0.00 +0.1 0.09 ± 20% perf-profile.self.cycles.native_write_msr
0.29 ± 11% +0.1 0.39 ± 2% perf-profile.self.cycles.wake_q_add
0.25 ± 9% +0.1 0.36 ± 17% perf-profile.self.cycles.link_path_walk
0.16 ± 5% +0.1 0.26 ± 13% perf-profile.self.cycles.do_sys_open
0.23 ± 4% +0.1 0.34 ± 14% perf-profile.self.cycles.xfs_trans_log_inode
0.28 ± 3% +0.1 0.39 ± 8% perf-profile.self.cycles.__alloc_file
0.19 ± 9% +0.1 0.30 ± 12% perf-profile.self.cycles.memset_erms
0.31 ± 6% +0.1 0.42 ± 7% perf-profile.self.cycles.inode_permission
0.23 ± 11% +0.1 0.34 ± 5% perf-profile.self.cycles.fsnotify
0.90 ± 3% +0.1 1.02 ± 4% perf-profile.self.cycles.rwsem_down_write_failed
0.26 ± 7% +0.1 0.38 ± 11% perf-profile.self.cycles.avc_has_perm_noaudit
0.30 ± 4% +0.1 0.43 ± 5% perf-profile.self.cycles.xfs_inode_item_format
0.32 ± 9% +0.2 0.48 ± 12% perf-profile.self.cycles.do_syscall_64
0.39 ± 4% +0.2 0.56 ± 4% perf-profile.self.cycles.__might_sleep
0.62 ± 4% +0.2 0.81 ± 5% perf-profile.self.cycles._raw_spin_lock
0.42 ± 4% +0.2 0.62 ± 6% perf-profile.self.cycles.selinux_inode_permission
0.44 ± 18% +0.2 0.65 ± 8% perf-profile.self.cycles.path_openat
0.39 ± 4% +0.2 0.60 ± 11% perf-profile.self.cycles.___might_sleep
0.75 ± 2% +0.2 0.96 ± 11% perf-profile.self.cycles.xlog_ungrant_log_space
0.31 ± 8% +0.2 0.52 ± 6% perf-profile.self.cycles.kmem_cache_free
0.93 ± 2% +0.2 1.14 ± 4% perf-profile.self.cycles.rwsem_spin_on_owner
1.42 ± 4% +0.2 1.64 ± 4% perf-profile.self.cycles.xfs_log_commit_cil
0.54 ± 2% +0.3 0.82 ± 4% perf-profile.self.cycles.kmem_cache_alloc
0.52 ± 3% +0.3 0.81 ± 7% perf-profile.self.cycles.xlog_grant_add_space
0.67 ± 3% +0.3 1.00 ± 7% perf-profile.self.cycles.syscall_return_via_sysret
0.83 ± 3% +0.4 1.22 ± 7% perf-profile.self.cycles.entry_SYSCALL_64
0.08 ± 10% +0.4 0.51 ± 34% perf-profile.self.cycles.intel_idle
aim7.jobs-per-min
70000 +-+-----------------------------------------------------------------+
|.+. +.+.+..+.+.+.+. + +.+.+..+ +.+..+ |
65000 +-+ |
| |
60000 +-+ |
| |
55000 +-+ |
| |
50000 +-+ |
| O O O O O O O |
45000 +-+ O O O O O O
O O O O O O O O O O |
40000 +-O O O O O O O O |
| |
35000 +-+-----------------------------------------------------------------+
aim7.time.user_time
130 +-+--------------------O----------------------------------------------+
| O O O O O |
120 O-+ O O O O O O O O O O
110 +-+ O O O O O O O O O O |
| O O O O |
100 +-+ |
| |
90 +-+ |
| |
80 +-+ |
70 +-+ |
| + |
60 +-+ .+. + + .+. .+. .+.. .+. .+. |
|.+. +.+..+ +..+ +..+.+.+. + + +. + |
50 +-+-------------------------------------------------------------------+
aim7.time.elapsed_time
230 +-+-------------------------------------------------------------------+
220 +-O O O O O O O O O |
O O O O O O |
210 +-+ O O O O O O O
200 +-+ O O |
190 +-+ O O O O O O O |
180 +-+ |
| |
170 +-+ |
160 +-+ |
150 +-+ |
140 +-+ |
|.+.. .+.+..+.+.+..+.+. .+. .+.+.+..+. .+..+. |
130 +-+ + +. +.+. + + |
120 +-+-------------------------------------------------------------------+
aim7.time.elapsed_time.max
230 +-+-------------------------------------------------------------------+
220 +-O O O O O O O O O |
O O O O O O |
210 +-+ O O O O O O O
200 +-+ O O |
190 +-+ O O O O O O O |
180 +-+ |
| |
170 +-+ |
160 +-+ |
150 +-+ |
140 +-+ |
|.+.. .+.+..+.+.+..+.+. .+. .+.+.+..+. .+..+. |
130 +-+ + +. +.+. + + |
120 +-+-------------------------------------------------------------------+
aim7.time.voluntary_context_switches
3e+06 +-+---------------------------------------------------------------+
2.8e+06 +-+ .+.. .+.+.. .+. |
|. .+. .+ +.+.+ +.+.+.+.+..+ +.+ |
2.6e+06 +-+. .+ + |
2.4e+06 +-+ +. |
2.2e+06 +-+ |
2e+06 +-+ |
| |
1.8e+06 O-+ O O O O O O O |
1.6e+06 +-O O O O O O O O O O |
1.4e+06 +-+ O O O
1.2e+06 +-+ |
| O O O |
1e+06 +-+ O O O O O O O |
800000 +-+---------------------------------------------------------------+
aim7.time.involuntary_context_switches
360000 +-+----------------------------------------------------------------+
340000 +-+ O O O O O O |
O O O O O O O O O O O O |
320000 +-+ O O O O O
300000 +-+ O O O O O O O |
| |
280000 +-+ O |
260000 +-+ |
240000 +-+ |
| |
220000 +-+ |
200000 +-+.+..+. |
| + +.+. .+. .+..+.+.+. |
180000 +-+ +..+.+ +..+.+.+ +.+..+.+ |
160000 +-+----------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode:
cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2019-05-14.cgz/900s/lkp-bdw-de1/TCP_MAERTS/netperf/0x7000017
commit:
fe146f762b ("sched: Remove rq->cpu_load[] update code")
cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()")
fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:2 -50% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x
:2 50% 1:2 dmesg.WARNING:at_ip__netif_receive_skb_core/0x
***************************************************************************************************
lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/pipe/x86_64-rhel-7.6/process/1600%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ex1/hackbench/0xb00002e
commit:
fe146f762b ("sched: Remove rq->cpu_load[] update code")
cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()")
fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:2 100% 2:2 dmesg.WARNING:stack_recursion
***************************************************************************************************
lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory
***************************************************************************************************
lkp-bdw-ep3: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
gcc-7/performance/socket/x86_64-rhel-7.6/process/1600%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep3/hackbench/0xb00002e
commit:
fe146f762b ("sched: Remove rq->cpu_load[] update code")
cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()")
fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
:2 100% 2:2 dmesg.WARNING:at_ip__slab_free/0x
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
1 year, 9 months
[nl80211] 52f9c100b1: WARNING:at_net/wireless/core.c:#wiphy_register
by kernel test robot
FYI, we noticed the following commit (built with gcc-6):
commit: 52f9c100b17e1d5d9696366d7401d14d3fd9c05c ("[PATCH] nl80211: require and validate vendor command policy")
url: https://github.com/0day-ci/linux/commits/Johannes-Berg/nl80211-require-an...
base: https://git.kernel.org/cgit/linux/kernel/git/jberg/mac80211.git master
in testcase: trinity
with following parameters:
runtime: 300s
test-description: Trinity is a linux system call fuzz tester.
test-url: http://codemonkey.org.uk/projects/trinity/
on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 2G
caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):
+------------------------------------------------+------------+------------+
| | 551842446e | 52f9c100b1 |
+------------------------------------------------+------------+------------+
| boot_successes | 7 | 0 |
| boot_failures | 1 | 8 |
| BUG:soft_lockup-CPU##stuck_for#s | 1 | |
| EIP:_raw_spin_unlock_irqrestore | 1 | |
| Kernel_panic-not_syncing:softlockup:hung_tasks | 1 | |
| WARNING:at_net/wireless/core.c:#wiphy_register | 0 | 8 |
| EIP:wiphy_register | 0 | 8 |
+------------------------------------------------+------------+------------+
If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen(a)intel.com>
[ 42.418616] WARNING: CPU: 1 PID: 1 at net/wireless/core.c:867 wiphy_register+0x6e8/0x950
[ 42.427495] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.1.0-10250-g52f9c10 #1
[ 42.427495] EIP: wiphy_register+0x6e8/0x950
[ 42.427495] Code: 74 8b 86 5c 03 00 00 8b 50 14 85 d2 74 1c 83 c0 0c 31 d2 8b 38 85 ff 74 49 83 c2 01 39 ca 74 55 83 c0 1c 8b 58 08 85 db 75 e9 <0f> 0b bb ea ff ff ff e9 0c fa ff ff 8d 74 26 00 0f 0b bb ea ff ff
[ 42.427495] EAX: c3377d4c EBX: c2cdcdba ECX: 00000001 EDX: 00000000
[ 42.427495] ESI: ee5f42e0 EDI: 00000003 EBP: c004be5c ESP: c004be04
[ 42.427495] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246
[ 42.427495] CR0: 80050033 CR2: ffffffff CR3: 034fd000 CR4: 000006b0
[ 42.427495] Call Trace:
[ 42.427495] ieee80211_register_hw+0x446/0xc00
[ 42.427495] mac80211_hwsim_new_radio+0x7c0/0xe20
[ 42.427495] init_mac80211_hwsim+0x15c/0x2c5
[ 42.427495] ? wl3501_driver_init+0xf/0xf
[ 42.427495] do_one_initcall+0x7b/0x148
[ 42.427495] kernel_init_freeable+0x16a/0x1e5
[ 42.427495] ? rest_init+0x120/0x120
[ 42.427495] kernel_init+0x8/0xf0
[ 42.427495] ret_from_fork+0x1e/0x28
[ 42.427495] random: get_random_bytes called from print_oops_end_marker+0x4f/0x60 with crng_init=0
[ 42.427495] ---[ end trace ca67d517de3b21fc ]---
To reproduce:
# build kernel
cd linux
cp config-5.1.0-10250-g52f9c10 .config
make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 olddefconfig
make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 prepare
make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 modules_prepare
make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 SHELL=/bin/bash
make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 bzImage
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email
Thanks,
Rong Chen
1 year, 9 months
[btrfs] 2996e1f8bc: aim7.jobs-per-min -13.2% regression
by kernel test robot
Greeting,
FYI, we noticed a -13.2% regression of aim7.jobs-per-min due to commit:
commit: 2996e1f8bcadf0436cc67b63af01523f6cf5d43f ("btrfs: factor our read/write stage off csum_tree_block into its callers")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
in testcase: aim7
on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory
with following parameters:
disk: 4BRD_12G
md: RAID0
fs: btrfs
test: sync_disk_rw
load: 20
ucode: 0x200005e
cpufreq_governor: performance
test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/
In addition to that, the commit also has significant impact on the following tests:
+------------------+-----------------------------------------------------------------------+
| testcase: change | aim7: aim7.jobs-per-min -19.1% regression |
| test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory |
| test parameters | cpufreq_governor=performance |
| | disk=4BRD_12G |
| | fs=btrfs |
| | load=20 |
| | md=RAID0 |
| | test=sync_disk_rw |
+------------------+-----------------------------------------------------------------------+
Details are as below:
-------------------------------------------------------------------------------------------------->
To reproduce:
git clone https://github.com/intel/lkp-tests.git
cd lkp-tests
bin/lkp install job.yaml # job file is attached in this email
bin/lkp run job.yaml
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode:
gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2019-05-14.cgz/lkp-skl-2sp7/sync_disk_rw/aim7/0x200005e
commit:
v5.1-rc7
2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers")
v5.1-rc7 2996e1f8bcadf0436cc67b63af0
---------------- ---------------------------
fail:runs %reproduction fail:runs
| | |
1:4 -25% :4 kmsg.Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#e)
%stddev %change %stddev
\ | \
1207 -13.2% 1047 aim7.jobs-per-min
99.41 +15.2% 114.56 aim7.time.elapsed_time
99.41 +15.2% 114.56 aim7.time.elapsed_time.max
884.50 ± 2% +13.5% 1004 ± 3% aim7.time.involuntary_context_switches
704.41 +18.2% 832.33 aim7.time.system_time
21484045 -10.5% 19231461 aim7.time.voluntary_context_switches
0.02 ± 38% -0.0 0.00 ± 76% mpstat.cpu.all.soft%
59.02 ± 6% -58.9% 24.26 boot-time.boot
3928 ± 8% -62.1% 1489 boot-time.idle
19798 -12.9% 17235 iostat.md0.w/s
448478 -13.1% 389592 iostat.md0.wkB/s
130905 ± 6% -20.8% 103683 ± 2% meminfo.AnonHugePages
100591 ± 2% -12.9% 87596 meminfo.max_used_kB
5058 ± 4% +17.1% 5925 ± 9% numa-meminfo.node1.KernelStack
1737 ± 33% +66.4% 2889 ± 34% numa-meminfo.node1.PageTables
1174778 ± 23% +37.5% 1615284 ± 2% numa-numastat.node1.local_node
1189955 ± 23% +36.7% 1626977 ± 2% numa-numastat.node1.numa_hit
344576 -12.9% 300018 vmstat.io.bo
558315 -21.3% 439168 vmstat.system.cs
194.50 ± 16% -29.7% 136.75 ± 15% numa-vmstat.node1.nr_dirty
5061 ± 4% +17.1% 5925 ± 9% numa-vmstat.node1.nr_kernel_stack
433.75 ± 33% +66.4% 721.75 ± 34% numa-vmstat.node1.nr_page_table_pages
1065659 ± 23% +41.9% 1511723 ± 8% numa-vmstat.node1.numa_hit
897602 ± 28% +50.0% 1346715 ± 10% numa-vmstat.node1.numa_local
3240 ± 5% -23.0% 2496 slabinfo.skbuff_head_cache.active_objs
3264 ± 4% -19.6% 2624 slabinfo.skbuff_head_cache.num_objs
1339 ± 2% -24.6% 1010 slabinfo.task_struct.active_objs
336.75 ± 2% +202.4% 1018 slabinfo.task_struct.active_slabs
1348 ± 2% -24.5% 1018 slabinfo.task_struct.num_objs
336.75 ± 2% +202.4% 1018 slabinfo.task_struct.num_slabs
23341140 ± 9% -23.6% 17828343 cpuidle.C1.usage
1.777e+09 ± 28% +215.6% 5.608e+09 ± 8% cpuidle.C1E.time
12674253 ± 20% +66.3% 21080014 ± 3% cpuidle.C1E.usage
3.182e+09 ± 12% -87.4% 4.02e+08 ±107% cpuidle.C6.time
3770499 ± 3% -83.8% 609866 ±127% cpuidle.C6.usage
6880882 ± 13% -46.1% 3706529 cpuidle.POLL.time
1441739 ± 10% -40.2% 862558 cpuidle.POLL.usage
23339885 ± 9% -23.6% 17822261 turbostat.C1
18.49 ± 15% -3.6 14.86 turbostat.C1%
12673355 ± 20% +66.3% 21078440 ± 3% turbostat.C1E
24.32 ± 28% +42.5 66.84 ± 7% turbostat.C1E%
3765573 ± 3% -84.0% 603105 ±128% turbostat.C6
43.48 ± 11% -38.7 4.77 ±108% turbostat.C6%
3.73 ± 42% -90.1% 0.37 ± 12% turbostat.CPU%c6
14813103 ± 2% +16.9% 17318955 turbostat.IRQ
1.59 ± 51% -100.0% 0.00 turbostat.Pkg%pc2
40.05 +1.7% 40.72 turbostat.RAMWatt
59758 -1.1% 59093 proc-vmstat.nr_anon_pages
48741 +1.0% 49245 proc-vmstat.nr_free_cma
4606 -4.6% 4393 proc-vmstat.nr_inactive_anon
6367 -2.2% 6227 proc-vmstat.nr_mapped
34775 +2.1% 35519 proc-vmstat.nr_slab_unreclaimable
4606 -4.6% 4393 proc-vmstat.nr_zone_inactive_anon
428.50 ± 91% +310.7% 1759 ± 63% proc-vmstat.numa_pages_migrated
578.25 ± 37% +2906.7% 17386 ± 85% proc-vmstat.numa_pte_updates
253947 +13.1% 287290 proc-vmstat.pgfault
717071 +5.7% 758253 proc-vmstat.pgfree
428.50 ± 91% +310.7% 1759 ± 63% proc-vmstat.pgmigrate_success
294.76 ± 12% -18.2% 241.15 ± 8% sched_debug.cfs_rq:/.load_avg.stddev
18953 ± 23% -39.3% 11499 ± 22% sched_debug.cfs_rq:/.min_vruntime.avg
12508 ± 15% -50.6% 6174 ± 36% sched_debug.cfs_rq:/.min_vruntime.min
3003 ± 15% +22.6% 3681 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev
250.34 ± 11% -19.7% 200.99 ± 12% sched_debug.cfs_rq:/.removed.load_avg.stddev
3092 ± 23% -37.1% 1945 ± 24% sched_debug.cfs_rq:/.removed.runnable_sum.avg
11538 ± 11% -19.9% 9247 ± 12% sched_debug.cfs_rq:/.removed.runnable_sum.stddev
98.26 ± 5% -19.7% 78.89 ± 24% sched_debug.cfs_rq:/.removed.util_avg.stddev
1.86 ± 15% +26.4% 2.36 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.avg
4.90 ± 4% +9.1% 5.34 ± 4% sched_debug.cfs_rq:/.runnable_load_avg.stddev
2166 ± 16% +24.6% 2699 ± 13% sched_debug.cfs_rq:/.runnable_weight.avg
18058 ± 2% +15.6% 20869 ± 5% sched_debug.cfs_rq:/.runnable_weight.max
5486 ± 6% +9.5% 6009 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev
2997 ± 15% +23.0% 3687 ± 7% sched_debug.cfs_rq:/.spread0.stddev
256.16 ± 10% +16.0% 297.04 ± 5% sched_debug.cfs_rq:/.util_avg.stddev
65601 ± 8% -53.8% 30291 sched_debug.cpu.clock.avg
65604 ± 8% -53.8% 30295 sched_debug.cpu.clock.max
65598 ± 8% -53.8% 30285 sched_debug.cpu.clock.min
1.71 ± 3% +21.8% 2.09 ± 15% sched_debug.cpu.clock.stddev
65601 ± 8% -53.8% 30291 sched_debug.cpu.clock_task.avg
65604 ± 8% -53.8% 30295 sched_debug.cpu.clock_task.max
65598 ± 8% -53.8% 30285 sched_debug.cpu.clock_task.min
1.71 ± 3% +21.8% 2.09 ± 15% sched_debug.cpu.clock_task.stddev
8296 ± 5% -12.4% 7266 sched_debug.cpu.nr_load_updates.avg
6306 ± 9% -21.8% 4931 ± 9% sched_debug.cpu.nr_load_updates.min
339.25 ± 12% -29.8% 238.25 ± 8% sched_debug.cpu.nr_switches.min
1141 ± 7% +8.3% 1236 ± 5% sched_debug.cpu.nr_switches.stddev
4.91 ± 20% -30.4% 3.42 ± 14% sched_debug.cpu.nr_uninterruptible.stddev
65598 ± 8% -53.8% 30288 sched_debug.cpu_clk
62902 ± 8% -56.1% 27590 sched_debug.ktime
66171 ± 8% -53.4% 30858 sched_debug.sched_clk
23.30 ± 14% -36.4% 14.81 perf-stat.i.MPKI
1.299e+09 +29.9% 1.687e+09 perf-stat.i.branch-instructions
2.54 ± 9% -0.9 1.64 ± 2% perf-stat.i.branch-miss-rate%
24240691 ± 2% +8.8% 26384355 ± 2% perf-stat.i.branch-misses
32469998 +4.7% 34002480 perf-stat.i.cache-misses
1.184e+08 ± 2% +4.3% 1.236e+08 perf-stat.i.cache-references
574477 -21.6% 450402 perf-stat.i.context-switches
4.71 ± 2% -21.4% 3.70 perf-stat.i.cpi
3.026e+10 -1.3% 2.987e+10 perf-stat.i.cpu-cycles
14.78 ± 12% -13.5% 12.78 ± 6% perf-stat.i.cpu-migrations
1091 ± 3% +21.9% 1331 ± 8% perf-stat.i.cycles-between-cache-misses
0.10 ± 28% -0.1 0.01 ± 21% perf-stat.i.dTLB-load-miss-rate%
265935 ± 9% -55.1% 119534 ± 5% perf-stat.i.dTLB-load-misses
1.798e+09 +12.4% 2.021e+09 perf-stat.i.dTLB-loads
0.02 ± 23% -0.0 0.00 ± 13% perf-stat.i.dTLB-store-miss-rate%
33528 ± 5% -50.3% 16673 ± 2% perf-stat.i.dTLB-store-misses
5.94 ± 24% -4.5 1.47 ± 21% perf-stat.i.iTLB-load-miss-rate%
8419599 -13.3% 7296814 perf-stat.i.iTLB-loads
6.595e+09 +20.3% 7.936e+09 perf-stat.i.instructions
0.22 ± 3% +25.4% 0.28 perf-stat.i.ipc
2351 -0.9% 2329 perf-stat.i.minor-faults
10288903 ± 2% -5.6% 9715506 perf-stat.i.node-load-misses
765100 +11.5% 852708 ± 3% perf-stat.i.node-loads
3887667 -10.6% 3474607 perf-stat.i.node-store-misses
754448 ± 5% -15.2% 639938 ± 2% perf-stat.i.node-stores
2351 -0.9% 2329 perf-stat.i.page-faults
17.96 ± 3% -13.3% 15.57 perf-stat.overall.MPKI
1.87 -0.3 1.56 ± 2% perf-stat.overall.branch-miss-rate%
4.59 -18.0% 3.76 perf-stat.overall.cpi
932.33 -5.7% 878.73 perf-stat.overall.cycles-between-cache-misses
0.01 ± 10% -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate%
0.00 ± 6% -0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate%
67735 ± 6% +83.2% 124066 ± 24% perf-stat.overall.instructions-per-iTLB-miss
0.22 +21.9% 0.27 perf-stat.overall.ipc
93.08 -1.1 91.93 perf-stat.overall.node-load-miss-rate%
1.286e+09 +30.0% 1.672e+09 perf-stat.ps.branch-instructions
23999328 ± 2% +9.0% 26157341 ± 2% perf-stat.ps.branch-misses
32137766 +4.9% 33699623 perf-stat.ps.cache-misses
1.172e+08 ± 2% +4.5% 1.225e+08 perf-stat.ps.cache-references
568582 -21.5% 446385 perf-stat.ps.context-switches
2.996e+10 -1.2% 2.96e+10 perf-stat.ps.cpu-cycles
263523 ± 9% -55.0% 118523 ± 5% perf-stat.ps.dTLB-load-misses
1.78e+09 +12.5% 2.003e+09 perf-stat.ps.dTLB-loads
33223 ± 5% -50.2% 16536 ± 2% perf-stat.ps.dTLB-store-misses
8333338 -13.2% 7231920 perf-stat.ps.iTLB-loads
6.528e+09 +20.5% 7.866e+09 perf-stat.ps.instructions
2328 -0.8% 2310 perf-stat.ps.minor-faults
10183467 ± 2% -5.4% 9628920 perf-stat.ps.node-load-misses
757265 +11.6% 845120 ± 3% perf-stat.ps.node-loads
3847829 -10.5% 3443639 perf-stat.ps.node-store-misses
746711 ± 5% -15.1% 634243 ± 2% perf-stat.ps.node-stores
2328 -0.8% 2310 perf-stat.ps.page-faults
6.55e+11 +38.9% 9.097e+11 perf-stat.total.instructions
26715 ± 6% +15.3% 30804 ± 3% softirqs.CPU0.RCU
25813 ± 3% +17.5% 30331 ± 2% softirqs.CPU1.RCU
26114 ± 2% +13.8% 29715 ± 4% softirqs.CPU10.RCU
16814 ± 6% +17.4% 19743 ± 5% softirqs.CPU10.SCHED
38584 +23.0% 47474 ± 19% softirqs.CPU10.TIMER
25226 ± 6% +24.3% 31347 ± 2% softirqs.CPU11.RCU
15728 ± 4% +20.0% 18878 ± 2% softirqs.CPU11.SCHED
37200 ± 2% +27.9% 47567 ± 21% softirqs.CPU11.TIMER
26744 ± 8% +17.8% 31517 ± 4% softirqs.CPU12.RCU
25992 ± 2% +26.8% 32949 ± 4% softirqs.CPU13.RCU
16769 ± 4% +13.3% 18999 ± 2% softirqs.CPU13.SCHED
25183 +23.4% 31086 ± 6% softirqs.CPU14.RCU
37036 ± 2% +24.4% 46059 ± 23% softirqs.CPU14.TIMER
24448 ± 2% +21.7% 29750 ± 2% softirqs.CPU15.RCU
26602 ± 6% +12.5% 29919 softirqs.CPU16.RCU
37821 ± 3% +9.5% 41426 ± 2% softirqs.CPU16.TIMER
28288 ± 2% +19.4% 33785 ± 4% softirqs.CPU18.RCU
38075 ± 2% +23.4% 46999 ± 21% softirqs.CPU19.TIMER
26162 ± 2% +20.0% 31395 ± 2% softirqs.CPU2.RCU
16553 ± 4% +15.5% 19112 ± 5% softirqs.CPU2.SCHED
37904 ± 2% +18.5% 44898 ± 6% softirqs.CPU2.TIMER
27787 ± 6% +15.0% 31953 ± 6% softirqs.CPU20.RCU
16907 ± 6% +8.9% 18409 ± 5% softirqs.CPU20.SCHED
38188 ± 2% +11.0% 42371 ± 3% softirqs.CPU20.TIMER
26947 +14.9% 30957 ± 6% softirqs.CPU21.RCU
37668 +10.8% 41744 ± 2% softirqs.CPU21.TIMER
27984 ± 6% +17.3% 32821 ± 5% softirqs.CPU22.RCU
37415 ± 3% +10.9% 41475 softirqs.CPU22.TIMER
26636 ± 9% +18.8% 31634 ± 6% softirqs.CPU23.RCU
26525 ± 2% +22.9% 32606 ± 8% softirqs.CPU24.RCU
37040 ± 3% +13.0% 41870 ± 2% softirqs.CPU24.TIMER
26441 +22.3% 32342 softirqs.CPU25.RCU
37168 ± 4% +12.8% 41929 ± 3% softirqs.CPU25.TIMER
26599 ± 5% +21.0% 32182 ± 4% softirqs.CPU26.RCU
26510 ± 4% +20.6% 31982 ± 2% softirqs.CPU27.RCU
16559 ± 4% +13.4% 18775 ± 2% softirqs.CPU27.SCHED
37518 +10.7% 41515 ± 2% softirqs.CPU28.TIMER
25728 ± 10% +27.6% 32830 ± 8% softirqs.CPU29.RCU
36817 ± 2% +11.6% 41102 softirqs.CPU29.TIMER
26141 ± 2% +13.2% 29590 ± 2% softirqs.CPU3.RCU
17273 ± 4% +20.2% 20755 softirqs.CPU3.SCHED
26476 ± 3% +18.2% 31287 ± 3% softirqs.CPU30.RCU
15921 ± 4% +19.6% 19042 ± 10% softirqs.CPU30.SCHED
37032 +11.8% 41391 ± 2% softirqs.CPU30.TIMER
27534 ± 6% +14.7% 31581 ± 4% softirqs.CPU31.RCU
36721 +13.4% 41633 ± 2% softirqs.CPU31.TIMER
25118 ± 3% +23.7% 31072 ± 3% softirqs.CPU32.RCU
15548 ± 2% +16.9% 18182 ± 6% softirqs.CPU33.SCHED
36529 ± 3% +12.9% 41256 softirqs.CPU33.TIMER
26802 ± 7% +15.8% 31044 ± 3% softirqs.CPU34.RCU
16424 ± 3% +12.1% 18409 ± 6% softirqs.CPU34.SCHED
27024 ± 6% +20.6% 32603 ± 2% softirqs.CPU35.RCU
37345 +11.2% 41546 softirqs.CPU35.TIMER
28719 ± 4% +12.8% 32400 ± 2% softirqs.CPU36.RCU
36406 ± 2% +9.0% 39681 softirqs.CPU36.TIMER
30989 ± 7% +23.5% 38266 softirqs.CPU38.RCU
15275 ± 3% +8.6% 16592 softirqs.CPU38.SCHED
28633 +18.3% 33883 softirqs.CPU39.RCU
26313 +17.6% 30942 softirqs.CPU4.RCU
37319 ± 3% +14.1% 42577 softirqs.CPU4.TIMER
28360 ± 3% +18.3% 33549 softirqs.CPU40.RCU
26803 ± 3% +22.6% 32857 ± 3% softirqs.CPU41.RCU
27921 ± 2% +21.5% 33914 ± 5% softirqs.CPU42.RCU
27840 ± 2% +23.2% 34308 softirqs.CPU43.RCU
36332 +9.6% 39810 ± 3% softirqs.CPU43.TIMER
27988 ± 2% +23.6% 34585 softirqs.CPU44.RCU
26964 ± 4% +23.0% 33161 ± 2% softirqs.CPU45.RCU
26647 ± 4% +19.6% 31866 ± 5% softirqs.CPU46.RCU
26733 ± 6% +27.6% 34120 ± 2% softirqs.CPU48.RCU
26780 ± 5% +26.4% 33860 ± 2% softirqs.CPU49.RCU
24665 ± 8% +24.2% 30646 softirqs.CPU5.RCU
15335 ± 4% +20.8% 18532 ± 5% softirqs.CPU5.SCHED
36270 ± 2% +12.6% 40855 ± 2% softirqs.CPU5.TIMER
26971 ± 2% +14.9% 30992 ± 6% softirqs.CPU50.RCU
27424 ± 2% +21.4% 33300 ± 2% softirqs.CPU51.RCU
15504 ± 4% +9.0% 16902 ± 2% softirqs.CPU51.SCHED
27430 +19.0% 32642 ± 6% softirqs.CPU52.RCU
27558 +19.3% 32887 ± 2% softirqs.CPU53.RCU
24817 ± 6% +24.4% 30881 ± 2% softirqs.CPU54.RCU
26529 ± 2% +13.3% 30048 ± 4% softirqs.CPU55.RCU
15454 ± 4% +16.1% 17936 ± 5% softirqs.CPU55.SCHED
25606 ± 8% +25.1% 32040 softirqs.CPU56.RCU
26139 ± 4% +24.1% 32437 ± 8% softirqs.CPU57.RCU
14558 ± 3% +19.2% 17360 ± 3% softirqs.CPU57.SCHED
26083 +21.6% 31717 ± 3% softirqs.CPU58.RCU
15101 ± 3% +13.3% 17105 ± 4% softirqs.CPU58.SCHED
14837 ± 6% +11.9% 16598 ± 6% softirqs.CPU59.SCHED
35970 ± 3% +12.0% 40295 softirqs.CPU59.TIMER
26271 +16.8% 30681 ± 2% softirqs.CPU6.RCU
15985 ± 6% +18.3% 18914 ± 8% softirqs.CPU6.SCHED
37813 ± 2% +10.9% 41946 softirqs.CPU6.TIMER
24423 ± 3% +24.7% 30453 ± 6% softirqs.CPU60.RCU
14485 ± 2% +20.9% 17519 ± 4% softirqs.CPU60.SCHED
36185 ± 2% +11.9% 40494 softirqs.CPU60.TIMER
23959 ± 2% +24.8% 29893 softirqs.CPU61.RCU
14903 ± 6% +12.5% 16767 ± 3% softirqs.CPU61.SCHED
24809 ± 2% +18.0% 29264 softirqs.CPU62.RCU
14321 ± 2% +18.1% 16916 ± 2% softirqs.CPU62.SCHED
36576 +9.0% 39877 softirqs.CPU62.TIMER
23487 ± 3% +25.9% 29568 ± 2% softirqs.CPU63.RCU
13582 ± 2% +25.3% 17014 ± 3% softirqs.CPU63.SCHED
35911 +12.2% 40280 softirqs.CPU63.TIMER
25668 ± 4% +18.4% 30390 ± 3% softirqs.CPU64.RCU
14666 ± 4% +13.6% 16660 ± 5% softirqs.CPU64.SCHED
24078 +29.4% 31156 ± 5% softirqs.CPU65.RCU
14177 ± 8% +20.5% 17090 ± 2% softirqs.CPU65.SCHED
35962 ± 3% +13.1% 40661 softirqs.CPU65.TIMER
23043 ± 7% +31.2% 30227 ± 8% softirqs.CPU66.RCU
36661 +10.5% 40517 ± 2% softirqs.CPU66.TIMER
25069 ± 4% +14.8% 28791 ± 2% softirqs.CPU67.RCU
13930 ± 3% +24.3% 17313 ± 6% softirqs.CPU67.SCHED
35788 +13.1% 40494 softirqs.CPU67.TIMER
23620 ± 3% +20.5% 28463 softirqs.CPU68.RCU
13693 ± 4% +26.3% 17294 ± 5% softirqs.CPU68.SCHED
36756 +11.4% 40930 softirqs.CPU68.TIMER
24380 ± 2% +19.4% 29104 ± 6% softirqs.CPU69.RCU
14512 ± 7% +23.9% 17978 ± 8% softirqs.CPU69.SCHED
25915 +18.1% 30605 softirqs.CPU7.RCU
15855 ± 2% +16.6% 18490 softirqs.CPU7.SCHED
36892 ± 2% +12.5% 41512 ± 2% softirqs.CPU7.TIMER
24291 +20.2% 29207 softirqs.CPU70.RCU
14647 ± 7% +20.2% 17609 softirqs.CPU70.SCHED
36029 ± 2% +12.8% 40633 softirqs.CPU70.TIMER
24761 ± 6% +19.6% 29619 softirqs.CPU71.RCU
36389 +10.2% 40114 softirqs.CPU71.TIMER
26860 ± 2% +13.7% 30535 ± 4% softirqs.CPU8.RCU
16226 ± 2% +20.1% 19492 ± 4% softirqs.CPU8.SCHED
37997 +11.7% 42458 ± 2% softirqs.CPU8.TIMER
25499 ± 2% +17.6% 29981 softirqs.CPU9.RCU
17382 ± 8% +11.0% 19298 ± 6% softirqs.CPU9.SCHED
36979 +9.1% 40348 softirqs.CPU9.TIMER
1901918 +19.3% 2268164 softirqs.RCU
1148813 +10.9% 1273763 softirqs.SCHED
2693395 +10.3% 2970253 softirqs.TIMER
93123 ± 2% +20.6% 112353 interrupts.CAL:Function_call_interrupts
1307 +21.0% 1581 interrupts.CPU0.CAL:Function_call_interrupts
196590 ± 3% +18.0% 231965 interrupts.CPU0.LOC:Local_timer_interrupts
1982 ± 18% -47.7% 1037 ± 54% interrupts.CPU0.NMI:Non-maskable_interrupts
1982 ± 18% -47.7% 1037 ± 54% interrupts.CPU0.PMI:Performance_monitoring_interrupts
1311 ± 2% +20.8% 1584 interrupts.CPU1.CAL:Function_call_interrupts
196507 ± 3% +17.9% 231706 interrupts.CPU1.LOC:Local_timer_interrupts
1315 ± 2% +20.3% 1581 interrupts.CPU10.CAL:Function_call_interrupts
196466 ± 2% +17.7% 231175 interrupts.CPU10.LOC:Local_timer_interrupts
1278 ± 6% +25.0% 1598 ± 2% interrupts.CPU11.CAL:Function_call_interrupts
196467 ± 2% +18.1% 232064 interrupts.CPU11.LOC:Local_timer_interrupts
2906 ± 34% +75.3% 5096 ± 7% interrupts.CPU11.RES:Rescheduling_interrupts
1312 ± 2% +16.6% 1530 ± 4% interrupts.CPU12.CAL:Function_call_interrupts
196661 ± 3% +17.9% 231954 interrupts.CPU12.LOC:Local_timer_interrupts
196407 ± 3% +17.7% 231133 interrupts.CPU13.LOC:Local_timer_interrupts
3966 ± 15% +32.6% 5258 ± 7% interrupts.CPU13.RES:Rescheduling_interrupts
196446 ± 3% +18.0% 231766 interrupts.CPU14.LOC:Local_timer_interrupts
196307 ± 3% +18.0% 231649 interrupts.CPU15.LOC:Local_timer_interrupts
1311 ± 2% +18.0% 1547 ± 3% interrupts.CPU16.CAL:Function_call_interrupts
196278 ± 3% +17.7% 230984 interrupts.CPU16.LOC:Local_timer_interrupts
2535 ± 36% -78.5% 545.25 ± 51% interrupts.CPU16.NMI:Non-maskable_interrupts
2535 ± 36% -78.5% 545.25 ± 51% interrupts.CPU16.PMI:Performance_monitoring_interrupts
1220 ± 11% +23.2% 1502 ± 6% interrupts.CPU17.CAL:Function_call_interrupts
196493 ± 3% +17.5% 230974 interrupts.CPU17.LOC:Local_timer_interrupts
1306 +21.1% 1582 interrupts.CPU18.CAL:Function_call_interrupts
196318 ± 2% +18.0% 231690 interrupts.CPU18.LOC:Local_timer_interrupts
1329 +18.7% 1578 interrupts.CPU19.CAL:Function_call_interrupts
197092 ± 2% +17.7% 231886 interrupts.CPU19.LOC:Local_timer_interrupts
1310 ± 2% +20.4% 1577 interrupts.CPU2.CAL:Function_call_interrupts
196499 ± 3% +17.8% 231564 interrupts.CPU2.LOC:Local_timer_interrupts
1328 ± 4% +20.5% 1601 ± 3% interrupts.CPU20.CAL:Function_call_interrupts
197064 ± 2% +17.6% 231838 interrupts.CPU20.LOC:Local_timer_interrupts
1311 ± 2% +20.4% 1578 interrupts.CPU21.CAL:Function_call_interrupts
196901 ± 2% +17.7% 231704 interrupts.CPU21.LOC:Local_timer_interrupts
1313 ± 2% +20.6% 1583 interrupts.CPU22.CAL:Function_call_interrupts
196919 ± 2% +17.5% 231471 interrupts.CPU22.LOC:Local_timer_interrupts
1309 ± 2% +17.7% 1540 ± 5% interrupts.CPU23.CAL:Function_call_interrupts
196812 ± 2% +17.7% 231745 interrupts.CPU23.LOC:Local_timer_interrupts
1311 ± 2% +20.6% 1582 interrupts.CPU24.CAL:Function_call_interrupts
196729 ± 2% +17.9% 231969 interrupts.CPU24.LOC:Local_timer_interrupts
1311 ± 2% +20.4% 1579 interrupts.CPU25.CAL:Function_call_interrupts
196696 ± 2% +17.8% 231715 interrupts.CPU25.LOC:Local_timer_interrupts
1152 ± 74% +184.9% 3281 ± 11% interrupts.CPU25.NMI:Non-maskable_interrupts
1152 ± 74% +184.9% 3281 ± 11% interrupts.CPU25.PMI:Performance_monitoring_interrupts
1311 ± 2% +20.6% 1581 interrupts.CPU26.CAL:Function_call_interrupts
197080 ± 2% +17.6% 231696 interrupts.CPU26.LOC:Local_timer_interrupts
1909 ± 42% +67.9% 3207 ± 10% interrupts.CPU26.NMI:Non-maskable_interrupts
1909 ± 42% +67.9% 3207 ± 10% interrupts.CPU26.PMI:Performance_monitoring_interrupts
1310 ± 2% +20.6% 1580 interrupts.CPU27.CAL:Function_call_interrupts
196958 ± 2% +17.6% 231702 interrupts.CPU27.LOC:Local_timer_interrupts
1309 ± 2% +20.7% 1580 interrupts.CPU28.CAL:Function_call_interrupts
196575 ± 2% +17.9% 231714 interrupts.CPU28.LOC:Local_timer_interrupts
1308 ± 2% +20.9% 1582 interrupts.CPU29.CAL:Function_call_interrupts
196943 ± 2% +17.3% 230984 interrupts.CPU29.LOC:Local_timer_interrupts
1310 ± 2% +21.3% 1588 interrupts.CPU3.CAL:Function_call_interrupts
196420 ± 2% +16.9% 229682 interrupts.CPU3.LOC:Local_timer_interrupts
2937 ± 21% -39.9% 1766 ± 23% interrupts.CPU3.NMI:Non-maskable_interrupts
2937 ± 21% -39.9% 1766 ± 23% interrupts.CPU3.PMI:Performance_monitoring_interrupts
4820 ± 15% +88.0% 9063 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts
1310 ± 2% +20.8% 1582 interrupts.CPU30.CAL:Function_call_interrupts
196831 ± 2% +17.7% 231691 interrupts.CPU30.LOC:Local_timer_interrupts
1209 ±113% +149.6% 3018 ± 34% interrupts.CPU30.NMI:Non-maskable_interrupts
1209 ±113% +149.6% 3018 ± 34% interrupts.CPU30.PMI:Performance_monitoring_interrupts
1304 ± 2% +23.0% 1604 interrupts.CPU31.CAL:Function_call_interrupts
196479 ± 2% +18.0% 231820 interrupts.CPU31.LOC:Local_timer_interrupts
1419 ± 73% +81.5% 2575 ± 26% interrupts.CPU31.NMI:Non-maskable_interrupts
1419 ± 73% +81.5% 2575 ± 26% interrupts.CPU31.PMI:Performance_monitoring_interrupts
1309 ± 2% +20.9% 1583 interrupts.CPU32.CAL:Function_call_interrupts
196861 ± 2% +17.7% 231732 interrupts.CPU32.LOC:Local_timer_interrupts
1159 ± 23% +36.6% 1583 interrupts.CPU33.CAL:Function_call_interrupts
196346 ± 2% +18.0% 231694 interrupts.CPU33.LOC:Local_timer_interrupts
1310 ± 2% +20.8% 1583 interrupts.CPU34.CAL:Function_call_interrupts
195339 +18.6% 231743 interrupts.CPU34.LOC:Local_timer_interrupts
1284 ± 2% +21.7% 1563 interrupts.CPU35.CAL:Function_call_interrupts
195354 +18.6% 231781 interrupts.CPU35.LOC:Local_timer_interrupts
1064 ± 25% +45.6% 1549 ± 4% interrupts.CPU36.CAL:Function_call_interrupts
196631 ± 2% +17.6% 231216 interrupts.CPU36.LOC:Local_timer_interrupts
2113 ± 33% -64.2% 756.50 ± 29% interrupts.CPU36.NMI:Non-maskable_interrupts
2113 ± 33% -64.2% 756.50 ± 29% interrupts.CPU36.PMI:Performance_monitoring_interrupts
1203 ± 9% +29.8% 1561 ± 2% interrupts.CPU37.CAL:Function_call_interrupts
196745 ± 3% +17.6% 231316 interrupts.CPU37.LOC:Local_timer_interrupts
1302 ± 2% +21.4% 1580 interrupts.CPU38.CAL:Function_call_interrupts
196339 ± 3% +17.6% 230859 interrupts.CPU38.LOC:Local_timer_interrupts
1290 ± 3% +22.4% 1580 interrupts.CPU39.CAL:Function_call_interrupts
196222 ± 3% +16.9% 229415 interrupts.CPU39.LOC:Local_timer_interrupts
1302 +19.1% 1551 ± 4% interrupts.CPU4.CAL:Function_call_interrupts
196114 ± 3% +18.2% 231711 interrupts.CPU4.LOC:Local_timer_interrupts
1282 ± 2% +23.2% 1579 interrupts.CPU40.CAL:Function_call_interrupts
196347 ± 3% +17.2% 230205 interrupts.CPU40.LOC:Local_timer_interrupts
1295 +21.9% 1579 interrupts.CPU41.CAL:Function_call_interrupts
196386 ± 2% +17.6% 230956 interrupts.CPU41.LOC:Local_timer_interrupts
2528 ± 16% -60.4% 1001 ± 26% interrupts.CPU41.NMI:Non-maskable_interrupts
2528 ± 16% -60.4% 1001 ± 26% interrupts.CPU41.PMI:Performance_monitoring_interrupts
5041 ± 17% -55.2% 2256 ± 35% interrupts.CPU41.RES:Rescheduling_interrupts
196437 ± 2% +17.7% 231202 interrupts.CPU42.LOC:Local_timer_interrupts
2533 ± 30% -65.8% 867.50 ± 67% interrupts.CPU42.NMI:Non-maskable_interrupts
2533 ± 30% -65.8% 867.50 ± 67% interrupts.CPU42.PMI:Performance_monitoring_interrupts
1287 ± 4% +22.8% 1580 interrupts.CPU43.CAL:Function_call_interrupts
196441 ± 2% +17.5% 230898 interrupts.CPU43.LOC:Local_timer_interrupts
1184 ± 15% +33.3% 1578 interrupts.CPU44.CAL:Function_call_interrupts
196438 ± 2% +18.0% 231740 interrupts.CPU44.LOC:Local_timer_interrupts
2587 ± 38% -63.2% 953.00 ± 57% interrupts.CPU44.NMI:Non-maskable_interrupts
2587 ± 38% -63.2% 953.00 ± 57% interrupts.CPU44.PMI:Performance_monitoring_interrupts
1307 ± 2% +20.7% 1579 interrupts.CPU45.CAL:Function_call_interrupts
196461 ± 2% +17.2% 230176 interrupts.CPU45.LOC:Local_timer_interrupts
1222 ± 10% +29.1% 1578 interrupts.CPU46.CAL:Function_call_interrupts
196498 ± 2% +17.5% 230967 interrupts.CPU46.LOC:Local_timer_interrupts
2619 ± 33% -63.7% 951.50 ±106% interrupts.CPU46.NMI:Non-maskable_interrupts
2619 ± 33% -63.7% 951.50 ±106% interrupts.CPU46.PMI:Performance_monitoring_interrupts
1305 ± 2% +21.0% 1580 interrupts.CPU47.CAL:Function_call_interrupts
196734 ± 3% +17.4% 230968 interrupts.CPU47.LOC:Local_timer_interrupts
1325 ± 2% +19.0% 1577 interrupts.CPU48.CAL:Function_call_interrupts
196228 ± 3% +18.1% 231730 interrupts.CPU48.LOC:Local_timer_interrupts
1331 ± 3% +18.5% 1578 interrupts.CPU49.CAL:Function_call_interrupts
196553 ± 3% +17.5% 230948 interrupts.CPU49.LOC:Local_timer_interrupts
1306 ± 2% +21.5% 1586 interrupts.CPU5.CAL:Function_call_interrupts
196436 ± 2% +17.6% 230955 interrupts.CPU5.LOC:Local_timer_interrupts
2656 ± 16% -52.5% 1260 ± 56% interrupts.CPU5.NMI:Non-maskable_interrupts
2656 ± 16% -52.5% 1260 ± 56% interrupts.CPU5.PMI:Performance_monitoring_interrupts
1307 ± 2% +20.7% 1578 interrupts.CPU50.CAL:Function_call_interrupts
196306 ± 3% +17.7% 231081 interrupts.CPU50.LOC:Local_timer_interrupts
1312 +20.3% 1578 interrupts.CPU51.CAL:Function_call_interrupts
196454 ± 2% +18.0% 231823 interrupts.CPU51.LOC:Local_timer_interrupts
1316 ± 2% +19.9% 1578 interrupts.CPU52.CAL:Function_call_interrupts
196419 ± 2% +17.7% 231185 interrupts.CPU52.LOC:Local_timer_interrupts
2242 ± 23% -74.4% 573.25 ± 57% interrupts.CPU52.NMI:Non-maskable_interrupts
2242 ± 23% -74.4% 573.25 ± 57% interrupts.CPU52.PMI:Performance_monitoring_interrupts
1308 +17.3% 1534 ± 3% interrupts.CPU53.CAL:Function_call_interrupts
196425 ± 2% +17.9% 231631 interrupts.CPU53.LOC:Local_timer_interrupts
1227 ± 13% +28.4% 1575 interrupts.CPU54.CAL:Function_call_interrupts
196179 ± 2% +18.0% 231486 interrupts.CPU54.LOC:Local_timer_interrupts
1288 ± 5% +19.5% 1539 ± 5% interrupts.CPU55.CAL:Function_call_interrupts
196633 ± 2% +17.8% 231726 interrupts.CPU55.LOC:Local_timer_interrupts
1308 ± 2% +21.0% 1582 interrupts.CPU56.CAL:Function_call_interrupts
196902 ± 2% +17.7% 231729 interrupts.CPU56.LOC:Local_timer_interrupts
196827 ± 2% +17.8% 231949 interrupts.CPU57.LOC:Local_timer_interrupts
2311 ± 41% +73.9% 4018 ± 29% interrupts.CPU57.RES:Rescheduling_interrupts
1314 ± 2% +18.3% 1554 interrupts.CPU58.CAL:Function_call_interrupts
196927 ± 2% +17.7% 231731 interrupts.CPU58.LOC:Local_timer_interrupts
1310 ± 2% +20.7% 1580 interrupts.CPU59.CAL:Function_call_interrupts
196866 ± 2% +17.7% 231802 interrupts.CPU59.LOC:Local_timer_interrupts
1311 ± 2% +20.9% 1585 interrupts.CPU6.CAL:Function_call_interrupts
196416 ± 2% +17.6% 230968 interrupts.CPU6.LOC:Local_timer_interrupts
2423 ± 31% -61.9% 922.25 ± 88% interrupts.CPU6.NMI:Non-maskable_interrupts
2423 ± 31% -61.9% 922.25 ± 88% interrupts.CPU6.PMI:Performance_monitoring_interrupts
1314 ± 2% +20.3% 1580 interrupts.CPU60.CAL:Function_call_interrupts
196951 ± 2% +17.7% 231714 interrupts.CPU60.LOC:Local_timer_interrupts
196924 ± 2% +17.7% 231716 interrupts.CPU61.LOC:Local_timer_interrupts
953.50 ± 81% +189.0% 2755 ± 31% interrupts.CPU61.NMI:Non-maskable_interrupts
953.50 ± 81% +189.0% 2755 ± 31% interrupts.CPU61.PMI:Performance_monitoring_interrupts
1314 ± 2% +19.9% 1576 interrupts.CPU62.CAL:Function_call_interrupts
196678 ± 2% +17.8% 231687 interrupts.CPU62.LOC:Local_timer_interrupts
1277 ± 5% +23.6% 1577 interrupts.CPU63.CAL:Function_call_interrupts
196543 ± 2% +17.9% 231706 interrupts.CPU63.LOC:Local_timer_interrupts
1869 ± 43% +108.0% 3888 ± 28% interrupts.CPU63.RES:Rescheduling_interrupts
1310 ± 2% +20.6% 1580 interrupts.CPU64.CAL:Function_call_interrupts
197006 ± 2% +17.6% 231724 interrupts.CPU64.LOC:Local_timer_interrupts
1312 ± 2% +20.1% 1575 interrupts.CPU65.CAL:Function_call_interrupts
196073 ± 2% +18.2% 231756 interrupts.CPU65.LOC:Local_timer_interrupts
1143 ± 95% +147.4% 2828 ± 24% interrupts.CPU65.NMI:Non-maskable_interrupts
1143 ± 95% +147.4% 2828 ± 24% interrupts.CPU65.PMI:Performance_monitoring_interrupts
1311 ± 2% +17.4% 1540 ± 4% interrupts.CPU66.CAL:Function_call_interrupts
196845 ± 2% +17.8% 231921 interrupts.CPU66.LOC:Local_timer_interrupts
1311 ± 2% +20.5% 1580 interrupts.CPU67.CAL:Function_call_interrupts
195708 ± 2% +18.4% 231743 interrupts.CPU67.LOC:Local_timer_interrupts
1310 ± 2% +20.7% 1581 interrupts.CPU68.CAL:Function_call_interrupts
196837 ± 2% +17.8% 231795 interrupts.CPU68.LOC:Local_timer_interrupts
1516 ± 59% +137.0% 3593 ± 26% interrupts.CPU68.RES:Rescheduling_interrupts
1311 ± 2% +20.5% 1581 interrupts.CPU69.CAL:Function_call_interrupts
196838 ± 2% +17.8% 231870 interrupts.CPU69.LOC:Local_timer_interrupts
1306 ± 2% +21.5% 1586 interrupts.CPU7.CAL:Function_call_interrupts
196395 ± 2% +17.7% 231082 interrupts.CPU7.LOC:Local_timer_interrupts
3299 ± 36% +68.7% 5567 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts
1310 +20.7% 1581 interrupts.CPU70.CAL:Function_call_interrupts
195366 +18.6% 231713 interrupts.CPU70.LOC:Local_timer_interrupts
1308 +18.1% 1545 ± 3% interrupts.CPU71.CAL:Function_call_interrupts
196019 ± 2% +18.2% 231692 interrupts.CPU71.LOC:Local_timer_interrupts
1280 ± 3% +23.9% 1586 interrupts.CPU8.CAL:Function_call_interrupts
196421 ± 2% +18.0% 231850 interrupts.CPU8.LOC:Local_timer_interrupts
2724 ± 33% -58.3% 1137 ± 62% interrupts.CPU8.NMI:Non-maskable_interrupts
2724 ± 33% -58.3% 1137 ± 62% interrupts.CPU8.PMI:Performance_monitoring_interrupts
3448 ± 10% +81.5% 6257 ± 16% interrupts.CPU8.RES:Rescheduling_interrupts
1300 +18.6% 1541 ± 5% interrupts.CPU9.CAL:Function_call_interrupts
196415 ± 2% +17.2% 230179 interrupts.CPU9.LOC:Local_timer_interrupts
14149246 ± 2% +17.8% 16663738 interrupts.LOC:Local_timer_interrupts
5.72 ± 6% -1.0 4.73 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
5.62 ± 7% -1.0 4.65 ± 6% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node
1.32 ± 36% -0.9 0.41 ± 57% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
13.67 -0.8 12.82 ± 5% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe
5.44 -0.6 4.88 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
5.43 -0.6 4.87 ± 6% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode
4.80 ± 3% -0.5 4.26 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
4.78 ± 3% -0.5 4.24 ± 6% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
2.84 ± 9% -0.5 2.33 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
2.78 ± 9% -0.5 2.29 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node
4.64 -0.4 4.21 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents
4.62 -0.4 4.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks
2.22 ± 3% -0.4 1.80 ± 7% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
1.51 ± 3% -0.4 1.12 ± 26% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
0.62 ± 4% -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
0.69 ± 6% -0.3 0.40 ± 57% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
2.47 -0.3 2.21 ± 4% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
2.48 -0.3 2.22 ± 4% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents
0.67 ± 4% -0.2 0.44 ± 58% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode.btrfs_log_inode_parent
0.65 ± 4% -0.2 0.42 ± 58% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode
0.66 ± 5% -0.2 0.44 ± 57% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
0.82 ± 7% -0.2 0.62 ± 8% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
1.44 -0.2 1.25 ± 6% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
0.82 ± 7% -0.2 0.62 ± 8% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work
1.93 ± 2% -0.2 1.75 ± 7% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
0.71 ± 6% -0.2 0.54 ± 3% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
1.53 ± 2% -0.1 1.40 ± 4% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
1.53 ± 2% -0.1 1.40 ± 4% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
1.55 ± 2% -0.1 1.42 ± 4% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
1.54 ± 2% -0.1 1.41 ± 4% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
1.54 ± 2% -0.1 1.41 ± 4% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.86 ± 3% -0.1 0.74 ± 4% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
0.67 ± 9% -0.1 0.55 ± 7% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
0.67 ± 9% -0.1 0.55 ± 7% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
1.03 ± 5% -0.1 0.91 ± 7% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
0.85 ± 3% -0.1 0.73 ± 4% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
0.75 ± 2% -0.1 0.64 perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
1.24 ± 5% +0.1 1.39 ± 2% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file
1.38 ± 5% +0.2 1.56 ± 4% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
1.59 ± 6% +0.2 1.80 ± 4% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.76 ± 7% +0.2 0.99 ± 4% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
0.83 ± 3% +0.2 1.08 ± 4% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
0.61 ± 8% +0.3 0.88 ± 5% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
0.59 ± 8% +0.3 0.85 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent
1.78 ± 7% +0.3 2.06 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
0.68 ± 3% +0.3 0.96 ± 4% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
0.66 ± 3% +0.3 0.93 ± 4% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents
0.55 ± 2% +0.3 0.88 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items
2.89 ± 3% +0.4 3.25 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
0.74 +0.4 1.13 ± 2% perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
0.71 ± 2% +0.4 1.12 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
0.27 ±100% +0.4 0.71 ± 14% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
0.39 ± 57% +0.5 0.85 ± 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot
0.94 ± 8% +0.5 1.48 ± 14% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
0.13 ±173% +0.6 0.73 ± 11% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
0.00 +0.6 0.64 ± 11% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
4.37 +0.6 5.01 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
0.00 +0.7 0.67 ± 11% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
0.00 +0.7 0.68 ± 35% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe
0.00 +0.7 0.72 ± 34% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert
1.56 ± 7% +0.7 2.30 ± 13% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
1.69 ± 7% +0.8 2.47 ± 13% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
4.09 ± 5% +0.8 4.90 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node
0.00 +0.8 0.81 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up
0.00 +1.0 0.97 ± 12% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
0.00 +1.0 1.00 ± 21% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents
0.00 +1.1 1.09 ± 11% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
0.00 +1.2 1.16 ± 11% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
0.00 +1.2 1.16 ± 11% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
0.00 +1.3 1.33 ± 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
48.26 +3.1 51.38 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
48.26 +3.1 51.38 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
48.24 +3.1 51.36 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
38.83 -2.2 36.66 ± 4% perf-profile.children.cycles-pp.btrfs_search_slot
22.19 -1.8 20.40 ± 4% perf-profile.children.cycles-pp.btrfs_read_lock_root_node
22.00 -1.8 20.24 ± 4% perf-profile.children.cycles-pp.btrfs_tree_read_lock
10.23 ± 3% -1.2 8.99 ± 5% perf-profile.children.cycles-pp.finish_wait
14.49 -1.0 13.45 ± 5% perf-profile.children.cycles-pp.btrfs_csum_file_blocks
16.59 -0.9 15.74 ± 4% perf-profile.children.cycles-pp.btrfs_insert_empty_items
2.97 ± 5% -0.7 2.29 ± 7% perf-profile.children.cycles-pp.queued_read_lock_slowpath
5.68 -0.6 5.06 ± 5% perf-profile.children.cycles-pp.btrfs_lookup_csum
2.98 ± 4% -0.5 2.45 ± 6% perf-profile.children.cycles-pp.queued_write_lock_slowpath
1.79 ± 7% -0.4 1.37 ± 3% perf-profile.children.cycles-pp.__schedule
1.14 ± 7% -0.3 0.86 ± 2% perf-profile.children.cycles-pp.schedule
0.43 ± 14% -0.2 0.18 ± 8% perf-profile.children.cycles-pp.poll_idle
2.35 ± 2% -0.2 2.10 ± 3% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
2.33 ± 2% -0.2 2.09 ± 3% perf-profile.children.cycles-pp.do_writepages
2.16 -0.2 1.92 ± 5% perf-profile.children.cycles-pp._raw_spin_lock
0.82 ± 7% -0.2 0.62 ± 8% perf-profile.children.cycles-pp.add_pending_csums
1.12 ± 3% -0.2 0.95 ± 3% perf-profile.children.cycles-pp.btrfs_cow_block
1.11 ± 3% -0.2 0.94 ± 3% perf-profile.children.cycles-pp.__btrfs_cow_block
0.71 ± 6% -0.2 0.55 ± 3% perf-profile.children.cycles-pp.schedule_idle
1.59 ± 2% -0.1 1.45 ± 4% perf-profile.children.cycles-pp.extent_writepages
1.55 ± 2% -0.1 1.42 ± 5% perf-profile.children.cycles-pp.btrfs_fdatawrite_range
1.55 ± 2% -0.1 1.42 ± 4% perf-profile.children.cycles-pp.start_ordered_ops
0.68 ± 9% -0.1 0.55 ± 7% perf-profile.children.cycles-pp.btrfs_run_delayed_refs
0.68 ± 9% -0.1 0.55 ± 7% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs
1.05 ± 5% -0.1 0.94 ± 7% perf-profile.children.cycles-pp.extent_write_cache_pages
0.53 ± 9% -0.1 0.41 perf-profile.children.cycles-pp.dequeue_task_fair
0.75 ± 2% -0.1 0.64 perf-profile.children.cycles-pp.btrfs_write_marked_extents
0.50 ± 4% -0.1 0.39 ± 2% perf-profile.children.cycles-pp.read_block_for_search
0.74 ± 2% -0.1 0.63 ± 2% perf-profile.children.cycles-pp.btree_write_cache_pages
0.54 ± 3% -0.1 0.43 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32
0.47 ± 8% -0.1 0.38 perf-profile.children.cycles-pp.dequeue_entity
1.02 ± 2% -0.1 0.93 ± 3% perf-profile.children.cycles-pp.flush_write_bio
1.02 ± 2% -0.1 0.93 ± 3% perf-profile.children.cycles-pp.submit_one_bio
0.54 ± 5% -0.1 0.45 ± 5% perf-profile.children.cycles-pp.btrfs_del_items
0.97 ± 2% -0.1 0.89 ± 3% perf-profile.children.cycles-pp.btrfs_map_bio
0.93 ± 3% -0.1 0.84 ± 2% perf-profile.children.cycles-pp.submit_bio
0.93 ± 3% -0.1 0.84 ± 2% perf-profile.children.cycles-pp.generic_make_request
0.38 ± 3% -0.1 0.29 ± 9% perf-profile.children.cycles-pp.__list_add_valid
0.56 ± 2% -0.1 0.47 ± 4% perf-profile.children.cycles-pp.brd_do_bvec
0.40 ± 2% -0.1 0.32 ± 3% perf-profile.children.cycles-pp.__list_del_entry_valid
0.34 ± 10% -0.1 0.26 ± 7% perf-profile.children.cycles-pp.update_load_avg
0.49 ± 2% -0.1 0.41 ± 4% perf-profile.children.cycles-pp.btree_submit_bio_hook
0.40 ± 6% -0.1 0.32 ± 4% perf-profile.children.cycles-pp._raw_read_lock
0.33 ± 4% -0.1 0.26 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_unlock
0.82 ± 3% -0.1 0.75 ± 3% perf-profile.children.cycles-pp.brd_make_request
0.31 ± 8% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.btrfs_set_token_32
0.38 ± 7% -0.1 0.31 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair
0.33 ± 4% -0.1 0.27 ± 4% perf-profile.children.cycles-pp.find_extent_buffer
0.35 ± 4% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair
0.37 ± 3% -0.1 0.31 ± 2% perf-profile.children.cycles-pp.copy_page
0.29 ± 4% -0.1 0.23 ± 6% perf-profile.children.cycles-pp.memmove
0.37 ± 3% -0.1 0.31 ± 2% perf-profile.children.cycles-pp.copy_extent_buffer_full
0.31 ± 3% -0.1 0.25 ± 5% perf-profile.children.cycles-pp.__set_extent_bit
0.19 ± 10% -0.1 0.13 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.42 ± 5% -0.1 0.37 ± 5% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush
0.33 ± 4% -0.1 0.28 ± 4% perf-profile.children.cycles-pp.generic_bin_search
0.42 ± 5% -0.1 0.37 ± 5% perf-profile.children.cycles-pp.btrfs_alloc_tree_block
0.23 ± 4% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.memmove_extent_buffer
0.12 ± 11% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.finish_task_switch
0.20 ± 6% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty
0.22 ± 5% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.btrfs_lookup_csums_range
0.17 ± 11% -0.0 0.12 ± 8% perf-profile.children.cycles-pp.update_rq_clock
0.21 ± 4% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.alloc_extent_buffer
0.20 ± 5% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.btrfs_tree_unlock
0.12 ± 8% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.brd_insert_page
0.22 ± 3% -0.0 0.18 ± 3% perf-profile.children.cycles-pp.update_curr
0.14 ± 10% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.set_extent_buffer_dirty
0.22 ± 5% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.map_private_extent_buffer
0.16 ± 4% -0.0 0.12 ± 12% perf-profile.children.cycles-pp.pagecache_get_page
0.06 -0.0 0.03 ±100% perf-profile.children.cycles-pp.___might_sleep
0.18 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.__radix_tree_lookup
0.17 ± 5% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.btrfs_set_path_blocking
0.16 ± 5% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.sched_clock
0.21 ± 8% -0.0 0.17 ± 10% perf-profile.children.cycles-pp.btrfs_wait_ordered_range
0.14 ± 11% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.update_ts_time_stats
0.21 ± 4% -0.0 0.18 ± 2% perf-profile.children.cycles-pp.set_extent_bit
0.13 ± 5% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.btrfs_set_lock_blocking_write
0.13 ± 10% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.btrfs_truncate_item
0.12 ± 11% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.truncate_one_csum
0.10 ± 10% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.ttwu_do_wakeup
0.13 ± 8% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.nr_iowait_cpu
0.15 ± 7% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc
0.13 ± 6% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.memcpy_extent_buffer
0.10 ± 7% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.lock_extent_bits
0.12 ± 9% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
0.17 ± 4% -0.0 0.14 perf-profile.children.cycles-pp.sched_clock_cpu
0.15 ± 5% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.native_sched_clock
0.06 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.__alloc_pages_nodemask
0.06 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.alloc_extent_state
0.09 ± 4% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.cpuacct_charge
0.17 ± 4% -0.0 0.15 ± 4% perf-profile.children.cycles-pp.write_one_eb
0.07 ± 11% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.reweight_entity
0.11 ± 7% -0.0 0.10 ± 5% perf-profile.children.cycles-pp.clear_state_bit
0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.read_extent_buffer
0.08 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.__lookup_extent_mapping
0.07 ± 10% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se
0.16 ± 5% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.add_delayed_ref_head
0.10 ± 5% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.__test_set_page_writeback
0.10 ± 4% -0.0 0.09 perf-profile.children.cycles-pp.__extent_writepage_io
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__etree_search
0.06 -0.0 0.05 perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
0.06 ± 7% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.load_balance
0.10 ± 15% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
0.12 ± 8% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.rebalance_domains
0.03 ±100% +0.0 0.06 ± 6% perf-profile.children.cycles-pp.run_rebalance_domains
0.33 ± 6% +0.0 0.36 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.12 ± 6% +0.0 0.17 ± 18% perf-profile.children.cycles-pp.rcu_core
0.06 ± 14% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.12 ± 10% +0.1 0.17 ± 8% perf-profile.children.cycles-pp.prepare_to_wait
0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats
0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.find_busiest_group
0.01 ±173% +0.1 0.08 ± 31% perf-profile.children.cycles-pp.run_local_timers
0.04 ± 60% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.cpu_load_update_active
0.14 ± 15% +0.1 0.24 ± 14% perf-profile.children.cycles-pp.scheduler_tick
0.00 +0.1 0.10 ± 7% perf-profile.children.cycles-pp.save_stack_address
0.00 +0.1 0.11 ± 11% perf-profile.children.cycles-pp.__unwind_start
0.37 ± 7% +0.1 0.49 ± 10% perf-profile.children.cycles-pp.__softirqentry_text_start
1.25 ± 5% +0.1 1.40 ± 2% perf-profile.children.cycles-pp.osq_lock
0.00 +0.2 0.16 ± 9% perf-profile.children.cycles-pp.__module_text_address
1.42 ± 5% +0.2 1.59 ± 3% perf-profile.children.cycles-pp.__mutex_lock
0.00 +0.2 0.17 ± 7% perf-profile.children.cycles-pp.is_module_text_address
0.32 ± 10% +0.2 0.53 ± 13% perf-profile.children.cycles-pp.tick_sched_handle
1.59 ± 6% +0.2 1.80 ± 4% perf-profile.children.cycles-pp.wait_log_commit
0.31 ± 10% +0.2 0.52 ± 12% perf-profile.children.cycles-pp.update_process_times
0.56 ± 8% +0.2 0.78 ± 13% perf-profile.children.cycles-pp.irq_exit
0.00 +0.2 0.23 ± 5% perf-profile.children.cycles-pp.kernel_text_address
0.41 ± 8% +0.2 0.65 ± 11% perf-profile.children.cycles-pp.tick_sched_timer
0.00 +0.3 0.26 ± 5% perf-profile.children.cycles-pp.__kernel_text_address
0.59 ± 8% +0.3 0.85 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.00 +0.3 0.27 ± 6% perf-profile.children.cycles-pp.unwind_get_return_address
0.00 +0.3 0.28 ± 7% perf-profile.children.cycles-pp.__module_address
0.40 ± 10% +0.3 0.68 ± 18% perf-profile.children.cycles-pp.clockevents_program_event
0.62 ± 8% +0.3 0.94 ± 17% perf-profile.children.cycles-pp.ktime_get
0.89 +0.4 1.32 perf-profile.children.cycles-pp.unlock_up
1.90 +0.4 2.34 ± 3% perf-profile.children.cycles-pp.setup_items_for_insert
1.14 ± 8% +0.5 1.67 ± 12% perf-profile.children.cycles-pp.hrtimer_interrupt
0.00 +0.6 0.56 ± 5% perf-profile.children.cycles-pp.__orc_find
1.40 ± 2% +0.6 1.97 ± 3% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.00 +0.7 0.72 ± 5% perf-profile.children.cycles-pp.orc_find
1.82 ± 7% +0.7 2.57 ± 12% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
1.98 ± 7% +0.8 2.76 ± 11% perf-profile.children.cycles-pp.apic_timer_interrupt
0.45 ± 8% +0.9 1.38 ± 11% perf-profile.children.cycles-pp.sched_ttwu_pending
1.77 ± 4% +1.1 2.92 ± 2% perf-profile.children.cycles-pp.__wake_up_common
1.66 ± 5% +1.2 2.81 ± 2% perf-profile.children.cycles-pp.autoremove_wake_function
1.58 ± 4% +1.3 2.83 ± 2% perf-profile.children.cycles-pp.try_to_wake_up
0.00 +1.3 1.32 ± 4% perf-profile.children.cycles-pp.unwind_next_frame
2.66 +1.4 4.01 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock
0.00 +1.8 1.75 ± 4% perf-profile.children.cycles-pp.__save_stack_trace
0.00 +1.8 1.85 ± 4% perf-profile.children.cycles-pp.save_stack_trace_tsk
0.61 ± 4% +2.5 3.07 ± 5% perf-profile.children.cycles-pp.ttwu_do_activate
0.58 ± 3% +2.5 3.04 ± 5% perf-profile.children.cycles-pp.enqueue_task_fair
0.44 ± 5% +2.5 2.92 ± 5% perf-profile.children.cycles-pp.enqueue_entity
0.00 +2.6 2.55 ± 5% perf-profile.children.cycles-pp.__account_scheduler_latency
48.26 +3.1 51.38 ± 3% perf-profile.children.cycles-pp.start_secondary
0.41 ± 16% -0.2 0.17 ± 10% perf-profile.self.cycles-pp.poll_idle
0.68 ± 8% -0.2 0.51 ± 8% perf-profile.self.cycles-pp.btrfs_tree_read_lock
1.22 -0.2 1.06 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.93 ± 3% -0.2 0.77 perf-profile.self.cycles-pp._raw_spin_lock
0.59 ± 3% -0.2 0.44 ± 5% perf-profile.self.cycles-pp.prepare_to_wait_event
0.45 ± 7% -0.1 0.32 ± 3% perf-profile.self.cycles-pp.__schedule
0.85 ± 3% -0.1 0.73 ± 6% perf-profile.self.cycles-pp.queued_write_lock_slowpath
0.52 ± 5% -0.1 0.42 ± 4% perf-profile.self.cycles-pp.queued_read_lock_slowpath
0.47 ± 4% -0.1 0.38 ± 7% perf-profile.self.cycles-pp.btrfs_get_token_32
0.38 ± 3% -0.1 0.29 ± 8% perf-profile.self.cycles-pp.__list_add_valid
0.40 ± 2% -0.1 0.31 ± 3% perf-profile.self.cycles-pp.__list_del_entry_valid
0.40 ± 4% -0.1 0.32 ± 3% perf-profile.self.cycles-pp._raw_read_lock
0.35 -0.1 0.27 ± 8% perf-profile.self.cycles-pp.btrfs_tree_lock
0.33 ± 5% -0.1 0.26 ± 10% perf-profile.self.cycles-pp.btrfs_tree_read_unlock
0.30 ± 7% -0.1 0.23 ± 11% perf-profile.self.cycles-pp.try_to_wake_up
0.29 ± 7% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.btrfs_set_token_32
0.29 ± 4% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.memmove
0.36 ± 4% -0.1 0.31 ± 2% perf-profile.self.cycles-pp.copy_page
0.10 ± 14% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.finish_task_switch
0.38 ± 5% -0.0 0.34 ± 3% perf-profile.self.cycles-pp.brd_do_bvec
0.15 ± 14% -0.0 0.11 ± 17% perf-profile.self.cycles-pp.update_load_avg
0.20 ± 5% -0.0 0.15 ± 7% perf-profile.self.cycles-pp.btrfs_tree_unlock
0.26 ± 5% -0.0 0.22 ± 10% perf-profile.self.cycles-pp.do_idle
0.21 ± 5% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.generic_bin_search
0.21 ± 6% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair
0.19 ± 9% -0.0 0.15 ± 5% perf-profile.self.cycles-pp.free_extent_buffer
0.18 ± 2% -0.0 0.15 ± 5% perf-profile.self.cycles-pp.__radix_tree_lookup
0.10 ± 8% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair
0.21 ± 5% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.map_private_extent_buffer
0.21 ± 5% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.btrfs_root_node
0.15 ± 7% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.native_sched_clock
0.13 ± 8% -0.0 0.10 ± 10% perf-profile.self.cycles-pp.nr_iowait_cpu
0.13 ± 3% -0.0 0.10 ± 10% perf-profile.self.cycles-pp.btrfs_set_lock_blocking_write
0.19 ± 2% -0.0 0.16 ± 6% perf-profile.self.cycles-pp.btrfs_search_slot
0.09 ± 4% -0.0 0.07 ± 10% perf-profile.self.cycles-pp.cpuacct_charge
0.08 ± 6% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.read_extent_buffer
0.11 ± 7% -0.0 0.09 ± 11% perf-profile.self.cycles-pp.sched_ttwu_pending
0.07 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.reweight_entity
0.11 ± 4% -0.0 0.10 perf-profile.self.cycles-pp.__wake_up_common
0.05 ± 8% +0.0 0.10 ± 15% perf-profile.self.cycles-pp.rcu_sched_clock_irq
0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.orc_find
0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.save_stack_address
0.00 +0.1 0.08 ± 31% perf-profile.self.cycles-pp.run_local_timers
0.00 +0.1 0.10 ± 5% perf-profile.self.cycles-pp.save_stack_trace_tsk
0.01 ±173% +0.1 0.12 ± 23% perf-profile.self.cycles-pp.cpu_load_update_active
1.24 ± 4% +0.2 1.39 ± 2% perf-profile.self.cycles-pp.osq_lock
0.00 +0.3 0.28 ± 7% perf-profile.self.cycles-pp.__module_address
0.45 ± 9% +0.3 0.79 ± 19% perf-profile.self.cycles-pp.ktime_get
0.00 +0.4 0.45 ± 7% perf-profile.self.cycles-pp.__account_scheduler_latency
0.00 +0.5 0.51 ± 5% perf-profile.self.cycles-pp.unwind_next_frame
0.00 +0.6 0.56 ± 5% perf-profile.self.cycles-pp.__orc_find
aim7.jobs-per-min
1250 +-+------------------------------------------------------------------+
1200 +-+.+.++.+.+.+.+.+.+.++.+.+.+.+ |
| |
1150 +-+ |
1100 +-+ O O O O OO O |
1050 +-+ O O O O O OO O O O
1000 +-+ |
| |
950 +-+ |
900 +-+ |
850 O-+ O O O O O O O O O O |
800 +-+ O O O O |
| O O |
750 +-+ O OO O |
700 +-+------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7
commit:
v5.1-rc7
2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers")
v5.1-rc7 2996e1f8bcadf0436cc67b63af0
---------------- ---------------------------
%stddev %change %stddev
\ | \
474.83 -19.1% 384.18 aim7.jobs-per-min
252.81 +23.6% 312.51 aim7.time.elapsed_time
252.81 +23.6% 312.51 aim7.time.elapsed_time.max
3369 ± 3% +15.9% 3905 ± 3% aim7.time.involuntary_context_switches
2829 +3.5% 2927 aim7.time.maximum_resident_set_size
8026 ± 4% +18.3% 9498 ± 3% aim7.time.minor_page_faults
1763 ± 2% +23.9% 2185 ± 2% aim7.time.system_time
27540068 -11.4% 24395342 aim7.time.voluntary_context_switches
39442 ± 2% -22.7% 30475 ± 2% meminfo.max_used_kB
0.11 ± 10% -0.0 0.08 ± 2% mpstat.cpu.all.usr%
5334 ± 8% -15.2% 4524 ± 10% numa-meminfo.node0.KernelStack
2058 ± 29% +43.2% 2949 ± 16% numa-meminfo.node1.PageTables
5335 ± 8% -15.2% 4524 ± 10% numa-vmstat.node0.nr_kernel_stack
514.25 ± 29% +43.3% 736.75 ± 16% numa-vmstat.node1.nr_page_table_pages
137189 -19.6% 110330 vmstat.io.bo
295976 -28.3% 212126 vmstat.system.cs
78.76 +1.6% 80.02 iostat.cpu.idle
21.12 -5.8% 19.89 iostat.cpu.system
7896 -19.5% 6356 iostat.md0.w/s
178487 -19.6% 143527 iostat.md0.wkB/s
20790 +3.9% 21601 proc-vmstat.nr_slab_unreclaimable
5524 ± 9% +24.1% 6853 ± 4% proc-vmstat.numa_hint_faults
3580 ± 12% +27.7% 4572 ± 6% proc-vmstat.numa_hint_faults_local
588396 +23.2% 724636 ± 2% proc-vmstat.pgfault
969401 +12.8% 1093038 proc-vmstat.pgfree
3.216e+09 ± 13% -19.2% 2.598e+09 ± 3% cpuidle.C1.time
34331442 ± 2% -20.2% 27399001 ± 2% cpuidle.C1.usage
5.21e+08 ± 52% +71.0% 8.911e+08 ± 26% cpuidle.C1E.time
5266261 ± 24% +46.0% 7687200 ± 16% cpuidle.C1E.usage
1.353e+09 ± 51% +98.2% 2.683e+09 ± 27% cpuidle.C3.time
6733067 ± 28% +53.5% 10335866 ± 16% cpuidle.C3.usage
5202587 ± 21% +45.4% 7564469 ± 7% cpuidle.C6.usage
170635 ± 77% -56.8% 73760 cpuidle.POLL.usage
4104 ± 3% -16.5% 3428 ± 10% slabinfo.btrfs_extent_buffer.active_objs
4111 ± 3% -16.5% 3434 ± 10% slabinfo.btrfs_extent_buffer.num_objs
11182 ± 2% -10.7% 9988 slabinfo.filp.active_objs
11535 -11.0% 10262 slabinfo.filp.num_objs
5266 +11.2% 5856 slabinfo.kmalloc-96.active_objs
5276 +11.1% 5863 slabinfo.kmalloc-96.num_objs
946.75 -13.2% 822.25 slabinfo.task_struct.active_objs
960.25 ± 2% -12.5% 840.50 slabinfo.task_struct.num_objs
309.00 -2.9% 300.00 turbostat.Avg_MHz
34327518 ± 2% -20.2% 27391739 ± 2% turbostat.C1
31.35 ± 13% -11.0 20.34 turbostat.C1%
5265915 ± 24% +46.0% 7686822 ± 16% turbostat.C1E
6732546 ± 28% +53.5% 10335310 ± 16% turbostat.C3
5199490 ± 21% +45.4% 7561891 ± 7% turbostat.C6
19750059 ± 13% +33.5% 26375491 ± 2% turbostat.IRQ
32.10 -1.7% 31.56 turbostat.RAMWatt
20520 +24.5% 25540 ± 2% turbostat.SMI
2739 ± 11% +115.5% 5902 ± 38% sched_debug.cfs_rq:/.MIN_vruntime.avg
109456 ± 12% +38.8% 151967 ± 11% sched_debug.cfs_rq:/.MIN_vruntime.max
17097 ± 11% +64.1% 28053 ± 17% sched_debug.cfs_rq:/.MIN_vruntime.stddev
2739 ± 11% +115.5% 5902 ± 38% sched_debug.cfs_rq:/.max_vruntime.avg
109456 ± 12% +38.8% 151967 ± 11% sched_debug.cfs_rq:/.max_vruntime.max
17097 ± 11% +64.1% 28053 ± 17% sched_debug.cfs_rq:/.max_vruntime.stddev
188026 +23.2% 231708 sched_debug.cfs_rq:/.min_vruntime.avg
224730 ± 2% +24.3% 279273 ± 3% sched_debug.cfs_rq:/.min_vruntime.max
155086 +22.2% 189563 sched_debug.cfs_rq:/.min_vruntime.min
15264 ± 9% +27.7% 19493 ± 12% sched_debug.cfs_rq:/.min_vruntime.stddev
15264 ± 9% +27.7% 19493 ± 12% sched_debug.cfs_rq:/.spread0.stddev
167631 +20.1% 201400 sched_debug.cpu.clock.avg
167635 +20.1% 201404 sched_debug.cpu.clock.max
167627 +20.1% 201395 sched_debug.cpu.clock.min
2.34 ± 4% +13.4% 2.65 ± 5% sched_debug.cpu.clock.stddev
167631 +20.1% 201400 sched_debug.cpu.clock_task.avg
167635 +20.1% 201404 sched_debug.cpu.clock_task.max
167627 +20.1% 201395 sched_debug.cpu.clock_task.min
2.34 ± 4% +13.4% 2.65 ± 5% sched_debug.cpu.clock_task.stddev
17.73 ± 10% -22.0% 13.83 ± 7% sched_debug.cpu.cpu_load[0].avg
44.72 ± 22% -29.1% 31.70 ± 21% sched_debug.cpu.cpu_load[0].stddev
15.81 ± 10% -21.7% 12.38 ± 11% sched_debug.cpu.cpu_load[4].avg
34.15 ± 32% -54.3% 15.61 ± 53% sched_debug.cpu.cpu_load[4].stddev
4754 +16.2% 5526 sched_debug.cpu.curr->pid.max
1004 ± 3% +12.1% 1126 sched_debug.cpu.curr->pid.stddev
130427 +23.1% 160557 sched_debug.cpu.nr_load_updates.avg
138214 +23.6% 170891 sched_debug.cpu.nr_load_updates.max
127564 +23.5% 157516 sched_debug.cpu.nr_load_updates.min
959187 -9.5% 867604 sched_debug.cpu.nr_switches.max
771024 -9.2% 700458 ± 4% sched_debug.cpu.nr_switches.min
167627 +20.1% 201395 sched_debug.cpu_clk
164058 +20.6% 197822 sched_debug.ktime
168073 +20.1% 201837 sched_debug.sched_clk
8.454e+08 ± 2% +20.1% 1.015e+09 perf-stat.i.branch-instructions
16630864 ± 7% +19.8% 19922156 ± 11% perf-stat.i.branch-misses
20.26 -4.1 16.20 ± 2% perf-stat.i.cache-miss-rate%
14888468 ± 2% -16.0% 12505705 perf-stat.i.cache-misses
74346170 ± 2% +5.5% 78449759 ± 2% perf-stat.i.cache-references
299258 -28.5% 214099 perf-stat.i.context-switches
3.21 -11.4% 2.85 perf-stat.i.cpi
218.00 ± 4% -10.1% 196.01 perf-stat.i.cpu-migrations
816.02 +17.2% 956.56 perf-stat.i.cycles-between-cache-misses
1.1e+09 ± 2% +7.0% 1.177e+09 perf-stat.i.dTLB-loads
0.11 ± 10% -0.0 0.09 ± 6% perf-stat.i.dTLB-store-miss-rate%
572872 ± 9% -19.6% 460348 ± 7% perf-stat.i.dTLB-store-misses
46.23 ± 3% +6.8 53.01 ± 2% perf-stat.i.iTLB-load-miss-rate%
2592240 -27.0% 1892449 ± 2% perf-stat.i.iTLB-loads
4.012e+09 ± 2% +13.9% 4.569e+09 perf-stat.i.instructions
1948 ± 6% +14.6% 2232 ± 4% perf-stat.i.instructions-per-iTLB-miss
0.32 +15.0% 0.37 perf-stat.i.ipc
6791873 -19.7% 5457038 perf-stat.i.node-load-misses
7294618 -19.3% 5886107 perf-stat.i.node-loads
4662051 ± 2% -19.9% 3733205 perf-stat.i.node-store-misses
6959162 ± 2% -19.3% 5614550 perf-stat.i.node-stores
18.53 ± 2% -7.4% 17.17 perf-stat.overall.MPKI
20.03 -4.1 15.95 perf-stat.overall.cache-miss-rate%
3.05 -13.5% 2.64 perf-stat.overall.cpi
822.35 +17.2% 964.05 perf-stat.overall.cycles-between-cache-misses
0.11 ± 11% -0.0 0.09 ± 5% perf-stat.overall.dTLB-store-miss-rate%
44.19 ± 3% +7.3 51.51 ± 2% perf-stat.overall.iTLB-load-miss-rate%
1959 ± 6% +16.2% 2275 ± 4% perf-stat.overall.instructions-per-iTLB-miss
0.33 +15.6% 0.38 perf-stat.overall.ipc
8.42e+08 ± 2% +20.2% 1.012e+09 perf-stat.ps.branch-instructions
16565866 ± 7% +19.9% 19859511 ± 11% perf-stat.ps.branch-misses
14829536 ± 2% -15.9% 12466228 perf-stat.ps.cache-misses
74051441 ± 2% +5.6% 78201714 ± 2% perf-stat.ps.cache-references
298067 -28.4% 213421 perf-stat.ps.context-switches
217.13 ± 4% -10.0% 195.39 perf-stat.ps.cpu-migrations
1.095e+09 ± 2% +7.1% 1.173e+09 perf-stat.ps.dTLB-loads
570616 ± 9% -19.6% 458896 ± 7% perf-stat.ps.dTLB-store-misses
2581925 -26.9% 1886455 ± 2% perf-stat.ps.iTLB-loads
3.996e+09 ± 2% +14.0% 4.555e+09 perf-stat.ps.instructions
6764857 -19.6% 5439764 perf-stat.ps.node-load-misses
7265612 -19.2% 5867478 perf-stat.ps.node-loads
4643500 ± 2% -19.9% 3721390 perf-stat.ps.node-store-misses
6931476 ± 2% -19.3% 5596795 perf-stat.ps.node-stores
1.02e+12 +42.1% 1.45e+12 perf-stat.total.instructions
151.25 ± 11% +94.5% 294.25 ± 46% interrupts.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
125.25 +24.6% 156.00 interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
99753 ± 7% +18.7% 118441 interrupts.CAL:Function_call_interrupts
2335 ± 9% +24.9% 2917 ± 8% interrupts.CPU0.CAL:Function_call_interrupts
500200 ± 4% +27.5% 638000 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts
2518 ± 9% +15.3% 2904 ± 4% interrupts.CPU1.CAL:Function_call_interrupts
501181 ± 3% +27.2% 637472 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts
2365 ± 8% +28.1% 3031 ± 6% interrupts.CPU10.CAL:Function_call_interrupts
498584 ± 4% +27.8% 637281 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts
14324 ± 23% +37.1% 19636 ± 14% interrupts.CPU10.RES:Rescheduling_interrupts
446594 ± 24% +42.8% 637802 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts
2446 ± 7% +24.3% 3039 ± 4% interrupts.CPU12.CAL:Function_call_interrupts
499544 ± 4% +27.6% 637595 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts
2265 ± 10% +37.6% 3117 ± 6% interrupts.CPU13.CAL:Function_call_interrupts
500218 ± 4% +27.5% 637698 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts
0.75 ±173% +81766.7% 614.00 ± 57% interrupts.CPU13.NMI:Non-maskable_interrupts
0.75 ±173% +81766.7% 614.00 ± 57% interrupts.CPU13.PMI:Performance_monitoring_interrupts
2413 ± 5% +24.0% 2993 ± 5% interrupts.CPU14.CAL:Function_call_interrupts
447745 ± 24% +42.4% 637743 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts
14160 ± 8% +30.9% 18541 ± 18% interrupts.CPU14.RES:Rescheduling_interrupts
2450 ± 8% +24.6% 3053 ± 3% interrupts.CPU15.CAL:Function_call_interrupts
447422 ± 24% +42.5% 637614 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts
499537 ± 4% +27.7% 637734 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts
2511 ± 8% +14.4% 2872 ± 4% interrupts.CPU17.CAL:Function_call_interrupts
501205 ± 3% +27.3% 637896 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts
2466 ± 5% +17.7% 2904 ± 7% interrupts.CPU18.CAL:Function_call_interrupts
498429 ± 4% +27.9% 637539 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts
2421 ± 8% +19.2% 2885 ± 2% interrupts.CPU19.CAL:Function_call_interrupts
447347 ± 24% +42.5% 637690 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts
14512 ± 10% +19.9% 17400 ± 16% interrupts.CPU19.RES:Rescheduling_interrupts
2472 ± 8% +23.8% 3061 ± 4% interrupts.CPU2.CAL:Function_call_interrupts
498740 ± 4% +27.9% 637673 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts
498931 ± 4% +27.8% 637604 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts
2457 ± 7% +26.2% 3102 ± 9% interrupts.CPU21.CAL:Function_call_interrupts
447732 ± 24% +42.4% 637663 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts
448745 ± 23% +42.1% 637616 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts
1062 ± 65% -61.6% 407.75 ±169% interrupts.CPU22.NMI:Non-maskable_interrupts
1062 ± 65% -61.6% 407.75 ±169% interrupts.CPU22.PMI:Performance_monitoring_interrupts
2525 ± 3% +15.0% 2905 ± 3% interrupts.CPU23.CAL:Function_call_interrupts
448647 ± 23% +42.2% 637857 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts
499423 ± 4% +27.7% 637620 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts
499966 ± 4% +27.6% 637919 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts
2545 ± 5% +16.7% 2971 ± 7% interrupts.CPU26.CAL:Function_call_interrupts
448780 ± 23% +42.1% 637593 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts
500205 ± 3% +27.5% 637903 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts
2401 ± 9% +21.7% 2923 ± 5% interrupts.CPU28.CAL:Function_call_interrupts
498973 ± 4% +27.8% 637717 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts
502124 ± 3% +27.0% 637676 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts
2474 ± 11% +21.2% 2998 ± 8% interrupts.CPU3.CAL:Function_call_interrupts
500168 ± 4% +27.5% 637840 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts
448818 ± 23% +42.0% 637276 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts
2535 ± 8% +19.8% 3037 ± 2% interrupts.CPU31.CAL:Function_call_interrupts
501670 ± 3% +27.0% 637350 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts
2444 ± 5% +20.1% 2936 ± 5% interrupts.CPU32.CAL:Function_call_interrupts
448601 ± 24% +42.1% 637465 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts
448644 ± 23% +42.1% 637433 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts
1298 ± 32% -68.5% 408.75 ±173% interrupts.CPU33.NMI:Non-maskable_interrupts
1298 ± 32% -68.5% 408.75 ±173% interrupts.CPU33.PMI:Performance_monitoring_interrupts
18952 ± 8% -18.8% 15386 ± 16% interrupts.CPU33.RES:Rescheduling_interrupts
499974 ± 4% +27.5% 637544 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts
446755 ± 24% +42.7% 637668 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts
1076 ± 36% -78.6% 230.00 ±157% interrupts.CPU35.NMI:Non-maskable_interrupts
1076 ± 36% -78.6% 230.00 ±157% interrupts.CPU35.PMI:Performance_monitoring_interrupts
2621 ± 4% +11.3% 2918 ± 5% interrupts.CPU36.CAL:Function_call_interrupts
447762 ± 24% +42.4% 637548 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts
2542 ± 10% +23.9% 3151 ± 3% interrupts.CPU37.CAL:Function_call_interrupts
447655 ± 24% +42.2% 636691 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts
16018 ± 16% +34.2% 21489 ± 4% interrupts.CPU37.RES:Rescheduling_interrupts
151.25 ± 11% +94.5% 294.25 ± 46% interrupts.CPU38.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6
448676 ± 24% +42.1% 637520 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts
500913 ± 3% +27.3% 637659 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts
448474 ± 24% +42.2% 637512 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts
2474 ± 8% +24.2% 3073 ± 3% interrupts.CPU5.CAL:Function_call_interrupts
448794 ± 23% +42.1% 637675 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts
2416 ± 7% +18.7% 2867 ± 6% interrupts.CPU6.CAL:Function_call_interrupts
499019 ± 4% +27.8% 637624 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts
449105 ± 23% +42.0% 637874 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts
2545 ± 4% +18.4% 3014 ± 4% interrupts.CPU8.CAL:Function_call_interrupts
448672 ± 24% +42.1% 637658 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts
447744 ± 24% +42.4% 637699 ± 2% interrupts.CPU9.LOC:Local_timer_interrupts
18961728 ± 13% +34.5% 25504957 ± 2% interrupts.LOC:Local_timer_interrupts
70856 ± 3% +27.7% 90455 ± 5% softirqs.CPU0.RCU
46993 ± 2% +19.6% 56188 ± 2% softirqs.CPU0.SCHED
90395 ± 4% +27.6% 115332 ± 2% softirqs.CPU0.TIMER
71106 ± 2% +27.3% 90543 ± 3% softirqs.CPU1.RCU
44000 +20.6% 53050 ± 2% softirqs.CPU1.SCHED
94521 ± 6% +16.0% 109649 softirqs.CPU1.TIMER
72447 ± 5% +23.7% 89595 ± 4% softirqs.CPU10.RCU
42542 +25.0% 53191 ± 4% softirqs.CPU10.SCHED
86643 ± 5% +30.5% 113030 softirqs.CPU10.TIMER
70388 ± 3% +27.8% 89934 ± 2% softirqs.CPU11.RCU
44629 ± 3% +19.4% 53308 softirqs.CPU11.SCHED
71989 ± 3% +30.1% 93643 ± 5% softirqs.CPU12.RCU
44508 ± 3% +19.9% 53353 ± 2% softirqs.CPU12.SCHED
90388 ± 3% +26.8% 114579 ± 3% softirqs.CPU12.TIMER
73320 +25.3% 91879 ± 3% softirqs.CPU13.RCU
44242 +20.1% 53144 softirqs.CPU13.SCHED
89009 ± 4% +27.3% 113304 ± 3% softirqs.CPU13.TIMER
70951 ± 6% +28.3% 91061 ± 6% softirqs.CPU14.RCU
44755 ± 3% +19.8% 53625 ± 2% softirqs.CPU14.SCHED
68460 ± 6% +33.5% 91393 ± 3% softirqs.CPU15.RCU
44863 ± 2% +19.2% 53481 softirqs.CPU15.SCHED
72151 ± 4% +26.3% 91161 ± 3% softirqs.CPU16.RCU
44009 ± 2% +20.9% 53186 ± 2% softirqs.CPU16.SCHED
90040 ± 5% +28.2% 115465 softirqs.CPU16.TIMER
73480 ± 2% +23.7% 90925 ± 2% softirqs.CPU17.RCU
43579 ± 2% +22.2% 53237 softirqs.CPU17.SCHED
89846 ± 5% +28.3% 115255 ± 2% softirqs.CPU17.TIMER
71778 ± 4% +27.0% 91150 ± 3% softirqs.CPU18.RCU
43811 ± 2% +22.3% 53560 ± 3% softirqs.CPU18.SCHED
89950 ± 5% +47.8% 132927 ± 22% softirqs.CPU18.TIMER
70427 ± 5% +28.7% 90642 ± 3% softirqs.CPU19.RCU
44467 ± 3% +19.6% 53172 softirqs.CPU19.SCHED
72448 ± 3% +25.8% 91106 ± 4% softirqs.CPU2.RCU
43284 ± 2% +22.8% 53170 ± 2% softirqs.CPU2.SCHED
88648 ± 5% +27.8% 113278 ± 4% softirqs.CPU2.TIMER
71858 ± 5% +28.6% 92375 ± 3% softirqs.CPU20.RCU
43172 +22.4% 52826 ± 2% softirqs.CPU20.SCHED
89040 ± 5% +26.3% 112429 ± 3% softirqs.CPU20.TIMER
72855 ± 5% +28.0% 93223 ± 2% softirqs.CPU21.RCU
44066 ± 3% +19.0% 52436 ± 3% softirqs.CPU21.SCHED
70991 ± 7% +32.5% 94074 ± 4% softirqs.CPU22.RCU
44490 ± 3% +19.0% 52958 ± 2% softirqs.CPU22.SCHED
71264 ± 5% +28.1% 91257 ± 3% softirqs.CPU23.RCU
44233 ± 2% +18.8% 52566 ± 2% softirqs.CPU23.SCHED
73463 ± 4% +25.3% 92068 ± 4% softirqs.CPU24.RCU
43552 ± 2% +21.6% 52956 softirqs.CPU24.SCHED
89944 ± 5% +27.3% 114465 softirqs.CPU24.TIMER
73945 +23.4% 91231 ± 2% softirqs.CPU25.RCU
43633 +21.9% 53174 ± 3% softirqs.CPU25.SCHED
89171 ± 4% +39.8% 124628 ± 20% softirqs.CPU25.TIMER
70343 ± 5% +29.4% 91058 ± 6% softirqs.CPU26.RCU
43709 ± 3% +20.8% 52784 ± 2% softirqs.CPU26.SCHED
72666 ± 3% +25.0% 90831 ± 6% softirqs.CPU27.RCU
43708 +20.6% 52709 softirqs.CPU27.SCHED
86809 ± 5% +24.5% 108113 ± 6% softirqs.CPU27.TIMER
71506 ± 3% +28.6% 91921 ± 5% softirqs.CPU28.RCU
43804 ± 2% +20.1% 52606 softirqs.CPU28.SCHED
73394 ± 3% +22.8% 90149 ± 5% softirqs.CPU29.RCU
43517 +20.8% 52588 softirqs.CPU29.SCHED
89568 ± 5% +22.1% 109326 ± 6% softirqs.CPU29.TIMER
73421 ± 2% +23.5% 90681 ± 4% softirqs.CPU3.RCU
44393 +21.5% 53943 ± 2% softirqs.CPU3.SCHED
89862 ± 4% +24.0% 111453 ± 2% softirqs.CPU3.TIMER
71663 ± 7% +29.0% 92457 ± 6% softirqs.CPU30.RCU
43602 ± 2% +20.8% 52683 ± 2% softirqs.CPU30.SCHED
73484 ± 2% +24.0% 91142 softirqs.CPU31.RCU
43471 +21.6% 52874 ± 2% softirqs.CPU31.SCHED
88937 ± 5% +26.5% 112542 ± 2% softirqs.CPU31.TIMER
64012 ± 4% +24.6% 79739 ± 3% softirqs.CPU32.RCU
44472 ± 5% +18.8% 52827 softirqs.CPU32.SCHED
61122 ± 2% +28.2% 78387 ± 3% softirqs.CPU33.RCU
44140 ± 3% +19.5% 52769 ± 2% softirqs.CPU33.SCHED
61441 +26.8% 77902 ± 5% softirqs.CPU34.RCU
43298 +21.4% 52570 ± 2% softirqs.CPU34.SCHED
90552 ± 6% +25.7% 113800 ± 3% softirqs.CPU34.TIMER
61123 ± 2% +30.8% 79962 ± 2% softirqs.CPU35.RCU
44252 ± 3% +19.5% 52863 ± 2% softirqs.CPU35.SCHED
59961 ± 3% +30.3% 78141 ± 4% softirqs.CPU36.RCU
43798 ± 2% +20.6% 52819 ± 2% softirqs.CPU36.SCHED
59914 +28.9% 77253 ± 2% softirqs.CPU37.RCU
44359 ± 3% +19.4% 52965 softirqs.CPU37.SCHED
60200 ± 3% +29.1% 77719 softirqs.CPU38.RCU
44765 ± 3% +18.8% 53159 ± 2% softirqs.CPU38.SCHED
60108 ± 2% +28.8% 77435 softirqs.CPU39.RCU
44041 +20.3% 52987 softirqs.CPU39.SCHED
88646 ± 4% +28.5% 113914 ± 3% softirqs.CPU39.TIMER
69245 ± 4% +31.6% 91096 ± 4% softirqs.CPU4.RCU
44364 ± 3% +19.7% 53100 ± 2% softirqs.CPU4.SCHED
69900 ± 3% +29.9% 90798 ± 3% softirqs.CPU5.RCU
44569 ± 3% +21.2% 54036 softirqs.CPU5.SCHED
71523 ± 4% +27.3% 91057 ± 5% softirqs.CPU6.RCU
43811 +21.1% 53061 ± 2% softirqs.CPU6.SCHED
90070 ± 4% +26.8% 114228 ± 2% softirqs.CPU6.TIMER
71179 ± 4% +28.2% 91267 ± 4% softirqs.CPU7.RCU
45398 ± 4% +19.0% 54043 softirqs.CPU7.SCHED
68555 ± 5% +32.1% 90537 ± 3% softirqs.CPU8.RCU
44109 ± 2% +19.9% 52878 ± 2% softirqs.CPU8.SCHED
69352 ± 3% +28.8% 89311 ± 3% softirqs.CPU9.RCU
44440 ± 3% +21.0% 53755 softirqs.CPU9.SCHED
2778312 ± 2% +27.7% 3546581 ± 3% softirqs.RCU
1764877 ± 2% +20.5% 2126626 ± 2% softirqs.SCHED
3831020 ± 12% +19.8% 4591281 ± 2% softirqs.TIMER
38.39 ± 4% -2.6 35.76 ± 3% perf-profile.calltrace.cycles-pp.write
37.65 ± 4% -2.5 35.17 ± 4% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
37.67 ± 4% -2.5 35.19 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
37.77 ± 4% -2.5 35.30 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
37.77 ± 4% -2.5 35.30 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
37.75 ± 4% -2.5 35.28 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
37.74 ± 4% -2.5 35.27 ± 4% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
36.01 ± 4% -2.4 33.62 ± 4% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
4.23 ± 8% -1.5 2.74 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
4.17 ± 8% -1.5 2.70 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node
6.75 ± 5% -1.1 5.60 ± 7% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
6.75 ± 5% -1.1 5.60 ± 7% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
2.65 ± 11% -1.0 1.60 ± 16% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
6.82 ± 5% -1.0 5.82 ± 4% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents
8.42 ± 5% -0.9 7.53 ± 6% perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread
2.46 ± 15% -0.9 1.58 ± 16% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node
8.12 ± 5% -0.9 7.27 ± 6% perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work
4.06 ± 9% -0.9 3.21 ± 7% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent
9.60 ± 4% -0.8 8.75 ± 4% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
9.58 ± 4% -0.8 8.74 ± 4% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread
1.25 ± 4% -0.8 0.43 ± 59% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
1.32 -0.8 0.56 ± 9% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
1.64 ± 6% -0.6 1.07 ± 28% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
2.04 ± 6% -0.5 1.50 ± 6% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
5.54 ± 5% -0.5 5.08 ± 4% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
5.53 ± 5% -0.5 5.08 ± 4% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
2.18 ± 4% -0.4 1.77 ± 6% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write
2.15 ± 4% -0.4 1.76 ± 6% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter
0.70 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter
0.70 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file
2.16 ± 4% -0.4 1.77 ± 6% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
0.69 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log
2.12 ± 4% -0.4 1.74 ± 6% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops
2.13 ± 4% -0.4 1.75 ± 6% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file
2.41 ± 7% -0.3 2.12 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents
1.61 ± 4% -0.3 1.32 ± 11% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode.btrfs_commit_inode_delayed_items
1.59 ± 4% -0.3 1.31 ± 11% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode
1.29 ± 4% -0.3 1.02 ± 8% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
0.70 ± 6% -0.3 0.43 ± 59% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write
1.59 ± 8% -0.2 1.35 ± 5% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
2.14 ± 2% -0.2 1.90 ± 7% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks
2.14 ± 2% -0.2 1.91 ± 7% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents
1.05 ± 7% -0.2 0.82 ± 8% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range
0.61 ± 8% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio
1.12 ± 9% -0.2 0.94 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.80 ± 8% -0.2 0.62 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages
1.08 ± 10% -0.2 0.91 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.84 ± 8% -0.2 0.68 ± 8% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages
0.77 ± 9% -0.2 0.61 ± 10% perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages
0.88 ± 4% -0.1 0.73 ± 4% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum
0.81 ± 4% -0.1 0.71 ± 7% perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.flush_write_bio.extent_writepages.do_writepages
0.82 ± 5% -0.1 0.72 ± 6% perf-profile.calltrace.cycles-pp.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range
0.81 ± 4% -0.1 0.72 ± 7% perf-profile.calltrace.cycles-pp.submit_one_bio.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range
2.63 ± 2% +0.2 2.85 ± 5% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
0.73 ± 6% +0.3 0.99 ± 6% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent
2.79 ± 2% +0.3 3.07 ± 5% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
0.63 ± 2% +0.3 0.95 ± 8% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io
0.62 ± 2% +0.3 0.94 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent
0.79 ± 3% +0.4 1.15 ± 9% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper
2.34 ± 4% +0.4 2.71 ± 6% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
0.42 ± 58% +0.4 0.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode
0.63 ± 8% +0.4 1.06 ± 6% perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent
0.61 ± 8% +0.4 1.05 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode
0.40 ± 58% +0.4 0.84 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents
2.74 ± 5% +0.5 3.28 ± 5% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items
0.28 ±100% +0.6 0.87 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up
0.30 ±100% +0.6 0.90 ± 7% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot
0.00 +0.6 0.59 ± 5% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
0.31 ±100% +0.6 0.91 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items
0.00 +0.6 0.61 ± 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
0.00 +0.6 0.62 ± 7% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
7.53 ± 3% +0.8 8.32 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
0.00 +0.8 0.82 ± 10% perf-profile.calltrace.cycles-pp.unwind_next_frame.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
7.34 ± 3% +0.8 8.17 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
4.47 ± 3% +0.9 5.40 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
4.33 ± 3% +0.9 5.28 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node
0.00 +1.2 1.24 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe
0.00 +1.3 1.31 ± 6% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
0.00 +1.3 1.31 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert
0.00 +1.3 1.34 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents
0.00 +1.4 1.37 ± 6% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate
0.80 ± 6% +1.7 2.55 ± 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
0.00 +1.8 1.83 ± 4% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending
0.27 ±100% +2.0 2.26 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
0.26 ±100% +2.0 2.25 ± 5% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
0.00 +2.2 2.16 ± 6% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle
49.01 ± 3% +2.3 51.35 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
49.01 ± 3% +2.3 51.35 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
48.97 ± 3% +2.4 51.33 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
49.91 ± 2% +2.5 52.45 perf-profile.calltrace.cycles-pp.secondary_startup_64
38.39 ± 4% -2.6 35.77 ± 3% perf-profile.children.cycles-pp.write
37.67 ± 4% -2.5 35.19 ± 4% perf-profile.children.cycles-pp.new_sync_write
37.65 ± 4% -2.5 35.17 ± 4% perf-profile.children.cycles-pp.btrfs_file_write_iter
37.76 ± 4% -2.5 35.28 ± 3% perf-profile.children.cycles-pp.ksys_write
37.74 ± 4% -2.5 35.27 ± 4% perf-profile.children.cycles-pp.vfs_write
36.01 ± 4% -2.4 33.62 ± 4% perf-profile.children.cycles-pp.btrfs_sync_file
36.52 ± 3% -2.3 34.18 ± 2% perf-profile.children.cycles-pp.btrfs_search_slot
19.62 ± 2% -2.1 17.56 ± 2% perf-profile.children.cycles-pp.btrfs_read_lock_root_node
19.50 ± 2% -2.0 17.47 ± 2% perf-profile.children.cycles-pp.btrfs_tree_read_lock
8.71 ± 2% -1.9 6.79 ± 3% perf-profile.children.cycles-pp.finish_wait
38.42 ± 4% -1.9 36.52 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
38.41 ± 4% -1.9 36.52 ± 2% perf-profile.children.cycles-pp.do_syscall_64
12.40 ± 4% -1.5 10.86 ± 4% perf-profile.children.cycles-pp.btrfs_lookup_file_extent
14.73 ± 3% -1.1 13.60 ± 3% perf-profile.children.cycles-pp.__btrfs_drop_extents
8.43 ± 5% -0.9 7.53 ± 6% perf-profile.children.cycles-pp.insert_reserved_file_extent
9.60 ± 4% -0.8 8.75 ± 4% perf-profile.children.cycles-pp.normal_work_helper
9.58 ± 4% -0.8 8.74 ± 4% perf-profile.children.cycles-pp.btrfs_finish_ordered_io
2.97 ± 5% -0.8 2.14 ± 4% perf-profile.children.cycles-pp.queued_read_lock_slowpath
3.06 ± 4% -0.7 2.34 ± 3% perf-profile.children.cycles-pp.queued_write_lock_slowpath
2.92 ± 4% -0.5 2.38 ± 6% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
2.89 ± 5% -0.5 2.37 ± 6% perf-profile.children.cycles-pp.do_writepages
2.08 ± 8% -0.4 1.64 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_inode
2.18 ± 8% -0.4 1.75 ± 8% perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
2.84 ± 4% -0.4 2.42 ± 2% perf-profile.children.cycles-pp.__schedule
2.18 ± 4% -0.4 1.77 ± 6% perf-profile.children.cycles-pp.start_ordered_ops
2.18 ± 4% -0.4 1.78 ± 6% perf-profile.children.cycles-pp.btrfs_fdatawrite_range
2.18 ± 5% -0.4 1.80 ± 5% perf-profile.children.cycles-pp.extent_writepages
1.33 ± 5% -0.3 1.06 ± 7% perf-profile.children.cycles-pp.extent_write_cache_pages
1.75 -0.3 1.49 ± 4% perf-profile.children.cycles-pp.schedule
1.06 ± 8% -0.2 0.83 ± 8% perf-profile.children.cycles-pp.__extent_writepage
0.66 ± 14% -0.2 0.43 ± 13% perf-profile.children.cycles-pp.btrfs_next_old_leaf
0.83 ± 2% -0.2 0.62 ± 7% perf-profile.children.cycles-pp.update_load_avg
1.14 ± 8% -0.2 0.95 perf-profile.children.cycles-pp.schedule_idle
0.80 ± 8% -0.2 0.62 ± 10% perf-profile.children.cycles-pp.btrfs_run_delalloc_range
0.30 ± 24% -0.2 0.12 ± 4% perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode
0.84 ± 8% -0.2 0.68 ± 8% perf-profile.children.cycles-pp.writepage_delalloc
1.24 ± 7% -0.2 1.08 ± 8% perf-profile.children.cycles-pp.flush_write_bio
0.77 ± 9% -0.2 0.61 ± 10% perf-profile.children.cycles-pp.cow_file_range
0.49 ± 6% -0.2 0.33 ± 10% perf-profile.children.cycles-pp.__list_del_entry_valid
1.04 ± 9% -0.1 0.89 ± 8% perf-profile.children.cycles-pp.generic_make_request
0.70 ± 7% -0.1 0.55 ± 12% perf-profile.children.cycles-pp.btree_write_cache_pages
0.70 ± 6% -0.1 0.56 ± 11% perf-profile.children.cycles-pp.btrfs_write_marked_extents
1.04 ± 9% -0.1 0.90 ± 8% perf-profile.children.cycles-pp.submit_bio
0.62 ± 4% -0.1 0.48 ± 6% perf-profile.children.cycles-pp.read_block_for_search
0.86 ± 11% -0.1 0.72 ± 8% perf-profile.children.cycles-pp.brd_make_request
0.51 ± 7% -0.1 0.40 ± 9% perf-profile.children.cycles-pp.select_task_rq_fair
0.84 ± 4% -0.1 0.74 ± 6% perf-profile.children.cycles-pp.btrfs_submit_bio_hook
0.49 ± 10% -0.1 0.39 ± 6% perf-profile.children.cycles-pp.brd_do_bvec
0.77 ± 6% -0.1 0.67 ± 4% perf-profile.children.cycles-pp.dequeue_entity
0.61 ± 8% -0.1 0.54 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair
0.27 ± 10% -0.1 0.20 ± 16% perf-profile.children.cycles-pp.map_private_extent_buffer
0.40 ± 8% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.find_extent_buffer
0.25 ± 4% -0.1 0.18 ± 13% perf-profile.children.cycles-pp.__update_load_avg_se
0.21 ± 2% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.write_one_eb
0.09 ± 13% -0.1 0.03 ±100% perf-profile.children.cycles-pp.create_io_em
0.21 ± 13% -0.1 0.15 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_unlock
0.35 ± 14% -0.1 0.30 ± 9% perf-profile.children.cycles-pp.__list_add_valid
0.21 ± 10% -0.1 0.15 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.10 ± 22% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.xas_load
0.12 ± 19% -0.0 0.08 ± 16% perf-profile.children.cycles-pp.__test_set_page_writeback
0.16 ± 15% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.block_group_cache_tree_search
0.29 ± 5% -0.0 0.25 ± 7% perf-profile.children.cycles-pp.memmove
0.29 ± 10% -0.0 0.25 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc
0.26 ± 7% -0.0 0.22 ± 8% perf-profile.children.cycles-pp._raw_read_lock
0.18 ± 8% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.__switch_to
0.13 ± 12% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.update_cfs_rq_h_load
0.16 ± 6% -0.0 0.13 ± 10% perf-profile.children.cycles-pp.___perf_sw_event
0.11 ± 17% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64
0.11 ± 7% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
0.25 ± 7% +0.0 0.28 ± 2% perf-profile.children.cycles-pp.copy_extent_buffer_full
0.24 ± 9% +0.0 0.28 ± 2% perf-profile.children.cycles-pp.copy_page
0.13 ± 9% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.queue_work_on
0.13 ± 12% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.__queue_work
0.07 ± 58% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking
0.00 +0.1 0.08 ± 34% perf-profile.children.cycles-pp.in_sched_functions
0.04 ± 57% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.reschedule_interrupt
0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.stack_access_ok
0.09 ± 37% +0.1 0.19 ± 17% perf-profile.children.cycles-pp.tick_irq_enter
0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.scheduler_ipi
0.15 ± 22% +0.1 0.26 ± 15% perf-profile.children.cycles-pp.irq_enter
0.58 ± 3% +0.1 0.71 ± 15% perf-profile.children.cycles-pp.update_process_times
1.26 ± 2% +0.2 1.43 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.43 ± 8% +0.2 0.61 ± 12% perf-profile.children.cycles-pp.btrfs_release_path
0.00 +0.2 0.22 ± 12% perf-profile.children.cycles-pp.save_stack_address
1.87 ± 2% +0.2 2.11 ± 7% perf-profile.children.cycles-pp.hrtimer_interrupt
0.00 +0.3 0.26 ± 16% perf-profile.children.cycles-pp.__module_text_address
0.00 +0.3 0.29 ± 13% perf-profile.children.cycles-pp.is_module_text_address
0.00 +0.3 0.29 ± 4% perf-profile.children.cycles-pp.__unwind_start
3.26 ± 2% +0.3 3.61 ± 4% perf-profile.children.cycles-pp.smp_apic_timer_interrupt
0.00 +0.4 0.40 ± 10% perf-profile.children.cycles-pp.kernel_text_address
3.47 ± 3% +0.4 3.90 ± 3% perf-profile.children.cycles-pp.apic_timer_interrupt
0.00 +0.4 0.43 ± 9% perf-profile.children.cycles-pp.__kernel_text_address
0.00 +0.5 0.46 ± 9% perf-profile.children.cycles-pp.unwind_get_return_address
0.00 +0.5 0.47 ± 7% perf-profile.children.cycles-pp.__module_address
0.78 ± 5% +0.5 1.25 ± 5% perf-profile.children.cycles-pp.unlock_up
1.80 ± 3% +0.6 2.42 ± 5% perf-profile.children.cycles-pp.setup_items_for_insert
1.26 +0.7 1.94 ± 4% perf-profile.children.cycles-pp.btrfs_unlock_up_safe
0.00 +0.8 0.85 ± 5% perf-profile.children.cycles-pp.__orc_find
0.00 +1.1 1.12 ± 4% perf-profile.children.cycles-pp.orc_find
1.87 ± 4% +1.4 3.30 ± 2% perf-profile.children.cycles-pp.__wake_up_common
1.80 ± 4% +1.4 3.23 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function
1.83 ± 4% +1.6 3.41 ± 3% perf-profile.children.cycles-pp.try_to_wake_up
2.43 ± 4% +1.6 4.06 perf-profile.children.cycles-pp.__wake_up_common_lock
14.82 ± 3% +1.7 16.56 ± 2% perf-profile.children.cycles-pp.prepare_to_wait_event
0.85 ± 4% +1.9 2.71 ± 2% perf-profile.children.cycles-pp.sched_ttwu_pending
0.00 +2.0 1.96 ± 5% perf-profile.children.cycles-pp.unwind_next_frame
49.01 ± 3% +2.3 51.35 ± 2% perf-profile.children.cycles-pp.start_secondary
49.91 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.secondary_startup_64
49.91 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.cpu_startup_entry
49.90 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.do_idle
0.00 +2.8 2.78 ± 4% perf-profile.children.cycles-pp.__save_stack_trace
0.00 +2.9 2.90 ± 4% perf-profile.children.cycles-pp.save_stack_trace_tsk
1.02 ± 10% +3.7 4.73 ± 2% perf-profile.children.cycles-pp.ttwu_do_activate
0.82 ± 9% +3.7 4.54 ± 2% perf-profile.children.cycles-pp.enqueue_entity
0.97 ± 9% +3.7 4.69 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair
0.00 +3.8 3.84 ± 2% perf-profile.children.cycles-pp.__account_scheduler_latency
0.54 ± 15% -0.2 0.35 ± 5% perf-profile.self.cycles-pp.btrfs_tree_read_lock
0.48 ± 9% -0.2 0.30 ± 7% perf-profile.self.cycles-pp.prepare_to_wait_event
0.49 ± 5% -0.2 0.32 ± 11% perf-profile.self.cycles-pp.__list_del_entry_valid
0.74 ± 7% -0.2 0.58 perf-profile.self.cycles-pp.queued_write_lock_slowpath
0.35 ± 8% -0.2 0.20 ± 7% perf-profile.self.cycles-pp.btrfs_tree_lock
0.48 ± 4% -0.1 0.37 ± 6% perf-profile.self.cycles-pp.__schedule
0.69 ± 7% -0.1 0.58 ± 4% perf-profile.self.cycles-pp._raw_spin_lock
0.44 ± 6% -0.1 0.33 ± 6% perf-profile.self.cycles-pp.queued_read_lock_slowpath
0.27 ± 6% -0.1 0.19 ± 16% perf-profile.self.cycles-pp.update_load_avg
0.23 ± 13% -0.1 0.17 ± 18% perf-profile.self.cycles-pp.brd_do_bvec
0.24 ± 6% -0.1 0.17 ± 11% perf-profile.self.cycles-pp.__update_load_avg_se
0.32 ± 2% -0.1 0.26 ± 7% perf-profile.self.cycles-pp.try_to_wake_up
0.29 ± 9% -0.1 0.23 ± 10% perf-profile.self.cycles-pp.select_task_rq_fair
0.24 ± 9% -0.1 0.19 ± 15% perf-profile.self.cycles-pp.map_private_extent_buffer
0.21 ± 14% -0.1 0.15 ± 16% perf-profile.self.cycles-pp.btrfs_tree_read_unlock
0.35 ± 14% -0.1 0.30 ± 9% perf-profile.self.cycles-pp.__list_add_valid
0.26 ± 7% -0.0 0.21 ± 8% perf-profile.self.cycles-pp._raw_read_lock
0.17 ± 9% -0.0 0.14 ± 8% perf-profile.self.cycles-pp.__switch_to
0.13 ± 12% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.update_cfs_rq_h_load
0.28 ± 6% -0.0 0.25 ± 7% perf-profile.self.cycles-pp.memmove
0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.___perf_sw_event
0.19 ± 9% +0.0 0.22 ± 11% perf-profile.self.cycles-pp.enqueue_entity
0.00 +0.1 0.06 ± 16% perf-profile.self.cycles-pp.__unwind_start
0.00 +0.1 0.07 ± 17% perf-profile.self.cycles-pp.kernel_text_address
0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.__save_stack_trace
0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.stack_access_ok
0.00 +0.1 0.12 ± 13% perf-profile.self.cycles-pp.save_stack_trace_tsk
0.00 +0.1 0.14 ± 21% perf-profile.self.cycles-pp.orc_find
0.00 +0.1 0.14 ± 18% perf-profile.self.cycles-pp.save_stack_address
0.00 +0.5 0.46 ± 7% perf-profile.self.cycles-pp.__module_address
0.00 +0.6 0.61 ± 7% perf-profile.self.cycles-pp.__account_scheduler_latency
0.00 +0.6 0.62 ± 7% perf-profile.self.cycles-pp.unwind_next_frame
0.00 +0.8 0.84 ± 5% perf-profile.self.cycles-pp.__orc_find
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Rong Chen
1 year, 9 months
Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression
by Xing Zhengjun
On 5/30/2019 10:00 AM, Trond Myklebust wrote:
> Hi Xing,
>
> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote:
>> Hi Trond,
>>
>> On 5/20/2019 1:54 PM, kernel test robot wrote:
>>> Greeting,
>>>
>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to
>>> commit:
>>>
>>>
>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert
>>> socket page send code to use iov_iter()")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git
>>> master
>>>
>>> in testcase: fsmark
>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @
>>> 3.00GHz with 384G memory
>>> with following parameters:
>>>
>>> iterations: 1x
>>> nr_threads: 64t
>>> disk: 1BRD_48G
>>> fs: xfs
>>> fs2: nfsv4
>>> filesize: 4M
>>> test_size: 40G
>>> sync_method: fsyncBeforeClose
>>> cpufreq_governor: performance
>>>
>>> test-description: The fsmark is a file system benchmark to test
>>> synchronous write workloads, for example, mail servers workload.
>>> test-url: https://sourceforge.net/projects/fsmark/
>>>
>>>
>>>
>>> Details are as below:
>>> -----------------------------------------------------------------
>>> --------------------------------->
>>>
>>>
>>> To reproduce:
>>>
>>> git clone https://github.com/intel/lkp-tests.git
>>> cd lkp-tests
>>> bin/lkp install job.yaml # job file is attached in this
>>> email
>>> bin/lkp run job.yaml
>>>
>>> ===================================================================
>>> ======================
>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/n
>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel-
>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb-
>>> ep01/40G/fsmark
>>>
>>> commit:
>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use
>>> iov_iter_kvec()")
>>> 0472e47660 ("SUNRPC: Convert socket page send code to use
>>> iov_iter()")
>>>
>>> e791f8e9380d945e 0472e476604998c127f3c80d291
>>> ---------------- ---------------------------
>>> fail:runs %reproduction fail:runs
>>> | | |
>>> :4 50% 2:4 dmesg.WARNING:at#for
>>> _ip_interrupt_entry/0x
>>> %stddev %change %stddev
>>> \ | \
>>> 15118573 ± 2% +16.0% 17538083 fsmark.app_overhead
>>> 510.93 -22.7% 395.12 fsmark.files_per_sec
>>> 24.90 +22.8% 30.57 fsmark.time.elapsed_
>>> time
>>> 24.90 +22.8% 30.57 fsmark.time.elapsed_
>>> time.max
>>> 288.00 ± 2% -
>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got
>>> 70.03 ± 2% -
>>> 11.3% 62.14 fsmark.time.system_time
>>>
>>
>> Do you have time to take a look at this regression?
>
> From your stats, it looks to me as if the problem is increased NUMA
> overhead. Pretty much everything else appears to be the same or
> actually performing better than previously. Am I interpreting that
> correctly?
The real regression is the throughput(fsmark.files_per_sec) is decreased
by 22.7%.
>
> If my interpretation above is correct, then I'm not seeing where this
> patch would be introducing new NUMA regressions. It is just converting
> from using one method of doing socket I/O to another. Could it perhaps
> be a memory artefact due to your running the NFS client and server on
> the same machine?
>
> Apologies for pushing back a little, but I just don't have the
> hardware available to test NUMA configurations, so I'm relying on
> external testing for the above kind of scenario.
>
Thanks for looking at this. If you need more information, please let me
know.
> Thanks
> Trond
>
--
Zhengjun Xing
1 year, 9 months