May 2019 - LKP - Ml01.01.Org

[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression

by kernel test robot

Greeting, FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit: commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory with following parameters: disk: 2pmem fs: btrfs runtime: 200s nr_task: 50% time_based: tb rw: randwrite bs: 4k ioengine: libaio test_size: 100G cpufreq_governor: performance ucode: 0x3d test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d commit: ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check") 8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio") ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.BUG:unable_to_handle_kernel :4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt :4 25% 1:4 dmesg.Oops:#[##] :4 25% 1:4 dmesg.RIP:cpuidle_enter_state :4 25% 1:4 dmesg.RIP:native_write_msr :4 25% 1:4 dmesg.RIP:perf_prepare_sample 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms% 4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms% 8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms% 1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms% 66.81 -5.9 60.90 ± 2% fio.latency_4ms% 0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us% 1.017e+08 -28.5% 72701525 fio.time.file_system_outputs 109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got 197.97 -26.4% 145.79 ± 2% fio.time.system_time 38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time 12388015 -28.5% 8858878 fio.time.voluntary_context_switches 12711059 -28.5% 9083954 fio.workload 247.91 -28.6% 176.96 fio.write_bw_MBps 309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us 17574 +40.0% 24609 fio.write_clat_mean_us 57989 ± 4% +82.2% 105661 fio.write_clat_stddev 63463 -28.6% 45302 fio.write_iops 564.61 +40.2% 791.54 fio.write_slat_mean_us 9375 ± 7% +96.3% 18406 fio.write_slat_stddev 21.25 +8.5% 23.06 ± 9% boot-time.dhcp 93.32 +1.4% 94.58 iostat.cpu.idle 6.43 -18.6% 5.23 ± 2% iostat.cpu.system 6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys% 0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr% 93.00 +1.1% 94.00 vmstat.cpu.id 591200 -30.0% 413921 vmstat.io.bo 31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache 11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free 5.00 -26.7% 3.67 ± 12% vmstat.procs.r 635910 ± 2% -27.6% 460384 vmstat.system.cs 7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node 7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss 242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node 7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign 7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit 9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time 53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage 1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time 16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage 5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time 16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage 9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage 19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time 10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active 10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file) 31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached 115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree 6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M 534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty 11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree 37604344 ± 2% -9.4% 34060483 meminfo.Memused 806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs 15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs 17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs 12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs 7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs 8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs 263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz 9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy% 53502248 ± 6% -31.8% 36503898 turbostat.C1 5.86 ± 8% -2.0 3.82 turbostat.C1% 16540396 ± 13% -45.0% 9102733 turbostat.C1E 6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E% 16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3 33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3% 9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6 18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3 53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp 4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2 56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp 11.84 -4.8% 11.28 turbostat.RAMWatt 7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active 7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file) 308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty 8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive 8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file) 572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback 15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages 11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive 105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon) 11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file) 5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack 6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree 18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed 4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables 857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim 109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem 986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab 575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max 100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev 830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max 29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg 591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max 107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev 24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg 541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max 93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev 519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max 87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev 480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max 80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev 408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max 71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev 381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max 64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev 106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly 2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file 33315360 -30.4% 23192301 proc-vmstat.nr_dirtied 133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty 7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages 28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma 2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages 5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file 430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable 197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback 33243480 -30.4% 23143756 proc-vmstat.nr_written 2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file 5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file 134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending 15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit 15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local 2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate 16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal 17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree 1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout 5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd 5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd 32760 -7.6% 30285 proc-vmstat.slabs_scanned 1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file 10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied 76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty 2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file 142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback 10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written 1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file 2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file 77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending 5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit 5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local 68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss 119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other 30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes 6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied 3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages 28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma 1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon 2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file 5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack 29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped 1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages 27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem 214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable 75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback 6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon 2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file 68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign 5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit 4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local 18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI 2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions 1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate% 37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses 28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses 2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references 645248 ± 2% -27.4% 468301 perf-stat.i.context-switches 1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi 1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles 1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate% 31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses 3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads 0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate% 2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses 2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores 13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads 1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions 8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss 0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc 16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses 8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads 2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses 1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores 17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI 1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate% 1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi 689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses 0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate% 0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate% 10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate% 7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss 0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc 202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length 2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions 36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses 28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses 2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references 642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches 1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles 31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses 3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads 2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses 2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores 13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads 1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions 15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses 8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads 2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses 1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores 2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions 36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU 37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU 37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU 39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU 38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU 37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU 37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU 33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU 34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU 34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU 44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU 43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU 36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU 67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER 39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU 42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU 40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU 42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU 42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU 41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU 39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU 40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU 40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU 41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU 36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU 39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU 39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU 38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU 39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU 39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU 39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU 40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU 39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU 41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU 38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU 36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU 40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU 40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU 40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU 41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU 38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU 39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU 42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU 40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU 38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU 37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU 39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU 40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU 40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU 39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU 40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU 41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU 38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU 39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU 39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU 41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU 36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU 37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU 36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU 35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU 35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU 35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU 36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU 37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU 36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU 35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU 35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU 37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU 36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU 36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU 37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU 37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU 2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7 150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49 10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts 12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts 13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts 13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts 13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts 7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts 2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts 2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts 2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts 10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts 2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts 12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts 2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts 9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts 11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts 11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts 2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts 8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts 2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts 2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts 10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts 2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts 10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts 1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts 1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts 1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts 12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts 2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts 2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts 2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts 150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49 11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts 11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts 2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts 2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts 10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts 2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts 9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts 8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts 9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6 12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts 7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts 9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts 2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts 2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts 8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts 7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts 8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7 9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts 8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts 12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts 749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread 5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit 5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit 5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64 6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit 5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums 1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread 0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread 0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter 0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space 1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write 1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range 1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages 0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread 0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space 0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work 49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes 61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode 55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page 0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages 7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage 6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64 6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one 5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums 5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write 4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write 5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc 5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range 5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow 2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block 2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block 2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping 1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit 2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items 3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit 2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule 1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits 1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit 1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search 2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs 1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule 1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em 0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need 1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock 0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages 1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio 1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search 1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer 0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum 1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range 1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range 1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common 0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit 1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right 0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item 1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function 1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache 0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path 0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32 1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page 0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page 0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc 1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io 1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes 0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done 1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle 0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc 0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages 0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit 0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent 0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path 1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache 1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit 0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free 0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair 0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket 0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup 0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load 0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent 0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items 0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity 0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate 0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook 0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head 0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit 0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio 0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc 0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback 0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range 0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist 0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer 0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty 0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range 0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user 0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry 0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent 0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc 0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state 0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin 0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode 0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state 0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc 0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer 0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty 0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up 0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block 0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref 0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free 0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right 0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback 0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent 0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark 0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer 0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map 0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent 0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic 0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head 0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start 0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio 0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase 0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start 0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock 0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space 0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map 0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used 0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref 0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist 0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep 0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock 0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent 0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node 0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed 0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid 0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref 0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling 0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents 0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents 0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages 0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search 0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map 0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked 0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking 0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent 0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node 0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable 0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to 0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached 0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u 0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu 0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock 0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock 0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit 0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid 0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events 0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile 0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle 0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events 0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping 0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret 0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys 0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance 0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter 0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains 1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer 2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt 0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction 0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64 1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32 0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn 50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle 61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages 56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary 1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page 1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages 1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb 1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio 0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook 0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf 0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio 2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping 1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search 2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search 0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page 0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc 0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32 0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule 0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup 0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load 0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer 0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head 0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free 0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free 0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer 0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot 0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head 0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state 0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se 0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock 0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc 0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map 0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg 0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item 0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep 0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid 0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid 0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking 0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed 0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair 0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked 0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node 0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock 0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio 0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete 0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u 0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock 0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock 0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to 0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit 0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev 0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu 0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep 0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer 0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event 0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right 0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty 0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance 0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline 0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret 0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state 0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8 0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer 0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf 0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64 1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32 0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer 49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle fio.write_bw_MBps 300 +-+-------------------------------------------------------------------+ | | 250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ | | : : +. +..+ + +..+ +. + | | : : : : : : | 200 +-+ : : : : : : O O | O O O: O :O O O O O O O O O O: O:O O: O: O O O O O 150 +-+ : : : : : : | | : : : : : : | 100 +-+ : : : : : : | | : : : : : : | | : : : : : : | 50 +-+ :: :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_iops 70000 +-+-----------------------------------------------------------------+ |..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ | 60000 +-+ : : +. : : + : | | : : : : : : | 50000 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 40000 +-+ : O: : : : : | | : : : : : : | 30000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 10000 +-+ : : : | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_mean_us 30000 +-+-----------------------------------------------------------------+ | | 25000 O-+O O O O O O O O O O O O O O O O O O O | | O O O O O | | 20000 +-+ | |..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ | 15000 +-+ : : : : : : | | : : : : : : | 10000 +-+ : : : : : : | | : : : : : : | | : : : : : : | 5000 +-+ : : : : :: | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_stddev 160000 +-+----------------------------------------------------------------+ | | 140000 O-+O O O O O O O O O O O O O O | 120000 +-+ O O | | | 100000 +-+ O O O O O O O O | | 80000 +-+ | | .+.. | 60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ | 40000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 0 +-+-------------------------------------------O------------O-------+ fio.write_slat_mean_us 900 +-+-------------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O | 800 +-+O O O O O O O 700 +-+ | | | 600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ | 500 +-+ : : : : : : | | : : : : : : | 400 +-+ : : : : : : | 300 +-+ : : : : : : | | : : : : : : | 200 +-+ : : : : : : | 100 +-+ : : :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_slat_stddev 25000 +-+-----O---------O-------------------------------------------------+ O O O O O O O O O O O O O O O | | | 20000 +-+ | | O O O O O O O O | | 15000 +-+ | | | 10000 +-++. .+.. .+.. .+..+ .+ .+..+ | |. + +..+..+ +..+ + : +. : +..+.+. | | : : : : : : | 5000 +-+ : : : : : : | | : : : : : : | | :: : :: | 0 +-+--------------------------------------------O------------O-------+ fio.latency_1000ms_ 0.4 +-+------------------------------------------------------------------+ | | 0.35 +-+ O | 0.3 +-+ | | O O | 0.25 +-+ O O | | O O O 0.2 O-+ O O O O O O O O O O | | O O | 0.15 +-+O O O O | 0.1 +-+ | | | 0.05 +-+ | | .+. .+..+..+.+..+.. .+. .+.+..+.. | 0 +-+---------------------------------------------O------------O-------+ fio.workload 1.4e+07 +-+---------------------------------------------------------------+ |..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ | 1.2e+07 +-+ : : : : + : | | : : : : : : | 1e+07 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 8e+06 +-+ : O: : : : : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | :: :: :: | 2e+06 +-+ : : : | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.voluntary_context_switches 1.4e+07 +-+---------------------------------------------------------------+ | | 1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | 1e+07 +-+ : : : : : : | | O O: : O O O : : :O : O O O O O O O 8e+06 O-+ : O:O O O O O O O: O:O O: : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | : : : : :: | 2e+06 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.file_system_outputs 1.2e+08 +-+---------------------------------------------------------------+ | | 1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | | : : : : : : | 8e+07 +-+ : : : : : : O O O O O O O O: O:O O O O O O O O O O: O:O O:O : O O | 6e+07 +-+ : : : : : : | | : : : : : : | 4e+07 +-+ : : : : : : | | : : : : : : | | : : : : :: | 2e+07 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 year, 6 months

4
4
0 / 0

Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression

by Xing Zhengjun

On 5/31/2019 3:10 AM, Trond Myklebust wrote: > On Thu, 2019-05-30 at 15:20 +0800, Xing Zhengjun wrote: >> >> On 5/30/2019 10:00 AM, Trond Myklebust wrote: >>> Hi Xing, >>> >>> On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote: >>>> Hi Trond, >>>> >>>> On 5/20/2019 1:54 PM, kernel test robot wrote: >>>>> Greeting, >>>>> >>>>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due >>>>> to >>>>> commit: >>>>> >>>>> >>>>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: >>>>> Convert >>>>> socket page send code to use iov_iter()") >>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git >>>>> master >>>>> >>>>> in testcase: fsmark >>>>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ >>>>> 3.00GHz with 384G memory >>>>> with following parameters: >>>>> >>>>> iterations: 1x >>>>> nr_threads: 64t >>>>> disk: 1BRD_48G >>>>> fs: xfs >>>>> fs2: nfsv4 >>>>> filesize: 4M >>>>> test_size: 40G >>>>> sync_method: fsyncBeforeClose >>>>> cpufreq_governor: performance >>>>> >>>>> test-description: The fsmark is a file system benchmark to test >>>>> synchronous write workloads, for example, mail servers >>>>> workload. >>>>> test-url: https://sourceforge.net/projects/fsmark/ >>>>> >>>>> >>>>> >>>>> Details are as below: >>>>> ------------------------------------------------------------- >>>>> ---- >>>>> ---------------------------------> >>>>> >>>>> >>>>> To reproduce: >>>>> >>>>> git clone https://github.com/intel/lkp-tests.git >>>>> cd lkp-tests >>>>> bin/lkp install job.yaml # job file is attached in >>>>> this >>>>> email >>>>> bin/lkp run job.yaml >>>>> >>>>> =============================================================== >>>>> ==== >>>>> ====================== >>>>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconf >>>>> ig/n >>>>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase: >>>>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel- >>>>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb- >>>>> ep01/40G/fsmark >>>>> >>>>> commit: >>>>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use >>>>> iov_iter_kvec()") >>>>> 0472e47660 ("SUNRPC: Convert socket page send code to use >>>>> iov_iter()") >>>>> >>>>> e791f8e9380d945e 0472e476604998c127f3c80d291 >>>>> ---------------- --------------------------- >>>>> fail:runs %reproduction fail:runs >>>>> | | | >>>>> :4 50% 2:4 dmesg.WARNING:a >>>>> t#for >>>>> _ip_interrupt_entry/0x >>>>> %stddev %change %stddev >>>>> \ | \ >>>>> 15118573 >>>>> ± 2% +16.0% 17538083 fsmark.app_overhead >>>>> 510.93 - >>>>> 22.7% 395.12 fsmark.files_per_sec >>>>> 24.90 +22.8% 30.57 fsmark.time.ela >>>>> psed_ >>>>> time >>>>> 24.90 +22.8% 30.57 fsmark.time.ela >>>>> psed_ >>>>> time.max >>>>> 288.00 ± 2% - >>>>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got >>>>> 70.03 ± 2% - >>>>> 11.3% 62.14 fsmark.time.system_time >>>>> >>>> >>>> Do you have time to take a look at this regression? >>> >>> From your stats, it looks to me as if the problem is increased >>> NUMA >>> overhead. Pretty much everything else appears to be the same or >>> actually performing better than previously. Am I interpreting that >>> correctly? >> The real regression is the throughput(fsmark.files_per_sec) is >> decreased >> by 22.7%. > > Understood, but I'm trying to make sense of why. I'm not able to > reproduce this, so I have to rely on your performance stats to > understand where the 22.7% regression is coming from. As far as I can > see, the only numbers in the stats you published that are showing a > performance regression (other than the fsmark number itself), are the > NUMA numbers. Is that a correct interpretation? > We re-test the case yesterday, the test result almost is the same. we will do more test and also check the test case itself, if you need more information, please let me know, thanks. >>> If my interpretation above is correct, then I'm not seeing where >>> this >>> patch would be introducing new NUMA regressions. It is just >>> converting >>> from using one method of doing socket I/O to another. Could it >>> perhaps >>> be a memory artefact due to your running the NFS client and server >>> on >>> the same machine? >>> >>> Apologies for pushing back a little, but I just don't have the >>> hardware available to test NUMA configurations, so I'm relying on >>> external testing for the above kind of scenario. >>> >> Thanks for looking at this. If you need more information, please let >> me >> know. >>> Thanks >>> Trond >>> -- Zhengjun Xing

2 years, 3 months

1
1
0 / 0

[btrfs] c8eaeac7b7: aim7.jobs-per-min -11.7% regression

by kernel test robot

Greeting, FYI, we noticed a -11.7% regression of aim7.jobs-per-min due to commit: commit: c8eaeac7b734347c3afba7008b7af62f37b9c140 ("btrfs: reserve delalloc metadata differently") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: aim7 on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: btrfs test: disk_rr load: 1500 cpufreq_governor: performance test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/1500/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_rr/aim7 commit: 4297ff84dc ("btrfs: track DIO bytes in flight") c8eaeac7b7 ("btrfs: reserve delalloc metadata differently") 4297ff84dc24d120 c8eaeac7b734347c3afba7008b7 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 75% 3:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 75% 3:4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 43654 -11.7% 38526 aim7.jobs-per-min 206.46 +13.3% 233.89 aim7.time.elapsed_time 206.46 +13.3% 233.89 aim7.time.elapsed_time.max 611427 ± 3% +36.6% 834981 ± 2% aim7.time.involuntary_context_switches 103580 +1.7% 105361 aim7.time.minor_page_faults 7636 +14.8% 8763 aim7.time.system_time 2731877 +7.6% 2940550 aim7.time.voluntary_context_switches 0.83 ± 2% -0.1 0.73 ± 2% mpstat.cpu.all.usr% 94.44 +7.0% 101.08 iostat.md0.w/s 16028 +7.2% 17182 iostat.md0.wkB/s 2040 +24.8% 2547 ± 18% meminfo.Inactive(file) 17154 -10.2% 15396 meminfo.max_used_kB 2.4e+08 ± 45% -87.4% 30268527 ±145% cpuidle.C3.time 731742 ± 18% -88.7% 82339 ±109% cpuidle.C3.usage 4.447e+08 ± 12% +35.0% 6.005e+08 ± 19% cpuidle.C6.time 10907 +6.9% 11662 vmstat.io.bo 55.33 ± 4% +13.9% 63.00 ± 2% vmstat.procs.r 24049 -6.9% 22394 vmstat.system.cs 10588 ± 30% -44.7% 5859 ± 25% numa-meminfo.node0.Inactive 9856 ± 36% -47.8% 5145 ± 34% numa-meminfo.node0.Inactive(anon) 4691 ± 9% +243.5% 16115 ± 69% numa-meminfo.node0.KernelStack 2202 ± 29% +1132.8% 27151 ± 92% numa-meminfo.node0.PageTables 11332 ± 29% +46.9% 16649 ± 8% numa-meminfo.node1.Inactive 10023 ± 36% +47.8% 14817 ± 12% numa-meminfo.node1.Inactive(anon) 1185 +15.5% 1369 ± 7% slabinfo.btrfs_ordered_extent.active_objs 1185 +15.5% 1369 ± 7% slabinfo.btrfs_ordered_extent.num_objs 2748 ± 3% +6.5% 2927 ± 3% slabinfo.kmalloc-rcl-64.active_objs 2748 ± 3% +6.5% 2927 ± 3% slabinfo.kmalloc-rcl-64.num_objs 367.33 ± 3% +29.2% 474.75 ± 10% slabinfo.skbuff_fclone_cache.active_objs 367.33 ± 3% +29.2% 474.75 ± 10% slabinfo.skbuff_fclone_cache.num_objs 1545 ± 11% +14.2% 1764 ± 13% slabinfo.xfrm_dst_cache.num_objs 2464 ± 36% -47.8% 1285 ± 34% numa-vmstat.node0.nr_inactive_anon 4690 ± 9% +243.7% 16120 ± 69% numa-vmstat.node0.nr_kernel_stack 550.00 ± 29% +1134.5% 6790 ± 92% numa-vmstat.node0.nr_page_table_pages 152291 ± 3% +19.0% 181153 ± 5% numa-vmstat.node0.nr_written 2464 ± 36% -47.8% 1285 ± 34% numa-vmstat.node0.nr_zone_inactive_anon 2505 ± 36% +47.9% 3705 ± 12% numa-vmstat.node1.nr_inactive_anon 151216 ± 3% +18.3% 178821 ± 4% numa-vmstat.node1.nr_written 2505 ± 36% +47.9% 3705 ± 12% numa-vmstat.node1.nr_zone_inactive_anon 493640 +1.0% 498686 proc-vmstat.nr_file_pages 510.00 +24.8% 636.50 ± 18% proc-vmstat.nr_inactive_file 584507 ± 2% +20.8% 706353 ± 2% proc-vmstat.nr_written 510.00 +24.8% 636.50 ± 18% proc-vmstat.nr_zone_inactive_file 29772 ± 5% +4.0% 30949 ± 5% proc-vmstat.numa_hint_faults 15567 ± 6% +5.4% 16416 ± 6% proc-vmstat.numa_hint_faults_local 13816 -1.7% 13583 proc-vmstat.numa_other 582286 +10.0% 640790 proc-vmstat.pgfault 2337841 ± 2% +20.9% 2825298 ± 2% proc-vmstat.pgpgout 2730 +5.2% 2872 turbostat.Avg_MHz 731158 ± 18% -88.8% 81753 ±110% turbostat.C3 2.80 ± 44% -2.5 0.32 ±146% turbostat.C3% 4.98 ± 25% -37.4% 3.12 ± 17% turbostat.CPU%c1 1.14 ± 78% -79.4% 0.24 ±160% turbostat.CPU%c3 135.74 +10.2% 149.62 turbostat.CorWatt 18209025 +13.7% 20695723 turbostat.IRQ 162.95 +8.5% 176.76 turbostat.PkgWatt 9586 ± 3% +8.6% 10410 ± 2% turbostat.SMI 24351 -6.9% 22675 perf-stat.i.context-switches 1.095e+11 +5.2% 1.151e+11 perf-stat.i.cpu-cycles 2942 -8.8% 2682 perf-stat.i.cpu-migrations 4026 ± 2% +5.5% 4248 perf-stat.i.cycles-between-cache-misses 0.11 ± 7% +0.0 0.15 ± 27% perf-stat.i.dTLB-store-miss-rate% 1.788e+09 -10.4% 1.602e+09 perf-stat.i.dTLB-stores 4679972 ± 2% -36.2% 2983497 ± 47% perf-stat.i.iTLB-loads 0.24 -3.5% 0.23 perf-stat.i.ipc 2646 -2.6% 2578 perf-stat.i.minor-faults 40.94 +0.4 41.38 perf-stat.i.node-store-miss-rate% 2646 -2.6% 2578 perf-stat.i.page-faults 4.07 +4.1% 4.24 perf-stat.overall.cpi 4196 +5.1% 4411 perf-stat.overall.cycles-between-cache-misses 0.11 ± 8% +0.0 0.15 ± 29% perf-stat.overall.dTLB-store-miss-rate% 0.25 -4.0% 0.24 perf-stat.overall.ipc 24233 -6.8% 22576 perf-stat.ps.context-switches 1.089e+11 +5.2% 1.146e+11 perf-stat.ps.cpu-cycles 2928 -8.8% 2670 perf-stat.ps.cpu-migrations 1.779e+09 -10.3% 1.595e+09 perf-stat.ps.dTLB-stores 4657162 ± 2% -36.2% 2970461 ± 47% perf-stat.ps.iTLB-loads 2635 -2.5% 2570 perf-stat.ps.minor-faults 2635 -2.5% 2570 perf-stat.ps.page-faults 5.677e+12 +14.2% 6.482e+12 perf-stat.total.instructions 23484 +27.9% 30030 ± 2% sched_debug.cfs_rq:/.load.avg 51446 ± 2% +492.7% 304901 ± 2% sched_debug.cfs_rq:/.load.max 11212 ± 5% +332.6% 48506 ± 2% sched_debug.cfs_rq:/.load.stddev 0.21 ± 14% -27.1% 0.15 ± 19% sched_debug.cfs_rq:/.nr_running.stddev 50.75 ± 10% +138.7% 121.12 ± 91% sched_debug.cfs_rq:/.runnable_load_avg.max 1.33 ± 93% +275.0% 5.00 ± 28% sched_debug.cfs_rq:/.runnable_load_avg.min 18594 ± 4% +39.0% 25853 ± 3% sched_debug.cfs_rq:/.runnable_weight.avg 41096 ± 7% +627.8% 299088 sched_debug.cfs_rq:/.runnable_weight.max 9009 ± 3% +431.5% 47888 ± 2% sched_debug.cfs_rq:/.runnable_weight.stddev -4276 -670.0% 24374 ± 56% sched_debug.cfs_rq:/.spread0.avg 74100 ± 9% +24.2% 92022 ± 12% sched_debug.cfs_rq:/.spread0.max -57421 -37.4% -35972 sched_debug.cfs_rq:/.spread0.min 463.62 +12.9% 523.36 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg 0.25 ± 81% +9225.0% 23.31 ±119% sched_debug.cfs_rq:/.util_est_enqueued.min 18.98 ± 4% +9.4% 20.76 ± 4% sched_debug.cpu.cpu_load[0].avg 47.67 ± 5% +20.8% 57.56 ± 8% sched_debug.cpu.cpu_load[0].max 1.92 ± 43% +154.3% 4.88 ± 40% sched_debug.cpu.cpu_load[0].min 44.75 +22.2% 54.69 ± 9% sched_debug.cpu.cpu_load[1].max 5.42 ± 15% +38.5% 7.50 ± 9% sched_debug.cpu.cpu_load[2].min 56.17 ± 5% +14.1% 64.06 ± 7% sched_debug.cpu.cpu_load[3].max 6.33 ± 9% +28.3% 8.12 ± 9% sched_debug.cpu.cpu_load[3].min 7.17 ± 4% +16.9% 8.38 ± 8% sched_debug.cpu.cpu_load[4].min 454.42 ± 86% +142.9% 1103 ± 21% sched_debug.cpu.curr->pid.min 23570 +20.9% 28493 ± 8% sched_debug.cpu.load.avg 51446 ± 2% +371.3% 242447 ± 45% sched_debug.cpu.load.max 4293 ± 80% +118.5% 9381 ± 21% sched_debug.cpu.load.min 2039 ± 7% +10.2% 2247 ± 2% sched_debug.cpu.nr_load_updates.stddev 1.06 ± 2% +22.2% 1.29 ± 5% sched_debug.cpu.nr_running.avg 0.25 ± 81% +150.0% 0.62 ± 20% sched_debug.cpu.nr_running.min 0.52 +13.5% 0.59 ± 10% sched_debug.cpu.nr_running.stddev 19645 ± 5% +9.4% 21489 ± 4% softirqs.CPU0.RCU 86145 +14.7% 98787 softirqs.CPU0.TIMER 87552 ± 7% +12.6% 98576 ± 4% softirqs.CPU1.TIMER 19349 ± 9% +13.9% 22048 ± 5% softirqs.CPU10.RCU 81218 ± 3% +29.7% 105305 ± 12% softirqs.CPU10.TIMER 19223 ± 2% +8.7% 20891 ± 7% softirqs.CPU11.RCU 83829 ± 2% +12.5% 94302 softirqs.CPU11.TIMER 84709 +16.0% 98222 ± 2% softirqs.CPU12.TIMER 85057 ± 2% +10.5% 93955 softirqs.CPU13.TIMER 20677 ± 7% +6.1% 21934 ± 7% softirqs.CPU14.RCU 84147 ± 2% +17.2% 98617 softirqs.CPU14.TIMER 19576 ± 4% +13.0% 22113 ± 7% softirqs.CPU16.RCU 84576 ± 2% +33.1% 112557 ± 14% softirqs.CPU16.TIMER 85248 ± 4% +14.5% 97595 ± 3% softirqs.CPU17.TIMER 19195 ± 6% +15.2% 22106 ± 11% softirqs.CPU18.RCU 85046 ± 2% +14.4% 97311 softirqs.CPU18.TIMER 85248 ± 3% +11.4% 94985 ± 2% softirqs.CPU19.TIMER 19415 ± 3% +11.4% 21619 ± 3% softirqs.CPU2.RCU 19095 ± 4% +10.6% 21115 ± 5% softirqs.CPU20.RCU 84097 ± 2% +15.0% 96686 softirqs.CPU20.TIMER 87892 ± 3% +13.1% 99386 ± 5% softirqs.CPU21.TIMER 19125 ± 3% +13.4% 21686 ± 5% softirqs.CPU22.RCU 14525 ± 5% -9.0% 13212 ± 4% softirqs.CPU22.SCHED 85902 ± 5% +15.4% 99114 softirqs.CPU22.TIMER 19616 ± 3% +12.4% 22042 ± 2% softirqs.CPU23.RCU 86349 ± 4% +12.0% 96741 softirqs.CPU23.TIMER 19747 ± 6% +13.5% 22407 ± 7% softirqs.CPU24.RCU 84333 +17.1% 98749 softirqs.CPU24.TIMER 19131 ± 2% +11.2% 21279 ± 5% softirqs.CPU25.RCU 19625 ± 5% +9.1% 21420 ± 5% softirqs.CPU26.RCU 84545 ± 3% +15.5% 97675 softirqs.CPU26.TIMER 20319 ± 7% +5.3% 21389 ± 6% softirqs.CPU27.RCU 86276 ± 4% +12.1% 96743 softirqs.CPU27.TIMER 18484 ± 4% +15.3% 21315 ± 7% softirqs.CPU28.RCU 83144 +16.5% 96862 softirqs.CPU28.TIMER 19786 ± 2% +12.0% 22157 ± 4% softirqs.CPU3.RCU 87487 ± 5% +12.2% 98117 softirqs.CPU3.TIMER 80763 ± 3% +20.0% 96896 ± 2% softirqs.CPU30.TIMER 83628 ± 2% +12.5% 94076 softirqs.CPU31.TIMER 85112 ± 4% +14.8% 97737 ± 3% softirqs.CPU32.TIMER 84626 ± 3% +10.9% 93812 softirqs.CPU33.TIMER 87944 ± 6% +12.1% 98575 softirqs.CPU34.TIMER 86859 +22.5% 106365 ± 13% softirqs.CPU36.TIMER 85336 ± 4% +16.8% 99655 ± 4% softirqs.CPU37.TIMER 85090 ± 2% +14.2% 97153 softirqs.CPU38.TIMER 19528 ± 4% +16.7% 22783 ± 8% softirqs.CPU4.RCU 84396 +18.0% 99614 softirqs.CPU4.TIMER 19042 ± 4% +8.8% 20714 ± 4% softirqs.CPU5.RCU 87389 ± 5% +12.7% 98473 ± 2% softirqs.CPU5.TIMER 20312 ± 3% +13.2% 22992 ± 6% softirqs.CPU6.RCU 84550 ± 3% +15.8% 97877 softirqs.CPU6.TIMER 18516 ± 7% +24.9% 23125 ± 5% softirqs.CPU8.RCU 82667 +28.2% 106005 ± 11% softirqs.CPU8.TIMER 764529 ± 2% +8.9% 832849 ± 5% softirqs.RCU 3442286 ± 2% +13.9% 3922206 softirqs.TIMER 275.00 ± 17% +64.0% 451.00 ± 28% interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0 204.00 ± 56% +41.4% 288.50 ± 52% interrupts.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3 103.67 +13.3% 117.50 interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 78055 +10.9% 86558 interrupts.CAL:Function_call_interrupts 425311 +13.1% 480959 interrupts.CPU0.LOC:Local_timer_interrupts 22301 ± 3% +25.5% 27997 interrupts.CPU0.RES:Rescheduling_interrupts 2033 ± 5% +12.1% 2278 ± 3% interrupts.CPU1.CAL:Function_call_interrupts 425286 +13.0% 480441 interrupts.CPU1.LOC:Local_timer_interrupts 22426 ± 3% +26.7% 28410 ± 2% interrupts.CPU1.RES:Rescheduling_interrupts 424918 +13.2% 480935 interrupts.CPU10.LOC:Local_timer_interrupts 22298 ± 2% +25.8% 28049 interrupts.CPU10.RES:Rescheduling_interrupts 1959 ± 5% +12.8% 2210 ± 6% interrupts.CPU11.CAL:Function_call_interrupts 424741 +13.1% 480460 interrupts.CPU11.LOC:Local_timer_interrupts 21914 ± 3% +27.7% 27983 interrupts.CPU11.RES:Rescheduling_interrupts 424836 +13.2% 480889 interrupts.CPU12.LOC:Local_timer_interrupts 21863 ± 3% +26.4% 27640 interrupts.CPU12.RES:Rescheduling_interrupts 1870 +21.7% 2276 ± 6% interrupts.CPU13.CAL:Function_call_interrupts 425017 +12.9% 479899 interrupts.CPU13.LOC:Local_timer_interrupts 1379 ±140% +232.7% 4590 ± 62% interrupts.CPU13.NMI:Non-maskable_interrupts 1379 ±140% +232.7% 4590 ± 62% interrupts.CPU13.PMI:Performance_monitoring_interrupts 21979 ± 2% +30.5% 28691 ± 3% interrupts.CPU13.RES:Rescheduling_interrupts 424883 +13.2% 480884 interrupts.CPU14.LOC:Local_timer_interrupts 21978 +27.1% 27940 ± 2% interrupts.CPU14.RES:Rescheduling_interrupts 1919 ± 8% +18.7% 2278 ± 6% interrupts.CPU15.CAL:Function_call_interrupts 425309 +13.0% 480679 interrupts.CPU15.LOC:Local_timer_interrupts 22330 ± 3% +25.9% 28118 interrupts.CPU15.RES:Rescheduling_interrupts 1925 ± 6% +18.0% 2271 ± 4% interrupts.CPU16.CAL:Function_call_interrupts 424967 +13.1% 480548 interrupts.CPU16.LOC:Local_timer_interrupts 21993 ± 3% +28.3% 28215 interrupts.CPU16.RES:Rescheduling_interrupts 2102 +9.7% 2305 ± 5% interrupts.CPU17.CAL:Function_call_interrupts 424760 +13.0% 480175 interrupts.CPU17.LOC:Local_timer_interrupts 22036 ± 2% +28.5% 28307 ± 2% interrupts.CPU17.RES:Rescheduling_interrupts 424881 +13.1% 480700 interrupts.CPU18.LOC:Local_timer_interrupts 22029 ± 3% +26.9% 27950 interrupts.CPU18.RES:Rescheduling_interrupts 424963 +13.1% 480489 interrupts.CPU19.LOC:Local_timer_interrupts 23004 ± 3% +27.6% 29354 ± 2% interrupts.CPU19.RES:Rescheduling_interrupts 424990 +13.1% 480714 interrupts.CPU2.LOC:Local_timer_interrupts 22102 ± 3% +26.5% 27967 interrupts.CPU2.RES:Rescheduling_interrupts 1757 ± 6% +17.1% 2057 ± 6% interrupts.CPU20.CAL:Function_call_interrupts 425145 +13.1% 480765 interrupts.CPU20.LOC:Local_timer_interrupts 22303 ± 2% +25.8% 28066 interrupts.CPU20.RES:Rescheduling_interrupts 425267 +12.9% 480312 interrupts.CPU21.LOC:Local_timer_interrupts 21813 ± 2% +27.1% 27723 interrupts.CPU21.RES:Rescheduling_interrupts 424618 +13.2% 480680 interrupts.CPU22.LOC:Local_timer_interrupts 22015 ± 3% +26.0% 27734 interrupts.CPU22.RES:Rescheduling_interrupts 1912 ± 8% +18.9% 2273 ± 5% interrupts.CPU23.CAL:Function_call_interrupts 424735 +13.1% 480469 interrupts.CPU23.LOC:Local_timer_interrupts 22040 +27.5% 28098 ± 3% interrupts.CPU23.RES:Rescheduling_interrupts 1924 ± 5% +6.4% 2048 ± 4% interrupts.CPU24.CAL:Function_call_interrupts 424618 +13.2% 480598 interrupts.CPU24.LOC:Local_timer_interrupts 21873 ± 2% +28.1% 28009 interrupts.CPU24.RES:Rescheduling_interrupts 424698 +13.1% 480416 interrupts.CPU25.LOC:Local_timer_interrupts 22041 +28.3% 28287 ± 2% interrupts.CPU25.RES:Rescheduling_interrupts 275.00 ± 17% +64.0% 451.00 ± 28% interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0 424902 +13.1% 480512 interrupts.CPU26.LOC:Local_timer_interrupts 22002 ± 3% +27.0% 27949 interrupts.CPU26.RES:Rescheduling_interrupts 424698 +13.2% 480806 interrupts.CPU27.LOC:Local_timer_interrupts 5859 ± 30% -81.3% 1093 ±173% interrupts.CPU27.NMI:Non-maskable_interrupts 5859 ± 30% -81.3% 1093 ±173% interrupts.CPU27.PMI:Performance_monitoring_interrupts 22290 +25.4% 27948 ± 3% interrupts.CPU27.RES:Rescheduling_interrupts 1904 ± 9% +12.6% 2144 ± 8% interrupts.CPU28.CAL:Function_call_interrupts 424963 +13.1% 480696 interrupts.CPU28.LOC:Local_timer_interrupts 21860 +28.7% 28145 interrupts.CPU28.RES:Rescheduling_interrupts 1933 ± 7% +18.4% 2288 ± 4% interrupts.CPU29.CAL:Function_call_interrupts 424503 +13.2% 480645 interrupts.CPU29.LOC:Local_timer_interrupts 22101 ± 4% +27.4% 28162 ± 3% interrupts.CPU29.RES:Rescheduling_interrupts 1994 ± 2% +11.0% 2214 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 424920 +13.1% 480485 interrupts.CPU3.LOC:Local_timer_interrupts 21988 ± 2% +27.7% 28074 ± 2% interrupts.CPU3.RES:Rescheduling_interrupts 425052 +13.1% 480827 interrupts.CPU30.LOC:Local_timer_interrupts 22012 ± 2% +26.6% 27873 interrupts.CPU30.RES:Rescheduling_interrupts 1986 ± 6% +12.8% 2240 ± 5% interrupts.CPU31.CAL:Function_call_interrupts 424997 +13.1% 480581 interrupts.CPU31.LOC:Local_timer_interrupts 21947 ± 3% +27.3% 27935 ± 2% interrupts.CPU31.RES:Rescheduling_interrupts 204.00 ± 56% +41.4% 288.50 ± 52% interrupts.CPU32.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3 424705 +13.1% 480486 interrupts.CPU32.LOC:Local_timer_interrupts 21887 +28.5% 28124 interrupts.CPU32.RES:Rescheduling_interrupts 425192 +13.0% 480284 interrupts.CPU33.LOC:Local_timer_interrupts 22206 ± 3% +25.4% 27853 ± 2% interrupts.CPU33.RES:Rescheduling_interrupts 1972 ± 6% +12.8% 2224 ± 6% interrupts.CPU34.CAL:Function_call_interrupts 424557 +13.2% 480684 interrupts.CPU34.LOC:Local_timer_interrupts 22117 ± 3% +25.6% 27777 interrupts.CPU34.RES:Rescheduling_interrupts 1894 +13.4% 2147 ± 4% interrupts.CPU35.CAL:Function_call_interrupts 424767 +13.1% 480496 interrupts.CPU35.LOC:Local_timer_interrupts 22043 ± 3% +28.0% 28225 interrupts.CPU35.RES:Rescheduling_interrupts 424967 +13.1% 480760 interrupts.CPU36.LOC:Local_timer_interrupts 21838 ± 3% +28.6% 28082 interrupts.CPU36.RES:Rescheduling_interrupts 1855 ± 3% +19.7% 2220 ± 7% interrupts.CPU37.CAL:Function_call_interrupts 424903 +13.0% 480330 interrupts.CPU37.LOC:Local_timer_interrupts 22221 ± 2% +27.8% 28392 ± 2% interrupts.CPU37.RES:Rescheduling_interrupts 1880 +15.3% 2167 ± 6% interrupts.CPU38.CAL:Function_call_interrupts 425042 +13.1% 480659 interrupts.CPU38.LOC:Local_timer_interrupts 21922 ± 3% +28.6% 28182 interrupts.CPU38.RES:Rescheduling_interrupts 1845 ± 2% +19.8% 2212 ± 6% interrupts.CPU39.CAL:Function_call_interrupts 425307 +13.0% 480459 interrupts.CPU39.LOC:Local_timer_interrupts 21737 ± 2% +30.3% 28324 ± 2% interrupts.CPU39.RES:Rescheduling_interrupts 1901 ± 8% +18.2% 2247 ± 4% interrupts.CPU4.CAL:Function_call_interrupts 424921 +13.2% 480871 interrupts.CPU4.LOC:Local_timer_interrupts 22205 ± 2% +26.1% 28002 interrupts.CPU4.RES:Rescheduling_interrupts 425096 +13.1% 480655 interrupts.CPU5.LOC:Local_timer_interrupts 22004 +28.7% 28319 ± 2% interrupts.CPU5.RES:Rescheduling_interrupts 103.67 +13.3% 117.50 interrupts.CPU6.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 425006 +13.1% 480843 interrupts.CPU6.LOC:Local_timer_interrupts 22177 +26.5% 28053 interrupts.CPU6.RES:Rescheduling_interrupts 1962 ± 5% +17.3% 2301 ± 7% interrupts.CPU7.CAL:Function_call_interrupts 424930 +13.1% 480702 interrupts.CPU7.LOC:Local_timer_interrupts 1402 ±138% +387.3% 6831 ± 49% interrupts.CPU7.NMI:Non-maskable_interrupts 1402 ±138% +387.3% 6831 ± 49% interrupts.CPU7.PMI:Performance_monitoring_interrupts 21891 ± 2% +28.1% 28048 ± 2% interrupts.CPU7.RES:Rescheduling_interrupts 1765 ± 6% +15.8% 2044 ± 5% interrupts.CPU8.CAL:Function_call_interrupts 423965 +13.4% 480881 interrupts.CPU8.LOC:Local_timer_interrupts 22606 ± 4% +23.5% 27917 interrupts.CPU8.RES:Rescheduling_interrupts 424961 +13.1% 480710 interrupts.CPU9.LOC:Local_timer_interrupts 22153 +27.1% 28146 ± 2% interrupts.CPU9.RES:Rescheduling_interrupts 16996312 +13.1% 19224399 interrupts.LOC:Local_timer_interrupts 883558 ± 2% +27.2% 1124085 interrupts.RES:Rescheduling_interrupts 43.50 -10.6 32.91 perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 42.95 -10.6 32.38 perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 41.58 -10.4 31.23 perf-profile.calltrace.cycles-pp._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 41.28 -10.3 30.96 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write 42.18 -10.3 31.90 perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter 42.48 -10.3 32.21 perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 42.65 -10.3 32.39 perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 42.08 -10.2 31.85 perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write 0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 0.87 ± 4% -0.6 0.27 ±100% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 0.88 ± 2% -0.5 0.40 ± 58% perf-profile.calltrace.cycles-pp.secondary_startup_64 2.01 -0.4 1.60 ± 3% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 1.55 ± 2% -0.3 1.22 ± 3% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 1.78 -0.3 1.46 perf-profile.calltrace.cycles-pp.read 1.53 ± 2% -0.3 1.21 ± 3% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write 1.10 ± 2% -0.3 0.81 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages 1.05 ± 2% -0.3 0.77 ± 6% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent 1.31 -0.2 1.06 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.32 -0.2 1.09 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 1.22 -0.2 1.01 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.16 -0.2 0.95 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.43 ± 4% -0.2 1.23 ± 3% perf-profile.calltrace.cycles-pp.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 0.76 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close 0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64 0.76 ± 2% -0.2 0.59 ± 3% perf-profile.calltrace.cycles-pp.close 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.btrfs_evict_inode.evict.__dentry_kill.dentry_kill.dput 0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop 0.75 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run 0.74 ± 2% -0.2 0.57 ± 3% perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput 0.86 -0.1 0.73 perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 ± 2% -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.lseek64 0.83 -0.1 0.70 perf-profile.calltrace.cycles-pp.generic_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64 0.74 ± 4% -0.1 0.61 ± 5% perf-profile.calltrace.cycles-pp.pagecache_get_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.64 ± 5% -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.59 ± 4% -0.0 0.54 ± 2% perf-profile.calltrace.cycles-pp.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter 0.58 ± 5% -0.0 0.53 ± 2% perf-profile.calltrace.cycles-pp.__extent_read_full_page.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write 94.94 +1.2 96.10 perf-profile.calltrace.cycles-pp.write 94.38 +1.2 95.60 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 94.47 +1.2 95.69 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 94.49 +1.2 95.72 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 94.29 +1.2 95.53 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 93.96 +1.3 95.23 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 93.90 +1.3 95.19 perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 93.59 +1.3 94.92 perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 41.79 +12.8 54.57 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release 3.76 +22.7 26.46 perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 0.00 +23.2 23.16 perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent 0.00 +23.2 23.20 perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit 0.83 ± 3% +23.3 24.09 perf-profile.calltrace.cycles-pp.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.81 ± 3% +23.3 24.09 perf-profile.calltrace.cycles-pp.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 0.58 ± 2% +23.3 23.88 perf-profile.calltrace.cycles-pp.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages 0.60 ± 2% +23.3 23.90 perf-profile.calltrace.cycles-pp.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write 0.00 +23.4 23.36 perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit 0.00 +23.5 23.48 perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit 43.13 -10.6 32.50 perf-profile.children.cycles-pp.reserve_metadata_bytes 43.51 -10.6 32.92 perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 2.02 -0.4 1.60 ± 3% perf-profile.children.cycles-pp.btrfs_get_extent 0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.secondary_startup_64 0.87 ± 4% -0.4 0.50 ± 9% perf-profile.children.cycles-pp.start_secondary 0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.cpu_startup_entry 0.88 ± 2% -0.4 0.51 ± 10% perf-profile.children.cycles-pp.do_idle 0.83 ± 3% -0.4 0.48 ± 9% perf-profile.children.cycles-pp.cpuidle_enter_state 0.82 ± 4% -0.3 0.47 ± 9% perf-profile.children.cycles-pp.intel_idle 1.64 ± 2% -0.3 1.30 ± 3% perf-profile.children.cycles-pp.btrfs_search_slot 1.55 ± 2% -0.3 1.22 ± 3% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 1.81 -0.3 1.50 perf-profile.children.cycles-pp.read 1.15 ± 3% -0.3 0.86 ± 4% perf-profile.children.cycles-pp.btrfs_read_lock_root_node 1.10 ± 4% -0.3 0.81 ± 6% perf-profile.children.cycles-pp.btrfs_tree_read_lock 1.01 ± 4% -0.3 0.74 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.23 -0.2 1.01 perf-profile.children.cycles-pp.ksys_read 1.43 ± 4% -0.2 1.24 ± 3% perf-profile.children.cycles-pp.prepare_pages 1.16 -0.2 0.96 perf-profile.children.cycles-pp.vfs_read 1.07 ± 3% -0.2 0.87 ± 3% perf-profile.children.cycles-pp.__set_extent_bit 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.task_work_run 0.65 ± 5% -0.2 0.48 ± 9% perf-profile.children.cycles-pp.prepare_to_wait_event 0.76 ± 2% -0.2 0.59 ± 3% perf-profile.children.cycles-pp.close 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.__fput 0.75 ± 2% -0.2 0.58 ± 3% perf-profile.children.cycles-pp.dput 0.74 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.btrfs_evict_inode 0.75 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.dentry_kill 0.75 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.__dentry_kill 0.75 ± 2% -0.2 0.58 ± 4% perf-profile.children.cycles-pp.exit_to_usermode_loop 0.74 ± 2% -0.2 0.57 ± 3% perf-profile.children.cycles-pp.evict 0.92 ± 2% -0.2 0.76 ± 3% perf-profile.children.cycles-pp.pagecache_get_page 0.75 ± 2% -0.1 0.61 perf-profile.children.cycles-pp.lseek64 0.87 -0.1 0.73 perf-profile.children.cycles-pp.new_sync_read 0.84 -0.1 0.70 perf-profile.children.cycles-pp.generic_file_read_iter 0.68 ± 2% -0.1 0.55 perf-profile.children.cycles-pp.entry_SYSCALL_64 0.46 -0.1 0.34 ± 5% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need 0.32 ± 6% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.creat 0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.do_sys_open 0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.do_filp_open 0.32 ± 5% -0.1 0.21 ± 6% perf-profile.children.cycles-pp.path_openat 1.00 ± 2% -0.1 0.90 ± 4% perf-profile.children.cycles-pp.can_overcommit 0.62 -0.1 0.52 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.43 ± 5% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.lock_extent_bits 0.57 ± 4% -0.1 0.48 ± 3% perf-profile.children.cycles-pp.set_extent_bit 0.31 ± 5% -0.1 0.22 ± 9% perf-profile.children.cycles-pp.finish_wait 0.43 ± 2% -0.1 0.35 perf-profile.children.cycles-pp.alloc_extent_state 0.16 ± 17% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.call_rwsem_down_write_failed 0.16 ± 17% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.rwsem_down_write_failed 0.20 ± 12% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.down_write 0.23 ± 2% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.do_unlinkat 0.23 -0.1 0.17 ± 3% perf-profile.children.cycles-pp.unlink 0.31 ± 3% -0.1 0.25 ± 2% perf-profile.children.cycles-pp.truncate_inode_pages_range 0.64 ± 5% -0.1 0.58 ± 2% perf-profile.children.cycles-pp.prepare_uptodate_page 0.46 -0.1 0.40 ± 3% perf-profile.children.cycles-pp.copy_page_to_iter 0.40 ± 5% -0.1 0.34 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 0.48 ± 2% -0.1 0.42 ± 2% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.46 -0.1 0.40 ± 2% perf-profile.children.cycles-pp.__do_readpage 0.38 ± 6% -0.1 0.32 ± 2% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc 0.31 -0.1 0.25 ± 5% perf-profile.children.cycles-pp.security_file_permission 0.29 ± 4% -0.1 0.24 perf-profile.children.cycles-pp.find_get_entry 0.17 ± 4% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.btrfs_create 0.18 ± 9% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.btrfs_trans_release_metadata 0.59 ± 4% -0.0 0.54 ± 2% perf-profile.children.cycles-pp.extent_read_full_page 0.17 ± 4% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.btrfs_unlink 0.18 ± 9% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.__btrfs_end_transaction 0.17 ± 4% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.vfs_unlink 0.31 ± 5% -0.0 0.27 ± 3% perf-profile.children.cycles-pp.btrfs_alloc_data_chunk_ondemand 0.14 ± 11% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.__btrfs_update_delayed_inode 0.33 ± 5% -0.0 0.29 ± 5% perf-profile.children.cycles-pp.btrfs_check_data_free_space 0.28 ± 7% -0.0 0.24 ± 5% perf-profile.children.cycles-pp.add_to_page_cache_lru 0.21 ± 8% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.memset_erms 0.11 ± 15% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.__btrfs_btree_balance_dirty 0.06 ± 7% -0.0 0.03 ±100% perf-profile.children.cycles-pp.__btrfs_unlink_inode 0.19 ± 4% -0.0 0.15 ± 2% perf-profile.children.cycles-pp.___might_sleep 0.16 ± 5% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.xas_load 0.11 ± 19% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.btrfs_delayed_inode_release_metadata 0.24 ± 5% -0.0 0.20 ± 3% perf-profile.children.cycles-pp.btrfs_copy_from_user 0.07 ± 17% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.21 ± 3% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.selinux_file_permission 0.21 ± 6% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.set_state_bits 0.36 ± 2% -0.0 0.33 ± 3% perf-profile.children.cycles-pp.copyout 0.10 ± 8% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.start_transaction 0.09 ± 18% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode 0.18 ± 4% -0.0 0.15 ± 11% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.17 ± 5% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.12 ± 10% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.free_extent_state 0.07 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.btrfs_delalloc_release_metadata 0.09 ± 10% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.btrfs_block_rsv_add 0.10 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.truncate_cleanup_page 0.18 ± 7% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size 0.06 ± 7% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.btrfs_unlink_inode 0.14 ± 3% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.copyin 0.18 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.08 ± 10% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.kfree 0.12 ± 3% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.release_pages 0.09 ± 9% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.xas_store 0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.split_state 0.07 -0.0 0.05 ± 8% perf-profile.children.cycles-pp.current_time 0.12 ± 4% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.__pagevec_release 0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.try_merge_map 0.10 -0.0 0.09 ± 4% perf-profile.children.cycles-pp.btrfs_add_extent_mapping 0.09 ± 24% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.uart_console_write 0.09 ± 24% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.serial8250_console_putchar 0.16 ± 7% +0.1 0.28 ± 3% perf-profile.children.cycles-pp.btrfs_free_reserved_data_space_noquota 97.42 +0.6 98.04 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 97.35 +0.6 97.97 perf-profile.children.cycles-pp.do_syscall_64 94.98 +1.2 96.13 perf-profile.children.cycles-pp.write 94.38 +1.2 95.61 perf-profile.children.cycles-pp.ksys_write 94.30 +1.2 95.54 perf-profile.children.cycles-pp.vfs_write 93.96 +1.3 95.24 perf-profile.children.cycles-pp.new_sync_write 93.91 +1.3 95.19 perf-profile.children.cycles-pp.btrfs_file_write_iter 93.60 +1.3 94.93 perf-profile.children.cycles-pp.btrfs_buffered_write 85.03 +2.0 87.03 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 86.13 +2.3 88.43 perf-profile.children.cycles-pp._raw_spin_lock 42.75 +12.7 55.42 perf-profile.children.cycles-pp.space_info_add_old_bytes 43.08 +12.8 55.90 perf-profile.children.cycles-pp.block_rsv_release_bytes 42.99 +12.9 55.92 perf-profile.children.cycles-pp.btrfs_inode_rsv_release 3.76 +22.7 26.47 perf-profile.children.cycles-pp.btrfs_dirty_pages 1.36 ± 3% +23.2 24.57 perf-profile.children.cycles-pp.__clear_extent_bit 0.89 ± 2% +23.2 24.14 perf-profile.children.cycles-pp.clear_extent_bit 0.94 ± 2% +23.3 24.21 perf-profile.children.cycles-pp.clear_state_bit 0.67 +23.3 23.95 perf-profile.children.cycles-pp.btrfs_clear_delalloc_extent 0.82 ± 4% -0.3 0.47 ± 9% perf-profile.self.cycles-pp.intel_idle 0.60 ± 2% -0.1 0.49 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.62 -0.1 0.52 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.30 ± 13% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.reserve_metadata_bytes 0.48 -0.1 0.42 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.51 ± 5% -0.1 0.45 ± 3% perf-profile.self.cycles-pp.can_overcommit 0.23 ± 7% -0.0 0.18 ± 8% perf-profile.self.cycles-pp.__set_extent_bit 0.23 ± 8% -0.0 0.19 ± 6% perf-profile.self.cycles-pp.kmem_cache_alloc 0.08 ± 12% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.btrfs_get_extent 0.20 ± 6% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.memset_erms 0.18 ± 10% -0.0 0.14 ± 5% perf-profile.self.cycles-pp.do_syscall_64 0.18 ± 6% -0.0 0.15 ± 2% perf-profile.self.cycles-pp.___might_sleep 0.07 ± 17% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.14 ± 6% -0.0 0.11 perf-profile.self.cycles-pp.btrfs_calculate_inode_block_rsv_size 0.11 ± 8% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.free_extent_state 0.16 ± 6% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.find_get_entry 0.10 ± 12% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.xas_load 0.14 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.selinux_file_permission 0.14 -0.0 0.12 ± 8% perf-profile.self.cycles-pp.btrfs_file_write_iter 0.15 ± 5% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.btrfs_buffered_write 0.11 ± 4% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.generic_file_read_iter 0.10 ± 4% -0.0 0.09 ± 4% perf-profile.self.cycles-pp.btrfs_dirty_pages 0.06 -0.0 0.05 perf-profile.self.cycles-pp.rb_next 0.16 ± 2% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.btrfs_drop_pages 0.05 ± 8% +0.0 0.09 ± 20% perf-profile.self.cycles-pp.btrfs_delalloc_reserve_metadata 0.00 +0.1 0.12 ± 8% perf-profile.self.cycles-pp.btrfs_free_reserved_data_space_noquota 84.44 +2.1 86.53 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath aim7.jobs-per-min 50000 +-+-----------------------------------------------------------------+ 45000 +-+ .+..+..+..+..+..+..+..+..+..+.+.. .+.. .+ | | +. +..+..+. +..+. : +..+..| 40000 O-+ O O O O O O O O O O O O O O O O O O | 35000 +-+O O : : | | : : | 30000 +-+ : : | 25000 +-+ : : | 20000 +-+ : : | | : : | 15000 +-+ : : | 10000 +-+ : : | | :: | 5000 +-+ : | 0 +-+------------------------------O----------------O-----------------+ aim7.time.system_time 10000 +-+-----------------------------------------------------------------+ 9000 O-+O O O O O | | O O O O O O O O O O O O O O O | 8000 +-++.. .+..+.. .+..+.. +..+..| 7000 +-+ +..+..+..+..+..+..+..+..+..+.+..+. +. + : | | : : | 6000 +-+ : : | 5000 +-+ : : | 4000 +-+ : : | | : : | 3000 +-+ : : | 2000 +-+ : : | | : : | 1000 +-+ : | 0 +-+------------------------------O----------------O-----------------+ aim7.time.elapsed_time 250 +-+------O------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O O O | | .+..+.. +..+..| 200 +-++..+..+..+..+..+..+..+..+..+..+...+..+. +..+..+..+ : | | : : | | : : | 150 +-+ : : | | : : | 100 +-+ : : | | : : | | : : | 50 +-+ : : | | : : | | : | 0 +-+------------------------------O------------------O-----------------+ aim7.time.elapsed_time.max 250 +-+------O------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O O O | | .+..+.. +..+..| 200 +-++..+..+..+..+..+..+..+..+..+..+...+..+. +..+..+..+ : | | : : | | : : | 150 +-+ : : | | : : | 100 +-+ : : | | : : | | : : | 50 +-+ : : | | : : | | : | 0 +-+------------------------------O------------------O-----------------+ aim7.time.involuntary_context_switches 1e+06 +-+----------------------------------------------------------------+ 900000 O-+O O | | O O O O O O O O O O O O O O | 800000 +-+ O O O O | 700000 +-+ | | .+.. +.. | 600000 +-++..+..+..+..+.+..+..+..+..+..+..+..+. +..+..+.+..+ : +..| 500000 +-+ : : | 400000 +-+ : : | | : : | 300000 +-+ : : | 200000 +-+ : : | | : : | 100000 +-+ : | 0 +-+-----------------------------O-----------------O----------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 4 months

2
3
0 / 0

[tcp] 8b27dae5a2: netperf.Throughput_Mbps -25.7% regression

by kernel test robot

Greeting, FYI, we noticed a -25.7% regression of netperf.Throughput_Mbps due to commit: commit: 8b27dae5a2e89a61c46c6dbc76c040c0e6d0ed4c ("tcp: add one skb cache for rx") https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master in testcase: netperf on test machine: 104 threads Skylake with 192G memory with following parameters: ip: ipv4 runtime: 900s nr_threads: 200% cluster: cs-localhost test: TCP_STREAM cpufreq_governor: performance test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_Mbps -25.6% regression | | test machine | 104 threads Skylake with 192G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=200% | | | runtime=900s | | | test=TCP_MAERTS | +------------------+----------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/900s/lkp-skl-fpga01/TCP_STREAM/netperf commit: 472c2e07ee ("tcp: add one skb cache for tx") 8b27dae5a2 ("tcp: add one skb cache for rx") 472c2e07eef04514 8b27dae5a2e89a61c46c6dbc76c ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x :4 25% 1:4 dmesg.WARNING:at_ip_ip_finish_output2/0x %stddev %change %stddev \ | \ 3939 -25.7% 2925 netperf.Throughput_Mbps 819330 -25.7% 608425 netperf.Throughput_total_Mbps 3.725e+09 -25.8% 2.764e+09 netperf.time.involuntary_context_switches 7399 -1.8% 7266 netperf.time.percent_of_cpu_this_job_got 65271 -1.0% 64597 netperf.time.system_time 1529 -34.0% 1009 netperf.time.user_time 5.626e+09 -25.7% 4.178e+09 netperf.workload 12.36 ± 4% +6.5 18.84 mpstat.cpu.all.soft% 2.95 ± 2% -0.8 2.15 mpstat.cpu.all.usr% 9.557e+08 +13.2% 1.082e+09 numa-numastat.node0.local_node 9.557e+08 +13.2% 1.082e+09 numa-numastat.node0.numa_hit 4.845e+08 ± 2% +11.6% 5.407e+08 numa-vmstat.node0.numa_hit 4.845e+08 ± 2% +11.6% 5.406e+08 numa-vmstat.node0.numa_local 2662 ± 2% +4.2% 2774 turbostat.Avg_MHz 1.87 ± 86% -66.1% 0.63 ± 9% turbostat.CPU%c1 94.75 ± 2% +2.4% 97.00 vmstat.cpu.sy 8145843 ± 2% -24.7% 6134835 vmstat.system.cs 224329 -0.8% 222470 vmstat.system.in 1.884e+09 +10.9% 2.09e+09 proc-vmstat.numa_hit 1.884e+09 +10.9% 2.089e+09 proc-vmstat.numa_local 1.506e+10 +10.9% 1.671e+10 proc-vmstat.pgalloc_normal 1.506e+10 +10.9% 1.671e+10 proc-vmstat.pgfree 2104 ± 8% +86.4% 3922 ± 58% sched_debug.cpu.avg_idle.min 35810547 -25.7% 26600377 sched_debug.cpu.nr_switches.avg 37423347 -25.1% 28047296 sched_debug.cpu.nr_switches.max 32860210 -25.5% 24491190 sched_debug.cpu.nr_switches.min 968.00 ± 4% -18.2% 792.00 ± 7% slabinfo.kmalloc-rcl-128.active_objs 968.00 ± 4% -18.2% 792.00 ± 7% slabinfo.kmalloc-rcl-128.num_objs 16611 ± 2% +6.1% 17628 ± 4% slabinfo.skbuff_fclone_cache.active_objs 16827 ± 2% +5.7% 17794 ± 4% slabinfo.skbuff_fclone_cache.num_objs 18.78 ± 5% +28.9% 24.21 perf-stat.i.MPKI 1.942e+10 ± 2% -21.1% 1.531e+10 perf-stat.i.branch-instructions 2.08 ± 6% -0.1 1.97 perf-stat.i.branch-miss-rate% 3.892e+08 ± 2% -22.6% 3.014e+08 perf-stat.i.branch-misses 1.767e+09 ± 2% +4.3% 1.842e+09 perf-stat.i.cache-references 8171306 ± 2% -24.7% 6151961 perf-stat.i.context-switches 2.88 ± 2% +31.0% 3.78 perf-stat.i.cpi 2.761e+11 ± 2% +4.2% 2.876e+11 perf-stat.i.cpu-cycles 1.10 ± 3% -0.1 0.97 perf-stat.i.dTLB-load-miss-rate% 3.254e+08 ± 4% -32.1% 2.208e+08 perf-stat.i.dTLB-load-misses 2.893e+10 ± 2% -21.9% 2.26e+10 perf-stat.i.dTLB-loads 58990104 ± 5% -25.8% 43766302 perf-stat.i.dTLB-store-misses 1.658e+10 ± 2% -23.6% 1.266e+10 perf-stat.i.dTLB-stores 55.76 -3.4 52.34 perf-stat.i.iTLB-load-miss-rate% 63191122 ± 2% -29.1% 44815478 perf-stat.i.iTLB-load-misses 49413835 ± 2% -17.7% 40661018 perf-stat.i.iTLB-loads 9.688e+10 ± 2% -21.5% 7.605e+10 perf-stat.i.instructions 1607 ± 3% +8.1% 1737 perf-stat.i.instructions-per-iTLB-miss 0.35 -24.3% 0.27 perf-stat.i.ipc 86243 ± 29% +52.4% 131400 ± 18% perf-stat.i.node-stores 18.24 +32.8% 24.23 perf-stat.overall.MPKI 2.00 -0.0 1.97 perf-stat.overall.branch-miss-rate% 2.85 +32.7% 3.78 perf-stat.overall.cpi 1.11 ± 3% -0.1 0.97 perf-stat.overall.dTLB-load-miss-rate% 56.12 -3.7 52.43 perf-stat.overall.iTLB-load-miss-rate% 1533 ± 2% +10.7% 1697 perf-stat.overall.instructions-per-iTLB-miss 0.35 -24.6% 0.26 perf-stat.overall.ipc 15748 +4.3% 16430 perf-stat.overall.path-length 1.939e+10 ± 2% -21.1% 1.53e+10 perf-stat.ps.branch-instructions 3.887e+08 ± 2% -22.5% 3.011e+08 perf-stat.ps.branch-misses 1.764e+09 ± 2% +4.3% 1.84e+09 perf-stat.ps.cache-references 8161286 ± 2% -24.7% 6144845 perf-stat.ps.context-switches 2.757e+11 ± 2% +4.2% 2.872e+11 perf-stat.ps.cpu-cycles 3.25e+08 ± 4% -32.1% 2.206e+08 perf-stat.ps.dTLB-load-misses 2.889e+10 ± 2% -21.8% 2.258e+10 perf-stat.ps.dTLB-loads 58916703 ± 5% -25.8% 43714633 perf-stat.ps.dTLB-store-misses 1.656e+10 ± 2% -23.6% 1.265e+10 perf-stat.ps.dTLB-stores 63112575 ± 2% -29.1% 44762927 perf-stat.ps.iTLB-load-misses 49353553 ± 2% -17.7% 40614004 perf-stat.ps.iTLB-loads 9.676e+10 ± 2% -21.5% 7.596e+10 perf-stat.ps.instructions 86182 ± 29% +52.3% 131291 ± 18% perf-stat.ps.node-stores 8.86e+13 -22.5% 6.864e+13 perf-stat.total.instructions 136251 ± 8% -18.5% 110991 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts 128756 ± 3% -18.6% 104869 ± 6% interrupts.CPU10.RES:Rescheduling_interrupts 119826 ± 6% -12.1% 105329 ± 4% interrupts.CPU100.RES:Rescheduling_interrupts 121133 ± 10% -17.9% 99457 interrupts.CPU101.RES:Rescheduling_interrupts 9051 ± 3% +14.1% 10328 interrupts.CPU102.CAL:Function_call_interrupts 131446 ± 7% -17.8% 108104 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts 124627 ± 2% -16.6% 103971 ± 9% interrupts.CPU12.RES:Rescheduling_interrupts 131072 ± 11% -19.9% 105033 ± 4% interrupts.CPU13.RES:Rescheduling_interrupts 127845 ± 8% -17.0% 106112 ± 9% interrupts.CPU14.RES:Rescheduling_interrupts 126159 ± 10% -20.6% 100111 ± 11% interrupts.CPU15.RES:Rescheduling_interrupts 132232 ± 5% -17.9% 108535 ± 7% interrupts.CPU17.RES:Rescheduling_interrupts 127157 ± 4% -17.3% 105165 ± 11% interrupts.CPU18.RES:Rescheduling_interrupts 127948 ± 2% -15.5% 108165 ± 11% interrupts.CPU19.RES:Rescheduling_interrupts 122716 ± 6% -14.0% 105519 ± 11% interrupts.CPU23.RES:Rescheduling_interrupts 130754 ± 5% -20.3% 104222 ± 9% interrupts.CPU24.RES:Rescheduling_interrupts 126283 ± 4% -16.6% 105320 ± 6% interrupts.CPU27.RES:Rescheduling_interrupts 8997 ± 3% -12.4% 7883 ± 9% interrupts.CPU28.CAL:Function_call_interrupts 128467 ± 5% -20.9% 101674 ± 11% interrupts.CPU29.RES:Rescheduling_interrupts 133914 ± 6% -26.0% 99089 ± 10% interrupts.CPU3.RES:Rescheduling_interrupts 8987 ± 3% -11.9% 7922 ± 7% interrupts.CPU31.CAL:Function_call_interrupts 115389 ± 9% -17.7% 94962 ± 5% interrupts.CPU31.RES:Rescheduling_interrupts 8830 ± 5% -12.1% 7759 ± 4% interrupts.CPU32.CAL:Function_call_interrupts 134198 ± 6% -19.6% 107954 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts 8940 ± 3% -8.8% 8154 ± 4% interrupts.CPU34.CAL:Function_call_interrupts 123678 ± 12% -17.1% 102582 ± 2% interrupts.CPU34.RES:Rescheduling_interrupts 8969 ± 3% -9.2% 8143 ± 4% interrupts.CPU35.CAL:Function_call_interrupts 118372 ± 8% -10.9% 105507 ± 9% interrupts.CPU35.RES:Rescheduling_interrupts 8958 ± 3% -9.2% 8131 ± 4% interrupts.CPU36.CAL:Function_call_interrupts 132196 ± 3% -16.2% 110761 ± 7% interrupts.CPU4.RES:Rescheduling_interrupts 123005 ± 6% -17.2% 101838 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts 132802 ± 15% -23.6% 101521 ± 11% interrupts.CPU43.RES:Rescheduling_interrupts 131107 ± 4% -25.6% 97566 ± 3% interrupts.CPU44.RES:Rescheduling_interrupts 127673 ± 8% -24.0% 97028 ± 6% interrupts.CPU47.RES:Rescheduling_interrupts 123709 ± 6% -14.3% 106030 ± 12% interrupts.CPU48.RES:Rescheduling_interrupts 123709 ± 7% -12.4% 108355 ± 6% interrupts.CPU49.RES:Rescheduling_interrupts 135382 ± 5% -20.7% 107334 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts 130424 ± 4% -17.4% 107704 ± 6% interrupts.CPU52.RES:Rescheduling_interrupts 129234 ± 8% -18.6% 105171 ± 10% interrupts.CPU53.RES:Rescheduling_interrupts 131374 ± 6% -18.8% 106699 ± 6% interrupts.CPU54.RES:Rescheduling_interrupts 126141 ± 10% -14.7% 107626 ± 11% interrupts.CPU57.RES:Rescheduling_interrupts 133750 ± 9% -19.9% 107102 interrupts.CPU6.RES:Rescheduling_interrupts 119663 ± 5% -16.7% 99633 ± 3% interrupts.CPU60.RES:Rescheduling_interrupts 121078 ± 7% -13.6% 104670 ± 9% interrupts.CPU61.RES:Rescheduling_interrupts 121662 ± 5% -15.3% 102992 ± 10% interrupts.CPU63.RES:Rescheduling_interrupts 118130 ± 2% -15.1% 100310 ± 5% interrupts.CPU65.RES:Rescheduling_interrupts 128075 ± 6% -18.4% 104495 ± 8% interrupts.CPU67.RES:Rescheduling_interrupts 124700 ± 6% -15.7% 105149 ± 4% interrupts.CPU68.RES:Rescheduling_interrupts 119607 ± 9% -12.7% 104432 ± 6% interrupts.CPU7.RES:Rescheduling_interrupts 119288 ± 4% -11.8% 105251 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts 125678 ± 7% -18.7% 102141 ± 8% interrupts.CPU72.RES:Rescheduling_interrupts 130174 ± 5% -19.8% 104394 ± 3% interrupts.CPU74.RES:Rescheduling_interrupts 9215 ± 2% +9.8% 10119 ± 4% interrupts.CPU76.CAL:Function_call_interrupts 113784 ± 8% -10.6% 101686 ± 7% interrupts.CPU78.RES:Rescheduling_interrupts 122103 ± 5% -16.2% 102305 ± 5% interrupts.CPU79.RES:Rescheduling_interrupts 122242 ± 3% -23.8% 93153 ± 8% interrupts.CPU81.RES:Rescheduling_interrupts 9154 ± 3% +10.3% 10092 ± 4% interrupts.CPU82.CAL:Function_call_interrupts 116414 ± 3% -18.8% 94476 ± 9% interrupts.CPU82.RES:Rescheduling_interrupts 128558 ± 12% -24.5% 97067 ± 2% interrupts.CPU83.RES:Rescheduling_interrupts 9147 ± 3% +10.2% 10078 ± 4% interrupts.CPU84.CAL:Function_call_interrupts 122842 ± 9% -19.8% 98521 ± 2% interrupts.CPU84.RES:Rescheduling_interrupts 9081 ± 3% +10.9% 10071 ± 4% interrupts.CPU85.CAL:Function_call_interrupts 9130 ± 2% +10.2% 10065 ± 4% interrupts.CPU86.CAL:Function_call_interrupts 9068 ± 3% +10.9% 10057 ± 5% interrupts.CPU87.CAL:Function_call_interrupts 9068 ± 3% +10.8% 10050 ± 5% interrupts.CPU88.CAL:Function_call_interrupts 122119 ± 5% -20.1% 97576 ± 3% interrupts.CPU88.RES:Rescheduling_interrupts 9065 ± 3% +12.8% 10223 ± 2% interrupts.CPU89.CAL:Function_call_interrupts 124799 ± 6% -20.4% 99390 ± 8% interrupts.CPU89.RES:Rescheduling_interrupts 130362 ± 2% -12.6% 113981 ± 5% interrupts.CPU9.RES:Rescheduling_interrupts 8902 +12.7% 10035 ± 5% interrupts.CPU90.CAL:Function_call_interrupts 9062 ± 3% +10.7% 10028 ± 5% interrupts.CPU91.CAL:Function_call_interrupts 119442 ± 8% -24.1% 90611 ± 2% interrupts.CPU91.RES:Rescheduling_interrupts 9058 ± 3% +10.7% 10023 ± 5% interrupts.CPU92.CAL:Function_call_interrupts 117081 ± 10% -21.2% 92294 ± 12% interrupts.CPU92.RES:Rescheduling_interrupts 9058 ± 3% +10.6% 10017 ± 5% interrupts.CPU93.CAL:Function_call_interrupts 130977 ± 17% -27.7% 94702 ± 6% interrupts.CPU94.RES:Rescheduling_interrupts 9055 ± 3% +10.5% 10003 ± 5% interrupts.CPU95.CAL:Function_call_interrupts 116952 ± 4% -12.4% 102488 ± 10% interrupts.CPU95.RES:Rescheduling_interrupts 9054 ± 3% +10.4% 9995 ± 5% interrupts.CPU96.CAL:Function_call_interrupts 9054 ± 3% +10.3% 9990 ± 5% interrupts.CPU97.CAL:Function_call_interrupts 118634 ± 5% -12.7% 103578 ± 9% interrupts.CPU98.RES:Rescheduling_interrupts 116356 ± 10% -16.5% 97115 ± 7% interrupts.CPU99.RES:Rescheduling_interrupts 12787758 ± 2% -14.8% 10896246 interrupts.RES:Rescheduling_interrupts 10.84 -10.8 0.00 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 10.56 -10.6 0.00 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 10.16 -10.2 0.00 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv 10.07 -10.1 0.00 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established 9.71 -9.7 0.00 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack 9.57 -9.6 0.00 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue 21.28 -6.8 14.49 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.19 -6.8 14.43 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.57 -6.6 14.00 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.46 -6.5 13.93 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 16.37 -3.0 13.34 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 16.18 -3.0 13.21 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto 7.01 ± 4% -2.4 4.61 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 6.90 ± 5% -2.4 4.54 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 7.71 ± 2% -2.2 5.51 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 7.69 ± 2% -2.2 5.49 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 6.19 ± 5% -2.1 4.10 perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 6.04 ± 5% -2.0 4.00 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 5.89 ± 6% -2.0 3.90 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv 7.03 ± 2% -2.0 5.04 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 5.71 ± 6% -1.9 3.78 perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established 13.46 -1.9 11.54 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 6.86 ± 2% -1.9 4.93 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 6.72 ± 2% -1.9 4.83 ± 2% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 13.26 -1.8 11.41 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg 5.49 -1.8 3.67 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 2.62 ± 14% -1.8 0.82 ± 5% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable 5.29 -1.7 3.61 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 4.80 -1.6 3.17 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 4.59 -1.6 3.02 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg 4.50 -1.5 2.96 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg 4.41 -1.5 2.90 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data 4.81 -1.5 3.31 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 10.13 -1.5 8.63 perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 9.79 -1.4 8.36 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 2.14 -1.4 0.73 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable 4.34 -1.4 2.96 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg 9.60 -1.4 8.23 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg 2.10 ± 34% -1.4 0.73 ± 6% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock 2.08 -1.4 0.71 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock 3.88 ± 2% -1.2 2.71 perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 3.71 -1.2 2.55 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg 3.56 ± 2% -1.1 2.49 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 3.32 ± 2% -1.0 2.34 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg 1.56 -0.9 0.61 ± 2% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 3.14 ± 2% -0.9 2.22 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked 1.77 -0.6 1.16 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken 7.43 ± 2% -0.6 6.87 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 7.41 ± 2% -0.6 6.85 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.39 -0.5 0.87 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.34 -0.5 0.84 perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.30 -0.5 0.81 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.09 -0.4 0.72 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.20 -0.4 0.84 perf-profile.calltrace.cycles-pp.__switch_to 1.09 -0.3 0.76 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 6.67 ± 2% -0.3 6.38 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg 0.83 -0.3 0.56 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg 0.84 -0.3 0.57 perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout.wait_woken 0.95 -0.3 0.69 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.75 ± 2% -0.2 0.52 perf-profile.calltrace.cycles-pp._cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 0.73 ± 2% -0.2 0.51 perf-profile.calltrace.cycles-pp.preempt_schedule_common._cond_resched.__release_sock.release_sock.tcp_sendmsg 9.41 +1.1 10.47 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb 9.29 +1.1 10.39 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data 95.31 +1.4 96.67 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 95.15 +1.4 96.56 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.02 +7.2 18.20 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 12.28 ± 2% +7.2 19.51 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 12.21 ± 2% +7.3 19.47 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit 12.00 ± 2% +7.3 19.34 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 11.98 ± 2% +7.3 19.32 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 11.74 ± 2% +7.4 19.14 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 11.43 ± 2% +7.4 18.87 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 11.37 ± 2% +7.5 18.83 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 9.15 +7.5 16.67 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim 9.26 +7.5 16.80 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb 9.43 +7.6 17.01 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked 9.79 +7.6 17.37 perf-profile.calltrace.cycles-pp.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 10.97 ± 2% +7.6 18.56 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 9.59 +7.6 17.22 perf-profile.calltrace.cycles-pp.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg 10.64 ± 3% +7.7 18.30 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 10.11 ± 3% +7.8 17.94 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 9.96 ± 3% +7.9 17.82 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 3.91 ± 2% +7.9 11.79 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 3.91 ± 2% +7.9 11.79 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 9.90 ± 3% +7.9 17.79 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 9.71 ± 3% +8.0 17.67 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 3.58 ± 2% +8.0 11.60 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv 3.34 ± 2% +8.1 11.45 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established 9.05 ± 2% +8.1 17.17 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 18.39 +8.7 27.12 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill 18.58 +8.7 27.31 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill 19.93 +8.7 28.67 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 19.92 +8.7 28.66 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 19.70 +8.7 28.45 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked 19.77 +8.8 28.54 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg 72.20 +8.8 81.00 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 72.09 +8.8 80.92 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 71.49 +9.0 80.48 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 71.15 +9.1 80.24 perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 0.00 +10.6 10.61 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv 0.00 +10.7 10.72 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +11.0 10.98 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.00 +11.1 11.11 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 54.40 +12.2 66.63 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 32407497 -26.7% 23748191 softirqs.CPU0.NET_RX 31936171 ± 3% -23.8% 24347695 softirqs.CPU1.NET_RX 31995720 ± 3% -23.5% 24474810 softirqs.CPU10.NET_RX 31851135 -28.4% 22807522 softirqs.CPU100.NET_RX 31165470 ± 2% -27.9% 22460444 ± 2% softirqs.CPU101.NET_RX 30427682 ± 3% -27.8% 21964060 ± 2% softirqs.CPU102.NET_RX 31822429 -30.3% 22190016 ± 2% softirqs.CPU103.NET_RX 31794681 ± 3% -24.0% 24170873 ± 2% softirqs.CPU11.NET_RX 31337588 ± 3% -25.0% 23487789 ± 2% softirqs.CPU12.NET_RX 30982384 ± 3% -21.6% 24285912 softirqs.CPU13.NET_RX 31985249 ± 2% -24.5% 24135878 ± 2% softirqs.CPU14.NET_RX 31569639 ± 3% -23.4% 24173977 ± 2% softirqs.CPU15.NET_RX 31672651 ± 2% -23.7% 24152941 ± 2% softirqs.CPU16.NET_RX 31845476 ± 2% -23.4% 24406051 softirqs.CPU17.NET_RX 32237888 -24.4% 24378972 softirqs.CPU18.NET_RX 32390163 -25.7% 24060765 softirqs.CPU19.NET_RX 32518154 -26.3% 23981309 ± 2% softirqs.CPU2.NET_RX 31625849 ± 2% -26.0% 23391632 ± 2% softirqs.CPU20.NET_RX 31800991 ± 2% -24.2% 24096267 ± 2% softirqs.CPU21.NET_RX 32165283 ± 2% -23.9% 24484882 softirqs.CPU22.NET_RX 32096347 ± 3% -24.6% 24198018 ± 2% softirqs.CPU23.NET_RX 32606868 -24.9% 24498919 softirqs.CPU24.NET_RX 31333059 ± 2% -23.5% 23972392 ± 2% softirqs.CPU25.NET_RX 29740104 ± 3% -23.9% 22620846 softirqs.CPU26.NET_RX 8933 ± 31% -24.6% 6736 ± 7% softirqs.CPU26.SCHED 31426927 -28.2% 22574373 ± 2% softirqs.CPU27.NET_RX 31494409 -27.6% 22811958 softirqs.CPU28.NET_RX 31749609 -29.8% 22274240 ± 2% softirqs.CPU29.NET_RX 31359195 ± 3% -23.3% 24058233 softirqs.CPU3.NET_RX 31891532 -28.5% 22801967 softirqs.CPU30.NET_RX 31022930 ± 3% -26.6% 22780850 softirqs.CPU31.NET_RX 31633941 -28.0% 22773366 softirqs.CPU32.NET_RX 31064582 ± 3% -28.2% 22291648 ± 2% softirqs.CPU33.NET_RX 30670424 ± 2% -26.0% 22706259 softirqs.CPU34.NET_RX 31155193 ± 3% -28.3% 22336447 ± 2% softirqs.CPU35.NET_RX 31767721 -29.3% 22462506 ± 2% softirqs.CPU36.NET_RX 31702765 -27.9% 22848963 softirqs.CPU37.NET_RX 31618979 -28.8% 22498908 ± 2% softirqs.CPU38.NET_RX 8157 ± 30% -22.3% 6342 softirqs.CPU38.SCHED 31178503 ± 2% -28.8% 22206116 ± 2% softirqs.CPU39.NET_RX 31543118 ± 3% -22.7% 24378208 softirqs.CPU4.NET_RX 29993167 ± 3% -26.6% 22006508 softirqs.CPU40.NET_RX 31389405 ± 2% -27.4% 22777491 softirqs.CPU41.NET_RX 31225889 ± 3% -28.1% 22466787 ± 2% softirqs.CPU42.NET_RX 31122849 ± 3% -27.5% 22548782 softirqs.CPU43.NET_RX 30449949 ± 3% -26.7% 22314973 ± 2% softirqs.CPU44.NET_RX 30708016 ± 3% -26.5% 22575709 softirqs.CPU45.NET_RX 30869514 ± 2% -27.1% 22517203 softirqs.CPU46.NET_RX 30349340 ± 3% -26.7% 22238899 ± 2% softirqs.CPU47.NET_RX 31690538 -27.9% 22844885 softirqs.CPU48.NET_RX 31442996 -27.6% 22774395 softirqs.CPU49.NET_RX 32539333 -24.8% 24475876 softirqs.CPU5.NET_RX 31777443 -28.1% 22839894 softirqs.CPU50.NET_RX 30319039 ± 4% -25.8% 22488386 softirqs.CPU51.NET_RX 32358234 -24.7% 24364666 softirqs.CPU52.NET_RX 31051127 ± 3% -23.5% 23764585 ± 2% softirqs.CPU53.NET_RX 31368731 ± 3% -22.3% 24384593 softirqs.CPU54.NET_RX 32739188 -26.5% 24063678 ± 2% softirqs.CPU55.NET_RX 32587893 -25.0% 24425586 softirqs.CPU56.NET_RX 31762403 ± 3% -24.6% 23947521 softirqs.CPU57.NET_RX 32342624 -25.5% 24095598 softirqs.CPU58.NET_RX 31850813 ± 3% -23.7% 24310413 softirqs.CPU59.NET_RX 32296552 -26.1% 23882250 ± 2% softirqs.CPU6.NET_RX 31208970 ± 3% -23.9% 23735529 ± 2% softirqs.CPU60.NET_RX 32655061 -25.7% 24259328 softirqs.CPU61.NET_RX 31834306 ± 3% -23.2% 24436288 softirqs.CPU62.NET_RX 32492263 -25.7% 24146807 ± 2% softirqs.CPU63.NET_RX 32160041 -24.6% 24237812 softirqs.CPU64.NET_RX 32146792 -25.1% 24083142 softirqs.CPU65.NET_RX 31886509 ± 3% -24.3% 24153287 ± 2% softirqs.CPU66.NET_RX 31124000 ± 2% -21.2% 24534615 softirqs.CPU67.NET_RX 31606731 ± 3% -23.7% 24115056 ± 2% softirqs.CPU68.NET_RX 31809649 ± 4% -27.3% 23140437 softirqs.CPU69.NET_RX 31832502 ± 3% -23.8% 24256886 softirqs.CPU7.NET_RX 31934664 ± 2% -23.5% 24445503 softirqs.CPU70.NET_RX 32283512 -24.5% 24374573 softirqs.CPU71.NET_RX 31559429 ± 3% -23.0% 24304361 softirqs.CPU72.NET_RX 32519181 -26.5% 23915099 ± 2% softirqs.CPU73.NET_RX 32064639 ± 3% -23.5% 24531236 softirqs.CPU74.NET_RX 32552635 -25.2% 24358616 softirqs.CPU75.NET_RX 32574031 -25.1% 24391309 softirqs.CPU76.NET_RX 29891561 -18.3% 24435768 softirqs.CPU77.NET_RX 30558797 ± 3% -27.1% 22290753 ± 2% softirqs.CPU78.NET_RX 30913524 ± 3% -27.1% 22542484 softirqs.CPU79.NET_RX 32338939 -25.2% 24184482 softirqs.CPU8.NET_RX 29930689 ± 3% -24.3% 22649621 softirqs.CPU80.NET_RX 31789053 -29.3% 22471154 ± 2% softirqs.CPU81.NET_RX 31080841 ± 3% -28.0% 22371709 ± 2% softirqs.CPU82.NET_RX 31004904 ± 3% -28.0% 22335427 softirqs.CPU83.NET_RX 30902184 ± 2% -26.4% 22746371 softirqs.CPU84.NET_RX 31508332 -28.4% 22567097 softirqs.CPU85.NET_RX 31437975 -29.0% 22331821 ± 2% softirqs.CPU86.NET_RX 31025912 ± 3% -26.1% 22913093 softirqs.CPU87.NET_RX 30270422 ± 2% -25.2% 22636977 ± 2% softirqs.CPU88.NET_RX 30385565 ± 4% -26.0% 22499521 ± 2% softirqs.CPU89.NET_RX 31577940 ± 3% -22.4% 24489124 softirqs.CPU9.NET_RX 30152844 ± 3% -24.7% 22717462 softirqs.CPU90.NET_RX 29900183 ± 2% -24.9% 22460973 softirqs.CPU91.NET_RX 31952707 -28.9% 22715130 softirqs.CPU92.NET_RX 30820437 ± 3% -26.6% 22619907 ± 2% softirqs.CPU93.NET_RX 30627580 ± 3% -25.6% 22790846 softirqs.CPU94.NET_RX 31367872 -27.7% 22670545 softirqs.CPU95.NET_RX 31045007 ± 3% -26.5% 22818368 softirqs.CPU96.NET_RX 30842779 ± 3% -28.2% 22143073 ± 2% softirqs.CPU97.NET_RX 30173663 ± 3% -24.8% 22686608 softirqs.CPU98.NET_RX 30750855 ± 3% -25.6% 22870339 softirqs.CPU99.NET_RX 3.273e+09 -25.8% 2.43e+09 softirqs.NET_RX netperf.Throughput_Mbps 4000 +-+------------------------------------------------------------------+ | | 3500 +-+ | 3000 +-+ | O O O OO OOO OO OO OOO OO OO OO OOO OO OO OO | 2500 +-+ | | | 2000 +-+ | | | 1500 +-+ | 1000 +-+ | | | 500 +-+ | | | 0 +O+O---O-------------------------------------------------------------+ netperf.Throughput_total_Mbps 900000 +-+----------------------------------------------------------------+ |+.++.+++.++.+++.++.+++.++.+++.++.+++.+++.++.+++.++.+++.++.+++.++.+| 800000 +-+ | 700000 +-+ | | O | 600000 O-+ O O O OO OOO OO OOO OO OOO O OOO OOO OO O | 500000 +-+ | | | 400000 +-+ | 300000 +-+ | | | 200000 +-+ | 100000 +-+ | | | 0 +O+O---O-----------------------------------------------------------+ netperf.workload 6e+09 +-+-----------------------------------------------------------------+ |+.++.++.+++.++.+++.++.++.+++.++.+++.++.++.+++.++.+++.++.++.+++.++.+| 5e+09 +-+ | | | O O O OOO OO OOO OO OO OOO OO OOO OO OO OOO | 4e+09 +-+ | | | 3e+09 +-+ | | | 2e+09 +-+ | | | | | 1e+09 +-+ | | | 0 +O+O---O------------------------------------------------------------+ netperf.time.user_time 2000 +-+------------------------------------------------------------------+ 1800 +-+ + + | | : : + : | 1600 +-+ +. + .++.++. + : +.+ .++. .++.++. + .++ + .++.+| 1400 +-+++.++.+ + + + + + +++ + +.++.++ +.++ | | | 1200 +-+ | 1000 O-+ O OO OOO OO OO OOO OO OO OO OOO OO OO OO | 800 +-+ | | | 600 +-+ | 400 +-+ | | | 200 +-+ | 0 +O+O--OO-------------------------------------------------------------+ netperf.time.percent_of_cpu_this_job_got 8000 +-+------------------------------------------------------------------+ O+.+O.++.OO.OOO.OO.OO.OOO.OO.OO.OO.OOO.OO.OO.OO+.++.++.++.+++.++.++.+| 7000 +-+ | 6000 +-+ | | | 5000 +-+ | | | 4000 +-+ | | | 3000 +-+ | 2000 +-+ | | | 1000 +-+ | | | 0 +O+O--OO-------------------------------------------------------------+ netperf.time.involuntary_context_switches 4e+09 +-+---------------------------------------------------------------+ |+.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+++.+| 3.5e+09 +-+ | 3e+09 +-+ | O O O OOO OO OOO OOO OO OOO OOO OO OOO OO | 2.5e+09 +-+ | | | 2e+09 +-+ | | | 1.5e+09 +-+ | 1e+09 +-+ | | | 5e+08 +-+ | | | 0 +O+O--OO----------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-skl-fpga01: 104 threads Skylake with 192G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2018-04-03.cgz/900s/lkp-skl-fpga01/TCP_MAERTS/netperf commit: 472c2e07ee ("tcp: add one skb cache for tx") 8b27dae5a2 ("tcp: add one skb cache for rx") 472c2e07eef04514 8b27dae5a2e89a61c46c6dbc76c ---------------- --------------------------- %stddev %change %stddev \ | \ 3933 -25.6% 2926 netperf.Throughput_Mbps 818124 -25.6% 608758 netperf.Throughput_total_Mbps 57423 ± 19% -41.1% 33817 netperf.time.involuntary_context_switches 3010 +3.7% 3122 netperf.time.percent_of_cpu_this_job_got 25432 +3.9% 26424 netperf.time.system_time 3.735e+09 -25.6% 2.778e+09 netperf.time.voluntary_context_switches 5.618e+09 -25.6% 4.18e+09 netperf.workload 194721 ± 22% +41.3% 275069 ± 2% cpuidle.C6.usage 13.00 ± 7% +5.9 18.86 mpstat.cpu.all.soft% 3.39 ± 5% -1.2 2.23 mpstat.cpu.all.usr% 9.539e+08 +13.2% 1.08e+09 numa-numastat.node0.local_node 9.539e+08 +13.2% 1.08e+09 numa-numastat.node0.numa_hit 4.767e+08 +13.1% 5.389e+08 numa-vmstat.node0.numa_hit 4.767e+08 +13.1% 5.389e+08 numa-vmstat.node0.numa_local 95.75 +1.3% 97.00 vmstat.cpu.sy 8245666 -25.5% 6142829 vmstat.system.cs 2696 +2.9% 2775 turbostat.Avg_MHz 0.89 ± 5% -32.4% 0.60 turbostat.CPU%c1 108.00 -1.6% 106.23 turbostat.RAMWatt 1.881e+09 +11.2% 2.091e+09 proc-vmstat.numa_hit 1.881e+09 +11.2% 2.091e+09 proc-vmstat.numa_local 1.504e+10 +11.2% 1.672e+10 proc-vmstat.pgalloc_normal 1.504e+10 +11.2% 1.672e+10 proc-vmstat.pgfree 4053 ± 15% +19.6% 4848 ± 13% slabinfo.eventpoll_pwq.active_objs 4053 ± 15% +19.6% 4848 ± 13% slabinfo.eventpoll_pwq.num_objs 545.50 ± 11% -17.1% 452.00 ± 11% slabinfo.kernfs_iattrs_cache.active_objs 545.50 ± 11% -17.1% 452.00 ± 11% slabinfo.kernfs_iattrs_cache.num_objs 12.31 ± 12% +25.1% 15.40 sched_debug.cfs_rq:/.load_avg.stddev 1449447 ± 23% -29.2% 1026327 ± 22% sched_debug.cfs_rq:/.min_vruntime.stddev 0.78 ± 66% +155.6% 2.00 ± 11% sched_debug.cfs_rq:/.removed.load_avg.avg 5.98 ± 60% +84.9% 11.05 ± 4% sched_debug.cfs_rq:/.removed.load_avg.stddev 35.94 ± 66% +156.2% 92.09 ± 11% sched_debug.cfs_rq:/.removed.runnable_sum.avg 274.98 ± 60% +85.4% 509.82 ± 4% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 0.39 ± 66% +115.0% 0.84 ± 11% sched_debug.cfs_rq:/.removed.util_avg.avg 1.75 ± 14% +32.1% 2.31 ± 12% sched_debug.cfs_rq:/.runnable_load_avg.min 98724 ± 85% -66.3% 33282 ± 85% sched_debug.cfs_rq:/.runnable_weight.max -3346961 -44.2% -1868641 sched_debug.cfs_rq:/.spread0.min 1449347 ± 23% -29.2% 1026220 ± 22% sched_debug.cfs_rq:/.spread0.stddev 156728 ± 15% +49.0% 233530 ± 14% sched_debug.cpu.avg_idle.avg 260031 ± 9% +33.3% 346707 ± 10% sched_debug.cpu.avg_idle.stddev 20.54 ± 5% -10.2% 18.44 ± 2% sched_debug.cpu.clock.stddev 20.54 ± 5% -10.2% 18.44 ± 2% sched_debug.cpu.clock_task.stddev 35742943 -25.6% 26602562 sched_debug.cpu.nr_switches.avg 37162589 -24.6% 28014480 sched_debug.cpu.nr_switches.max 32371186 -24.3% 24512619 sched_debug.cpu.nr_switches.min 18.74 +29.7% 24.31 perf-stat.i.MPKI 1.938e+10 -20.9% 1.534e+10 perf-stat.i.branch-instructions 2.05 -0.1 1.98 perf-stat.i.branch-miss-rate% 3.961e+08 -23.4% 3.033e+08 perf-stat.i.branch-misses 0.74 ± 31% -0.4 0.38 ± 18% perf-stat.i.cache-miss-rate% 12929011 ± 32% -49.1% 6575616 ± 19% perf-stat.i.cache-misses 1.813e+09 +2.3% 1.854e+09 perf-stat.i.cache-references 8267879 -25.5% 6157361 perf-stat.i.context-switches 2.89 +30.6% 3.77 perf-stat.i.cpi 2.794e+11 +2.9% 2.876e+11 perf-stat.i.cpu-cycles 23942 ± 24% +97.5% 47291 ± 21% perf-stat.i.cycles-between-cache-misses 1.13 -0.1 0.99 perf-stat.i.dTLB-load-miss-rate% 3.331e+08 -31.8% 2.274e+08 perf-stat.i.dTLB-load-misses 2.9e+10 -22.0% 2.263e+10 perf-stat.i.dTLB-loads 60002134 ± 2% -26.3% 44193691 ± 2% perf-stat.i.dTLB-store-misses 1.664e+10 -23.6% 1.271e+10 perf-stat.i.dTLB-stores 58.26 -4.7 53.57 perf-stat.i.iTLB-load-miss-rate% 70291090 -31.0% 48469793 perf-stat.i.iTLB-load-misses 50169394 -16.5% 41890643 perf-stat.i.iTLB-loads 9.677e+10 -21.2% 7.624e+10 perf-stat.i.instructions 1410 +13.6% 1602 perf-stat.i.instructions-per-iTLB-miss 0.35 -23.4% 0.27 perf-stat.i.ipc 2150902 ± 41% -42.7% 1233268 ± 27% perf-stat.i.node-load-misses 652857 ± 24% -39.8% 393122 ± 15% perf-stat.i.node-loads 77.08 ± 11% -20.6 56.48 ± 14% perf-stat.i.node-store-miss-rate% 386848 ± 40% -64.7% 136565 ± 27% perf-stat.i.node-store-misses 18.74 +29.8% 24.32 perf-stat.overall.MPKI 2.04 -0.1 1.98 perf-stat.overall.branch-miss-rate% 0.71 ± 32% -0.4 0.35 ± 20% perf-stat.overall.cache-miss-rate% 2.89 +30.6% 3.77 perf-stat.overall.cpi 23413 ± 23% +95.0% 45644 ± 21% perf-stat.overall.cycles-between-cache-misses 1.14 -0.1 0.99 perf-stat.overall.dTLB-load-miss-rate% 58.35 -4.7 53.64 perf-stat.overall.iTLB-load-miss-rate% 1376 +14.3% 1573 perf-stat.overall.instructions-per-iTLB-miss 0.35 -23.4% 0.27 perf-stat.overall.ipc 75.96 ± 13% -20.1 55.89 ± 14% perf-stat.overall.node-store-miss-rate% 15545 +5.9% 16457 perf-stat.overall.path-length 1.936e+10 -20.9% 1.532e+10 perf-stat.ps.branch-instructions 3.956e+08 -23.4% 3.03e+08 perf-stat.ps.branch-misses 12917167 ± 32% -49.1% 6568704 ± 19% perf-stat.ps.cache-misses 1.811e+09 +2.3% 1.852e+09 perf-stat.ps.cache-references 8258274 -25.5% 6150289 perf-stat.ps.context-switches 2.791e+11 +2.9% 2.872e+11 perf-stat.ps.cpu-cycles 3.328e+08 -31.8% 2.271e+08 perf-stat.ps.dTLB-load-misses 2.897e+10 -22.0% 2.26e+10 perf-stat.ps.dTLB-loads 59931265 ± 2% -26.3% 44142618 ± 2% perf-stat.ps.dTLB-store-misses 1.662e+10 -23.6% 1.269e+10 perf-stat.ps.dTLB-stores 70208114 -31.0% 48413951 perf-stat.ps.iTLB-load-misses 50111323 -16.5% 41842519 perf-stat.ps.iTLB-loads 9.666e+10 -21.2% 7.616e+10 perf-stat.ps.instructions 2148559 ± 41% -42.7% 1231893 ± 27% perf-stat.ps.node-load-misses 653248 ± 24% -39.9% 392846 ± 15% perf-stat.ps.node-loads 386418 ± 40% -64.7% 136417 ± 27% perf-stat.ps.node-store-misses 8.733e+13 -21.2% 6.879e+13 perf-stat.total.instructions 887.50 ± 21% +961.1% 9417 ±122% interrupts.39:PCI-MSI.67633154-edge.eth0-TxRx-1 906772 +4.4% 946334 interrupts.CAL:Function_call_interrupts 128451 ± 3% -20.1% 102684 ± 11% interrupts.CPU1.RES:Rescheduling_interrupts 122297 ± 12% -20.1% 97767 ± 5% interrupts.CPU100.RES:Rescheduling_interrupts 114996 ± 8% -12.8% 100315 ± 5% interrupts.CPU101.RES:Rescheduling_interrupts 8676 ± 2% +14.9% 9972 ± 3% interrupts.CPU102.CAL:Function_call_interrupts 145573 ± 15% -30.4% 101298 ± 9% interrupts.CPU102.RES:Rescheduling_interrupts 8795 +13.2% 9956 ± 3% interrupts.CPU103.CAL:Function_call_interrupts 129834 ± 7% -15.4% 109820 ± 9% interrupts.CPU11.RES:Rescheduling_interrupts 122266 ± 5% -18.3% 99893 ± 4% interrupts.CPU13.RES:Rescheduling_interrupts 133695 ± 4% -23.7% 101984 ± 5% interrupts.CPU14.RES:Rescheduling_interrupts 115773 ± 7% -18.6% 94202 ± 8% interrupts.CPU18.RES:Rescheduling_interrupts 120005 ± 3% -15.7% 101149 ± 8% interrupts.CPU27.RES:Rescheduling_interrupts 119584 ± 3% -19.5% 96286 ± 6% interrupts.CPU29.RES:Rescheduling_interrupts 127064 ± 7% -15.1% 107858 ± 5% interrupts.CPU30.RES:Rescheduling_interrupts 887.50 ± 21% +961.1% 9417 ±122% interrupts.CPU31.39:PCI-MSI.67633154-edge.eth0-TxRx-1 118974 ± 11% -17.1% 98609 ± 7% interrupts.CPU33.RES:Rescheduling_interrupts 125463 ± 13% -23.9% 95477 ± 5% interrupts.CPU34.RES:Rescheduling_interrupts 125126 ± 16% -21.7% 97997 ± 7% interrupts.CPU35.RES:Rescheduling_interrupts 133035 ± 12% -28.2% 95517 ± 10% interrupts.CPU37.RES:Rescheduling_interrupts 120167 ± 11% -24.5% 90782 ± 8% interrupts.CPU38.RES:Rescheduling_interrupts 125040 ± 3% -19.8% 100223 ± 7% interrupts.CPU42.RES:Rescheduling_interrupts 119768 ± 15% -16.6% 99882 interrupts.CPU43.RES:Rescheduling_interrupts 125986 ± 7% -18.9% 102226 ± 8% interrupts.CPU45.RES:Rescheduling_interrupts 124516 ± 10% -15.7% 104961 ± 5% interrupts.CPU48.RES:Rescheduling_interrupts 125647 ± 5% -22.8% 96970 ± 3% interrupts.CPU49.RES:Rescheduling_interrupts 131345 ± 5% -17.2% 108796 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts 127858 ± 3% -18.9% 103683 ± 5% interrupts.CPU53.RES:Rescheduling_interrupts 125241 ± 15% -18.7% 101797 ± 5% interrupts.CPU54.RES:Rescheduling_interrupts 133398 ± 7% -22.1% 103961 ± 3% interrupts.CPU56.RES:Rescheduling_interrupts 129732 ± 10% -24.2% 98324 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts 132858 ± 12% -20.5% 105623 ± 9% interrupts.CPU60.RES:Rescheduling_interrupts 123442 ± 5% -20.7% 97934 ± 4% interrupts.CPU64.RES:Rescheduling_interrupts 132863 ± 3% -20.2% 106040 ± 4% interrupts.CPU65.RES:Rescheduling_interrupts 9002 +10.7% 9967 ± 2% interrupts.CPU66.CAL:Function_call_interrupts 8999 +10.7% 9958 ± 2% interrupts.CPU67.CAL:Function_call_interrupts 8774 ± 5% +13.4% 9951 ± 2% interrupts.CPU68.CAL:Function_call_interrupts 8991 +10.6% 9945 ± 2% interrupts.CPU69.CAL:Function_call_interrupts 125949 ± 9% -15.0% 107048 ± 4% interrupts.CPU69.RES:Rescheduling_interrupts 123443 ± 8% -15.7% 104008 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts 8987 +10.6% 9938 ± 2% interrupts.CPU70.CAL:Function_call_interrupts 8983 +10.5% 9927 ± 2% interrupts.CPU71.CAL:Function_call_interrupts 122925 ± 12% -18.0% 100786 ± 8% interrupts.CPU71.RES:Rescheduling_interrupts 8925 +11.0% 9910 ± 2% interrupts.CPU73.CAL:Function_call_interrupts 126258 ± 7% -19.4% 101773 ± 7% interrupts.CPU73.RES:Rescheduling_interrupts 129281 ± 6% -18.1% 105945 ± 5% interrupts.CPU74.RES:Rescheduling_interrupts 8969 +10.3% 9893 ± 2% interrupts.CPU75.CAL:Function_call_interrupts 8966 +10.3% 9887 ± 2% interrupts.CPU76.CAL:Function_call_interrupts 8965 +10.1% 9870 ± 2% interrupts.CPU77.CAL:Function_call_interrupts 129057 ± 9% -24.7% 97240 ± 3% interrupts.CPU77.RES:Rescheduling_interrupts 8683 ± 5% +13.7% 9874 ± 2% interrupts.CPU78.CAL:Function_call_interrupts 129114 ± 9% -27.6% 93483 ± 4% interrupts.CPU78.RES:Rescheduling_interrupts 8685 ± 4% +13.6% 9865 ± 2% interrupts.CPU79.CAL:Function_call_interrupts 121386 ± 7% -18.7% 98627 ± 5% interrupts.CPU8.RES:Rescheduling_interrupts 8956 +10.1% 9864 ± 2% interrupts.CPU80.CAL:Function_call_interrupts 125015 ± 11% -24.1% 94918 ± 5% interrupts.CPU80.RES:Rescheduling_interrupts 8633 ± 7% +14.2% 9856 ± 2% interrupts.CPU81.CAL:Function_call_interrupts 132492 ± 4% -20.7% 105014 ± 4% interrupts.CPU81.RES:Rescheduling_interrupts 8857 ± 2% +11.2% 9848 ± 2% interrupts.CPU82.CAL:Function_call_interrupts 111387 ± 6% -13.8% 96026 ± 8% interrupts.CPU82.RES:Rescheduling_interrupts 112445 ± 6% -9.6% 101699 ± 7% interrupts.CPU83.RES:Rescheduling_interrupts 121973 ± 7% -21.1% 96217 ± 6% interrupts.CPU84.RES:Rescheduling_interrupts 8883 +10.6% 9822 ± 2% interrupts.CPU85.CAL:Function_call_interrupts 115045 ± 6% -16.3% 96249 ± 4% interrupts.CPU85.RES:Rescheduling_interrupts 8668 ± 3% +12.6% 9762 interrupts.CPU86.CAL:Function_call_interrupts 8579 ± 3% +14.3% 9806 ± 2% interrupts.CPU87.CAL:Function_call_interrupts 8719 ± 2% +12.4% 9797 ± 2% interrupts.CPU88.CAL:Function_call_interrupts 8769 +11.6% 9788 ± 2% interrupts.CPU89.CAL:Function_call_interrupts 131248 ± 9% -27.5% 95130 ± 9% interrupts.CPU9.RES:Rescheduling_interrupts 8766 +11.6% 9780 ± 2% interrupts.CPU90.CAL:Function_call_interrupts 121573 ± 6% -14.4% 104044 ± 8% interrupts.CPU90.RES:Rescheduling_interrupts 8762 +11.5% 9768 ± 2% interrupts.CPU91.CAL:Function_call_interrupts 125741 ± 6% -21.2% 99116 ± 10% interrupts.CPU91.RES:Rescheduling_interrupts 8764 +11.3% 9754 ± 2% interrupts.CPU92.CAL:Function_call_interrupts 125340 ± 7% -21.1% 98921 ± 6% interrupts.CPU92.RES:Rescheduling_interrupts 8620 ± 4% +13.0% 9743 ± 2% interrupts.CPU93.CAL:Function_call_interrupts 129358 ± 10% -25.0% 97082 ± 6% interrupts.CPU93.RES:Rescheduling_interrupts 8753 +11.2% 9730 ± 2% interrupts.CPU94.CAL:Function_call_interrupts 8750 +11.1% 9719 ± 2% interrupts.CPU95.CAL:Function_call_interrupts 115441 ± 6% -15.6% 97484 ± 8% interrupts.CPU95.RES:Rescheduling_interrupts 121213 ± 5% -17.0% 100662 ± 12% interrupts.CPU99.RES:Rescheduling_interrupts 12650754 ± 3% -14.8% 10783909 interrupts.RES:Rescheduling_interrupts 176.50 ± 59% -79.7% 35.75 ± 47% interrupts.TLB:TLB_shootdowns 10.98 -11.0 0.00 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 10.70 -10.7 0.00 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 10.29 -10.3 0.00 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv 10.20 -10.2 0.00 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established 9.82 -9.8 0.00 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue.tcp_ack 9.68 -9.7 0.00 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_clean_rtx_queue 21.27 -6.8 14.52 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.18 -6.7 14.46 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.54 -6.5 14.04 perf-profile.calltrace.cycles-pp.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.43 -6.5 13.96 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 16.41 -3.1 13.34 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 16.22 -3.0 13.21 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto 7.69 -2.2 5.51 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 7.67 -2.2 5.50 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 6.75 ± 6% -2.2 4.60 ± 2% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 13.64 -2.1 11.53 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 13.43 -2.0 11.40 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg 7.00 -2.0 5.04 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 6.83 -1.9 4.93 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 6.70 -1.9 4.83 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 6.39 ± 3% -1.9 4.53 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 5.43 -1.7 3.69 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 5.31 -1.7 3.59 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 5.68 ± 3% -1.6 4.11 ± 2% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 4.76 -1.6 3.19 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 4.84 -1.5 3.30 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 4.56 -1.5 3.04 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg 5.51 ± 3% -1.5 4.01 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 4.48 -1.5 2.99 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg 4.39 -1.5 2.92 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_woken.sk_wait_data 5.36 ± 3% -1.5 3.90 ± 2% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv 5.19 ± 3% -1.4 3.78 ± 2% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established 4.37 -1.4 2.96 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg.inet_recvmsg 2.14 -1.4 0.73 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable 10.04 -1.4 8.65 perf-profile.calltrace.cycles-pp._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 2.08 -1.4 0.70 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.__wake_up_common.__wake_up_common_lock 9.73 -1.3 8.42 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 9.56 -1.3 8.29 perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin._copy_from_iter_full.tcp_sendmsg_locked.tcp_sendmsg 3.73 -1.2 2.55 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg 3.78 ± 2% -1.1 2.68 perf-profile.calltrace.cycles-pp.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 1.86 ± 30% -1.0 0.84 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable 3.46 ± 2% -1.0 2.46 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.51 ± 6% -0.9 0.61 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 3.22 ± 2% -0.9 2.33 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked.tcp_sendmsg 3.02 ± 2% -0.8 2.21 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.tcp_sendmsg_locked 1.78 -0.6 1.16 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken 7.18 ± 2% -0.6 6.58 ± 2% perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 7.16 ± 2% -0.6 6.56 ± 2% perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.37 -0.5 0.88 perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.33 -0.5 0.84 perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.29 -0.5 0.82 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.71 -0.5 0.25 ±100% perf-profile.calltrace.cycles-pp.__sched_text_start.preempt_schedule_common._cond_resched.__release_sock.release_sock 1.20 -0.4 0.82 perf-profile.calltrace.cycles-pp.__switch_to 1.09 -0.4 0.72 perf-profile.calltrace.cycles-pp.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.09 -0.3 0.77 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 6.42 ± 2% -0.3 6.09 ± 2% perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg 0.83 -0.3 0.56 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg 0.84 -0.3 0.58 ± 3% perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__sched_text_start.schedule.schedule_timeout.wait_woken 0.94 -0.3 0.68 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.74 -0.2 0.53 perf-profile.calltrace.cycles-pp._cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 0.73 -0.2 0.52 perf-profile.calltrace.cycles-pp.preempt_schedule_common._cond_resched.__release_sock.release_sock.tcp_sendmsg 9.51 +0.7 10.17 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb 9.38 +0.7 10.08 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.skb_release_data 95.32 +1.4 96.69 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 95.16 +1.4 96.58 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.17 +7.3 18.47 perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 11.81 ± 2% +7.4 19.18 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 11.74 ± 2% +7.4 19.15 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output.__ip_queue_xmit 11.52 ± 2% +7.5 18.98 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 11.55 ± 2% +7.5 19.01 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 11.27 ± 2% +7.5 18.81 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 10.95 ± 2% +7.6 18.55 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack.do_softirq 9.28 +7.6 16.91 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim 9.39 +7.7 17.05 perf-profile.calltrace.cycles-pp._raw_spin_lock.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb 10.83 ± 2% +7.7 18.52 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 9.57 +7.7 17.27 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked 9.94 +7.7 17.64 perf-profile.calltrace.cycles-pp.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 10.49 ± 2% +7.7 18.23 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.do_softirq_own_stack 9.73 +7.8 17.49 perf-profile.calltrace.cycles-pp.__free_pages_ok.___pskb_trim.sk_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg 10.16 ± 2% +7.8 17.98 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 3.79 +8.0 11.77 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 3.78 +8.0 11.77 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 9.61 ± 2% +8.0 17.61 perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 9.45 ± 2% +8.1 17.50 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 9.40 ± 2% +8.1 17.47 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 9.21 ± 2% +8.1 17.35 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 3.44 +8.1 11.58 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv 8.65 ± 2% +8.2 16.88 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 3.21 +8.2 11.44 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established 18.55 ± 2% +8.6 27.14 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill 20.09 ± 2% +8.6 28.68 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 19.87 ± 2% +8.6 28.45 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked 20.08 ± 2% +8.6 28.67 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 18.74 ± 2% +8.6 27.34 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill 19.94 ± 2% +8.6 28.53 perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg 72.24 +8.7 80.98 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 72.13 +8.8 80.91 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 71.52 +9.0 80.48 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 71.18 +9.1 80.25 perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 0.00 +10.3 10.29 perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv 0.00 +10.4 10.41 perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +10.7 10.66 perf-profile.calltrace.cycles-pp.skb_release_data.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 0.00 +10.8 10.81 perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 54.39 +12.2 66.64 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 31487988 ± 3% -23.2% 24183132 softirqs.CPU0.NET_RX 32151929 -26.1% 23765617 ± 2% softirqs.CPU1.NET_RX 31962684 ± 3% -23.5% 24458409 softirqs.CPU10.NET_RX 31110228 ± 3% -26.2% 22964939 softirqs.CPU100.NET_RX 31626866 -27.6% 22906397 softirqs.CPU101.NET_RX 29779153 ± 3% -22.9% 22969929 softirqs.CPU102.NET_RX 31513116 -27.3% 22920216 softirqs.CPU103.NET_RX 31713243 ± 2% -24.9% 23826688 ± 2% softirqs.CPU11.NET_RX 32072474 -26.1% 23696606 ± 2% softirqs.CPU12.NET_RX 32279430 -24.8% 24260634 softirqs.CPU13.NET_RX 31697375 ± 3% -23.0% 24406003 softirqs.CPU14.NET_RX 32278026 -24.7% 24317293 softirqs.CPU15.NET_RX 31162122 ± 3% -21.7% 24396540 softirqs.CPU16.NET_RX 32209863 -24.2% 24418327 softirqs.CPU17.NET_RX 32329002 -25.6% 24054387 ± 2% softirqs.CPU18.NET_RX 32127509 -24.5% 24258223 softirqs.CPU19.NET_RX 32217120 -25.2% 24089001 softirqs.CPU2.NET_RX 31415502 ± 3% -22.5% 24349881 softirqs.CPU20.NET_RX 32186243 -25.0% 24137967 softirqs.CPU21.NET_RX 32271735 -26.9% 23586468 ± 2% softirqs.CPU22.NET_RX 31871656 ± 3% -23.8% 24292511 softirqs.CPU23.NET_RX 32321894 -25.2% 24177260 softirqs.CPU24.NET_RX 31969188 -24.7% 24057006 softirqs.CPU25.NET_RX 30938777 -27.7% 22360933 ± 2% softirqs.CPU26.NET_RX 30118051 ± 3% -25.9% 22321500 ± 2% softirqs.CPU27.NET_RX 31079253 -28.1% 22337317 ± 2% softirqs.CPU28.NET_RX 31429166 -27.6% 22746093 ± 2% softirqs.CPU29.NET_RX 30485045 ± 3% -20.5% 24221128 ± 2% softirqs.CPU3.NET_RX 30912223 ± 2% -26.8% 22622183 ± 2% softirqs.CPU30.NET_RX 31357027 -27.6% 22717601 softirqs.CPU31.NET_RX 31052178 -27.3% 22565776 ± 2% softirqs.CPU32.NET_RX 31101262 -28.2% 22323348 ± 2% softirqs.CPU33.NET_RX 31331660 -27.7% 22656208 softirqs.CPU34.NET_RX 30833278 ± 3% -25.4% 22998777 softirqs.CPU35.NET_RX 30738161 ± 3% -25.2% 22998898 softirqs.CPU36.NET_RX 30881460 ± 4% -26.6% 22671426 softirqs.CPU37.NET_RX 31140748 -27.4% 22621903 softirqs.CPU38.NET_RX 31218654 -27.3% 22684067 softirqs.CPU39.NET_RX 32583070 -24.8% 24513384 softirqs.CPU4.NET_RX 31462139 -28.4% 22517351 softirqs.CPU40.NET_RX 31566697 -27.9% 22768461 softirqs.CPU41.NET_RX 30975279 ± 2% -27.2% 22555119 softirqs.CPU42.NET_RX 30718280 ± 3% -26.2% 22661320 ± 2% softirqs.CPU43.NET_RX 31283332 -27.3% 22740697 softirqs.CPU44.NET_RX 30983467 -26.4% 22794814 softirqs.CPU45.NET_RX 30658730 ± 4% -28.4% 21946934 ± 2% softirqs.CPU46.NET_RX 31181856 -26.5% 22929226 softirqs.CPU47.NET_RX 31037938 ± 2% -26.9% 22691892 ± 2% softirqs.CPU48.NET_RX 30911500 ± 2% -26.3% 22777693 softirqs.CPU49.NET_RX 31802923 ± 3% -23.9% 24204592 softirqs.CPU5.NET_RX 31517206 -28.1% 22648510 softirqs.CPU50.NET_RX 30660453 ± 3% -27.2% 22327925 ± 2% softirqs.CPU51.NET_RX 31819962 ± 2% -23.9% 24206194 softirqs.CPU52.NET_RX 32142649 -26.4% 23668373 ± 2% softirqs.CPU53.NET_RX 32149559 -24.2% 24360340 softirqs.CPU54.NET_RX 32425853 -26.4% 23854082 ± 2% softirqs.CPU55.NET_RX 31817157 ± 2% -24.3% 24096785 softirqs.CPU56.NET_RX 31640490 ± 3% -24.4% 23920278 ± 2% softirqs.CPU57.NET_RX 31650729 ± 3% -23.1% 24327421 softirqs.CPU58.NET_RX 31269148 ± 3% -22.3% 24307789 softirqs.CPU59.NET_RX 31030416 ± 4% -22.0% 24203694 softirqs.CPU6.NET_RX 32255424 -25.2% 24141973 ± 2% softirqs.CPU60.NET_RX 32245715 -25.1% 24167151 ± 2% softirqs.CPU61.NET_RX 31763163 ± 2% -23.1% 24415760 softirqs.CPU62.NET_RX 31632095 ± 3% -22.7% 24451397 softirqs.CPU63.NET_RX 31490930 ± 2% -24.2% 23879322 softirqs.CPU64.NET_RX 31516754 ± 3% -23.5% 24108737 softirqs.CPU65.NET_RX 32359236 -24.5% 24440756 softirqs.CPU66.NET_RX 31784935 ± 3% -23.9% 24195033 ± 2% softirqs.CPU67.NET_RX 30967426 ± 4% -21.8% 24208434 softirqs.CPU68.NET_RX 32208978 -27.9% 23223363 softirqs.CPU69.NET_RX 31534273 ± 3% -23.6% 24083955 softirqs.CPU7.NET_RX 31800791 ± 2% -23.3% 24375677 softirqs.CPU70.NET_RX 31511220 ± 3% -23.1% 24225232 softirqs.CPU71.NET_RX 32055308 -25.2% 23985501 ± 2% softirqs.CPU72.NET_RX 32255683 -24.3% 24425926 softirqs.CPU73.NET_RX 31808226 ± 2% -23.7% 24280351 ± 2% softirqs.CPU74.NET_RX 31994746 ± 3% -25.5% 23828432 ± 2% softirqs.CPU75.NET_RX 31732779 ± 3% -24.0% 24121773 ± 2% softirqs.CPU76.NET_RX 31249558 ± 2% -23.8% 23816726 ± 2% softirqs.CPU77.NET_RX 29102709 ± 3% -24.1% 22087804 ± 2% softirqs.CPU78.NET_RX 31050911 -26.3% 22879670 softirqs.CPU79.NET_RX 32385972 -25.5% 24114907 softirqs.CPU8.NET_RX 31250746 -26.8% 22865137 softirqs.CPU80.NET_RX 30970512 ± 3% -27.5% 22460428 ± 2% softirqs.CPU81.NET_RX 30769035 ± 3% -26.4% 22640114 softirqs.CPU82.NET_RX 31163906 -28.0% 22426881 ± 2% softirqs.CPU83.NET_RX 30833151 ± 2% -26.9% 22535706 softirqs.CPU84.NET_RX 31213448 -27.8% 22530613 ± 2% softirqs.CPU85.NET_RX 31299014 -27.8% 22598945 ± 2% softirqs.CPU86.NET_RX 31380574 -26.8% 22966556 softirqs.CPU87.NET_RX 31419156 -26.9% 22966072 softirqs.CPU88.NET_RX 31485946 -29.0% 22360125 ± 2% softirqs.CPU89.NET_RX 32266157 -25.9% 23914440 ± 2% softirqs.CPU9.NET_RX 30336933 ± 3% -25.9% 22489963 softirqs.CPU90.NET_RX 30595348 ± 3% -26.1% 22614928 softirqs.CPU91.NET_RX 30924249 ± 2% -26.0% 22894665 softirqs.CPU92.NET_RX 30593902 ± 2% -24.8% 23001426 softirqs.CPU93.NET_RX 31556110 -29.2% 22330354 ± 2% softirqs.CPU94.NET_RX 30668287 ± 3% -29.0% 21759470 softirqs.CPU95.NET_RX 31245897 -26.9% 22847267 softirqs.CPU96.NET_RX 31136421 -28.7% 22213048 ± 2% softirqs.CPU97.NET_RX 30013471 ± 4% -23.9% 22844117 softirqs.CPU98.NET_RX 31291344 -28.5% 22378868 ± 2% softirqs.CPU99.NET_RX 3.269e+09 -25.6% 2.431e+09 softirqs.NET_RX Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 4 months

3
5
0 / 0

CKI hackfest @Plumbers invite

by Veronika Kabatova

Hi, as some of you have heard, CKI Project is planning hackfest CI meetings after Plumbers conference this year (Sept. 12-13). We would like to invite everyone who has interest in CI for kernel to come and join us. The early agenda with summary is at the end of the email. If you think there's something important missing let us know! Also let us know in case you'd want to lead any of the sessions, we'd be happy to delegate out some work :) Please send us an email as soon as you decide to come and feel free to invite other people who should be present. We are not planning to cap the attendance right now but need to solve the logistics based on the interest. The event is free to attend, no additional registration except letting us know is needed. Feel free to contact us if you have any questions, Veronika CKI Project ----------------------------------------------------------- Here is an early agenda we put together: - Introductions - Common place for upstream results, result publishing in general - The discussion on the mailing list is going strong so we might be able to substitute this session for a different one in case everything is solved by September. - Test result interpretation and bug detection - How to autodetect infrastructure failures, regressions/new bugs and test bugs? How to handle continuous failures due to known bugs in both tests and kernel? What's your solution? Can people always trust the results they receive? - Getting results to developers/maintainers - Aimed at kernel developers and maintainers, share your feedback and expectations. - How much data should be sent in the initial communication vs. a click away in a dashboard? Do you want incremental emails with new results as they come in? - What about adding checks to tested patches in Patchwork when patch series are being tested? - Providing enough data/script to reproduce the failure. What if special HW is needed? - Onboarding new kernel trees to test - Aimed at kernel developers and maintainers. - Which trees are most prone to bring in new problems? Which are the most critical ones? Do you want them to be tested? Which tests do you feel are most beneficial for specific trees or in general? - Security when testing untrusted patches - How do we merge, compile, and test patches that have untrusted code in them and have not yet been reviewed? How do we avoid abuse of systems, information theft, or other damage? - Check out the original patch that sparked the discussion at https://patchwork.ozlabs.org/patch/862123/ - Avoiding effort duplication - Food for thought by GregKH - X different CI systems running ${TEST} on latest stable kernel on x86_64 might look useless on the first look but is it? AMD/Intel CPUs, different network cards, different graphic drivers, compilers, kernel configuration... How do we distribute the workload to avoid doing the same thing all over again while still running in enough different environments to get the most coverage? - Common hardware pools - Is this something people are interested in? Would be helpful especially for HW that's hard to access, eg. ppc64le or s390x systems. Companies could also sing up to share their HW for testing to ensure kernel works with their products.

2 years, 4 months

6
6
0 / 0

Re: [LKP] [rcu] kernel BUG at include/linux/pagemap.h:149!

by Frederic Weisbecker

On Fri, Sep 11, 2015 at 10:19:47AM +0800, Boqun Feng wrote: > Subject: [PATCH 01/27] rcu: Don't disable preemption for Tiny and Tree RCU > readers > > Because preempt_disable() maps to barrier() for non-debug builds, > it forces the compiler to spill and reload registers. Because Tree > RCU and Tiny RCU now only appear in CONFIG_PREEMPT=n builds, these > barrier() instances generate needless extra code for each instance of > rcu_read_lock() and rcu_read_unlock(). This extra code slows down Tree > RCU and bloats Tiny RCU. > > This commit therefore removes the preempt_disable() and preempt_enable() > from the non-preemptible implementations of __rcu_read_lock() and > __rcu_read_unlock(), respectively. > > For debug purposes, preempt_disable() and preempt_enable() are still > kept if CONFIG_PREEMPT_COUNT=y, which makes the detection of sleeping > inside atomic sections still work in non-preemptible kernels. > > Signed-off-by: Boqun Feng <boqun.feng(a)gmail.com> > Signed-off-by: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> > --- > include/linux/rcupdate.h | 6 ++++-- > include/linux/rcutiny.h | 1 + > kernel/rcu/tree.c | 9 +++++++++ > 3 files changed, 14 insertions(+), 2 deletions(-) > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h > index d63bb77..6c3cece 100644 > --- a/include/linux/rcupdate.h > +++ b/include/linux/rcupdate.h > @@ -297,12 +297,14 @@ void synchronize_rcu(void); > > static inline void __rcu_read_lock(void) > { > - preempt_disable(); > + if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) > + preempt_disable(); preempt_disable() is a no-op when !CONFIG_PREEMPT_COUNT, right? Or rather it's a barrier(), which is anyway implied by rcu_read_lock(). So perhaps we can get rid of the IS_ENABLED() check?

2 years, 4 months

4
4
0 / 0

[sched/fair] cca2ae2c00: aim7.jobs-per-min -35.9% regression

by kernel test robot

Greeting, FYI, we noticed a -35.9% regression of aim7.jobs-per-min due to commit: commit: cca2ae2c00b5fc7893f1a184d822812ee3425b7e ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()") git://linux-arm.org/linux-de.git upstream/rem_rq_load in testcase: aim7 on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory with following parameters: disk: 1BRD_48G fs: xfs test: creat-clo load: 1500 cpufreq_governor: performance test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase: gcc-7/performance/1BRD_48G/xfs/x86_64-rhel-7.6/1500/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/creat-clo/aim7 commit: fe146f762b ("sched: Remove rq->cpu_load[] update code") cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()") fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 68031 -35.9% 43605 ± 5% aim7.jobs-per-min 132.55 +56.4% 207.34 ± 5% aim7.time.elapsed_time 132.55 +56.4% 207.34 ± 5% aim7.time.elapsed_time.max 173584 +89.5% 328925 ± 3% aim7.time.involuntary_context_switches 92120 ± 2% +10.0% 101340 ± 4% aim7.time.minor_page_faults 4210 +36.0% 5725 ± 7% aim7.time.system_time 57.74 +101.6% 116.39 ± 5% aim7.time.user_time 2692874 ± 3% -42.8% 1540853 ± 7% aim7.time.voluntary_context_switches 22.16 +38.0% 30.58 ± 5% iostat.cpu.idle 76.74 -11.4% 67.98 ± 2% iostat.cpu.system 1.10 ± 2% +30.8% 1.44 ± 2% iostat.cpu.user 21.03 +8.9 29.93 ± 5% mpstat.cpu.all.idle% 77.85 -9.2 68.61 ± 2% mpstat.cpu.all.sys% 1.11 ± 2% +0.3 1.45 ± 2% mpstat.cpu.all.usr% 21.75 +37.9% 30.00 ± 5% vmstat.cpu.id 76.00 -11.2% 67.50 ± 2% vmstat.cpu.sy 41880 ± 3% -59.0% 17180 vmstat.system.cs 8.722e+08 ± 13% +131.5% 2.019e+09 ± 8% cpuidle.C6.time 1282389 ± 8% +136.9% 3038476 ± 17% cpuidle.C6.usage 1315243 ± 4% -93.1% 91226 ± 19% cpuidle.POLL.time 1772414 ± 4% -98.6% 24472 ± 17% cpuidle.POLL.usage 123678 +24.8% 154380 ± 2% meminfo.AnonHugePages 242624 ± 9% -18.6% 197568 ± 5% meminfo.DirectMap4k 40068 ± 8% -24.5% 30237 ± 2% meminfo.Shmem 16032 ± 2% -36.2% 10229 ± 3% meminfo.max_used_kB 1191648 ± 2% -21.3% 937552 numa-numastat.node0.local_node 1197443 ± 2% -21.5% 939550 numa-numastat.node0.numa_hit 1169975 ± 2% -26.9% 855025 ± 11% numa-numastat.node1.local_node 1172017 ± 2% -26.5% 860874 ± 11% numa-numastat.node1.numa_hit 933658 ± 6% -10.2% 838153 ± 3% numa-vmstat.node0.numa_hit 927596 ± 6% -24.4% 700855 ± 4% numa-vmstat.node0.numa_local 6061 ± 54% +2165.1% 137297 ± 2% numa-vmstat.node0.numa_other 906672 ± 5% -14.9% 771132 ± 7% numa-vmstat.node1.numa_hit 167353 -78.6% 35802 ± 7% numa-vmstat.node1.numa_other 1914 ± 4% +10.3% 2111 ± 3% slabinfo.eventpoll_pwq.active_objs 1914 ± 4% +10.3% 2111 ± 3% slabinfo.eventpoll_pwq.num_objs 68433 ± 4% -37.8% 42579 ± 3% slabinfo.filp.active_objs 2291 ± 4% -32.8% 1539 slabinfo.filp.active_slabs 73346 ± 4% -32.8% 49281 slabinfo.filp.num_objs 2291 ± 4% -32.8% 1539 slabinfo.filp.num_slabs 2867 ± 3% +9.2% 3130 ± 2% slabinfo.kmalloc-128.active_objs 2867 ± 3% +9.2% 3130 ± 2% slabinfo.kmalloc-128.num_objs 922.00 ± 9% +17.6% 1084 ± 4% slabinfo.task_group.active_objs 922.00 ± 9% +17.6% 1084 ± 4% slabinfo.task_group.num_objs 2334 -56.6% 1012 ± 3% turbostat.Avg_MHz 79.29 -8.5 70.79 ± 2% turbostat.Busy% 2943 -51.4% 1430 ± 3% turbostat.Bzy_MHz 1279422 ± 8% +137.3% 3035612 ± 17% turbostat.C6 15.91 ± 12% +8.0 23.87 ± 7% turbostat.C6% 12.74 ± 8% +46.4% 18.66 ± 9% turbostat.CPU%c1 127.15 -60.3% 50.54 ± 3% turbostat.CorWatt 68.75 -12.4% 60.25 turbostat.CoreTmp 11323915 +51.8% 17189663 ± 5% turbostat.IRQ 1.89 ± 34% -64.0% 0.68 ± 87% turbostat.Pkg%pc6 68.75 -12.4% 60.25 turbostat.PkgTmp 154.35 -50.2% 76.80 ± 2% turbostat.PkgWatt 36.98 -9.2% 33.59 turbostat.RAMWatt 10930 +54.8% 16920 ± 5% turbostat.SMI 79583 -2.1% 77935 proc-vmstat.nr_active_anon 275358 -1.3% 271685 proc-vmstat.nr_file_pages 5012 -2.0% 4914 proc-vmstat.nr_inactive_anon 30851 +1.5% 31301 proc-vmstat.nr_kernel_stack 6172 -2.0% 6046 proc-vmstat.nr_mapped 10015 ± 8% -24.5% 7558 ± 2% proc-vmstat.nr_shmem 32115 -4.7% 30603 proc-vmstat.nr_slab_unreclaimable 79583 -2.1% 77935 proc-vmstat.nr_zone_active_anon 5012 -2.0% 4914 proc-vmstat.nr_zone_inactive_anon 16174 +58.8% 25684 ± 13% proc-vmstat.numa_hint_faults 6212 ± 4% +117.6% 13517 ± 15% proc-vmstat.numa_hint_faults_local 2394858 -23.7% 1826339 ± 4% proc-vmstat.numa_hit 2387014 -23.8% 1818486 ± 4% proc-vmstat.numa_local 9960 +31.0% 13047 ± 10% proc-vmstat.numa_pages_migrated 17309 +188.4% 49918 ± 27% proc-vmstat.numa_pte_updates 5225 ± 16% -56.6% 2269 ± 9% proc-vmstat.pgactivate 4457838 -28.2% 3200155 ± 5% proc-vmstat.pgalloc_normal 413171 +41.1% 583119 ± 4% proc-vmstat.pgfault 4428619 -28.3% 3174333 ± 5% proc-vmstat.pgfree 9960 +31.0% 13047 ± 10% proc-vmstat.pgmigrate_success 4427 ± 4% +32.5% 5865 ± 3% proc-vmstat.pgpgout 932.00 ± 3% +3.4% 963.25 ± 4% proc-vmstat.unevictable_pgs_culled 23687 ± 91% +95.1% 46206 ± 72% sched_debug.cfs_rq:/.load.stddev 1212 ± 68% -70.6% 356.25 ± 11% sched_debug.cfs_rq:/.load_avg.max 13.08 ± 11% -49.8% 6.56 ± 30% sched_debug.cfs_rq:/.load_avg.min 0.35 ± 5% +19.6% 0.42 ± 4% sched_debug.cfs_rq:/.nr_running.stddev 15.34 +25.5% 19.25 ± 17% sched_debug.cfs_rq:/.runnable_load_avg.avg 8.52 ± 6% +290.6% 33.28 ± 67% sched_debug.cfs_rq:/.runnable_load_avg.stddev 22196 ± 96% +104.1% 45294 ± 75% sched_debug.cfs_rq:/.runnable_weight.stddev 23081 ± 26% -212.8% -26032 sched_debug.cfs_rq:/.spread0.avg 102531 ± 17% -46.3% 55090 ± 36% sched_debug.cfs_rq:/.spread0.max -7190 +952.0% -75647 sched_debug.cfs_rq:/.spread0.min 439.42 ± 14% -23.7% 335.08 ± 15% sched_debug.cfs_rq:/.util_avg.min 173.42 ± 11% +32.0% 228.90 ± 6% sched_debug.cfs_rq:/.util_avg.stddev 284592 ± 6% +90.7% 542802 ± 4% sched_debug.cpu.avg_idle.avg 6069 ± 33% +1188.9% 78224 ± 32% sched_debug.cpu.avg_idle.min 228220 ± 5% +15.8% 264189 ± 3% sched_debug.cpu.avg_idle.stddev 3.07 ± 11% +97.8% 6.08 ± 10% sched_debug.cpu.clock.stddev 3.07 ± 11% +97.8% 6.08 ± 10% sched_debug.cpu.clock_task.stddev 4032 +17.3% 4729 ± 7% sched_debug.cpu.curr->pid.max 851.30 ± 2% +25.1% 1065 ± 3% sched_debug.cpu.curr->pid.stddev 0.00 ± 2% +9.0% 0.00 ± 2% sched_debug.cpu.next_balance.stddev 0.40 ± 4% +21.8% 0.49 ± 7% sched_debug.cpu.nr_running.stddev 61876 ± 2% -39.5% 37429 ± 15% sched_debug.cpu.nr_switches.avg 72542 ± 3% -32.2% 49204 ± 10% sched_debug.cpu.nr_switches.max 55154 ± 3% -39.5% 33393 ± 16% sched_debug.cpu.nr_switches.min 4093 ± 18% -21.0% 3234 ± 14% sched_debug.cpu.nr_switches.stddev 23.68 +13.1% 26.78 ± 4% sched_debug.cpu.nr_uninterruptible.avg 170.25 ± 11% +136.5% 402.67 ± 20% sched_debug.cpu.nr_uninterruptible.max 46.30 ± 10% +97.2% 91.30 ± 20% sched_debug.cpu.nr_uninterruptible.stddev 4118075 -50.0% 2059067 sched_debug.sysctl_sched.sysctl_sched_features 9.08e+09 -57.2% 3.887e+09 ± 2% perf-stat.i.branch-instructions 39772561 -30.2% 27751148 ± 4% perf-stat.i.branch-misses 30.87 -4.6 26.27 ± 2% perf-stat.i.cache-miss-rate% 34133998 -42.8% 19525491 ± 4% perf-stat.i.cache-misses 1.1e+08 ± 2% -31.0% 75953637 ± 3% perf-stat.i.cache-references 42802 ± 3% -59.5% 17355 perf-stat.i.context-switches 9.388e+10 -57.1% 4.024e+10 ± 3% perf-stat.i.cpu-cycles 1480 ± 2% +161.3% 3868 ± 3% perf-stat.i.cpu-migrations 2576 ± 2% -19.5% 2073 ± 4% perf-stat.i.cycles-between-cache-misses 1.461e+10 -57.9% 6.147e+09 ± 2% perf-stat.i.dTLB-loads 2.598e+09 -34.4% 1.704e+09 ± 5% perf-stat.i.dTLB-stores 3340458 ± 2% -27.2% 2431112 ± 4% perf-stat.i.iTLB-load-misses 4.759e+10 -57.5% 2.021e+10 ± 2% perf-stat.i.instructions 13273 ± 3% -39.0% 8098 ± 4% perf-stat.i.instructions-per-iTLB-miss 2917 -8.8% 2660 perf-stat.i.minor-faults 47.27 +1.2 48.42 perf-stat.i.node-load-miss-rate% 17915543 -42.3% 10339302 ± 5% perf-stat.i.node-load-misses 19859340 -45.4% 10837882 ± 5% perf-stat.i.node-loads 40.22 +1.5 41.74 perf-stat.i.node-store-miss-rate% 9587423 -42.5% 5512775 ± 5% perf-stat.i.node-store-misses 13693791 -45.2% 7500888 ± 5% perf-stat.i.node-stores 2917 -8.8% 2660 perf-stat.i.page-faults 2.31 ± 3% +62.6% 3.76 ± 3% perf-stat.overall.MPKI 0.44 ± 2% +0.3 0.71 ± 4% perf-stat.overall.branch-miss-rate% 31.03 -5.3 25.73 ± 4% perf-stat.overall.cache-miss-rate% 2751 ± 2% -25.1% 2062 perf-stat.overall.cycles-between-cache-misses 14261 ± 4% -41.6% 8325 ± 4% perf-stat.overall.instructions-per-iTLB-miss 47.43 +1.4 48.82 perf-stat.overall.node-load-miss-rate% 41.18 +1.2 42.36 perf-stat.overall.node-store-miss-rate% 9.011e+09 -57.1% 3.868e+09 ± 2% perf-stat.ps.branch-instructions 39471063 -30.0% 27616773 ± 4% perf-stat.ps.branch-misses 33874716 -42.6% 19430430 ± 4% perf-stat.ps.cache-misses 1.092e+08 ± 2% -30.8% 75583753 ± 3% perf-stat.ps.cache-references 42475 ± 3% -59.3% 17270 perf-stat.ps.context-switches 9.317e+10 -57.0% 4.004e+10 ± 3% perf-stat.ps.cpu-cycles 1469 ± 2% +162.0% 3849 ± 3% perf-stat.ps.cpu-migrations 1.45e+10 -57.8% 6.117e+09 ± 2% perf-stat.ps.dTLB-loads 2.578e+09 -34.2% 1.696e+09 ± 5% perf-stat.ps.dTLB-stores 3315089 ± 2% -27.0% 2419259 ± 4% perf-stat.ps.iTLB-load-misses 4.723e+10 -57.4% 2.011e+10 ± 2% perf-stat.ps.instructions 2898 -8.6% 2649 perf-stat.ps.minor-faults 17778569 -42.1% 10288528 ± 5% perf-stat.ps.node-load-misses 19707496 -45.3% 10784655 ± 5% perf-stat.ps.node-loads 9514153 -42.3% 5485718 ± 5% perf-stat.ps.node-store-misses 13589365 -45.1% 7464266 ± 5% perf-stat.ps.node-stores 2898 -8.6% 2649 perf-stat.ps.page-faults 6.414e+12 ± 2% -34.0% 4.231e+12 ± 4% perf-stat.total.instructions 13476 ± 3% +40.8% 18978 ± 2% softirqs.CPU0.SCHED 50649 ± 3% +55.9% 78958 ± 7% softirqs.CPU0.TIMER 10154 ± 4% +69.5% 17215 ± 4% softirqs.CPU1.SCHED 49566 ± 4% +96.4% 97347 ± 16% softirqs.CPU1.TIMER 10195 +57.5% 16054 ± 3% softirqs.CPU10.SCHED 50586 ± 2% +53.2% 77512 ± 8% softirqs.CPU10.TIMER 9932 +67.0% 16585 ± 3% softirqs.CPU11.SCHED 49178 ± 6% +72.3% 84755 ± 4% softirqs.CPU11.TIMER 9750 ± 2% +65.0% 16089 ± 4% softirqs.CPU12.SCHED 54719 ± 18% +42.4% 77900 ± 5% softirqs.CPU12.TIMER 154622 ± 4% -14.5% 132171 ± 7% softirqs.CPU13.RCU 10067 ± 2% +63.7% 16475 ± 3% softirqs.CPU13.SCHED 49427 ± 5% +67.0% 82545 ± 4% softirqs.CPU13.TIMER 10123 ± 3% +59.9% 16185 ± 3% softirqs.CPU14.SCHED 50112 ± 2% +58.2% 79263 ± 6% softirqs.CPU14.TIMER 153886 ± 3% -14.4% 131761 ± 8% softirqs.CPU15.RCU 9982 ± 2% +64.2% 16390 ± 2% softirqs.CPU15.SCHED 49303 ± 5% +69.2% 83407 ± 3% softirqs.CPU15.TIMER 10189 +60.1% 16315 ± 2% softirqs.CPU16.SCHED 50924 ± 2% +53.7% 78274 ± 7% softirqs.CPU16.TIMER 149727 ± 6% -12.1% 131573 ± 7% softirqs.CPU17.RCU 10052 ± 2% +64.1% 16493 ± 2% softirqs.CPU17.SCHED 49666 ± 4% +67.0% 82929 ± 4% softirqs.CPU17.TIMER 10146 ± 2% +61.4% 16375 ± 2% softirqs.CPU18.SCHED 57432 ± 20% +36.9% 78602 ± 7% softirqs.CPU18.TIMER 149562 ± 5% -11.6% 132252 ± 7% softirqs.CPU19.RCU 10032 ± 2% +63.9% 16446 ± 3% softirqs.CPU19.SCHED 49273 ± 4% +67.2% 82393 ± 4% softirqs.CPU19.TIMER 10206 ± 2% +57.5% 16075 ± 4% softirqs.CPU2.SCHED 51506 +51.9% 78234 ± 7% softirqs.CPU2.TIMER 9769 ± 2% +64.9% 16110 ± 3% softirqs.CPU20.SCHED 49181 ± 4% +59.6% 78481 ± 6% softirqs.CPU20.TIMER 152472 ± 4% -13.5% 131909 ± 6% softirqs.CPU21.RCU 11128 ± 9% +45.9% 16237 ± 2% softirqs.CPU21.SCHED 52047 ± 6% +76.1% 91663 ± 9% softirqs.CPU21.TIMER 9992 +61.5% 16142 ± 4% softirqs.CPU22.SCHED 49187 ± 3% +60.2% 78791 ± 8% softirqs.CPU22.TIMER 146898 ± 4% -11.3% 130257 ± 7% softirqs.CPU23.RCU 10528 ± 6% +56.5% 16472 ± 2% softirqs.CPU23.SCHED 50357 ± 3% +68.0% 84618 ± 6% softirqs.CPU23.TIMER 10130 +60.0% 16211 ± 3% softirqs.CPU24.SCHED 55993 ± 16% +40.1% 78467 ± 8% softirqs.CPU24.TIMER 151281 ± 5% -13.8% 130379 ± 6% softirqs.CPU25.RCU 10544 ± 8% +56.3% 16483 ± 2% softirqs.CPU25.SCHED 56974 ± 12% +46.4% 83428 ± 5% softirqs.CPU25.TIMER 9988 +62.1% 16192 ± 4% softirqs.CPU26.SCHED 49650 +82.2% 90457 ± 13% softirqs.CPU26.TIMER 149485 -11.9% 131759 ± 6% softirqs.CPU27.RCU 9873 +67.1% 16493 ± 3% softirqs.CPU27.SCHED 48333 ± 4% +71.3% 82778 ± 4% softirqs.CPU27.TIMER 9977 +60.9% 16053 ± 3% softirqs.CPU28.SCHED 49167 ± 3% +63.7% 80466 ± 5% softirqs.CPU28.TIMER 147449 ± 6% -11.7% 130131 ± 6% softirqs.CPU29.RCU 9834 +66.0% 16325 softirqs.CPU29.SCHED 48376 ± 4% +68.1% 81326 ± 5% softirqs.CPU29.TIMER 154504 ± 6% -12.7% 134808 ± 7% softirqs.CPU3.RCU 9968 +88.7% 18814 ± 4% softirqs.CPU3.SCHED 49472 ± 3% +79.9% 88977 ± 8% softirqs.CPU3.TIMER 9917 ± 2% +62.1% 16074 ± 5% softirqs.CPU30.SCHED 49682 ± 3% +54.7% 76867 ± 8% softirqs.CPU30.TIMER 152680 ± 3% -14.9% 129949 ± 7% softirqs.CPU31.RCU 9983 +64.8% 16453 ± 2% softirqs.CPU31.SCHED 48960 ± 6% +70.4% 83422 ± 5% softirqs.CPU31.TIMER 10167 +57.0% 15962 ± 3% softirqs.CPU32.SCHED 51076 ± 2% +51.2% 77243 ± 6% softirqs.CPU32.TIMER 151683 ± 5% -15.6% 128070 ± 6% softirqs.CPU33.RCU 10089 ± 3% +62.3% 16379 ± 2% softirqs.CPU33.SCHED 49549 ± 5% +66.7% 82575 ± 5% softirqs.CPU33.TIMER 10213 +58.4% 16174 ± 3% softirqs.CPU34.SCHED 52055 ± 5% +50.9% 78542 ± 6% softirqs.CPU34.TIMER 141065 ± 5% -10.0% 126977 ± 6% softirqs.CPU35.RCU 10022 +64.8% 16519 ± 3% softirqs.CPU35.SCHED 53261 ± 14% +57.0% 83625 ± 3% softirqs.CPU35.TIMER 10083 +61.2% 16257 ± 3% softirqs.CPU36.SCHED 51476 +51.1% 77803 ± 6% softirqs.CPU36.TIMER 148739 ± 2% -14.3% 127470 ± 7% softirqs.CPU37.RCU 9978 ± 2% +64.5% 16411 ± 2% softirqs.CPU37.SCHED 49715 ± 3% +66.3% 82678 ± 4% softirqs.CPU37.TIMER 10154 ± 2% +58.7% 16110 ± 2% softirqs.CPU38.SCHED 53442 ± 6% +59.1% 85001 ± 18% softirqs.CPU38.TIMER 150024 ± 4% -13.1% 130420 ± 8% softirqs.CPU39.RCU 10042 +59.0% 15967 softirqs.CPU39.SCHED 49248 ± 5% +66.3% 81906 ± 4% softirqs.CPU39.TIMER 10462 ± 5% +54.3% 16142 ± 3% softirqs.CPU4.SCHED 50777 ± 2% +53.9% 78169 ± 7% softirqs.CPU4.TIMER 150834 ± 2% -10.3% 135341 ± 7% softirqs.CPU5.RCU 10002 +67.0% 16704 ± 5% softirqs.CPU5.SCHED 49621 +69.4% 84052 ± 5% softirqs.CPU5.TIMER 10015 +61.4% 16164 ± 5% softirqs.CPU6.SCHED 49679 +63.2% 81070 ± 5% softirqs.CPU6.TIMER 150360 ± 5% -11.3% 133386 ± 6% softirqs.CPU7.RCU 9951 +65.1% 16432 ± 2% softirqs.CPU7.SCHED 53815 ± 20% +52.8% 82214 ± 5% softirqs.CPU7.TIMER 9958 +62.2% 16148 ± 3% softirqs.CPU8.SCHED 48872 ± 2% +61.7% 79048 ± 6% softirqs.CPU8.TIMER 152788 ± 3% -12.8% 133236 ± 7% softirqs.CPU9.RCU 9863 ± 3% +65.9% 16367 ± 2% softirqs.CPU9.SCHED 48422 ± 5% +68.2% 81429 ± 5% softirqs.CPU9.TIMER 406958 +61.6% 657494 ± 2% softirqs.SCHED 2030752 ± 2% +61.4% 3277241 ± 5% softirqs.TIMER 94.75 ± 13% +335.1% 412.25 ± 67% interrupts.41:IR-PCI-MSI.524291-edge.eth0-TxRx-2 90.75 ± 19% +64.5% 149.25 ± 18% interrupts.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5 101.00 ± 23% +72.0% 173.75 ± 32% interrupts.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6 66.25 +56.2% 103.50 ± 4% interrupts.96:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 53986 +51.2% 81621 ± 8% interrupts.CAL:Function_call_interrupts 1354 ± 6% +54.4% 2091 ± 4% interrupts.CPU0.CAL:Function_call_interrupts 267296 +58.0% 422236 ± 5% interrupts.CPU0.LOC:Local_timer_interrupts 6383 ± 8% -41.4% 3739 ± 9% interrupts.CPU0.RES:Rescheduling_interrupts 1358 ± 4% +46.0% 1982 ± 11% interrupts.CPU1.CAL:Function_call_interrupts 269685 ± 2% +56.6% 422237 ± 5% interrupts.CPU1.LOC:Local_timer_interrupts 7255 ± 7% -56.2% 3179 ± 9% interrupts.CPU1.RES:Rescheduling_interrupts 66.25 +56.2% 103.50 ± 4% interrupts.CPU10.96:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 1366 ± 5% +45.2% 1984 ± 14% interrupts.CPU10.CAL:Function_call_interrupts 267276 +58.0% 422206 ± 5% interrupts.CPU10.LOC:Local_timer_interrupts 6675 ± 10% -57.6% 2827 ± 8% interrupts.CPU10.RES:Rescheduling_interrupts 1309 ± 5% +53.9% 2015 ± 11% interrupts.CPU11.CAL:Function_call_interrupts 266188 +58.7% 422319 ± 5% interrupts.CPU11.LOC:Local_timer_interrupts 6719 ± 6% -56.8% 2906 ± 13% interrupts.CPU11.RES:Rescheduling_interrupts 1351 ± 4% +45.8% 1971 ± 8% interrupts.CPU12.CAL:Function_call_interrupts 269142 ± 2% +56.8% 422008 ± 5% interrupts.CPU12.LOC:Local_timer_interrupts 6698 ± 11% -57.6% 2839 ± 7% interrupts.CPU12.RES:Rescheduling_interrupts 1359 ± 4% +54.4% 2098 ± 12% interrupts.CPU13.CAL:Function_call_interrupts 266898 +58.2% 422104 ± 5% interrupts.CPU13.LOC:Local_timer_interrupts 6765 ± 7% -58.4% 2815 ± 17% interrupts.CPU13.RES:Rescheduling_interrupts 1340 ± 7% +50.0% 2010 ± 7% interrupts.CPU14.CAL:Function_call_interrupts 268735 ± 2% +57.1% 422054 ± 5% interrupts.CPU14.LOC:Local_timer_interrupts 6450 ± 7% -55.5% 2872 ± 8% interrupts.CPU14.RES:Rescheduling_interrupts 1376 ± 3% +48.0% 2037 ± 12% interrupts.CPU15.CAL:Function_call_interrupts 269196 ± 2% +56.8% 421995 ± 5% interrupts.CPU15.LOC:Local_timer_interrupts 6730 ± 5% -59.2% 2744 ± 13% interrupts.CPU15.RES:Rescheduling_interrupts 1286 ± 13% +59.2% 2046 ± 12% interrupts.CPU16.CAL:Function_call_interrupts 269168 ± 2% +56.7% 421841 ± 5% interrupts.CPU16.LOC:Local_timer_interrupts 6273 ± 10% -55.9% 2768 ± 9% interrupts.CPU16.RES:Rescheduling_interrupts 1361 ± 3% +48.3% 2019 ± 6% interrupts.CPU17.CAL:Function_call_interrupts 269146 ± 2% +56.9% 422160 ± 5% interrupts.CPU17.LOC:Local_timer_interrupts 6946 ± 3% -62.1% 2635 ± 10% interrupts.CPU17.RES:Rescheduling_interrupts 1326 ± 4% +51.3% 2006 ± 12% interrupts.CPU18.CAL:Function_call_interrupts 268729 ± 2% +57.0% 421848 ± 5% interrupts.CPU18.LOC:Local_timer_interrupts 6064 ± 12% -53.2% 2839 ± 9% interrupts.CPU18.RES:Rescheduling_interrupts 1275 +56.1% 1990 ± 4% interrupts.CPU19.CAL:Function_call_interrupts 269182 ± 2% +56.8% 422201 ± 5% interrupts.CPU19.LOC:Local_timer_interrupts 23.25 ±138% +6069.9% 1434 ±105% interrupts.CPU19.NMI:Non-maskable_interrupts 23.25 ±138% +6069.9% 1434 ±105% interrupts.CPU19.PMI:Performance_monitoring_interrupts 6504 ± 3% -57.0% 2798 ± 8% interrupts.CPU19.RES:Rescheduling_interrupts 90.75 ± 19% +64.5% 149.25 ± 18% interrupts.CPU2.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5 1305 ± 9% +54.9% 2022 ± 14% interrupts.CPU2.CAL:Function_call_interrupts 266564 +58.3% 421875 ± 5% interrupts.CPU2.LOC:Local_timer_interrupts 6681 ± 17% -51.8% 3219 ± 14% interrupts.CPU2.RES:Rescheduling_interrupts 1379 ± 6% +40.5% 1937 ± 10% interrupts.CPU20.CAL:Function_call_interrupts 269083 ± 2% +56.8% 421789 ± 5% interrupts.CPU20.LOC:Local_timer_interrupts 6676 ± 12% -57.5% 2838 ± 9% interrupts.CPU20.RES:Rescheduling_interrupts 1376 ± 8% +51.3% 2083 ± 10% interrupts.CPU21.CAL:Function_call_interrupts 268308 +57.3% 421982 ± 5% interrupts.CPU21.LOC:Local_timer_interrupts 7344 ± 6% -63.1% 2711 ± 10% interrupts.CPU21.RES:Rescheduling_interrupts 1367 ± 5% +48.8% 2033 ± 5% interrupts.CPU22.CAL:Function_call_interrupts 267131 +57.9% 421846 ± 5% interrupts.CPU22.LOC:Local_timer_interrupts 6569 ± 12% -57.7% 2779 ± 7% interrupts.CPU22.RES:Rescheduling_interrupts 1347 ± 7% +51.1% 2036 ± 15% interrupts.CPU23.CAL:Function_call_interrupts 267612 +57.7% 422027 ± 5% interrupts.CPU23.LOC:Local_timer_interrupts 6903 ± 4% -59.9% 2771 ± 16% interrupts.CPU23.RES:Rescheduling_interrupts 1299 ± 7% +47.1% 1910 ± 6% interrupts.CPU24.CAL:Function_call_interrupts 266620 +58.2% 421923 ± 5% interrupts.CPU24.LOC:Local_timer_interrupts 3745 ± 66% -90.9% 340.00 ±173% interrupts.CPU24.NMI:Non-maskable_interrupts 3745 ± 66% -90.9% 340.00 ±173% interrupts.CPU24.PMI:Performance_monitoring_interrupts 6605 ± 9% -57.5% 2804 ± 5% interrupts.CPU24.RES:Rescheduling_interrupts 1392 ± 2% +50.3% 2092 ± 11% interrupts.CPU25.CAL:Function_call_interrupts 267144 +57.9% 421926 ± 5% interrupts.CPU25.LOC:Local_timer_interrupts 7062 ± 2% -60.9% 2761 ± 14% interrupts.CPU25.RES:Rescheduling_interrupts 1445 +32.0% 1907 ± 9% interrupts.CPU26.CAL:Function_call_interrupts 266423 +58.3% 421705 ± 5% interrupts.CPU26.LOC:Local_timer_interrupts 5617 ± 32% -99.8% 10.25 ±156% interrupts.CPU26.NMI:Non-maskable_interrupts 5617 ± 32% -99.8% 10.25 ±156% interrupts.CPU26.PMI:Performance_monitoring_interrupts 6698 ± 12% -58.1% 2803 ± 8% interrupts.CPU26.RES:Rescheduling_interrupts 1349 ± 7% +47.1% 1985 ± 10% interrupts.CPU27.CAL:Function_call_interrupts 267001 +58.1% 422055 ± 5% interrupts.CPU27.LOC:Local_timer_interrupts 3567 ± 9% -70.7% 1046 ±131% interrupts.CPU27.NMI:Non-maskable_interrupts 3567 ± 9% -70.7% 1046 ±131% interrupts.CPU27.PMI:Performance_monitoring_interrupts 6626 ± 4% -58.2% 2767 ± 18% interrupts.CPU27.RES:Rescheduling_interrupts 1343 ± 5% +53.3% 2058 ± 9% interrupts.CPU28.CAL:Function_call_interrupts 266397 +58.3% 421762 ± 5% interrupts.CPU28.LOC:Local_timer_interrupts 6308 ± 9% -55.8% 2789 ± 11% interrupts.CPU28.RES:Rescheduling_interrupts 1348 ± 5% +52.7% 2059 ± 9% interrupts.CPU29.CAL:Function_call_interrupts 269137 ± 2% +56.8% 422139 ± 5% interrupts.CPU29.LOC:Local_timer_interrupts 6912 ± 3% -60.0% 2764 ± 14% interrupts.CPU29.RES:Rescheduling_interrupts 1331 ± 4% +54.6% 2057 ± 5% interrupts.CPU3.CAL:Function_call_interrupts 269405 ± 2% +56.7% 422183 ± 5% interrupts.CPU3.LOC:Local_timer_interrupts 7171 ± 6% -61.2% 2785 ± 7% interrupts.CPU3.RES:Rescheduling_interrupts 1337 ± 6% +51.8% 2029 ± 5% interrupts.CPU30.CAL:Function_call_interrupts 266174 +58.6% 422198 ± 5% interrupts.CPU30.LOC:Local_timer_interrupts 6522 ± 16% -54.6% 2961 ± 12% interrupts.CPU30.RES:Rescheduling_interrupts 1397 ± 3% +51.0% 2110 ± 11% interrupts.CPU31.CAL:Function_call_interrupts 266913 +58.2% 422156 ± 5% interrupts.CPU31.LOC:Local_timer_interrupts 7094 ± 3% -60.9% 2771 ± 14% interrupts.CPU31.RES:Rescheduling_interrupts 1347 ± 7% +59.8% 2153 ± 8% interrupts.CPU32.CAL:Function_call_interrupts 266250 +58.1% 421070 ± 4% interrupts.CPU32.LOC:Local_timer_interrupts 6519 ± 14% -55.5% 2900 ± 9% interrupts.CPU32.RES:Rescheduling_interrupts 1351 ± 4% +47.0% 1985 ± 9% interrupts.CPU33.CAL:Function_call_interrupts 266078 +58.6% 421991 ± 5% interrupts.CPU33.LOC:Local_timer_interrupts 6204 ± 3% -58.2% 2590 ± 14% interrupts.CPU33.RES:Rescheduling_interrupts 94.75 ± 13% +335.1% 412.25 ± 67% interrupts.CPU34.41:IR-PCI-MSI.524291-edge.eth0-TxRx-2 1315 ± 8% +65.9% 2183 ± 8% interrupts.CPU34.CAL:Function_call_interrupts 266341 +58.3% 421683 ± 5% interrupts.CPU34.LOC:Local_timer_interrupts 6533 ± 15% -53.6% 3033 ± 14% interrupts.CPU34.RES:Rescheduling_interrupts 1373 ± 6% +55.2% 2132 ± 5% interrupts.CPU35.CAL:Function_call_interrupts 266140 +58.6% 422149 ± 5% interrupts.CPU35.LOC:Local_timer_interrupts 6663 ± 5% -58.7% 2753 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts 1381 ± 5% +49.1% 2059 ± 6% interrupts.CPU36.CAL:Function_call_interrupts 265961 +58.6% 421688 ± 5% interrupts.CPU36.LOC:Local_timer_interrupts 6267 ± 14% -54.7% 2840 ± 7% interrupts.CPU36.RES:Rescheduling_interrupts 1385 ± 4% +53.2% 2122 ± 11% interrupts.CPU37.CAL:Function_call_interrupts 266929 +58.2% 422152 ± 5% interrupts.CPU37.LOC:Local_timer_interrupts 7480 ± 34% -77.2% 1703 ± 40% interrupts.CPU37.NMI:Non-maskable_interrupts 7480 ± 34% -77.2% 1703 ± 40% interrupts.CPU37.PMI:Performance_monitoring_interrupts 7124 ± 2% -61.8% 2721 ± 17% interrupts.CPU37.RES:Rescheduling_interrupts 1405 ± 6% +40.8% 1979 ± 5% interrupts.CPU38.CAL:Function_call_interrupts 266168 +58.5% 421978 ± 5% interrupts.CPU38.LOC:Local_timer_interrupts 6570 ± 13% -56.6% 2854 ± 10% interrupts.CPU38.RES:Rescheduling_interrupts 36.50 ± 68% +575.3% 246.50 ± 65% interrupts.CPU38.TLB:TLB_shootdowns 1453 ± 2% +41.0% 2048 ± 12% interrupts.CPU39.CAL:Function_call_interrupts 266343 +58.1% 421194 ± 5% interrupts.CPU39.LOC:Local_timer_interrupts 6526 ± 24% -72.9% 1767 ±102% interrupts.CPU39.NMI:Non-maskable_interrupts 6526 ± 24% -72.9% 1767 ±102% interrupts.CPU39.PMI:Performance_monitoring_interrupts 6869 ± 3% -60.8% 2690 ± 14% interrupts.CPU39.RES:Rescheduling_interrupts 64.75 ± 46% +556.0% 424.75 ± 74% interrupts.CPU39.TLB:TLB_shootdowns 101.00 ± 23% +72.0% 173.75 ± 32% interrupts.CPU4.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6 1350 ± 5% +60.3% 2165 ± 8% interrupts.CPU4.CAL:Function_call_interrupts 267121 +57.9% 421817 ± 5% interrupts.CPU4.LOC:Local_timer_interrupts 6429 ± 15% -52.2% 3072 ± 9% interrupts.CPU4.RES:Rescheduling_interrupts 1331 ± 4% +47.1% 1959 ± 15% interrupts.CPU5.CAL:Function_call_interrupts 269514 ± 2% +56.8% 422468 ± 5% interrupts.CPU5.LOC:Local_timer_interrupts 6580 ± 6% -57.6% 2790 ± 13% interrupts.CPU5.RES:Rescheduling_interrupts 1273 ± 3% +67.9% 2139 ± 9% interrupts.CPU6.CAL:Function_call_interrupts 269080 ± 2% +56.8% 421874 ± 5% interrupts.CPU6.LOC:Local_timer_interrupts 6094 ± 12% -49.5% 3079 ± 16% interrupts.CPU6.RES:Rescheduling_interrupts 1281 ± 6% +64.3% 2106 ± 8% interrupts.CPU7.CAL:Function_call_interrupts 269338 ± 2% +56.8% 422190 ± 5% interrupts.CPU7.LOC:Local_timer_interrupts 6606 ± 5% -58.8% 2724 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts 1320 ± 4% +50.6% 1989 ± 12% interrupts.CPU8.CAL:Function_call_interrupts 269056 ± 2% +56.9% 422204 ± 5% interrupts.CPU8.LOC:Local_timer_interrupts 6206 ± 10% -54.1% 2850 ± 6% interrupts.CPU8.RES:Rescheduling_interrupts 1330 ± 6% +51.7% 2017 ± 8% interrupts.CPU9.CAL:Function_call_interrupts 266031 +58.7% 422174 ± 5% interrupts.CPU9.LOC:Local_timer_interrupts 6742 ± 7% -60.8% 2642 ± 7% interrupts.CPU9.RES:Rescheduling_interrupts 10704917 +57.7% 16879423 ± 5% interrupts.LOC:Local_timer_interrupts 149849 ± 17% -61.7% 57361 ± 11% interrupts.NMI:Non-maskable_interrupts 149849 ± 17% -61.7% 57361 ± 11% interrupts.PMI:Performance_monitoring_interrupts 266053 ± 5% -57.1% 114037 ± 10% interrupts.RES:Rescheduling_interrupts 1072 ± 9% +60.8% 1725 ± 20% interrupts.TLB:TLB_shootdowns 65.19 -11.9 53.32 ± 4% perf-profile.calltrace.cycles.osq_lock.rwsem_down_write_failed.down_write.path_openat.do_filp_open 67.76 -11.4 56.37 ± 4% perf-profile.calltrace.cycles.rwsem_down_write_failed.down_write.path_openat.do_filp_open.do_sys_open 68.21 -11.4 56.86 ± 4% perf-profile.calltrace.cycles.down_write.path_openat.do_filp_open.do_sys_open.do_syscall_64 31.80 ±120% -9.2 22.57 ±173% perf-profile.calltrace.cycles.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.22 ±120% -9.2 24.01 ±173% perf-profile.calltrace.cycles.do_syscall_64.entry_SYSCALL_64_after_hwframe 32.26 ±120% -9.2 23.06 ±173% perf-profile.calltrace.cycles.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.26 ±120% -9.2 24.06 ±173% perf-profile.calltrace.cycles.entry_SYSCALL_64_after_hwframe 93.10 -3.7 89.35 perf-profile.calltrace.cycles.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.45 ± 57% +0.2 0.69 ± 5% perf-profile.calltrace.cycles.osq_unlock.rwsem_down_write_failed.down_write.path_openat.do_filp_open 0.69 ± 15% +0.3 0.98 ± 12% perf-profile.calltrace.cycles.xlog_ungrant_log_space.xfs_log_done.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize 0.84 ± 18% +0.3 1.15 ± 3% perf-profile.calltrace.cycles.rwsem_spin_on_owner.rwsem_down_write_failed.down_write.path_openat.do_filp_open 0.43 ± 58% +0.3 0.75 ± 4% perf-profile.calltrace.cycles.complete_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.78 ± 16% +0.4 1.17 ± 6% perf-profile.calltrace.cycles.__alloc_file.alloc_empty_file.path_openat.do_filp_open.do_sys_open 0.26 ±100% +0.4 0.66 ± 7% perf-profile.calltrace.cycles.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize 0.83 ± 16% +0.4 1.25 ± 6% perf-profile.calltrace.cycles.alloc_empty_file.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.40 ± 57% +0.4 0.82 ± 8% perf-profile.calltrace.cycles.xlog_grant_add_space.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize 0.47 ± 57% +0.4 0.89 ± 6% perf-profile.calltrace.cycles.selinux_inode_permission.security_inode_permission.link_path_walk.path_openat.do_filp_open 0.89 ± 17% +0.4 1.31 ± 10% perf-profile.calltrace.cycles.xfs_log_done.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_setattr_size 0.28 ±100% +0.4 0.72 ± 4% perf-profile.calltrace.cycles.unlazy_walk.complete_walk.path_openat.do_filp_open.do_sys_open 0.50 ± 57% +0.5 0.96 ± 5% perf-profile.calltrace.cycles.security_inode_permission.link_path_walk.path_openat.do_filp_open.do_sys_open 0.77 ± 18% +0.5 1.23 ± 3% perf-profile.calltrace.cycles.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.14 ±173% +0.5 0.61 ± 12% perf-profile.calltrace.cycles.dput.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.14 ±173% +0.5 0.65 ± 3% perf-profile.calltrace.cycles.legitimize_path.unlazy_walk.complete_walk.path_openat.do_filp_open 0.00 +0.6 0.55 ± 3% perf-profile.calltrace.cycles.strncpy_from_user.getname_flags.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.57 ± 5% perf-profile.calltrace.cycles.smp_apic_timer_interrupt.apic_timer_interrupt.osq_lock.rwsem_down_write_failed.down_write 0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles.down_read.xfs_log_commit_cil.__xfs_trans_commit.xfs_setattr_nonsize.xfs_setattr_size 0.00 +0.6 0.61 ± 4% perf-profile.calltrace.cycles.apic_timer_interrupt.osq_lock.rwsem_down_write_failed.down_write.path_openat 0.93 ± 19% +0.6 1.54 ± 3% perf-profile.calltrace.cycles.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.14 ±173% +0.6 0.76 ± 9% perf-profile.calltrace.cycles.walk_component.link_path_walk.path_openat.do_filp_open.do_sys_open 0.00 +0.6 0.63 ± 8% perf-profile.calltrace.cycles.may_open.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.00 +0.6 0.63 ± 9% perf-profile.calltrace.cycles.lookup_fast.walk_component.link_path_walk.path_openat.do_filp_open 1.59 +0.7 2.25 ± 6% perf-profile.calltrace.cycles.xfs_log_reserve.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size 1.64 +0.7 2.32 ± 5% perf-profile.calltrace.cycles.xfs_trans_reserve.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr 1.75 ± 3% +0.8 2.54 ± 6% perf-profile.calltrace.cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.do_syscall_64 2.35 +1.0 3.31 ± 6% perf-profile.calltrace.cycles.xfs_trans_alloc.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr.notify_change 0.00 +1.0 0.98 ± 26% perf-profile.calltrace.cycles.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 0.00 +1.0 0.98 ± 26% perf-profile.calltrace.cycles.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.76 ± 19% +1.0 1.76 ± 7% perf-profile.calltrace.cycles._raw_spin_lock_irqsave.try_to_wake_up.wake_up_q.rwsem_wake.path_openat 0.45 ± 58% +1.0 1.47 ± 8% perf-profile.calltrace.cycles.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.try_to_wake_up.wake_up_q.rwsem_wake 0.94 ± 19% +1.1 2.06 ± 5% perf-profile.calltrace.cycles.try_to_wake_up.wake_up_q.rwsem_wake.path_openat.do_filp_open 1.02 ± 19% +1.2 2.18 ± 6% perf-profile.calltrace.cycles.wake_up_q.rwsem_wake.path_openat.do_filp_open.do_sys_open 0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +1.2 1.20 ± 22% perf-profile.calltrace.cycles.start_secondary.secondary_startup_64 1.99 ± 2% +1.2 3.22 ± 4% perf-profile.calltrace.cycles.rwsem_wake.path_openat.do_filp_open.do_sys_open.do_syscall_64 0.00 +1.2 1.25 ± 22% perf-profile.calltrace.cycles.secondary_startup_64 13.78 ± 9% +3.2 16.98 ± 9% perf-profile.calltrace.cycles.xfs_setattr_nonsize.xfs_setattr_size.xfs_vn_setattr.notify_change.do_truncate 13.95 ± 9% +3.2 17.19 ± 9% perf-profile.calltrace.cycles.xfs_setattr_size.xfs_vn_setattr.notify_change.do_truncate.path_openat 14.60 ± 8% +3.5 18.06 ± 8% perf-profile.calltrace.cycles.xfs_vn_setattr.notify_change.do_truncate.path_openat.do_filp_open 15.35 ± 8% +3.7 19.04 ± 8% perf-profile.calltrace.cycles.notify_change.do_truncate.path_openat.do_filp_open.do_sys_open 15.55 ± 8% +3.8 19.31 ± 8% perf-profile.calltrace.cycles.do_truncate.path_openat.do_filp_open.do_sys_open.do_syscall_64 65.26 -11.9 53.37 ± 4% perf-profile.children.cycles.osq_lock 67.82 -11.4 56.42 ± 4% perf-profile.children.cycles.rwsem_down_write_failed 68.58 -11.3 57.31 ± 3% perf-profile.children.cycles.down_write 93.10 -3.7 89.36 perf-profile.children.cycles.path_openat 93.19 -3.7 89.52 perf-profile.children.cycles.do_filp_open 94.57 -3.0 91.53 perf-profile.children.cycles.do_sys_open 97.37 -1.9 95.45 perf-profile.children.cycles.do_syscall_64 97.49 -1.9 95.62 perf-profile.children.cycles.entry_SYSCALL_64_after_hwframe 0.10 ± 10% +0.0 0.12 ± 4% perf-profile.children.cycles.xfs_inode_item_size 0.06 ± 11% +0.0 0.08 ± 10% perf-profile.children.cycles.dax_layout_busy_page 0.05 ± 9% +0.0 0.08 ± 10% perf-profile.children.cycles.xfs_trans_ijoin 0.07 ± 5% +0.0 0.10 ± 15% perf-profile.children.cycles.xfs_da3_node_lookup_int 0.20 ± 6% +0.0 0.22 ± 4% perf-profile.children.cycles.run_ksoftirqd 0.07 ± 6% +0.0 0.10 ± 15% perf-profile.children.cycles.xfs_break_leased_layouts 0.07 ± 6% +0.0 0.10 ± 5% perf-profile.children.cycles.percpu_counter_add_batch 0.20 ± 3% +0.0 0.23 ± 3% perf-profile.children.cycles.__list_del_entry_valid 0.21 ± 5% +0.0 0.24 ± 3% perf-profile.children.cycles.smpboot_thread_fn 0.08 ± 5% +0.0 0.11 ± 9% perf-profile.children.cycles.__slab_free 0.07 ± 5% +0.0 0.10 ± 8% perf-profile.children.cycles.xfs_dir3_data_check 0.07 ± 5% +0.0 0.10 ± 8% perf-profile.children.cycles.__xfs_dir3_data_check 0.11 ± 4% +0.0 0.15 ± 7% perf-profile.children.cycles.xfs_dir_removename 0.22 ± 7% +0.0 0.25 perf-profile.children.cycles.terminate_walk 0.07 ± 12% +0.0 0.10 ± 18% perf-profile.children.cycles.selinux_task_getsecid 0.13 ± 3% +0.0 0.16 ± 12% perf-profile.children.cycles.xfs_create 0.11 ± 4% +0.0 0.15 ± 7% perf-profile.children.cycles.xfs_dir2_node_removename 0.06 ± 14% +0.0 0.10 ± 5% perf-profile.children.cycles.xfs_inode_item_format_data_fork 0.04 ± 58% +0.0 0.08 ± 8% perf-profile.children.cycles.__check_heap_object 0.11 ± 6% +0.0 0.15 ± 8% perf-profile.children.cycles.xfs_log_calc_unit_res 0.07 ± 12% +0.0 0.11 ± 12% perf-profile.children.cycles.xfs_inode_item_unlock 0.11 ± 7% +0.0 0.15 ± 5% perf-profile.children.cycles.__list_add_valid 0.11 ± 6% +0.0 0.15 ± 11% perf-profile.children.cycles.rcu_all_qs 0.13 ± 3% +0.0 0.17 ± 12% perf-profile.children.cycles.xfs_generic_create 0.13 ± 6% +0.0 0.17 ± 9% perf-profile.children.cycles.__call_rcu 0.09 ± 13% +0.0 0.14 ± 11% perf-profile.children.cycles.__legitimize_mnt 0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles.lockref_get 0.01 ±173% +0.0 0.06 ± 20% perf-profile.children.cycles.xfs_difree 0.11 ± 9% +0.0 0.16 ± 11% perf-profile.children.cycles.xfs_trans_del_item 0.25 ± 5% +0.0 0.30 ± 6% perf-profile.children.cycles.xfs_break_layouts 0.06 ± 14% +0.0 0.11 ± 24% perf-profile.children.cycles.io_serial_in 0.00 +0.1 0.05 perf-profile.children.cycles.__update_load_avg_se 0.10 ± 8% +0.1 0.16 ± 7% perf-profile.children.cycles.xfs_log_space_wake 0.01 ±173% +0.1 0.07 ± 17% perf-profile.children.cycles.update_curr 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles.ktime_get_update_offsets_now 0.11 ± 7% +0.1 0.16 ± 18% perf-profile.children.cycles.__x86_indirect_thunk_rax 0.04 ± 57% +0.1 0.09 ± 14% perf-profile.children.cycles.setattr_prepare 0.01 ±173% +0.1 0.07 ± 6% perf-profile.children.cycles.__fd_install 0.01 ±173% +0.1 0.07 ± 10% perf-profile.children.cycles.rcu_segcblist_enqueue 0.17 ± 3% +0.1 0.22 ± 3% perf-profile.children.cycles.vfs_unlink 0.16 ± 4% +0.1 0.22 ± 7% perf-profile.children.cycles.xfs_remove 0.13 ± 3% +0.1 0.18 ± 9% perf-profile.children.cycles.schedule 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles.inode_has_perm 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles.xfs_log_ticket_put 0.01 ±173% +0.1 0.07 ± 17% perf-profile.children.cycles.xfs_file_release 0.09 ± 7% +0.1 0.15 ± 13% perf-profile.children.cycles.security_task_getsecid 0.18 ± 2% +0.1 0.24 ± 21% perf-profile.children.cycles.path_init 0.18 ± 5% +0.1 0.24 ± 5% perf-profile.children.cycles.__mnt_want_write 0.16 ± 4% +0.1 0.22 ± 5% perf-profile.children.cycles.xfs_vn_unlink 0.00 +0.1 0.06 ± 16% perf-profile.children.cycles.delay_tsc 0.32 ± 4% +0.1 0.38 ± 6% perf-profile.children.cycles.xlog_grant_push_ail 0.15 ± 5% +0.1 0.21 ± 11% perf-profile.children.cycles.xfs_iunlock 0.31 ± 3% +0.1 0.38 ± 5% perf-profile.children.cycles.up_write 0.16 ± 2% +0.1 0.23 ± 10% perf-profile.children.cycles.ima_file_check 0.15 ± 7% +0.1 0.22 ± 13% perf-profile.children.cycles.__fsnotify_parent 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles.update_sd_lb_stats 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles.read_tsc 0.09 ± 4% +0.1 0.16 ± 11% perf-profile.children.cycles._raw_spin_unlock_irqrestore 0.19 ± 14% +0.1 0.25 ± 7% perf-profile.children.cycles.__schedule 0.00 +0.1 0.07 ± 19% perf-profile.children.cycles.lapic_next_deadline 0.23 ± 8% +0.1 0.30 ± 8% perf-profile.children.cycles.ret_from_fork 0.11 ± 4% +0.1 0.18 ± 12% perf-profile.children.cycles.xfs_vn_setattr_size 0.42 ± 2% +0.1 0.49 ± 10% perf-profile.children.cycles.__x64_sys_close 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles.find_busiest_group 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles.xfs_can_free_eofblocks 0.23 ± 6% +0.1 0.30 ± 8% perf-profile.children.cycles.kthread 0.13 ± 14% +0.1 0.20 ± 12% perf-profile.children.cycles.avc_has_perm 0.10 ± 17% +0.1 0.17 ± 21% perf-profile.children.cycles.uart_console_write 0.09 ± 15% +0.1 0.17 ± 19% perf-profile.children.cycles.serial8250_console_putchar 0.09 ± 13% +0.1 0.16 ± 14% perf-profile.children.cycles.generic_permission 0.07 ± 21% +0.1 0.14 ± 13% perf-profile.children.cycles.xlog_grant_head_check 0.10 ± 15% +0.1 0.17 ± 19% perf-profile.children.cycles.serial8250_console_write 0.26 ± 5% +0.1 0.33 ± 7% perf-profile.children.cycles.up_read 0.15 ± 6% +0.1 0.22 ± 6% perf-profile.children.cycles.__virt_addr_valid 0.01 ±173% +0.1 0.09 ± 7% perf-profile.children.cycles.xlog_verify_grant_tail 0.33 +0.1 0.40 ± 7% perf-profile.children.cycles.do_unlinkat 0.20 ± 6% +0.1 0.28 ± 16% perf-profile.children.cycles.inode_security_rcu 0.10 ± 13% +0.1 0.17 ± 19% perf-profile.children.cycles.wait_for_xmitr 0.36 ± 4% +0.1 0.44 ± 8% perf-profile.children.cycles.lockref_get_not_dead 0.33 ± 2% +0.1 0.41 ± 8% perf-profile.children.cycles.xlog_space_left 0.26 ± 9% +0.1 0.34 ± 6% perf-profile.children.cycles.security_file_alloc 0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.irq_work_interrupt 0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.smp_irq_work_interrupt 0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.irq_work_run 0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.printk 0.09 ± 17% +0.1 0.17 ± 17% perf-profile.children.cycles.vprintk_emit 0.03 ±100% +0.1 0.11 ± 23% perf-profile.children.cycles.pick_next_task_fair 0.49 ± 2% +0.1 0.57 ± 3% perf-profile.children.cycles.down_read 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles.native_write_msr 0.61 ± 2% +0.1 0.69 ± 5% perf-profile.children.cycles.osq_unlock 0.26 ± 7% +0.1 0.35 ± 10% perf-profile.children.cycles._cond_resched 0.11 ± 17% +0.1 0.20 ± 20% perf-profile.children.cycles.irq_work_run_list 0.29 ± 11% +0.1 0.39 ± 2% perf-profile.children.cycles.wake_q_add 0.27 ± 7% +0.1 0.36 ± 7% perf-profile.children.cycles.rcu_core 0.26 ± 5% +0.1 0.35 ± 11% perf-profile.children.cycles.xfs_ilock 0.17 +0.1 0.26 ± 4% perf-profile.children.cycles.__d_lookup_rcu 0.00 +0.1 0.09 ± 19% perf-profile.children.cycles.load_balance 0.10 ± 18% +0.1 0.20 ± 19% perf-profile.children.cycles.console_unlock 0.26 ± 5% +0.1 0.35 ± 11% perf-profile.children.cycles.__alloc_fd 0.03 ±100% +0.1 0.12 ± 18% perf-profile.children.cycles.clockevents_program_event 0.00 +0.1 0.10 ± 8% perf-profile.children.cycles.menu_select 0.23 ± 10% +0.1 0.33 ± 8% perf-profile.children.cycles.selinux_inode_setattr 0.15 ± 2% +0.1 0.26 ± 6% perf-profile.children.cycles.xfs_release 0.00 +0.1 0.11 ± 16% perf-profile.children.cycles.ktime_get 0.24 ± 7% +0.1 0.35 ± 14% perf-profile.children.cycles.xfs_trans_log_inode 0.00 +0.1 0.11 ± 17% perf-profile.children.cycles.perf_mux_hrtimer_handler 0.31 ± 10% +0.1 0.42 ± 5% perf-profile.children.cycles.__check_object_size 0.33 ± 11% +0.1 0.45 ± 3% perf-profile.children.cycles.__rwsem_mark_wake 0.19 ± 9% +0.1 0.31 ± 12% perf-profile.children.cycles.memset_erms 0.23 ± 10% +0.1 0.34 ± 5% perf-profile.children.cycles.fsnotify 0.26 ± 9% +0.1 0.38 ± 8% perf-profile.children.cycles.security_inode_setattr 0.26 ± 5% +0.1 0.39 ± 9% perf-profile.children.cycles.avc_has_perm_noaudit 0.28 ± 2% +0.1 0.41 ± 13% perf-profile.children.cycles.selinux_file_open 0.29 ± 5% +0.1 0.43 ± 8% perf-profile.children.cycles.xfs_trans_free_items 0.13 ± 11% +0.1 0.27 ± 14% perf-profile.children.cycles.task_tick_fair 0.52 ± 9% +0.1 0.65 ± 3% perf-profile.children.cycles.legitimize_path 0.41 ± 6% +0.1 0.56 ± 4% perf-profile.children.cycles.strncpy_from_user 0.28 ± 3% +0.1 0.42 ± 7% perf-profile.children.cycles.mnt_want_write 0.20 ± 6% +0.1 0.34 ± 9% perf-profile.children.cycles.__sb_start_write 0.30 ± 3% +0.1 0.45 ± 13% perf-profile.children.cycles.__inode_security_revalidate 0.51 ± 7% +0.2 0.67 ± 8% perf-profile.children.cycles.xlog_ticket_alloc 0.59 ± 9% +0.2 0.75 ± 4% perf-profile.children.cycles.complete_walk 0.55 ± 10% +0.2 0.72 ± 4% perf-profile.children.cycles.unlazy_walk 0.16 ± 9% +0.2 0.34 ± 9% perf-profile.children.cycles.scheduler_tick 0.16 ± 10% +0.2 0.34 ± 15% perf-profile.children.cycles.irq_exit 0.36 ± 9% +0.2 0.54 ± 5% perf-profile.children.cycles.xfs_inode_item_format 0.35 ± 3% +0.2 0.52 ± 10% perf-profile.children.cycles.security_file_open 0.35 ± 7% +0.2 0.52 ± 7% perf-profile.children.cycles.__softirqentry_text_start 0.39 ± 4% +0.2 0.57 ± 4% perf-profile.children.cycles.__might_sleep 0.45 ± 9% +0.2 0.64 ± 8% perf-profile.children.cycles.lookup_fast 0.40 ± 2% +0.2 0.59 ± 9% perf-profile.children.cycles.inode_permission 0.74 ± 8% +0.2 0.94 ± 10% perf-profile.children.cycles.dput 0.43 ± 7% +0.2 0.63 ± 8% perf-profile.children.cycles.may_open 0.39 ± 4% +0.2 0.60 ± 12% perf-profile.children.cycles.___might_sleep 0.93 ± 2% +0.2 1.15 ± 3% perf-profile.children.cycles.rwsem_spin_on_owner 0.76 ± 2% +0.2 0.99 ± 12% perf-profile.children.cycles.xlog_ungrant_log_space 0.52 ± 8% +0.2 0.76 ± 9% perf-profile.children.cycles.walk_component 0.41 ± 7% +0.3 0.68 ± 4% perf-profile.children.cycles.kmem_cache_free 0.24 ± 11% +0.3 0.51 ± 9% perf-profile.children.cycles.update_process_times 0.25 ± 10% +0.3 0.52 ± 9% perf-profile.children.cycles.tick_sched_handle 0.53 ± 4% +0.3 0.82 ± 8% perf-profile.children.cycles.xlog_grant_add_space 0.59 ± 6% +0.3 0.88 ± 7% perf-profile.children.cycles.kmem_zone_alloc 0.70 ± 4% +0.3 1.00 ± 4% perf-profile.children.cycles.getname_flags 0.27 ± 11% +0.3 0.57 ± 8% perf-profile.children.cycles.tick_sched_timer 0.86 ± 4% +0.3 1.18 ± 5% perf-profile.children.cycles.__alloc_file 0.67 ± 3% +0.3 1.00 ± 7% perf-profile.children.cycles.syscall_return_via_sysret 0.91 ± 4% +0.3 1.25 ± 6% perf-profile.children.cycles.alloc_empty_file 0.98 ± 2% +0.3 1.31 ± 10% perf-profile.children.cycles.xfs_log_done 0.84 ± 2% +0.4 1.23 ± 3% perf-profile.children.cycles.do_dentry_open 0.83 ± 3% +0.4 1.22 ± 7% perf-profile.children.cycles.entry_SYSCALL_64 0.89 ± 2% +0.4 1.28 ± 5% perf-profile.children.cycles.selinux_inode_permission 0.09 ± 9% +0.4 0.51 ± 34% perf-profile.children.cycles.intel_idle 0.97 ± 3% +0.4 1.39 ± 4% perf-profile.children.cycles.security_inode_permission 0.36 ± 8% +0.5 0.82 ± 9% perf-profile.children.cycles.__hrtimer_run_queues 1.02 ± 2% +0.5 1.54 ± 3% perf-profile.children.cycles.__fput 1.09 +0.6 1.67 ± 6% perf-profile.children.cycles.kmem_cache_alloc 0.46 ± 7% +0.6 1.08 ± 9% perf-profile.children.cycles.hrtimer_interrupt 1.41 +0.6 2.06 ± 4% perf-profile.children.cycles.task_work_run 1.60 +0.7 2.26 ± 6% perf-profile.children.cycles.xfs_log_reserve 1.64 +0.7 2.33 ± 5% perf-profile.children.cycles.xfs_trans_reserve 1.56 +0.7 2.26 ± 4% perf-profile.children.cycles.exit_to_usermode_loop 0.25 ± 5% +0.8 1.02 ± 26% perf-profile.children.cycles.cpuidle_enter 0.25 ± 5% +0.8 1.02 ± 26% perf-profile.children.cycles.cpuidle_enter_state 1.76 ± 3% +0.8 2.54 ± 6% perf-profile.children.cycles.link_path_walk 0.39 ± 6% +0.8 1.20 ± 22% perf-profile.children.cycles.start_secondary 0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.do_idle 0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.secondary_startup_64 0.40 ± 4% +0.9 1.25 ± 22% perf-profile.children.cycles.cpu_startup_entry 0.65 ± 6% +0.9 1.51 ± 11% perf-profile.children.cycles.smp_apic_timer_interrupt 0.69 ± 6% +0.9 1.62 ± 10% perf-profile.children.cycles.apic_timer_interrupt 0.86 ± 5% +0.9 1.80 ± 7% perf-profile.children.cycles._raw_spin_lock_irqsave 2.35 +1.0 3.33 ± 6% perf-profile.children.cycles.xfs_trans_alloc 1.04 ± 4% +1.0 2.08 ± 5% perf-profile.children.cycles.try_to_wake_up 1.13 ± 4% +1.1 2.18 ± 6% perf-profile.children.cycles.wake_up_q 1.99 ± 2% +1.2 3.23 ± 4% perf-profile.children.cycles.rwsem_wake 13.78 ± 9% +3.2 16.98 ± 9% perf-profile.children.cycles.xfs_setattr_nonsize 13.95 ± 9% +3.2 17.19 ± 9% perf-profile.children.cycles.xfs_setattr_size 14.60 ± 8% +3.5 18.06 ± 8% perf-profile.children.cycles.xfs_vn_setattr 15.34 ± 8% +3.7 19.04 ± 8% perf-profile.children.cycles.notify_change 15.55 ± 8% +3.8 19.31 ± 8% perf-profile.children.cycles.do_truncate 64.82 -12.2 52.65 ± 4% perf-profile.self.cycles.osq_lock 0.06 ± 11% +0.0 0.08 ± 12% perf-profile.self.cycles.dax_layout_busy_page 0.08 ± 5% +0.0 0.10 ± 10% perf-profile.self.cycles.__call_rcu 0.20 ± 3% +0.0 0.22 ± 3% perf-profile.self.cycles.__list_del_entry_valid 0.06 ± 6% +0.0 0.08 ± 13% perf-profile.self.cycles.xfs_trans_unreserve_and_mod_sb 0.07 ± 10% +0.0 0.10 ± 9% perf-profile.self.cycles.selinux_inode_setattr 0.07 ± 6% +0.0 0.09 ± 8% perf-profile.self.cycles.percpu_counter_add_batch 0.04 ± 58% +0.0 0.07 ± 12% perf-profile.self.cycles.xfs_trans_ijoin 0.08 ± 10% +0.0 0.11 ± 4% perf-profile.self.cycles.xfs_vn_setattr 0.08 ± 5% +0.0 0.11 ± 9% perf-profile.self.cycles.__slab_free 0.07 ± 7% +0.0 0.10 ± 15% perf-profile.self.cycles.xfs_break_leased_layouts 0.09 +0.0 0.12 ± 12% perf-profile.self.cycles.wake_up_q 0.06 ± 11% +0.0 0.09 ± 14% perf-profile.self.cycles.xfs_trans_free_items 0.04 ± 57% +0.0 0.07 ± 10% perf-profile.self.cycles.xfs_trans_reserve 0.07 ± 12% +0.0 0.10 ± 18% perf-profile.self.cycles.selinux_task_getsecid 0.12 ± 3% +0.0 0.16 ± 5% perf-profile.self.cycles.xfs_setattr_size 0.10 ± 9% +0.0 0.13 ± 10% perf-profile.self.cycles.lookup_fast 0.04 ± 58% +0.0 0.08 ± 5% perf-profile.self.cycles.__check_heap_object 0.12 ± 3% +0.0 0.15 ± 11% perf-profile.self.cycles.selinux_file_open 0.09 ± 8% +0.0 0.13 ± 5% perf-profile.self.cycles.exit_to_usermode_loop 0.04 ± 58% +0.0 0.08 ± 15% perf-profile.self.cycles.xfs_inode_item_format_data_fork 0.07 ± 7% +0.0 0.11 ± 10% perf-profile.self.cycles.__xfs_trans_commit 0.11 ± 4% +0.0 0.15 ± 5% perf-profile.self.cycles.xfs_log_calc_unit_res 0.08 ± 8% +0.0 0.12 ± 6% perf-profile.self.cycles.try_to_wake_up 0.11 ± 7% +0.0 0.15 ± 5% perf-profile.self.cycles.__list_add_valid 0.12 ± 9% +0.0 0.16 ± 6% perf-profile.self.cycles.task_work_run 0.09 ± 13% +0.0 0.13 ± 11% perf-profile.self.cycles.__legitimize_mnt 0.06 ± 13% +0.0 0.11 ± 8% perf-profile.self.cycles.do_filp_open 0.11 ± 6% +0.0 0.15 ± 11% perf-profile.self.cycles.rcu_all_qs 0.43 +0.0 0.48 ± 2% perf-profile.self.cycles.down_read 0.13 ± 6% +0.0 0.17 ± 9% perf-profile.self.cycles._cond_resched 0.11 ± 7% +0.0 0.15 ± 12% perf-profile.self.cycles.xfs_trans_del_item 0.08 ± 5% +0.0 0.13 ± 12% perf-profile.self.cycles._raw_spin_unlock_irqrestore 0.01 ±173% +0.0 0.06 ± 11% perf-profile.self.cycles.__rwsem_mark_wake 0.01 ±173% +0.0 0.06 ± 20% perf-profile.self.cycles.filp_close 0.07 ± 11% +0.0 0.12 ± 22% perf-profile.self.cycles.walk_component 0.06 ± 14% +0.0 0.11 ± 24% perf-profile.self.cycles.io_serial_in 0.01 ±173% +0.1 0.06 ± 13% perf-profile.self.cycles.do_truncate 0.11 ± 7% +0.1 0.16 ± 18% perf-profile.self.cycles.__alloc_fd 0.61 ± 2% +0.1 0.66 ± 4% perf-profile.self.cycles.osq_unlock 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles.current_time 0.10 ± 7% +0.1 0.15 ± 9% perf-profile.self.cycles.xfs_log_space_wake 0.23 ± 3% +0.1 0.28 ± 5% perf-profile.self.cycles.rwsem_wake 0.11 ± 7% +0.1 0.16 ± 18% perf-profile.self.cycles.__x86_indirect_thunk_rax 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles.inode_has_perm 0.01 ±173% +0.1 0.07 ± 6% perf-profile.self.cycles.__fd_install 0.01 ±173% +0.1 0.07 ± 10% perf-profile.self.cycles.rcu_segcblist_enqueue 0.01 ±173% +0.1 0.07 ± 16% perf-profile.self.cycles.xlog_grant_head_check 0.18 ± 7% +0.1 0.23 ± 4% perf-profile.self.cycles.__mnt_want_write 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles.ktime_get 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles.xfs_log_ticket_put 0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles.xfs_file_release 0.11 ± 9% +0.1 0.17 ± 13% perf-profile.self.cycles.entry_SYSCALL_64_after_hwframe 0.00 +0.1 0.06 ± 16% perf-profile.self.cycles.delay_tsc 0.31 ± 3% +0.1 0.37 ± 6% perf-profile.self.cycles.up_write 0.15 ± 7% +0.1 0.21 ± 12% perf-profile.self.cycles.__fsnotify_parent 0.00 +0.1 0.06 ± 13% perf-profile.self.cycles.menu_select 0.00 +0.1 0.07 ± 7% perf-profile.self.cycles.security_file_open 0.00 +0.1 0.07 ± 17% perf-profile.self.cycles.setattr_prepare 0.00 +0.1 0.07 ± 23% perf-profile.self.cycles.read_tsc 0.18 ± 6% +0.1 0.25 ± 13% perf-profile.self.cycles.do_dentry_open 0.09 ± 13% +0.1 0.16 ± 15% perf-profile.self.cycles.generic_permission 0.00 +0.1 0.07 ± 14% perf-profile.self.cycles.xfs_can_free_eofblocks 0.15 ± 6% +0.1 0.22 ± 8% perf-profile.self.cycles.__virt_addr_valid 0.13 ± 14% +0.1 0.20 ± 12% perf-profile.self.cycles.avc_has_perm 0.25 ± 5% +0.1 0.33 ± 7% perf-profile.self.cycles._raw_spin_lock_irqsave 0.01 ±173% +0.1 0.09 ± 9% perf-profile.self.cycles.xlog_verify_grant_tail 0.36 ± 4% +0.1 0.44 ± 8% perf-profile.self.cycles.lockref_get_not_dead 0.26 ± 6% +0.1 0.33 ± 6% perf-profile.self.cycles.up_read 0.20 ± 4% +0.1 0.27 ± 5% perf-profile.self.cycles.__fput 0.03 ±100% +0.1 0.10 ± 4% perf-profile.self.cycles.apic_timer_interrupt 0.32 +0.1 0.40 ± 7% perf-profile.self.cycles.xlog_space_left 0.15 ± 8% +0.1 0.23 ± 14% perf-profile.self.cycles.__inode_security_revalidate 0.08 ± 15% +0.1 0.16 ± 9% perf-profile.self.cycles.__sb_start_write 0.13 ± 13% +0.1 0.21 ± 12% perf-profile.self.cycles.xfs_setattr_nonsize 0.17 +0.1 0.26 ± 5% perf-profile.self.cycles.__d_lookup_rcu 0.00 +0.1 0.09 ± 20% perf-profile.self.cycles.native_write_msr 0.29 ± 11% +0.1 0.39 ± 2% perf-profile.self.cycles.wake_q_add 0.25 ± 9% +0.1 0.36 ± 17% perf-profile.self.cycles.link_path_walk 0.16 ± 5% +0.1 0.26 ± 13% perf-profile.self.cycles.do_sys_open 0.23 ± 4% +0.1 0.34 ± 14% perf-profile.self.cycles.xfs_trans_log_inode 0.28 ± 3% +0.1 0.39 ± 8% perf-profile.self.cycles.__alloc_file 0.19 ± 9% +0.1 0.30 ± 12% perf-profile.self.cycles.memset_erms 0.31 ± 6% +0.1 0.42 ± 7% perf-profile.self.cycles.inode_permission 0.23 ± 11% +0.1 0.34 ± 5% perf-profile.self.cycles.fsnotify 0.90 ± 3% +0.1 1.02 ± 4% perf-profile.self.cycles.rwsem_down_write_failed 0.26 ± 7% +0.1 0.38 ± 11% perf-profile.self.cycles.avc_has_perm_noaudit 0.30 ± 4% +0.1 0.43 ± 5% perf-profile.self.cycles.xfs_inode_item_format 0.32 ± 9% +0.2 0.48 ± 12% perf-profile.self.cycles.do_syscall_64 0.39 ± 4% +0.2 0.56 ± 4% perf-profile.self.cycles.__might_sleep 0.62 ± 4% +0.2 0.81 ± 5% perf-profile.self.cycles._raw_spin_lock 0.42 ± 4% +0.2 0.62 ± 6% perf-profile.self.cycles.selinux_inode_permission 0.44 ± 18% +0.2 0.65 ± 8% perf-profile.self.cycles.path_openat 0.39 ± 4% +0.2 0.60 ± 11% perf-profile.self.cycles.___might_sleep 0.75 ± 2% +0.2 0.96 ± 11% perf-profile.self.cycles.xlog_ungrant_log_space 0.31 ± 8% +0.2 0.52 ± 6% perf-profile.self.cycles.kmem_cache_free 0.93 ± 2% +0.2 1.14 ± 4% perf-profile.self.cycles.rwsem_spin_on_owner 1.42 ± 4% +0.2 1.64 ± 4% perf-profile.self.cycles.xfs_log_commit_cil 0.54 ± 2% +0.3 0.82 ± 4% perf-profile.self.cycles.kmem_cache_alloc 0.52 ± 3% +0.3 0.81 ± 7% perf-profile.self.cycles.xlog_grant_add_space 0.67 ± 3% +0.3 1.00 ± 7% perf-profile.self.cycles.syscall_return_via_sysret 0.83 ± 3% +0.4 1.22 ± 7% perf-profile.self.cycles.entry_SYSCALL_64 0.08 ± 10% +0.4 0.51 ± 34% perf-profile.self.cycles.intel_idle aim7.jobs-per-min 70000 +-+-----------------------------------------------------------------+ |.+. +.+.+..+.+.+.+. + +.+.+..+ +.+..+ | 65000 +-+ | | | 60000 +-+ | | | 55000 +-+ | | | 50000 +-+ | | O O O O O O O | 45000 +-+ O O O O O O O O O O O O O O O O | 40000 +-O O O O O O O O | | | 35000 +-+-----------------------------------------------------------------+ aim7.time.user_time 130 +-+--------------------O----------------------------------------------+ | O O O O O | 120 O-+ O O O O O O O O O O 110 +-+ O O O O O O O O O O | | O O O O | 100 +-+ | | | 90 +-+ | | | 80 +-+ | 70 +-+ | | + | 60 +-+ .+. + + .+. .+. .+.. .+. .+. | |.+. +.+..+ +..+ +..+.+.+. + + +. + | 50 +-+-------------------------------------------------------------------+ aim7.time.elapsed_time 230 +-+-------------------------------------------------------------------+ 220 +-O O O O O O O O O | O O O O O O | 210 +-+ O O O O O O O 200 +-+ O O | 190 +-+ O O O O O O O | 180 +-+ | | | 170 +-+ | 160 +-+ | 150 +-+ | 140 +-+ | |.+.. .+.+..+.+.+..+.+. .+. .+.+.+..+. .+..+. | 130 +-+ + +. +.+. + + | 120 +-+-------------------------------------------------------------------+ aim7.time.elapsed_time.max 230 +-+-------------------------------------------------------------------+ 220 +-O O O O O O O O O | O O O O O O | 210 +-+ O O O O O O O 200 +-+ O O | 190 +-+ O O O O O O O | 180 +-+ | | | 170 +-+ | 160 +-+ | 150 +-+ | 140 +-+ | |.+.. .+.+..+.+.+..+.+. .+. .+.+.+..+. .+..+. | 130 +-+ + +. +.+. + + | 120 +-+-------------------------------------------------------------------+ aim7.time.voluntary_context_switches 3e+06 +-+---------------------------------------------------------------+ 2.8e+06 +-+ .+.. .+.+.. .+. | |. .+. .+ +.+.+ +.+.+.+.+..+ +.+ | 2.6e+06 +-+. .+ + | 2.4e+06 +-+ +. | 2.2e+06 +-+ | 2e+06 +-+ | | | 1.8e+06 O-+ O O O O O O O | 1.6e+06 +-O O O O O O O O O O | 1.4e+06 +-+ O O O 1.2e+06 +-+ | | O O O | 1e+06 +-+ O O O O O O O | 800000 +-+---------------------------------------------------------------+ aim7.time.involuntary_context_switches 360000 +-+----------------------------------------------------------------+ 340000 +-+ O O O O O O | O O O O O O O O O O O O | 320000 +-+ O O O O O 300000 +-+ O O O O O O O | | | 280000 +-+ O | 260000 +-+ | 240000 +-+ | | | 220000 +-+ | 200000 +-+.+..+. | | + +.+. .+. .+..+.+.+. | 180000 +-+ +..+.+ +..+.+.+ +.+..+.+ | 160000 +-+----------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase/ucode: cs-localhost/gcc-7/performance/ipv4/x86_64-rhel-7.6/200%/debian-x86_64-2019-05-14.cgz/900s/lkp-bdw-de1/TCP_MAERTS/netperf/0x7000017 commit: fe146f762b ("sched: Remove rq->cpu_load[] update code") cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()") fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:2 -50% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x :2 50% 1:2 dmesg.WARNING:at_ip__netif_receive_skb_core/0x *************************************************************************************************** lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/pipe/x86_64-rhel-7.6/process/1600%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ex1/hackbench/0xb00002e commit: fe146f762b ("sched: Remove rq->cpu_load[] update code") cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()") fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 100% 2:2 dmesg.WARNING:stack_recursion *************************************************************************************************** lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory *************************************************************************************************** lkp-bdw-ep3: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory ========================================================================================= compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode: gcc-7/performance/socket/x86_64-rhel-7.6/process/1600%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep3/hackbench/0xb00002e commit: fe146f762b ("sched: Remove rq->cpu_load[] update code") cca2ae2c00 ("sched/fair: Replace source_load() & target_load() w/ weighted_cpuload()") fe146f762b5800ff cca2ae2c00b5fc7893f1a184d82 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :2 100% 2:2 dmesg.WARNING:at_ip__slab_free/0x Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 4 months

2
1
0 / 0

[nl80211] 52f9c100b1: WARNING:at_net/wireless/core.c:#wiphy_register

by kernel test robot

FYI, we noticed the following commit (built with gcc-6): commit: 52f9c100b17e1d5d9696366d7401d14d3fd9c05c ("[PATCH] nl80211: require and validate vendor command policy") url: https://github.com/0day-ci/linux/commits/Johannes-Berg/nl80211-require-an... base: https://git.kernel.org/cgit/linux/kernel/git/jberg/mac80211.git master in testcase: trinity with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 2G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------------+------------+------------+ | | 551842446e | 52f9c100b1 | +------------------------------------------------+------------+------------+ | boot_successes | 7 | 0 | | boot_failures | 1 | 8 | | BUG:soft_lockup-CPU##stuck_for#s | 1 | | | EIP:_raw_spin_unlock_irqrestore | 1 | | | Kernel_panic-not_syncing:softlockup:hung_tasks | 1 | | | WARNING:at_net/wireless/core.c:#wiphy_register | 0 | 8 | | EIP:wiphy_register | 0 | 8 | +------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 42.418616] WARNING: CPU: 1 PID: 1 at net/wireless/core.c:867 wiphy_register+0x6e8/0x950 [ 42.427495] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.1.0-10250-g52f9c10 #1 [ 42.427495] EIP: wiphy_register+0x6e8/0x950 [ 42.427495] Code: 74 8b 86 5c 03 00 00 8b 50 14 85 d2 74 1c 83 c0 0c 31 d2 8b 38 85 ff 74 49 83 c2 01 39 ca 74 55 83 c0 1c 8b 58 08 85 db 75 e9 <0f> 0b bb ea ff ff ff e9 0c fa ff ff 8d 74 26 00 0f 0b bb ea ff ff [ 42.427495] EAX: c3377d4c EBX: c2cdcdba ECX: 00000001 EDX: 00000000 [ 42.427495] ESI: ee5f42e0 EDI: 00000003 EBP: c004be5c ESP: c004be04 [ 42.427495] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010246 [ 42.427495] CR0: 80050033 CR2: ffffffff CR3: 034fd000 CR4: 000006b0 [ 42.427495] Call Trace: [ 42.427495] ieee80211_register_hw+0x446/0xc00 [ 42.427495] mac80211_hwsim_new_radio+0x7c0/0xe20 [ 42.427495] init_mac80211_hwsim+0x15c/0x2c5 [ 42.427495] ? wl3501_driver_init+0xf/0xf [ 42.427495] do_one_initcall+0x7b/0x148 [ 42.427495] kernel_init_freeable+0x16a/0x1e5 [ 42.427495] ? rest_init+0x120/0x120 [ 42.427495] kernel_init+0x8/0xf0 [ 42.427495] ret_from_fork+0x1e/0x28 [ 42.427495] random: get_random_bytes called from print_oops_end_marker+0x4f/0x60 with crng_init=0 [ 42.427495] ---[ end trace ca67d517de3b21fc ]--- To reproduce: # build kernel cd linux cp config-5.1.0-10250-g52f9c10 .config make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 olddefconfig make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 prepare make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 modules_prepare make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 SHELL=/bin/bash make HOSTCC=gcc-6 CC=gcc-6 ARCH=i386 bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

2 years, 4 months

1
0
0 / 0

[btrfs] 2996e1f8bc: aim7.jobs-per-min -13.2% regression

by kernel test robot

Greeting, FYI, we noticed a -13.2% regression of aim7.jobs-per-min due to commit: commit: 2996e1f8bcadf0436cc67b63af01523f6cf5d43f ("btrfs: factor our read/write stage off csum_tree_block into its callers") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: aim7 on test machine: 72 threads Intel(R) Xeon(R) Gold 6139 CPU @ 2.30GHz with 128G memory with following parameters: disk: 4BRD_12G md: RAID0 fs: btrfs test: sync_disk_rw load: 20 ucode: 0x200005e cpufreq_governor: performance test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ In addition to that, the commit also has significant impact on the following tests: +------------------+-----------------------------------------------------------------------+ | testcase: change | aim7: aim7.jobs-per-min -19.1% regression | | test machine | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory | | test parameters | cpufreq_governor=performance | | | disk=4BRD_12G | | | fs=btrfs | | | load=20 | | | md=RAID0 | | | test=sync_disk_rw | +------------------+-----------------------------------------------------------------------+ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode: gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2019-05-14.cgz/lkp-skl-2sp7/sync_disk_rw/aim7/0x200005e commit: v5.1-rc7 2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers") v5.1-rc7 2996e1f8bcadf0436cc67b63af0 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 kmsg.Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#e) %stddev %change %stddev \ | \ 1207 -13.2% 1047 aim7.jobs-per-min 99.41 +15.2% 114.56 aim7.time.elapsed_time 99.41 +15.2% 114.56 aim7.time.elapsed_time.max 884.50 ± 2% +13.5% 1004 ± 3% aim7.time.involuntary_context_switches 704.41 +18.2% 832.33 aim7.time.system_time 21484045 -10.5% 19231461 aim7.time.voluntary_context_switches 0.02 ± 38% -0.0 0.00 ± 76% mpstat.cpu.all.soft% 59.02 ± 6% -58.9% 24.26 boot-time.boot 3928 ± 8% -62.1% 1489 boot-time.idle 19798 -12.9% 17235 iostat.md0.w/s 448478 -13.1% 389592 iostat.md0.wkB/s 130905 ± 6% -20.8% 103683 ± 2% meminfo.AnonHugePages 100591 ± 2% -12.9% 87596 meminfo.max_used_kB 5058 ± 4% +17.1% 5925 ± 9% numa-meminfo.node1.KernelStack 1737 ± 33% +66.4% 2889 ± 34% numa-meminfo.node1.PageTables 1174778 ± 23% +37.5% 1615284 ± 2% numa-numastat.node1.local_node 1189955 ± 23% +36.7% 1626977 ± 2% numa-numastat.node1.numa_hit 344576 -12.9% 300018 vmstat.io.bo 558315 -21.3% 439168 vmstat.system.cs 194.50 ± 16% -29.7% 136.75 ± 15% numa-vmstat.node1.nr_dirty 5061 ± 4% +17.1% 5925 ± 9% numa-vmstat.node1.nr_kernel_stack 433.75 ± 33% +66.4% 721.75 ± 34% numa-vmstat.node1.nr_page_table_pages 1065659 ± 23% +41.9% 1511723 ± 8% numa-vmstat.node1.numa_hit 897602 ± 28% +50.0% 1346715 ± 10% numa-vmstat.node1.numa_local 3240 ± 5% -23.0% 2496 slabinfo.skbuff_head_cache.active_objs 3264 ± 4% -19.6% 2624 slabinfo.skbuff_head_cache.num_objs 1339 ± 2% -24.6% 1010 slabinfo.task_struct.active_objs 336.75 ± 2% +202.4% 1018 slabinfo.task_struct.active_slabs 1348 ± 2% -24.5% 1018 slabinfo.task_struct.num_objs 336.75 ± 2% +202.4% 1018 slabinfo.task_struct.num_slabs 23341140 ± 9% -23.6% 17828343 cpuidle.C1.usage 1.777e+09 ± 28% +215.6% 5.608e+09 ± 8% cpuidle.C1E.time 12674253 ± 20% +66.3% 21080014 ± 3% cpuidle.C1E.usage 3.182e+09 ± 12% -87.4% 4.02e+08 ±107% cpuidle.C6.time 3770499 ± 3% -83.8% 609866 ±127% cpuidle.C6.usage 6880882 ± 13% -46.1% 3706529 cpuidle.POLL.time 1441739 ± 10% -40.2% 862558 cpuidle.POLL.usage 23339885 ± 9% -23.6% 17822261 turbostat.C1 18.49 ± 15% -3.6 14.86 turbostat.C1% 12673355 ± 20% +66.3% 21078440 ± 3% turbostat.C1E 24.32 ± 28% +42.5 66.84 ± 7% turbostat.C1E% 3765573 ± 3% -84.0% 603105 ±128% turbostat.C6 43.48 ± 11% -38.7 4.77 ±108% turbostat.C6% 3.73 ± 42% -90.1% 0.37 ± 12% turbostat.CPU%c6 14813103 ± 2% +16.9% 17318955 turbostat.IRQ 1.59 ± 51% -100.0% 0.00 turbostat.Pkg%pc2 40.05 +1.7% 40.72 turbostat.RAMWatt 59758 -1.1% 59093 proc-vmstat.nr_anon_pages 48741 +1.0% 49245 proc-vmstat.nr_free_cma 4606 -4.6% 4393 proc-vmstat.nr_inactive_anon 6367 -2.2% 6227 proc-vmstat.nr_mapped 34775 +2.1% 35519 proc-vmstat.nr_slab_unreclaimable 4606 -4.6% 4393 proc-vmstat.nr_zone_inactive_anon 428.50 ± 91% +310.7% 1759 ± 63% proc-vmstat.numa_pages_migrated 578.25 ± 37% +2906.7% 17386 ± 85% proc-vmstat.numa_pte_updates 253947 +13.1% 287290 proc-vmstat.pgfault 717071 +5.7% 758253 proc-vmstat.pgfree 428.50 ± 91% +310.7% 1759 ± 63% proc-vmstat.pgmigrate_success 294.76 ± 12% -18.2% 241.15 ± 8% sched_debug.cfs_rq:/.load_avg.stddev 18953 ± 23% -39.3% 11499 ± 22% sched_debug.cfs_rq:/.min_vruntime.avg 12508 ± 15% -50.6% 6174 ± 36% sched_debug.cfs_rq:/.min_vruntime.min 3003 ± 15% +22.6% 3681 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 250.34 ± 11% -19.7% 200.99 ± 12% sched_debug.cfs_rq:/.removed.load_avg.stddev 3092 ± 23% -37.1% 1945 ± 24% sched_debug.cfs_rq:/.removed.runnable_sum.avg 11538 ± 11% -19.9% 9247 ± 12% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 98.26 ± 5% -19.7% 78.89 ± 24% sched_debug.cfs_rq:/.removed.util_avg.stddev 1.86 ± 15% +26.4% 2.36 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.avg 4.90 ± 4% +9.1% 5.34 ± 4% sched_debug.cfs_rq:/.runnable_load_avg.stddev 2166 ± 16% +24.6% 2699 ± 13% sched_debug.cfs_rq:/.runnable_weight.avg 18058 ± 2% +15.6% 20869 ± 5% sched_debug.cfs_rq:/.runnable_weight.max 5486 ± 6% +9.5% 6009 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev 2997 ± 15% +23.0% 3687 ± 7% sched_debug.cfs_rq:/.spread0.stddev 256.16 ± 10% +16.0% 297.04 ± 5% sched_debug.cfs_rq:/.util_avg.stddev 65601 ± 8% -53.8% 30291 sched_debug.cpu.clock.avg 65604 ± 8% -53.8% 30295 sched_debug.cpu.clock.max 65598 ± 8% -53.8% 30285 sched_debug.cpu.clock.min 1.71 ± 3% +21.8% 2.09 ± 15% sched_debug.cpu.clock.stddev 65601 ± 8% -53.8% 30291 sched_debug.cpu.clock_task.avg 65604 ± 8% -53.8% 30295 sched_debug.cpu.clock_task.max 65598 ± 8% -53.8% 30285 sched_debug.cpu.clock_task.min 1.71 ± 3% +21.8% 2.09 ± 15% sched_debug.cpu.clock_task.stddev 8296 ± 5% -12.4% 7266 sched_debug.cpu.nr_load_updates.avg 6306 ± 9% -21.8% 4931 ± 9% sched_debug.cpu.nr_load_updates.min 339.25 ± 12% -29.8% 238.25 ± 8% sched_debug.cpu.nr_switches.min 1141 ± 7% +8.3% 1236 ± 5% sched_debug.cpu.nr_switches.stddev 4.91 ± 20% -30.4% 3.42 ± 14% sched_debug.cpu.nr_uninterruptible.stddev 65598 ± 8% -53.8% 30288 sched_debug.cpu_clk 62902 ± 8% -56.1% 27590 sched_debug.ktime 66171 ± 8% -53.4% 30858 sched_debug.sched_clk 23.30 ± 14% -36.4% 14.81 perf-stat.i.MPKI 1.299e+09 +29.9% 1.687e+09 perf-stat.i.branch-instructions 2.54 ± 9% -0.9 1.64 ± 2% perf-stat.i.branch-miss-rate% 24240691 ± 2% +8.8% 26384355 ± 2% perf-stat.i.branch-misses 32469998 +4.7% 34002480 perf-stat.i.cache-misses 1.184e+08 ± 2% +4.3% 1.236e+08 perf-stat.i.cache-references 574477 -21.6% 450402 perf-stat.i.context-switches 4.71 ± 2% -21.4% 3.70 perf-stat.i.cpi 3.026e+10 -1.3% 2.987e+10 perf-stat.i.cpu-cycles 14.78 ± 12% -13.5% 12.78 ± 6% perf-stat.i.cpu-migrations 1091 ± 3% +21.9% 1331 ± 8% perf-stat.i.cycles-between-cache-misses 0.10 ± 28% -0.1 0.01 ± 21% perf-stat.i.dTLB-load-miss-rate% 265935 ± 9% -55.1% 119534 ± 5% perf-stat.i.dTLB-load-misses 1.798e+09 +12.4% 2.021e+09 perf-stat.i.dTLB-loads 0.02 ± 23% -0.0 0.00 ± 13% perf-stat.i.dTLB-store-miss-rate% 33528 ± 5% -50.3% 16673 ± 2% perf-stat.i.dTLB-store-misses 5.94 ± 24% -4.5 1.47 ± 21% perf-stat.i.iTLB-load-miss-rate% 8419599 -13.3% 7296814 perf-stat.i.iTLB-loads 6.595e+09 +20.3% 7.936e+09 perf-stat.i.instructions 0.22 ± 3% +25.4% 0.28 perf-stat.i.ipc 2351 -0.9% 2329 perf-stat.i.minor-faults 10288903 ± 2% -5.6% 9715506 perf-stat.i.node-load-misses 765100 +11.5% 852708 ± 3% perf-stat.i.node-loads 3887667 -10.6% 3474607 perf-stat.i.node-store-misses 754448 ± 5% -15.2% 639938 ± 2% perf-stat.i.node-stores 2351 -0.9% 2329 perf-stat.i.page-faults 17.96 ± 3% -13.3% 15.57 perf-stat.overall.MPKI 1.87 -0.3 1.56 ± 2% perf-stat.overall.branch-miss-rate% 4.59 -18.0% 3.76 perf-stat.overall.cpi 932.33 -5.7% 878.73 perf-stat.overall.cycles-between-cache-misses 0.01 ± 10% -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 6% -0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate% 67735 ± 6% +83.2% 124066 ± 24% perf-stat.overall.instructions-per-iTLB-miss 0.22 +21.9% 0.27 perf-stat.overall.ipc 93.08 -1.1 91.93 perf-stat.overall.node-load-miss-rate% 1.286e+09 +30.0% 1.672e+09 perf-stat.ps.branch-instructions 23999328 ± 2% +9.0% 26157341 ± 2% perf-stat.ps.branch-misses 32137766 +4.9% 33699623 perf-stat.ps.cache-misses 1.172e+08 ± 2% +4.5% 1.225e+08 perf-stat.ps.cache-references 568582 -21.5% 446385 perf-stat.ps.context-switches 2.996e+10 -1.2% 2.96e+10 perf-stat.ps.cpu-cycles 263523 ± 9% -55.0% 118523 ± 5% perf-stat.ps.dTLB-load-misses 1.78e+09 +12.5% 2.003e+09 perf-stat.ps.dTLB-loads 33223 ± 5% -50.2% 16536 ± 2% perf-stat.ps.dTLB-store-misses 8333338 -13.2% 7231920 perf-stat.ps.iTLB-loads 6.528e+09 +20.5% 7.866e+09 perf-stat.ps.instructions 2328 -0.8% 2310 perf-stat.ps.minor-faults 10183467 ± 2% -5.4% 9628920 perf-stat.ps.node-load-misses 757265 +11.6% 845120 ± 3% perf-stat.ps.node-loads 3847829 -10.5% 3443639 perf-stat.ps.node-store-misses 746711 ± 5% -15.1% 634243 ± 2% perf-stat.ps.node-stores 2328 -0.8% 2310 perf-stat.ps.page-faults 6.55e+11 +38.9% 9.097e+11 perf-stat.total.instructions 26715 ± 6% +15.3% 30804 ± 3% softirqs.CPU0.RCU 25813 ± 3% +17.5% 30331 ± 2% softirqs.CPU1.RCU 26114 ± 2% +13.8% 29715 ± 4% softirqs.CPU10.RCU 16814 ± 6% +17.4% 19743 ± 5% softirqs.CPU10.SCHED 38584 +23.0% 47474 ± 19% softirqs.CPU10.TIMER 25226 ± 6% +24.3% 31347 ± 2% softirqs.CPU11.RCU 15728 ± 4% +20.0% 18878 ± 2% softirqs.CPU11.SCHED 37200 ± 2% +27.9% 47567 ± 21% softirqs.CPU11.TIMER 26744 ± 8% +17.8% 31517 ± 4% softirqs.CPU12.RCU 25992 ± 2% +26.8% 32949 ± 4% softirqs.CPU13.RCU 16769 ± 4% +13.3% 18999 ± 2% softirqs.CPU13.SCHED 25183 +23.4% 31086 ± 6% softirqs.CPU14.RCU 37036 ± 2% +24.4% 46059 ± 23% softirqs.CPU14.TIMER 24448 ± 2% +21.7% 29750 ± 2% softirqs.CPU15.RCU 26602 ± 6% +12.5% 29919 softirqs.CPU16.RCU 37821 ± 3% +9.5% 41426 ± 2% softirqs.CPU16.TIMER 28288 ± 2% +19.4% 33785 ± 4% softirqs.CPU18.RCU 38075 ± 2% +23.4% 46999 ± 21% softirqs.CPU19.TIMER 26162 ± 2% +20.0% 31395 ± 2% softirqs.CPU2.RCU 16553 ± 4% +15.5% 19112 ± 5% softirqs.CPU2.SCHED 37904 ± 2% +18.5% 44898 ± 6% softirqs.CPU2.TIMER 27787 ± 6% +15.0% 31953 ± 6% softirqs.CPU20.RCU 16907 ± 6% +8.9% 18409 ± 5% softirqs.CPU20.SCHED 38188 ± 2% +11.0% 42371 ± 3% softirqs.CPU20.TIMER 26947 +14.9% 30957 ± 6% softirqs.CPU21.RCU 37668 +10.8% 41744 ± 2% softirqs.CPU21.TIMER 27984 ± 6% +17.3% 32821 ± 5% softirqs.CPU22.RCU 37415 ± 3% +10.9% 41475 softirqs.CPU22.TIMER 26636 ± 9% +18.8% 31634 ± 6% softirqs.CPU23.RCU 26525 ± 2% +22.9% 32606 ± 8% softirqs.CPU24.RCU 37040 ± 3% +13.0% 41870 ± 2% softirqs.CPU24.TIMER 26441 +22.3% 32342 softirqs.CPU25.RCU 37168 ± 4% +12.8% 41929 ± 3% softirqs.CPU25.TIMER 26599 ± 5% +21.0% 32182 ± 4% softirqs.CPU26.RCU 26510 ± 4% +20.6% 31982 ± 2% softirqs.CPU27.RCU 16559 ± 4% +13.4% 18775 ± 2% softirqs.CPU27.SCHED 37518 +10.7% 41515 ± 2% softirqs.CPU28.TIMER 25728 ± 10% +27.6% 32830 ± 8% softirqs.CPU29.RCU 36817 ± 2% +11.6% 41102 softirqs.CPU29.TIMER 26141 ± 2% +13.2% 29590 ± 2% softirqs.CPU3.RCU 17273 ± 4% +20.2% 20755 softirqs.CPU3.SCHED 26476 ± 3% +18.2% 31287 ± 3% softirqs.CPU30.RCU 15921 ± 4% +19.6% 19042 ± 10% softirqs.CPU30.SCHED 37032 +11.8% 41391 ± 2% softirqs.CPU30.TIMER 27534 ± 6% +14.7% 31581 ± 4% softirqs.CPU31.RCU 36721 +13.4% 41633 ± 2% softirqs.CPU31.TIMER 25118 ± 3% +23.7% 31072 ± 3% softirqs.CPU32.RCU 15548 ± 2% +16.9% 18182 ± 6% softirqs.CPU33.SCHED 36529 ± 3% +12.9% 41256 softirqs.CPU33.TIMER 26802 ± 7% +15.8% 31044 ± 3% softirqs.CPU34.RCU 16424 ± 3% +12.1% 18409 ± 6% softirqs.CPU34.SCHED 27024 ± 6% +20.6% 32603 ± 2% softirqs.CPU35.RCU 37345 +11.2% 41546 softirqs.CPU35.TIMER 28719 ± 4% +12.8% 32400 ± 2% softirqs.CPU36.RCU 36406 ± 2% +9.0% 39681 softirqs.CPU36.TIMER 30989 ± 7% +23.5% 38266 softirqs.CPU38.RCU 15275 ± 3% +8.6% 16592 softirqs.CPU38.SCHED 28633 +18.3% 33883 softirqs.CPU39.RCU 26313 +17.6% 30942 softirqs.CPU4.RCU 37319 ± 3% +14.1% 42577 softirqs.CPU4.TIMER 28360 ± 3% +18.3% 33549 softirqs.CPU40.RCU 26803 ± 3% +22.6% 32857 ± 3% softirqs.CPU41.RCU 27921 ± 2% +21.5% 33914 ± 5% softirqs.CPU42.RCU 27840 ± 2% +23.2% 34308 softirqs.CPU43.RCU 36332 +9.6% 39810 ± 3% softirqs.CPU43.TIMER 27988 ± 2% +23.6% 34585 softirqs.CPU44.RCU 26964 ± 4% +23.0% 33161 ± 2% softirqs.CPU45.RCU 26647 ± 4% +19.6% 31866 ± 5% softirqs.CPU46.RCU 26733 ± 6% +27.6% 34120 ± 2% softirqs.CPU48.RCU 26780 ± 5% +26.4% 33860 ± 2% softirqs.CPU49.RCU 24665 ± 8% +24.2% 30646 softirqs.CPU5.RCU 15335 ± 4% +20.8% 18532 ± 5% softirqs.CPU5.SCHED 36270 ± 2% +12.6% 40855 ± 2% softirqs.CPU5.TIMER 26971 ± 2% +14.9% 30992 ± 6% softirqs.CPU50.RCU 27424 ± 2% +21.4% 33300 ± 2% softirqs.CPU51.RCU 15504 ± 4% +9.0% 16902 ± 2% softirqs.CPU51.SCHED 27430 +19.0% 32642 ± 6% softirqs.CPU52.RCU 27558 +19.3% 32887 ± 2% softirqs.CPU53.RCU 24817 ± 6% +24.4% 30881 ± 2% softirqs.CPU54.RCU 26529 ± 2% +13.3% 30048 ± 4% softirqs.CPU55.RCU 15454 ± 4% +16.1% 17936 ± 5% softirqs.CPU55.SCHED 25606 ± 8% +25.1% 32040 softirqs.CPU56.RCU 26139 ± 4% +24.1% 32437 ± 8% softirqs.CPU57.RCU 14558 ± 3% +19.2% 17360 ± 3% softirqs.CPU57.SCHED 26083 +21.6% 31717 ± 3% softirqs.CPU58.RCU 15101 ± 3% +13.3% 17105 ± 4% softirqs.CPU58.SCHED 14837 ± 6% +11.9% 16598 ± 6% softirqs.CPU59.SCHED 35970 ± 3% +12.0% 40295 softirqs.CPU59.TIMER 26271 +16.8% 30681 ± 2% softirqs.CPU6.RCU 15985 ± 6% +18.3% 18914 ± 8% softirqs.CPU6.SCHED 37813 ± 2% +10.9% 41946 softirqs.CPU6.TIMER 24423 ± 3% +24.7% 30453 ± 6% softirqs.CPU60.RCU 14485 ± 2% +20.9% 17519 ± 4% softirqs.CPU60.SCHED 36185 ± 2% +11.9% 40494 softirqs.CPU60.TIMER 23959 ± 2% +24.8% 29893 softirqs.CPU61.RCU 14903 ± 6% +12.5% 16767 ± 3% softirqs.CPU61.SCHED 24809 ± 2% +18.0% 29264 softirqs.CPU62.RCU 14321 ± 2% +18.1% 16916 ± 2% softirqs.CPU62.SCHED 36576 +9.0% 39877 softirqs.CPU62.TIMER 23487 ± 3% +25.9% 29568 ± 2% softirqs.CPU63.RCU 13582 ± 2% +25.3% 17014 ± 3% softirqs.CPU63.SCHED 35911 +12.2% 40280 softirqs.CPU63.TIMER 25668 ± 4% +18.4% 30390 ± 3% softirqs.CPU64.RCU 14666 ± 4% +13.6% 16660 ± 5% softirqs.CPU64.SCHED 24078 +29.4% 31156 ± 5% softirqs.CPU65.RCU 14177 ± 8% +20.5% 17090 ± 2% softirqs.CPU65.SCHED 35962 ± 3% +13.1% 40661 softirqs.CPU65.TIMER 23043 ± 7% +31.2% 30227 ± 8% softirqs.CPU66.RCU 36661 +10.5% 40517 ± 2% softirqs.CPU66.TIMER 25069 ± 4% +14.8% 28791 ± 2% softirqs.CPU67.RCU 13930 ± 3% +24.3% 17313 ± 6% softirqs.CPU67.SCHED 35788 +13.1% 40494 softirqs.CPU67.TIMER 23620 ± 3% +20.5% 28463 softirqs.CPU68.RCU 13693 ± 4% +26.3% 17294 ± 5% softirqs.CPU68.SCHED 36756 +11.4% 40930 softirqs.CPU68.TIMER 24380 ± 2% +19.4% 29104 ± 6% softirqs.CPU69.RCU 14512 ± 7% +23.9% 17978 ± 8% softirqs.CPU69.SCHED 25915 +18.1% 30605 softirqs.CPU7.RCU 15855 ± 2% +16.6% 18490 softirqs.CPU7.SCHED 36892 ± 2% +12.5% 41512 ± 2% softirqs.CPU7.TIMER 24291 +20.2% 29207 softirqs.CPU70.RCU 14647 ± 7% +20.2% 17609 softirqs.CPU70.SCHED 36029 ± 2% +12.8% 40633 softirqs.CPU70.TIMER 24761 ± 6% +19.6% 29619 softirqs.CPU71.RCU 36389 +10.2% 40114 softirqs.CPU71.TIMER 26860 ± 2% +13.7% 30535 ± 4% softirqs.CPU8.RCU 16226 ± 2% +20.1% 19492 ± 4% softirqs.CPU8.SCHED 37997 +11.7% 42458 ± 2% softirqs.CPU8.TIMER 25499 ± 2% +17.6% 29981 softirqs.CPU9.RCU 17382 ± 8% +11.0% 19298 ± 6% softirqs.CPU9.SCHED 36979 +9.1% 40348 softirqs.CPU9.TIMER 1901918 +19.3% 2268164 softirqs.RCU 1148813 +10.9% 1273763 softirqs.SCHED 2693395 +10.3% 2970253 softirqs.TIMER 93123 ± 2% +20.6% 112353 interrupts.CAL:Function_call_interrupts 1307 +21.0% 1581 interrupts.CPU0.CAL:Function_call_interrupts 196590 ± 3% +18.0% 231965 interrupts.CPU0.LOC:Local_timer_interrupts 1982 ± 18% -47.7% 1037 ± 54% interrupts.CPU0.NMI:Non-maskable_interrupts 1982 ± 18% -47.7% 1037 ± 54% interrupts.CPU0.PMI:Performance_monitoring_interrupts 1311 ± 2% +20.8% 1584 interrupts.CPU1.CAL:Function_call_interrupts 196507 ± 3% +17.9% 231706 interrupts.CPU1.LOC:Local_timer_interrupts 1315 ± 2% +20.3% 1581 interrupts.CPU10.CAL:Function_call_interrupts 196466 ± 2% +17.7% 231175 interrupts.CPU10.LOC:Local_timer_interrupts 1278 ± 6% +25.0% 1598 ± 2% interrupts.CPU11.CAL:Function_call_interrupts 196467 ± 2% +18.1% 232064 interrupts.CPU11.LOC:Local_timer_interrupts 2906 ± 34% +75.3% 5096 ± 7% interrupts.CPU11.RES:Rescheduling_interrupts 1312 ± 2% +16.6% 1530 ± 4% interrupts.CPU12.CAL:Function_call_interrupts 196661 ± 3% +17.9% 231954 interrupts.CPU12.LOC:Local_timer_interrupts 196407 ± 3% +17.7% 231133 interrupts.CPU13.LOC:Local_timer_interrupts 3966 ± 15% +32.6% 5258 ± 7% interrupts.CPU13.RES:Rescheduling_interrupts 196446 ± 3% +18.0% 231766 interrupts.CPU14.LOC:Local_timer_interrupts 196307 ± 3% +18.0% 231649 interrupts.CPU15.LOC:Local_timer_interrupts 1311 ± 2% +18.0% 1547 ± 3% interrupts.CPU16.CAL:Function_call_interrupts 196278 ± 3% +17.7% 230984 interrupts.CPU16.LOC:Local_timer_interrupts 2535 ± 36% -78.5% 545.25 ± 51% interrupts.CPU16.NMI:Non-maskable_interrupts 2535 ± 36% -78.5% 545.25 ± 51% interrupts.CPU16.PMI:Performance_monitoring_interrupts 1220 ± 11% +23.2% 1502 ± 6% interrupts.CPU17.CAL:Function_call_interrupts 196493 ± 3% +17.5% 230974 interrupts.CPU17.LOC:Local_timer_interrupts 1306 +21.1% 1582 interrupts.CPU18.CAL:Function_call_interrupts 196318 ± 2% +18.0% 231690 interrupts.CPU18.LOC:Local_timer_interrupts 1329 +18.7% 1578 interrupts.CPU19.CAL:Function_call_interrupts 197092 ± 2% +17.7% 231886 interrupts.CPU19.LOC:Local_timer_interrupts 1310 ± 2% +20.4% 1577 interrupts.CPU2.CAL:Function_call_interrupts 196499 ± 3% +17.8% 231564 interrupts.CPU2.LOC:Local_timer_interrupts 1328 ± 4% +20.5% 1601 ± 3% interrupts.CPU20.CAL:Function_call_interrupts 197064 ± 2% +17.6% 231838 interrupts.CPU20.LOC:Local_timer_interrupts 1311 ± 2% +20.4% 1578 interrupts.CPU21.CAL:Function_call_interrupts 196901 ± 2% +17.7% 231704 interrupts.CPU21.LOC:Local_timer_interrupts 1313 ± 2% +20.6% 1583 interrupts.CPU22.CAL:Function_call_interrupts 196919 ± 2% +17.5% 231471 interrupts.CPU22.LOC:Local_timer_interrupts 1309 ± 2% +17.7% 1540 ± 5% interrupts.CPU23.CAL:Function_call_interrupts 196812 ± 2% +17.7% 231745 interrupts.CPU23.LOC:Local_timer_interrupts 1311 ± 2% +20.6% 1582 interrupts.CPU24.CAL:Function_call_interrupts 196729 ± 2% +17.9% 231969 interrupts.CPU24.LOC:Local_timer_interrupts 1311 ± 2% +20.4% 1579 interrupts.CPU25.CAL:Function_call_interrupts 196696 ± 2% +17.8% 231715 interrupts.CPU25.LOC:Local_timer_interrupts 1152 ± 74% +184.9% 3281 ± 11% interrupts.CPU25.NMI:Non-maskable_interrupts 1152 ± 74% +184.9% 3281 ± 11% interrupts.CPU25.PMI:Performance_monitoring_interrupts 1311 ± 2% +20.6% 1581 interrupts.CPU26.CAL:Function_call_interrupts 197080 ± 2% +17.6% 231696 interrupts.CPU26.LOC:Local_timer_interrupts 1909 ± 42% +67.9% 3207 ± 10% interrupts.CPU26.NMI:Non-maskable_interrupts 1909 ± 42% +67.9% 3207 ± 10% interrupts.CPU26.PMI:Performance_monitoring_interrupts 1310 ± 2% +20.6% 1580 interrupts.CPU27.CAL:Function_call_interrupts 196958 ± 2% +17.6% 231702 interrupts.CPU27.LOC:Local_timer_interrupts 1309 ± 2% +20.7% 1580 interrupts.CPU28.CAL:Function_call_interrupts 196575 ± 2% +17.9% 231714 interrupts.CPU28.LOC:Local_timer_interrupts 1308 ± 2% +20.9% 1582 interrupts.CPU29.CAL:Function_call_interrupts 196943 ± 2% +17.3% 230984 interrupts.CPU29.LOC:Local_timer_interrupts 1310 ± 2% +21.3% 1588 interrupts.CPU3.CAL:Function_call_interrupts 196420 ± 2% +16.9% 229682 interrupts.CPU3.LOC:Local_timer_interrupts 2937 ± 21% -39.9% 1766 ± 23% interrupts.CPU3.NMI:Non-maskable_interrupts 2937 ± 21% -39.9% 1766 ± 23% interrupts.CPU3.PMI:Performance_monitoring_interrupts 4820 ± 15% +88.0% 9063 ± 8% interrupts.CPU3.RES:Rescheduling_interrupts 1310 ± 2% +20.8% 1582 interrupts.CPU30.CAL:Function_call_interrupts 196831 ± 2% +17.7% 231691 interrupts.CPU30.LOC:Local_timer_interrupts 1209 ±113% +149.6% 3018 ± 34% interrupts.CPU30.NMI:Non-maskable_interrupts 1209 ±113% +149.6% 3018 ± 34% interrupts.CPU30.PMI:Performance_monitoring_interrupts 1304 ± 2% +23.0% 1604 interrupts.CPU31.CAL:Function_call_interrupts 196479 ± 2% +18.0% 231820 interrupts.CPU31.LOC:Local_timer_interrupts 1419 ± 73% +81.5% 2575 ± 26% interrupts.CPU31.NMI:Non-maskable_interrupts 1419 ± 73% +81.5% 2575 ± 26% interrupts.CPU31.PMI:Performance_monitoring_interrupts 1309 ± 2% +20.9% 1583 interrupts.CPU32.CAL:Function_call_interrupts 196861 ± 2% +17.7% 231732 interrupts.CPU32.LOC:Local_timer_interrupts 1159 ± 23% +36.6% 1583 interrupts.CPU33.CAL:Function_call_interrupts 196346 ± 2% +18.0% 231694 interrupts.CPU33.LOC:Local_timer_interrupts 1310 ± 2% +20.8% 1583 interrupts.CPU34.CAL:Function_call_interrupts 195339 +18.6% 231743 interrupts.CPU34.LOC:Local_timer_interrupts 1284 ± 2% +21.7% 1563 interrupts.CPU35.CAL:Function_call_interrupts 195354 +18.6% 231781 interrupts.CPU35.LOC:Local_timer_interrupts 1064 ± 25% +45.6% 1549 ± 4% interrupts.CPU36.CAL:Function_call_interrupts 196631 ± 2% +17.6% 231216 interrupts.CPU36.LOC:Local_timer_interrupts 2113 ± 33% -64.2% 756.50 ± 29% interrupts.CPU36.NMI:Non-maskable_interrupts 2113 ± 33% -64.2% 756.50 ± 29% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1203 ± 9% +29.8% 1561 ± 2% interrupts.CPU37.CAL:Function_call_interrupts 196745 ± 3% +17.6% 231316 interrupts.CPU37.LOC:Local_timer_interrupts 1302 ± 2% +21.4% 1580 interrupts.CPU38.CAL:Function_call_interrupts 196339 ± 3% +17.6% 230859 interrupts.CPU38.LOC:Local_timer_interrupts 1290 ± 3% +22.4% 1580 interrupts.CPU39.CAL:Function_call_interrupts 196222 ± 3% +16.9% 229415 interrupts.CPU39.LOC:Local_timer_interrupts 1302 +19.1% 1551 ± 4% interrupts.CPU4.CAL:Function_call_interrupts 196114 ± 3% +18.2% 231711 interrupts.CPU4.LOC:Local_timer_interrupts 1282 ± 2% +23.2% 1579 interrupts.CPU40.CAL:Function_call_interrupts 196347 ± 3% +17.2% 230205 interrupts.CPU40.LOC:Local_timer_interrupts 1295 +21.9% 1579 interrupts.CPU41.CAL:Function_call_interrupts 196386 ± 2% +17.6% 230956 interrupts.CPU41.LOC:Local_timer_interrupts 2528 ± 16% -60.4% 1001 ± 26% interrupts.CPU41.NMI:Non-maskable_interrupts 2528 ± 16% -60.4% 1001 ± 26% interrupts.CPU41.PMI:Performance_monitoring_interrupts 5041 ± 17% -55.2% 2256 ± 35% interrupts.CPU41.RES:Rescheduling_interrupts 196437 ± 2% +17.7% 231202 interrupts.CPU42.LOC:Local_timer_interrupts 2533 ± 30% -65.8% 867.50 ± 67% interrupts.CPU42.NMI:Non-maskable_interrupts 2533 ± 30% -65.8% 867.50 ± 67% interrupts.CPU42.PMI:Performance_monitoring_interrupts 1287 ± 4% +22.8% 1580 interrupts.CPU43.CAL:Function_call_interrupts 196441 ± 2% +17.5% 230898 interrupts.CPU43.LOC:Local_timer_interrupts 1184 ± 15% +33.3% 1578 interrupts.CPU44.CAL:Function_call_interrupts 196438 ± 2% +18.0% 231740 interrupts.CPU44.LOC:Local_timer_interrupts 2587 ± 38% -63.2% 953.00 ± 57% interrupts.CPU44.NMI:Non-maskable_interrupts 2587 ± 38% -63.2% 953.00 ± 57% interrupts.CPU44.PMI:Performance_monitoring_interrupts 1307 ± 2% +20.7% 1579 interrupts.CPU45.CAL:Function_call_interrupts 196461 ± 2% +17.2% 230176 interrupts.CPU45.LOC:Local_timer_interrupts 1222 ± 10% +29.1% 1578 interrupts.CPU46.CAL:Function_call_interrupts 196498 ± 2% +17.5% 230967 interrupts.CPU46.LOC:Local_timer_interrupts 2619 ± 33% -63.7% 951.50 ±106% interrupts.CPU46.NMI:Non-maskable_interrupts 2619 ± 33% -63.7% 951.50 ±106% interrupts.CPU46.PMI:Performance_monitoring_interrupts 1305 ± 2% +21.0% 1580 interrupts.CPU47.CAL:Function_call_interrupts 196734 ± 3% +17.4% 230968 interrupts.CPU47.LOC:Local_timer_interrupts 1325 ± 2% +19.0% 1577 interrupts.CPU48.CAL:Function_call_interrupts 196228 ± 3% +18.1% 231730 interrupts.CPU48.LOC:Local_timer_interrupts 1331 ± 3% +18.5% 1578 interrupts.CPU49.CAL:Function_call_interrupts 196553 ± 3% +17.5% 230948 interrupts.CPU49.LOC:Local_timer_interrupts 1306 ± 2% +21.5% 1586 interrupts.CPU5.CAL:Function_call_interrupts 196436 ± 2% +17.6% 230955 interrupts.CPU5.LOC:Local_timer_interrupts 2656 ± 16% -52.5% 1260 ± 56% interrupts.CPU5.NMI:Non-maskable_interrupts 2656 ± 16% -52.5% 1260 ± 56% interrupts.CPU5.PMI:Performance_monitoring_interrupts 1307 ± 2% +20.7% 1578 interrupts.CPU50.CAL:Function_call_interrupts 196306 ± 3% +17.7% 231081 interrupts.CPU50.LOC:Local_timer_interrupts 1312 +20.3% 1578 interrupts.CPU51.CAL:Function_call_interrupts 196454 ± 2% +18.0% 231823 interrupts.CPU51.LOC:Local_timer_interrupts 1316 ± 2% +19.9% 1578 interrupts.CPU52.CAL:Function_call_interrupts 196419 ± 2% +17.7% 231185 interrupts.CPU52.LOC:Local_timer_interrupts 2242 ± 23% -74.4% 573.25 ± 57% interrupts.CPU52.NMI:Non-maskable_interrupts 2242 ± 23% -74.4% 573.25 ± 57% interrupts.CPU52.PMI:Performance_monitoring_interrupts 1308 +17.3% 1534 ± 3% interrupts.CPU53.CAL:Function_call_interrupts 196425 ± 2% +17.9% 231631 interrupts.CPU53.LOC:Local_timer_interrupts 1227 ± 13% +28.4% 1575 interrupts.CPU54.CAL:Function_call_interrupts 196179 ± 2% +18.0% 231486 interrupts.CPU54.LOC:Local_timer_interrupts 1288 ± 5% +19.5% 1539 ± 5% interrupts.CPU55.CAL:Function_call_interrupts 196633 ± 2% +17.8% 231726 interrupts.CPU55.LOC:Local_timer_interrupts 1308 ± 2% +21.0% 1582 interrupts.CPU56.CAL:Function_call_interrupts 196902 ± 2% +17.7% 231729 interrupts.CPU56.LOC:Local_timer_interrupts 196827 ± 2% +17.8% 231949 interrupts.CPU57.LOC:Local_timer_interrupts 2311 ± 41% +73.9% 4018 ± 29% interrupts.CPU57.RES:Rescheduling_interrupts 1314 ± 2% +18.3% 1554 interrupts.CPU58.CAL:Function_call_interrupts 196927 ± 2% +17.7% 231731 interrupts.CPU58.LOC:Local_timer_interrupts 1310 ± 2% +20.7% 1580 interrupts.CPU59.CAL:Function_call_interrupts 196866 ± 2% +17.7% 231802 interrupts.CPU59.LOC:Local_timer_interrupts 1311 ± 2% +20.9% 1585 interrupts.CPU6.CAL:Function_call_interrupts 196416 ± 2% +17.6% 230968 interrupts.CPU6.LOC:Local_timer_interrupts 2423 ± 31% -61.9% 922.25 ± 88% interrupts.CPU6.NMI:Non-maskable_interrupts 2423 ± 31% -61.9% 922.25 ± 88% interrupts.CPU6.PMI:Performance_monitoring_interrupts 1314 ± 2% +20.3% 1580 interrupts.CPU60.CAL:Function_call_interrupts 196951 ± 2% +17.7% 231714 interrupts.CPU60.LOC:Local_timer_interrupts 196924 ± 2% +17.7% 231716 interrupts.CPU61.LOC:Local_timer_interrupts 953.50 ± 81% +189.0% 2755 ± 31% interrupts.CPU61.NMI:Non-maskable_interrupts 953.50 ± 81% +189.0% 2755 ± 31% interrupts.CPU61.PMI:Performance_monitoring_interrupts 1314 ± 2% +19.9% 1576 interrupts.CPU62.CAL:Function_call_interrupts 196678 ± 2% +17.8% 231687 interrupts.CPU62.LOC:Local_timer_interrupts 1277 ± 5% +23.6% 1577 interrupts.CPU63.CAL:Function_call_interrupts 196543 ± 2% +17.9% 231706 interrupts.CPU63.LOC:Local_timer_interrupts 1869 ± 43% +108.0% 3888 ± 28% interrupts.CPU63.RES:Rescheduling_interrupts 1310 ± 2% +20.6% 1580 interrupts.CPU64.CAL:Function_call_interrupts 197006 ± 2% +17.6% 231724 interrupts.CPU64.LOC:Local_timer_interrupts 1312 ± 2% +20.1% 1575 interrupts.CPU65.CAL:Function_call_interrupts 196073 ± 2% +18.2% 231756 interrupts.CPU65.LOC:Local_timer_interrupts 1143 ± 95% +147.4% 2828 ± 24% interrupts.CPU65.NMI:Non-maskable_interrupts 1143 ± 95% +147.4% 2828 ± 24% interrupts.CPU65.PMI:Performance_monitoring_interrupts 1311 ± 2% +17.4% 1540 ± 4% interrupts.CPU66.CAL:Function_call_interrupts 196845 ± 2% +17.8% 231921 interrupts.CPU66.LOC:Local_timer_interrupts 1311 ± 2% +20.5% 1580 interrupts.CPU67.CAL:Function_call_interrupts 195708 ± 2% +18.4% 231743 interrupts.CPU67.LOC:Local_timer_interrupts 1310 ± 2% +20.7% 1581 interrupts.CPU68.CAL:Function_call_interrupts 196837 ± 2% +17.8% 231795 interrupts.CPU68.LOC:Local_timer_interrupts 1516 ± 59% +137.0% 3593 ± 26% interrupts.CPU68.RES:Rescheduling_interrupts 1311 ± 2% +20.5% 1581 interrupts.CPU69.CAL:Function_call_interrupts 196838 ± 2% +17.8% 231870 interrupts.CPU69.LOC:Local_timer_interrupts 1306 ± 2% +21.5% 1586 interrupts.CPU7.CAL:Function_call_interrupts 196395 ± 2% +17.7% 231082 interrupts.CPU7.LOC:Local_timer_interrupts 3299 ± 36% +68.7% 5567 ± 8% interrupts.CPU7.RES:Rescheduling_interrupts 1310 +20.7% 1581 interrupts.CPU70.CAL:Function_call_interrupts 195366 +18.6% 231713 interrupts.CPU70.LOC:Local_timer_interrupts 1308 +18.1% 1545 ± 3% interrupts.CPU71.CAL:Function_call_interrupts 196019 ± 2% +18.2% 231692 interrupts.CPU71.LOC:Local_timer_interrupts 1280 ± 3% +23.9% 1586 interrupts.CPU8.CAL:Function_call_interrupts 196421 ± 2% +18.0% 231850 interrupts.CPU8.LOC:Local_timer_interrupts 2724 ± 33% -58.3% 1137 ± 62% interrupts.CPU8.NMI:Non-maskable_interrupts 2724 ± 33% -58.3% 1137 ± 62% interrupts.CPU8.PMI:Performance_monitoring_interrupts 3448 ± 10% +81.5% 6257 ± 16% interrupts.CPU8.RES:Rescheduling_interrupts 1300 +18.6% 1541 ± 5% interrupts.CPU9.CAL:Function_call_interrupts 196415 ± 2% +17.2% 230179 interrupts.CPU9.LOC:Local_timer_interrupts 14149246 ± 2% +17.8% 16663738 interrupts.LOC:Local_timer_interrupts 5.72 ± 6% -1.0 4.73 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 5.62 ± 7% -1.0 4.65 ± 6% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node 1.32 ± 36% -0.9 0.41 ± 57% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 13.67 -0.8 12.82 ± 5% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe 5.44 -0.6 4.88 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 5.43 -0.6 4.87 ± 6% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents.btrfs_log_inode 4.80 ± 3% -0.5 4.26 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent 4.78 ± 3% -0.5 4.24 ± 6% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode 2.84 ± 9% -0.5 2.33 ± 7% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 2.78 ± 9% -0.5 2.29 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node 4.64 -0.4 4.21 ± 5% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.btrfs_log_changed_extents 4.62 -0.4 4.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks 2.22 ± 3% -0.4 1.80 ± 7% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 1.51 ± 3% -0.4 1.12 ± 26% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 0.62 ± 4% -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 0.69 ± 6% -0.3 0.40 ± 57% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 2.47 -0.3 2.21 ± 4% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 2.48 -0.3 2.22 ± 4% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents 0.67 ± 4% -0.2 0.44 ± 58% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode.btrfs_log_inode_parent 0.65 ± 4% -0.2 0.42 ± 58% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_log_all_xattrs.btrfs_log_inode 0.66 ± 5% -0.2 0.44 ± 57% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 0.82 ± 7% -0.2 0.62 ± 8% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 1.44 -0.2 1.25 ± 6% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum 0.82 ± 7% -0.2 0.62 ± 8% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work 1.93 ± 2% -0.2 1.75 ± 7% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 0.71 ± 6% -0.2 0.54 ± 3% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.53 ± 2% -0.1 1.40 ± 4% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file 1.53 ± 2% -0.1 1.40 ± 4% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops 1.55 ± 2% -0.1 1.42 ± 4% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write 1.54 ± 2% -0.1 1.41 ± 4% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter 1.54 ± 2% -0.1 1.41 ± 4% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.86 ± 3% -0.1 0.74 ± 4% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent 0.67 ± 9% -0.1 0.55 ± 7% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 0.67 ± 9% -0.1 0.55 ± 7% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 1.03 ± 5% -0.1 0.91 ± 7% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range 0.85 ± 3% -0.1 0.73 ± 4% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode 0.75 ± 2% -0.1 0.64 perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file 0.74 ± 2% -0.1 0.63 ± 2% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log 1.24 ± 5% +0.1 1.39 ± 2% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file 1.38 ± 5% +0.2 1.56 ± 4% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 1.59 ± 6% +0.2 1.80 ± 4% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.76 ± 7% +0.2 0.99 ± 4% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper 0.83 ± 3% +0.2 1.08 ± 4% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 0.61 ± 8% +0.3 0.88 ± 5% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io 0.59 ± 8% +0.3 0.85 ± 5% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent 1.78 ± 7% +0.3 2.06 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 0.68 ± 3% +0.3 0.96 ± 4% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode 0.66 ± 3% +0.3 0.93 ± 4% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents 0.55 ± 2% +0.3 0.88 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items 2.89 ± 3% +0.4 3.25 ± 2% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 0.74 +0.4 1.13 ± 2% perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent 0.71 ± 2% +0.4 1.12 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode 0.27 ±100% +0.4 0.71 ± 14% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.39 ± 57% +0.5 0.85 ± 2% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot 0.94 ± 8% +0.5 1.48 ± 14% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 0.13 ±173% +0.6 0.73 ± 11% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +0.6 0.64 ± 11% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 4.37 +0.6 5.01 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 0.00 +0.7 0.67 ± 11% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate 0.00 +0.7 0.68 ± 35% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe 0.00 +0.7 0.72 ± 34% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert 1.56 ± 7% +0.7 2.30 ± 13% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 1.69 ± 7% +0.8 2.47 ± 13% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 4.09 ± 5% +0.8 4.90 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node 0.00 +0.8 0.81 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up 0.00 +1.0 0.97 ± 12% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending 0.00 +1.0 1.00 ± 21% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents 0.00 +1.1 1.09 ± 11% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle 0.00 +1.2 1.16 ± 11% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry 0.00 +1.2 1.16 ± 11% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary 0.00 +1.3 1.33 ± 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 48.26 +3.1 51.38 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 48.26 +3.1 51.38 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 48.24 +3.1 51.36 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 38.83 -2.2 36.66 ± 4% perf-profile.children.cycles-pp.btrfs_search_slot 22.19 -1.8 20.40 ± 4% perf-profile.children.cycles-pp.btrfs_read_lock_root_node 22.00 -1.8 20.24 ± 4% perf-profile.children.cycles-pp.btrfs_tree_read_lock 10.23 ± 3% -1.2 8.99 ± 5% perf-profile.children.cycles-pp.finish_wait 14.49 -1.0 13.45 ± 5% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 16.59 -0.9 15.74 ± 4% perf-profile.children.cycles-pp.btrfs_insert_empty_items 2.97 ± 5% -0.7 2.29 ± 7% perf-profile.children.cycles-pp.queued_read_lock_slowpath 5.68 -0.6 5.06 ± 5% perf-profile.children.cycles-pp.btrfs_lookup_csum 2.98 ± 4% -0.5 2.45 ± 6% perf-profile.children.cycles-pp.queued_write_lock_slowpath 1.79 ± 7% -0.4 1.37 ± 3% perf-profile.children.cycles-pp.__schedule 1.14 ± 7% -0.3 0.86 ± 2% perf-profile.children.cycles-pp.schedule 0.43 ± 14% -0.2 0.18 ± 8% perf-profile.children.cycles-pp.poll_idle 2.35 ± 2% -0.2 2.10 ± 3% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 2.33 ± 2% -0.2 2.09 ± 3% perf-profile.children.cycles-pp.do_writepages 2.16 -0.2 1.92 ± 5% perf-profile.children.cycles-pp._raw_spin_lock 0.82 ± 7% -0.2 0.62 ± 8% perf-profile.children.cycles-pp.add_pending_csums 1.12 ± 3% -0.2 0.95 ± 3% perf-profile.children.cycles-pp.btrfs_cow_block 1.11 ± 3% -0.2 0.94 ± 3% perf-profile.children.cycles-pp.__btrfs_cow_block 0.71 ± 6% -0.2 0.55 ± 3% perf-profile.children.cycles-pp.schedule_idle 1.59 ± 2% -0.1 1.45 ± 4% perf-profile.children.cycles-pp.extent_writepages 1.55 ± 2% -0.1 1.42 ± 5% perf-profile.children.cycles-pp.btrfs_fdatawrite_range 1.55 ± 2% -0.1 1.42 ± 4% perf-profile.children.cycles-pp.start_ordered_ops 0.68 ± 9% -0.1 0.55 ± 7% perf-profile.children.cycles-pp.btrfs_run_delayed_refs 0.68 ± 9% -0.1 0.55 ± 7% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs 1.05 ± 5% -0.1 0.94 ± 7% perf-profile.children.cycles-pp.extent_write_cache_pages 0.53 ± 9% -0.1 0.41 perf-profile.children.cycles-pp.dequeue_task_fair 0.75 ± 2% -0.1 0.64 perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.50 ± 4% -0.1 0.39 ± 2% perf-profile.children.cycles-pp.read_block_for_search 0.74 ± 2% -0.1 0.63 ± 2% perf-profile.children.cycles-pp.btree_write_cache_pages 0.54 ± 3% -0.1 0.43 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32 0.47 ± 8% -0.1 0.38 perf-profile.children.cycles-pp.dequeue_entity 1.02 ± 2% -0.1 0.93 ± 3% perf-profile.children.cycles-pp.flush_write_bio 1.02 ± 2% -0.1 0.93 ± 3% perf-profile.children.cycles-pp.submit_one_bio 0.54 ± 5% -0.1 0.45 ± 5% perf-profile.children.cycles-pp.btrfs_del_items 0.97 ± 2% -0.1 0.89 ± 3% perf-profile.children.cycles-pp.btrfs_map_bio 0.93 ± 3% -0.1 0.84 ± 2% perf-profile.children.cycles-pp.submit_bio 0.93 ± 3% -0.1 0.84 ± 2% perf-profile.children.cycles-pp.generic_make_request 0.38 ± 3% -0.1 0.29 ± 9% perf-profile.children.cycles-pp.__list_add_valid 0.56 ± 2% -0.1 0.47 ± 4% perf-profile.children.cycles-pp.brd_do_bvec 0.40 ± 2% -0.1 0.32 ± 3% perf-profile.children.cycles-pp.__list_del_entry_valid 0.34 ± 10% -0.1 0.26 ± 7% perf-profile.children.cycles-pp.update_load_avg 0.49 ± 2% -0.1 0.41 ± 4% perf-profile.children.cycles-pp.btree_submit_bio_hook 0.40 ± 6% -0.1 0.32 ± 4% perf-profile.children.cycles-pp._raw_read_lock 0.33 ± 4% -0.1 0.26 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_unlock 0.82 ± 3% -0.1 0.75 ± 3% perf-profile.children.cycles-pp.brd_make_request 0.31 ± 8% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.btrfs_set_token_32 0.38 ± 7% -0.1 0.31 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair 0.33 ± 4% -0.1 0.27 ± 4% perf-profile.children.cycles-pp.find_extent_buffer 0.35 ± 4% -0.1 0.29 ± 5% perf-profile.children.cycles-pp.select_task_rq_fair 0.37 ± 3% -0.1 0.31 ± 2% perf-profile.children.cycles-pp.copy_page 0.29 ± 4% -0.1 0.23 ± 6% perf-profile.children.cycles-pp.memmove 0.37 ± 3% -0.1 0.31 ± 2% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.31 ± 3% -0.1 0.25 ± 5% perf-profile.children.cycles-pp.__set_extent_bit 0.19 ± 10% -0.1 0.13 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.42 ± 5% -0.1 0.37 ± 5% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 0.33 ± 4% -0.1 0.28 ± 4% perf-profile.children.cycles-pp.generic_bin_search 0.42 ± 5% -0.1 0.37 ± 5% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 0.23 ± 4% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.memmove_extent_buffer 0.12 ± 11% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.finish_task_switch 0.20 ± 6% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty 0.22 ± 5% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.btrfs_lookup_csums_range 0.17 ± 11% -0.0 0.12 ± 8% perf-profile.children.cycles-pp.update_rq_clock 0.21 ± 4% -0.0 0.17 ± 9% perf-profile.children.cycles-pp.alloc_extent_buffer 0.20 ± 5% -0.0 0.16 ± 6% perf-profile.children.cycles-pp.btrfs_tree_unlock 0.12 ± 8% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.brd_insert_page 0.22 ± 3% -0.0 0.18 ± 3% perf-profile.children.cycles-pp.update_curr 0.14 ± 10% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.set_extent_buffer_dirty 0.22 ± 5% -0.0 0.18 ± 4% perf-profile.children.cycles-pp.map_private_extent_buffer 0.16 ± 4% -0.0 0.12 ± 12% perf-profile.children.cycles-pp.pagecache_get_page 0.06 -0.0 0.03 ±100% perf-profile.children.cycles-pp.___might_sleep 0.18 ± 2% -0.0 0.15 ± 5% perf-profile.children.cycles-pp.__radix_tree_lookup 0.17 ± 5% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.btrfs_set_path_blocking 0.16 ± 5% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.sched_clock 0.21 ± 8% -0.0 0.17 ± 10% perf-profile.children.cycles-pp.btrfs_wait_ordered_range 0.14 ± 11% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.update_ts_time_stats 0.21 ± 4% -0.0 0.18 ± 2% perf-profile.children.cycles-pp.set_extent_bit 0.13 ± 5% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.btrfs_set_lock_blocking_write 0.13 ± 10% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.btrfs_truncate_item 0.12 ± 11% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.truncate_one_csum 0.10 ± 10% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.ttwu_do_wakeup 0.13 ± 8% -0.0 0.10 ± 10% perf-profile.children.cycles-pp.nr_iowait_cpu 0.15 ± 7% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 0.13 ± 6% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.memcpy_extent_buffer 0.10 ± 7% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.lock_extent_bits 0.12 ± 9% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.17 ± 4% -0.0 0.14 perf-profile.children.cycles-pp.sched_clock_cpu 0.15 ± 5% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.native_sched_clock 0.06 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.06 ± 6% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.alloc_extent_state 0.09 ± 4% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.cpuacct_charge 0.17 ± 4% -0.0 0.15 ± 4% perf-profile.children.cycles-pp.write_one_eb 0.07 ± 11% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.reweight_entity 0.11 ± 7% -0.0 0.10 ± 5% perf-profile.children.cycles-pp.clear_state_bit 0.08 ± 6% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.read_extent_buffer 0.08 ± 5% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.__lookup_extent_mapping 0.07 ± 10% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se 0.16 ± 5% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.add_delayed_ref_head 0.10 ± 5% -0.0 0.08 ± 8% perf-profile.children.cycles-pp.__test_set_page_writeback 0.10 ± 4% -0.0 0.09 perf-profile.children.cycles-pp.__extent_writepage_io 0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__etree_search 0.06 -0.0 0.05 perf-profile.children.cycles-pp.btrfs_set_extent_delalloc 0.06 ± 7% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.load_balance 0.10 ± 15% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.12 ± 8% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.rebalance_domains 0.03 ±100% +0.0 0.06 ± 6% perf-profile.children.cycles-pp.run_rebalance_domains 0.33 ± 6% +0.0 0.36 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.12 ± 6% +0.0 0.17 ± 18% perf-profile.children.cycles-pp.rcu_core 0.06 ± 14% +0.0 0.10 ± 14% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.12 ± 10% +0.1 0.17 ± 8% perf-profile.children.cycles-pp.prepare_to_wait 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.find_busiest_group 0.01 ±173% +0.1 0.08 ± 31% perf-profile.children.cycles-pp.run_local_timers 0.04 ± 60% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.cpu_load_update_active 0.14 ± 15% +0.1 0.24 ± 14% perf-profile.children.cycles-pp.scheduler_tick 0.00 +0.1 0.10 ± 7% perf-profile.children.cycles-pp.save_stack_address 0.00 +0.1 0.11 ± 11% perf-profile.children.cycles-pp.__unwind_start 0.37 ± 7% +0.1 0.49 ± 10% perf-profile.children.cycles-pp.__softirqentry_text_start 1.25 ± 5% +0.1 1.40 ± 2% perf-profile.children.cycles-pp.osq_lock 0.00 +0.2 0.16 ± 9% perf-profile.children.cycles-pp.__module_text_address 1.42 ± 5% +0.2 1.59 ± 3% perf-profile.children.cycles-pp.__mutex_lock 0.00 +0.2 0.17 ± 7% perf-profile.children.cycles-pp.is_module_text_address 0.32 ± 10% +0.2 0.53 ± 13% perf-profile.children.cycles-pp.tick_sched_handle 1.59 ± 6% +0.2 1.80 ± 4% perf-profile.children.cycles-pp.wait_log_commit 0.31 ± 10% +0.2 0.52 ± 12% perf-profile.children.cycles-pp.update_process_times 0.56 ± 8% +0.2 0.78 ± 13% perf-profile.children.cycles-pp.irq_exit 0.00 +0.2 0.23 ± 5% perf-profile.children.cycles-pp.kernel_text_address 0.41 ± 8% +0.2 0.65 ± 11% perf-profile.children.cycles-pp.tick_sched_timer 0.00 +0.3 0.26 ± 5% perf-profile.children.cycles-pp.__kernel_text_address 0.59 ± 8% +0.3 0.85 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.00 +0.3 0.27 ± 6% perf-profile.children.cycles-pp.unwind_get_return_address 0.00 +0.3 0.28 ± 7% perf-profile.children.cycles-pp.__module_address 0.40 ± 10% +0.3 0.68 ± 18% perf-profile.children.cycles-pp.clockevents_program_event 0.62 ± 8% +0.3 0.94 ± 17% perf-profile.children.cycles-pp.ktime_get 0.89 +0.4 1.32 perf-profile.children.cycles-pp.unlock_up 1.90 +0.4 2.34 ± 3% perf-profile.children.cycles-pp.setup_items_for_insert 1.14 ± 8% +0.5 1.67 ± 12% perf-profile.children.cycles-pp.hrtimer_interrupt 0.00 +0.6 0.56 ± 5% perf-profile.children.cycles-pp.__orc_find 1.40 ± 2% +0.6 1.97 ± 3% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.00 +0.7 0.72 ± 5% perf-profile.children.cycles-pp.orc_find 1.82 ± 7% +0.7 2.57 ± 12% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 1.98 ± 7% +0.8 2.76 ± 11% perf-profile.children.cycles-pp.apic_timer_interrupt 0.45 ± 8% +0.9 1.38 ± 11% perf-profile.children.cycles-pp.sched_ttwu_pending 1.77 ± 4% +1.1 2.92 ± 2% perf-profile.children.cycles-pp.__wake_up_common 1.66 ± 5% +1.2 2.81 ± 2% perf-profile.children.cycles-pp.autoremove_wake_function 1.58 ± 4% +1.3 2.83 ± 2% perf-profile.children.cycles-pp.try_to_wake_up 0.00 +1.3 1.32 ± 4% perf-profile.children.cycles-pp.unwind_next_frame 2.66 +1.4 4.01 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 0.00 +1.8 1.75 ± 4% perf-profile.children.cycles-pp.__save_stack_trace 0.00 +1.8 1.85 ± 4% perf-profile.children.cycles-pp.save_stack_trace_tsk 0.61 ± 4% +2.5 3.07 ± 5% perf-profile.children.cycles-pp.ttwu_do_activate 0.58 ± 3% +2.5 3.04 ± 5% perf-profile.children.cycles-pp.enqueue_task_fair 0.44 ± 5% +2.5 2.92 ± 5% perf-profile.children.cycles-pp.enqueue_entity 0.00 +2.6 2.55 ± 5% perf-profile.children.cycles-pp.__account_scheduler_latency 48.26 +3.1 51.38 ± 3% perf-profile.children.cycles-pp.start_secondary 0.41 ± 16% -0.2 0.17 ± 10% perf-profile.self.cycles-pp.poll_idle 0.68 ± 8% -0.2 0.51 ± 8% perf-profile.self.cycles-pp.btrfs_tree_read_lock 1.22 -0.2 1.06 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.93 ± 3% -0.2 0.77 perf-profile.self.cycles-pp._raw_spin_lock 0.59 ± 3% -0.2 0.44 ± 5% perf-profile.self.cycles-pp.prepare_to_wait_event 0.45 ± 7% -0.1 0.32 ± 3% perf-profile.self.cycles-pp.__schedule 0.85 ± 3% -0.1 0.73 ± 6% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.52 ± 5% -0.1 0.42 ± 4% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.47 ± 4% -0.1 0.38 ± 7% perf-profile.self.cycles-pp.btrfs_get_token_32 0.38 ± 3% -0.1 0.29 ± 8% perf-profile.self.cycles-pp.__list_add_valid 0.40 ± 2% -0.1 0.31 ± 3% perf-profile.self.cycles-pp.__list_del_entry_valid 0.40 ± 4% -0.1 0.32 ± 3% perf-profile.self.cycles-pp._raw_read_lock 0.35 -0.1 0.27 ± 8% perf-profile.self.cycles-pp.btrfs_tree_lock 0.33 ± 5% -0.1 0.26 ± 10% perf-profile.self.cycles-pp.btrfs_tree_read_unlock 0.30 ± 7% -0.1 0.23 ± 11% perf-profile.self.cycles-pp.try_to_wake_up 0.29 ± 7% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.btrfs_set_token_32 0.29 ± 4% -0.1 0.23 ± 6% perf-profile.self.cycles-pp.memmove 0.36 ± 4% -0.1 0.31 ± 2% perf-profile.self.cycles-pp.copy_page 0.10 ± 14% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.finish_task_switch 0.38 ± 5% -0.0 0.34 ± 3% perf-profile.self.cycles-pp.brd_do_bvec 0.15 ± 14% -0.0 0.11 ± 17% perf-profile.self.cycles-pp.update_load_avg 0.20 ± 5% -0.0 0.15 ± 7% perf-profile.self.cycles-pp.btrfs_tree_unlock 0.26 ± 5% -0.0 0.22 ± 10% perf-profile.self.cycles-pp.do_idle 0.21 ± 5% -0.0 0.18 ± 4% perf-profile.self.cycles-pp.generic_bin_search 0.21 ± 6% -0.0 0.17 ± 4% perf-profile.self.cycles-pp.select_task_rq_fair 0.19 ± 9% -0.0 0.15 ± 5% perf-profile.self.cycles-pp.free_extent_buffer 0.18 ± 2% -0.0 0.15 ± 5% perf-profile.self.cycles-pp.__radix_tree_lookup 0.10 ± 8% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair 0.21 ± 5% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.map_private_extent_buffer 0.21 ± 5% -0.0 0.18 ± 6% perf-profile.self.cycles-pp.btrfs_root_node 0.15 ± 7% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.native_sched_clock 0.13 ± 8% -0.0 0.10 ± 10% perf-profile.self.cycles-pp.nr_iowait_cpu 0.13 ± 3% -0.0 0.10 ± 10% perf-profile.self.cycles-pp.btrfs_set_lock_blocking_write 0.19 ± 2% -0.0 0.16 ± 6% perf-profile.self.cycles-pp.btrfs_search_slot 0.09 ± 4% -0.0 0.07 ± 10% perf-profile.self.cycles-pp.cpuacct_charge 0.08 ± 6% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.read_extent_buffer 0.11 ± 7% -0.0 0.09 ± 11% perf-profile.self.cycles-pp.sched_ttwu_pending 0.07 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.reweight_entity 0.11 ± 4% -0.0 0.10 perf-profile.self.cycles-pp.__wake_up_common 0.05 ± 8% +0.0 0.10 ± 15% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.orc_find 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.save_stack_address 0.00 +0.1 0.08 ± 31% perf-profile.self.cycles-pp.run_local_timers 0.00 +0.1 0.10 ± 5% perf-profile.self.cycles-pp.save_stack_trace_tsk 0.01 ±173% +0.1 0.12 ± 23% perf-profile.self.cycles-pp.cpu_load_update_active 1.24 ± 4% +0.2 1.39 ± 2% perf-profile.self.cycles-pp.osq_lock 0.00 +0.3 0.28 ± 7% perf-profile.self.cycles-pp.__module_address 0.45 ± 9% +0.3 0.79 ± 19% perf-profile.self.cycles-pp.ktime_get 0.00 +0.4 0.45 ± 7% perf-profile.self.cycles-pp.__account_scheduler_latency 0.00 +0.5 0.51 ± 5% perf-profile.self.cycles-pp.unwind_next_frame 0.00 +0.6 0.56 ± 5% perf-profile.self.cycles-pp.__orc_find aim7.jobs-per-min 1250 +-+------------------------------------------------------------------+ 1200 +-+.+.++.+.+.+.+.+.+.++.+.+.+.+ | | | 1150 +-+ | 1100 +-+ O O O O OO O | 1050 +-+ O O O O O OO O O O 1000 +-+ | | | 950 +-+ | 900 +-+ | 850 O-+ O O O O O O O O O O | 800 +-+ O O O O | | O O | 750 +-+ O OO O | 700 +-+------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase: gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/20/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/sync_disk_rw/aim7 commit: v5.1-rc7 2996e1f8bc ("btrfs: factor our read/write stage off csum_tree_block into its callers") v5.1-rc7 2996e1f8bcadf0436cc67b63af0 ---------------- --------------------------- %stddev %change %stddev \ | \ 474.83 -19.1% 384.18 aim7.jobs-per-min 252.81 +23.6% 312.51 aim7.time.elapsed_time 252.81 +23.6% 312.51 aim7.time.elapsed_time.max 3369 ± 3% +15.9% 3905 ± 3% aim7.time.involuntary_context_switches 2829 +3.5% 2927 aim7.time.maximum_resident_set_size 8026 ± 4% +18.3% 9498 ± 3% aim7.time.minor_page_faults 1763 ± 2% +23.9% 2185 ± 2% aim7.time.system_time 27540068 -11.4% 24395342 aim7.time.voluntary_context_switches 39442 ± 2% -22.7% 30475 ± 2% meminfo.max_used_kB 0.11 ± 10% -0.0 0.08 ± 2% mpstat.cpu.all.usr% 5334 ± 8% -15.2% 4524 ± 10% numa-meminfo.node0.KernelStack 2058 ± 29% +43.2% 2949 ± 16% numa-meminfo.node1.PageTables 5335 ± 8% -15.2% 4524 ± 10% numa-vmstat.node0.nr_kernel_stack 514.25 ± 29% +43.3% 736.75 ± 16% numa-vmstat.node1.nr_page_table_pages 137189 -19.6% 110330 vmstat.io.bo 295976 -28.3% 212126 vmstat.system.cs 78.76 +1.6% 80.02 iostat.cpu.idle 21.12 -5.8% 19.89 iostat.cpu.system 7896 -19.5% 6356 iostat.md0.w/s 178487 -19.6% 143527 iostat.md0.wkB/s 20790 +3.9% 21601 proc-vmstat.nr_slab_unreclaimable 5524 ± 9% +24.1% 6853 ± 4% proc-vmstat.numa_hint_faults 3580 ± 12% +27.7% 4572 ± 6% proc-vmstat.numa_hint_faults_local 588396 +23.2% 724636 ± 2% proc-vmstat.pgfault 969401 +12.8% 1093038 proc-vmstat.pgfree 3.216e+09 ± 13% -19.2% 2.598e+09 ± 3% cpuidle.C1.time 34331442 ± 2% -20.2% 27399001 ± 2% cpuidle.C1.usage 5.21e+08 ± 52% +71.0% 8.911e+08 ± 26% cpuidle.C1E.time 5266261 ± 24% +46.0% 7687200 ± 16% cpuidle.C1E.usage 1.353e+09 ± 51% +98.2% 2.683e+09 ± 27% cpuidle.C3.time 6733067 ± 28% +53.5% 10335866 ± 16% cpuidle.C3.usage 5202587 ± 21% +45.4% 7564469 ± 7% cpuidle.C6.usage 170635 ± 77% -56.8% 73760 cpuidle.POLL.usage 4104 ± 3% -16.5% 3428 ± 10% slabinfo.btrfs_extent_buffer.active_objs 4111 ± 3% -16.5% 3434 ± 10% slabinfo.btrfs_extent_buffer.num_objs 11182 ± 2% -10.7% 9988 slabinfo.filp.active_objs 11535 -11.0% 10262 slabinfo.filp.num_objs 5266 +11.2% 5856 slabinfo.kmalloc-96.active_objs 5276 +11.1% 5863 slabinfo.kmalloc-96.num_objs 946.75 -13.2% 822.25 slabinfo.task_struct.active_objs 960.25 ± 2% -12.5% 840.50 slabinfo.task_struct.num_objs 309.00 -2.9% 300.00 turbostat.Avg_MHz 34327518 ± 2% -20.2% 27391739 ± 2% turbostat.C1 31.35 ± 13% -11.0 20.34 turbostat.C1% 5265915 ± 24% +46.0% 7686822 ± 16% turbostat.C1E 6732546 ± 28% +53.5% 10335310 ± 16% turbostat.C3 5199490 ± 21% +45.4% 7561891 ± 7% turbostat.C6 19750059 ± 13% +33.5% 26375491 ± 2% turbostat.IRQ 32.10 -1.7% 31.56 turbostat.RAMWatt 20520 +24.5% 25540 ± 2% turbostat.SMI 2739 ± 11% +115.5% 5902 ± 38% sched_debug.cfs_rq:/.MIN_vruntime.avg 109456 ± 12% +38.8% 151967 ± 11% sched_debug.cfs_rq:/.MIN_vruntime.max 17097 ± 11% +64.1% 28053 ± 17% sched_debug.cfs_rq:/.MIN_vruntime.stddev 2739 ± 11% +115.5% 5902 ± 38% sched_debug.cfs_rq:/.max_vruntime.avg 109456 ± 12% +38.8% 151967 ± 11% sched_debug.cfs_rq:/.max_vruntime.max 17097 ± 11% +64.1% 28053 ± 17% sched_debug.cfs_rq:/.max_vruntime.stddev 188026 +23.2% 231708 sched_debug.cfs_rq:/.min_vruntime.avg 224730 ± 2% +24.3% 279273 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 155086 +22.2% 189563 sched_debug.cfs_rq:/.min_vruntime.min 15264 ± 9% +27.7% 19493 ± 12% sched_debug.cfs_rq:/.min_vruntime.stddev 15264 ± 9% +27.7% 19493 ± 12% sched_debug.cfs_rq:/.spread0.stddev 167631 +20.1% 201400 sched_debug.cpu.clock.avg 167635 +20.1% 201404 sched_debug.cpu.clock.max 167627 +20.1% 201395 sched_debug.cpu.clock.min 2.34 ± 4% +13.4% 2.65 ± 5% sched_debug.cpu.clock.stddev 167631 +20.1% 201400 sched_debug.cpu.clock_task.avg 167635 +20.1% 201404 sched_debug.cpu.clock_task.max 167627 +20.1% 201395 sched_debug.cpu.clock_task.min 2.34 ± 4% +13.4% 2.65 ± 5% sched_debug.cpu.clock_task.stddev 17.73 ± 10% -22.0% 13.83 ± 7% sched_debug.cpu.cpu_load[0].avg 44.72 ± 22% -29.1% 31.70 ± 21% sched_debug.cpu.cpu_load[0].stddev 15.81 ± 10% -21.7% 12.38 ± 11% sched_debug.cpu.cpu_load[4].avg 34.15 ± 32% -54.3% 15.61 ± 53% sched_debug.cpu.cpu_load[4].stddev 4754 +16.2% 5526 sched_debug.cpu.curr->pid.max 1004 ± 3% +12.1% 1126 sched_debug.cpu.curr->pid.stddev 130427 +23.1% 160557 sched_debug.cpu.nr_load_updates.avg 138214 +23.6% 170891 sched_debug.cpu.nr_load_updates.max 127564 +23.5% 157516 sched_debug.cpu.nr_load_updates.min 959187 -9.5% 867604 sched_debug.cpu.nr_switches.max 771024 -9.2% 700458 ± 4% sched_debug.cpu.nr_switches.min 167627 +20.1% 201395 sched_debug.cpu_clk 164058 +20.6% 197822 sched_debug.ktime 168073 +20.1% 201837 sched_debug.sched_clk 8.454e+08 ± 2% +20.1% 1.015e+09 perf-stat.i.branch-instructions 16630864 ± 7% +19.8% 19922156 ± 11% perf-stat.i.branch-misses 20.26 -4.1 16.20 ± 2% perf-stat.i.cache-miss-rate% 14888468 ± 2% -16.0% 12505705 perf-stat.i.cache-misses 74346170 ± 2% +5.5% 78449759 ± 2% perf-stat.i.cache-references 299258 -28.5% 214099 perf-stat.i.context-switches 3.21 -11.4% 2.85 perf-stat.i.cpi 218.00 ± 4% -10.1% 196.01 perf-stat.i.cpu-migrations 816.02 +17.2% 956.56 perf-stat.i.cycles-between-cache-misses 1.1e+09 ± 2% +7.0% 1.177e+09 perf-stat.i.dTLB-loads 0.11 ± 10% -0.0 0.09 ± 6% perf-stat.i.dTLB-store-miss-rate% 572872 ± 9% -19.6% 460348 ± 7% perf-stat.i.dTLB-store-misses 46.23 ± 3% +6.8 53.01 ± 2% perf-stat.i.iTLB-load-miss-rate% 2592240 -27.0% 1892449 ± 2% perf-stat.i.iTLB-loads 4.012e+09 ± 2% +13.9% 4.569e+09 perf-stat.i.instructions 1948 ± 6% +14.6% 2232 ± 4% perf-stat.i.instructions-per-iTLB-miss 0.32 +15.0% 0.37 perf-stat.i.ipc 6791873 -19.7% 5457038 perf-stat.i.node-load-misses 7294618 -19.3% 5886107 perf-stat.i.node-loads 4662051 ± 2% -19.9% 3733205 perf-stat.i.node-store-misses 6959162 ± 2% -19.3% 5614550 perf-stat.i.node-stores 18.53 ± 2% -7.4% 17.17 perf-stat.overall.MPKI 20.03 -4.1 15.95 perf-stat.overall.cache-miss-rate% 3.05 -13.5% 2.64 perf-stat.overall.cpi 822.35 +17.2% 964.05 perf-stat.overall.cycles-between-cache-misses 0.11 ± 11% -0.0 0.09 ± 5% perf-stat.overall.dTLB-store-miss-rate% 44.19 ± 3% +7.3 51.51 ± 2% perf-stat.overall.iTLB-load-miss-rate% 1959 ± 6% +16.2% 2275 ± 4% perf-stat.overall.instructions-per-iTLB-miss 0.33 +15.6% 0.38 perf-stat.overall.ipc 8.42e+08 ± 2% +20.2% 1.012e+09 perf-stat.ps.branch-instructions 16565866 ± 7% +19.9% 19859511 ± 11% perf-stat.ps.branch-misses 14829536 ± 2% -15.9% 12466228 perf-stat.ps.cache-misses 74051441 ± 2% +5.6% 78201714 ± 2% perf-stat.ps.cache-references 298067 -28.4% 213421 perf-stat.ps.context-switches 217.13 ± 4% -10.0% 195.39 perf-stat.ps.cpu-migrations 1.095e+09 ± 2% +7.1% 1.173e+09 perf-stat.ps.dTLB-loads 570616 ± 9% -19.6% 458896 ± 7% perf-stat.ps.dTLB-store-misses 2581925 -26.9% 1886455 ± 2% perf-stat.ps.iTLB-loads 3.996e+09 ± 2% +14.0% 4.555e+09 perf-stat.ps.instructions 6764857 -19.6% 5439764 perf-stat.ps.node-load-misses 7265612 -19.2% 5867478 perf-stat.ps.node-loads 4643500 ± 2% -19.9% 3721390 perf-stat.ps.node-store-misses 6931476 ± 2% -19.3% 5596795 perf-stat.ps.node-stores 1.02e+12 +42.1% 1.45e+12 perf-stat.total.instructions 151.25 ± 11% +94.5% 294.25 ± 46% interrupts.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6 125.25 +24.6% 156.00 interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2] 99753 ± 7% +18.7% 118441 interrupts.CAL:Function_call_interrupts 2335 ± 9% +24.9% 2917 ± 8% interrupts.CPU0.CAL:Function_call_interrupts 500200 ± 4% +27.5% 638000 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts 2518 ± 9% +15.3% 2904 ± 4% interrupts.CPU1.CAL:Function_call_interrupts 501181 ± 3% +27.2% 637472 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts 2365 ± 8% +28.1% 3031 ± 6% interrupts.CPU10.CAL:Function_call_interrupts 498584 ± 4% +27.8% 637281 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts 14324 ± 23% +37.1% 19636 ± 14% interrupts.CPU10.RES:Rescheduling_interrupts 446594 ± 24% +42.8% 637802 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts 2446 ± 7% +24.3% 3039 ± 4% interrupts.CPU12.CAL:Function_call_interrupts 499544 ± 4% +27.6% 637595 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts 2265 ± 10% +37.6% 3117 ± 6% interrupts.CPU13.CAL:Function_call_interrupts 500218 ± 4% +27.5% 637698 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts 0.75 ±173% +81766.7% 614.00 ± 57% interrupts.CPU13.NMI:Non-maskable_interrupts 0.75 ±173% +81766.7% 614.00 ± 57% interrupts.CPU13.PMI:Performance_monitoring_interrupts 2413 ± 5% +24.0% 2993 ± 5% interrupts.CPU14.CAL:Function_call_interrupts 447745 ± 24% +42.4% 637743 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts 14160 ± 8% +30.9% 18541 ± 18% interrupts.CPU14.RES:Rescheduling_interrupts 2450 ± 8% +24.6% 3053 ± 3% interrupts.CPU15.CAL:Function_call_interrupts 447422 ± 24% +42.5% 637614 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts 499537 ± 4% +27.7% 637734 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts 2511 ± 8% +14.4% 2872 ± 4% interrupts.CPU17.CAL:Function_call_interrupts 501205 ± 3% +27.3% 637896 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts 2466 ± 5% +17.7% 2904 ± 7% interrupts.CPU18.CAL:Function_call_interrupts 498429 ± 4% +27.9% 637539 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts 2421 ± 8% +19.2% 2885 ± 2% interrupts.CPU19.CAL:Function_call_interrupts 447347 ± 24% +42.5% 637690 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts 14512 ± 10% +19.9% 17400 ± 16% interrupts.CPU19.RES:Rescheduling_interrupts 2472 ± 8% +23.8% 3061 ± 4% interrupts.CPU2.CAL:Function_call_interrupts 498740 ± 4% +27.9% 637673 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts 498931 ± 4% +27.8% 637604 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts 2457 ± 7% +26.2% 3102 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 447732 ± 24% +42.4% 637663 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts 448745 ± 23% +42.1% 637616 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts 1062 ± 65% -61.6% 407.75 ±169% interrupts.CPU22.NMI:Non-maskable_interrupts 1062 ± 65% -61.6% 407.75 ±169% interrupts.CPU22.PMI:Performance_monitoring_interrupts 2525 ± 3% +15.0% 2905 ± 3% interrupts.CPU23.CAL:Function_call_interrupts 448647 ± 23% +42.2% 637857 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts 499423 ± 4% +27.7% 637620 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts 499966 ± 4% +27.6% 637919 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts 2545 ± 5% +16.7% 2971 ± 7% interrupts.CPU26.CAL:Function_call_interrupts 448780 ± 23% +42.1% 637593 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts 500205 ± 3% +27.5% 637903 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts 2401 ± 9% +21.7% 2923 ± 5% interrupts.CPU28.CAL:Function_call_interrupts 498973 ± 4% +27.8% 637717 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts 502124 ± 3% +27.0% 637676 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts 2474 ± 11% +21.2% 2998 ± 8% interrupts.CPU3.CAL:Function_call_interrupts 500168 ± 4% +27.5% 637840 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts 448818 ± 23% +42.0% 637276 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts 2535 ± 8% +19.8% 3037 ± 2% interrupts.CPU31.CAL:Function_call_interrupts 501670 ± 3% +27.0% 637350 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts 2444 ± 5% +20.1% 2936 ± 5% interrupts.CPU32.CAL:Function_call_interrupts 448601 ± 24% +42.1% 637465 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts 448644 ± 23% +42.1% 637433 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts 1298 ± 32% -68.5% 408.75 ±173% interrupts.CPU33.NMI:Non-maskable_interrupts 1298 ± 32% -68.5% 408.75 ±173% interrupts.CPU33.PMI:Performance_monitoring_interrupts 18952 ± 8% -18.8% 15386 ± 16% interrupts.CPU33.RES:Rescheduling_interrupts 499974 ± 4% +27.5% 637544 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts 446755 ± 24% +42.7% 637668 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts 1076 ± 36% -78.6% 230.00 ±157% interrupts.CPU35.NMI:Non-maskable_interrupts 1076 ± 36% -78.6% 230.00 ±157% interrupts.CPU35.PMI:Performance_monitoring_interrupts 2621 ± 4% +11.3% 2918 ± 5% interrupts.CPU36.CAL:Function_call_interrupts 447762 ± 24% +42.4% 637548 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts 2542 ± 10% +23.9% 3151 ± 3% interrupts.CPU37.CAL:Function_call_interrupts 447655 ± 24% +42.2% 636691 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts 16018 ± 16% +34.2% 21489 ± 4% interrupts.CPU37.RES:Rescheduling_interrupts 151.25 ± 11% +94.5% 294.25 ± 46% interrupts.CPU38.43:IR-PCI-MSI.524295-edge.eth0-TxRx-6 448676 ± 24% +42.1% 637520 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts 500913 ± 3% +27.3% 637659 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts 448474 ± 24% +42.2% 637512 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts 2474 ± 8% +24.2% 3073 ± 3% interrupts.CPU5.CAL:Function_call_interrupts 448794 ± 23% +42.1% 637675 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts 2416 ± 7% +18.7% 2867 ± 6% interrupts.CPU6.CAL:Function_call_interrupts 499019 ± 4% +27.8% 637624 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts 449105 ± 23% +42.0% 637874 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts 2545 ± 4% +18.4% 3014 ± 4% interrupts.CPU8.CAL:Function_call_interrupts 448672 ± 24% +42.1% 637658 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts 447744 ± 24% +42.4% 637699 ± 2% interrupts.CPU9.LOC:Local_timer_interrupts 18961728 ± 13% +34.5% 25504957 ± 2% interrupts.LOC:Local_timer_interrupts 70856 ± 3% +27.7% 90455 ± 5% softirqs.CPU0.RCU 46993 ± 2% +19.6% 56188 ± 2% softirqs.CPU0.SCHED 90395 ± 4% +27.6% 115332 ± 2% softirqs.CPU0.TIMER 71106 ± 2% +27.3% 90543 ± 3% softirqs.CPU1.RCU 44000 +20.6% 53050 ± 2% softirqs.CPU1.SCHED 94521 ± 6% +16.0% 109649 softirqs.CPU1.TIMER 72447 ± 5% +23.7% 89595 ± 4% softirqs.CPU10.RCU 42542 +25.0% 53191 ± 4% softirqs.CPU10.SCHED 86643 ± 5% +30.5% 113030 softirqs.CPU10.TIMER 70388 ± 3% +27.8% 89934 ± 2% softirqs.CPU11.RCU 44629 ± 3% +19.4% 53308 softirqs.CPU11.SCHED 71989 ± 3% +30.1% 93643 ± 5% softirqs.CPU12.RCU 44508 ± 3% +19.9% 53353 ± 2% softirqs.CPU12.SCHED 90388 ± 3% +26.8% 114579 ± 3% softirqs.CPU12.TIMER 73320 +25.3% 91879 ± 3% softirqs.CPU13.RCU 44242 +20.1% 53144 softirqs.CPU13.SCHED 89009 ± 4% +27.3% 113304 ± 3% softirqs.CPU13.TIMER 70951 ± 6% +28.3% 91061 ± 6% softirqs.CPU14.RCU 44755 ± 3% +19.8% 53625 ± 2% softirqs.CPU14.SCHED 68460 ± 6% +33.5% 91393 ± 3% softirqs.CPU15.RCU 44863 ± 2% +19.2% 53481 softirqs.CPU15.SCHED 72151 ± 4% +26.3% 91161 ± 3% softirqs.CPU16.RCU 44009 ± 2% +20.9% 53186 ± 2% softirqs.CPU16.SCHED 90040 ± 5% +28.2% 115465 softirqs.CPU16.TIMER 73480 ± 2% +23.7% 90925 ± 2% softirqs.CPU17.RCU 43579 ± 2% +22.2% 53237 softirqs.CPU17.SCHED 89846 ± 5% +28.3% 115255 ± 2% softirqs.CPU17.TIMER 71778 ± 4% +27.0% 91150 ± 3% softirqs.CPU18.RCU 43811 ± 2% +22.3% 53560 ± 3% softirqs.CPU18.SCHED 89950 ± 5% +47.8% 132927 ± 22% softirqs.CPU18.TIMER 70427 ± 5% +28.7% 90642 ± 3% softirqs.CPU19.RCU 44467 ± 3% +19.6% 53172 softirqs.CPU19.SCHED 72448 ± 3% +25.8% 91106 ± 4% softirqs.CPU2.RCU 43284 ± 2% +22.8% 53170 ± 2% softirqs.CPU2.SCHED 88648 ± 5% +27.8% 113278 ± 4% softirqs.CPU2.TIMER 71858 ± 5% +28.6% 92375 ± 3% softirqs.CPU20.RCU 43172 +22.4% 52826 ± 2% softirqs.CPU20.SCHED 89040 ± 5% +26.3% 112429 ± 3% softirqs.CPU20.TIMER 72855 ± 5% +28.0% 93223 ± 2% softirqs.CPU21.RCU 44066 ± 3% +19.0% 52436 ± 3% softirqs.CPU21.SCHED 70991 ± 7% +32.5% 94074 ± 4% softirqs.CPU22.RCU 44490 ± 3% +19.0% 52958 ± 2% softirqs.CPU22.SCHED 71264 ± 5% +28.1% 91257 ± 3% softirqs.CPU23.RCU 44233 ± 2% +18.8% 52566 ± 2% softirqs.CPU23.SCHED 73463 ± 4% +25.3% 92068 ± 4% softirqs.CPU24.RCU 43552 ± 2% +21.6% 52956 softirqs.CPU24.SCHED 89944 ± 5% +27.3% 114465 softirqs.CPU24.TIMER 73945 +23.4% 91231 ± 2% softirqs.CPU25.RCU 43633 +21.9% 53174 ± 3% softirqs.CPU25.SCHED 89171 ± 4% +39.8% 124628 ± 20% softirqs.CPU25.TIMER 70343 ± 5% +29.4% 91058 ± 6% softirqs.CPU26.RCU 43709 ± 3% +20.8% 52784 ± 2% softirqs.CPU26.SCHED 72666 ± 3% +25.0% 90831 ± 6% softirqs.CPU27.RCU 43708 +20.6% 52709 softirqs.CPU27.SCHED 86809 ± 5% +24.5% 108113 ± 6% softirqs.CPU27.TIMER 71506 ± 3% +28.6% 91921 ± 5% softirqs.CPU28.RCU 43804 ± 2% +20.1% 52606 softirqs.CPU28.SCHED 73394 ± 3% +22.8% 90149 ± 5% softirqs.CPU29.RCU 43517 +20.8% 52588 softirqs.CPU29.SCHED 89568 ± 5% +22.1% 109326 ± 6% softirqs.CPU29.TIMER 73421 ± 2% +23.5% 90681 ± 4% softirqs.CPU3.RCU 44393 +21.5% 53943 ± 2% softirqs.CPU3.SCHED 89862 ± 4% +24.0% 111453 ± 2% softirqs.CPU3.TIMER 71663 ± 7% +29.0% 92457 ± 6% softirqs.CPU30.RCU 43602 ± 2% +20.8% 52683 ± 2% softirqs.CPU30.SCHED 73484 ± 2% +24.0% 91142 softirqs.CPU31.RCU 43471 +21.6% 52874 ± 2% softirqs.CPU31.SCHED 88937 ± 5% +26.5% 112542 ± 2% softirqs.CPU31.TIMER 64012 ± 4% +24.6% 79739 ± 3% softirqs.CPU32.RCU 44472 ± 5% +18.8% 52827 softirqs.CPU32.SCHED 61122 ± 2% +28.2% 78387 ± 3% softirqs.CPU33.RCU 44140 ± 3% +19.5% 52769 ± 2% softirqs.CPU33.SCHED 61441 +26.8% 77902 ± 5% softirqs.CPU34.RCU 43298 +21.4% 52570 ± 2% softirqs.CPU34.SCHED 90552 ± 6% +25.7% 113800 ± 3% softirqs.CPU34.TIMER 61123 ± 2% +30.8% 79962 ± 2% softirqs.CPU35.RCU 44252 ± 3% +19.5% 52863 ± 2% softirqs.CPU35.SCHED 59961 ± 3% +30.3% 78141 ± 4% softirqs.CPU36.RCU 43798 ± 2% +20.6% 52819 ± 2% softirqs.CPU36.SCHED 59914 +28.9% 77253 ± 2% softirqs.CPU37.RCU 44359 ± 3% +19.4% 52965 softirqs.CPU37.SCHED 60200 ± 3% +29.1% 77719 softirqs.CPU38.RCU 44765 ± 3% +18.8% 53159 ± 2% softirqs.CPU38.SCHED 60108 ± 2% +28.8% 77435 softirqs.CPU39.RCU 44041 +20.3% 52987 softirqs.CPU39.SCHED 88646 ± 4% +28.5% 113914 ± 3% softirqs.CPU39.TIMER 69245 ± 4% +31.6% 91096 ± 4% softirqs.CPU4.RCU 44364 ± 3% +19.7% 53100 ± 2% softirqs.CPU4.SCHED 69900 ± 3% +29.9% 90798 ± 3% softirqs.CPU5.RCU 44569 ± 3% +21.2% 54036 softirqs.CPU5.SCHED 71523 ± 4% +27.3% 91057 ± 5% softirqs.CPU6.RCU 43811 +21.1% 53061 ± 2% softirqs.CPU6.SCHED 90070 ± 4% +26.8% 114228 ± 2% softirqs.CPU6.TIMER 71179 ± 4% +28.2% 91267 ± 4% softirqs.CPU7.RCU 45398 ± 4% +19.0% 54043 softirqs.CPU7.SCHED 68555 ± 5% +32.1% 90537 ± 3% softirqs.CPU8.RCU 44109 ± 2% +19.9% 52878 ± 2% softirqs.CPU8.SCHED 69352 ± 3% +28.8% 89311 ± 3% softirqs.CPU9.RCU 44440 ± 3% +21.0% 53755 softirqs.CPU9.SCHED 2778312 ± 2% +27.7% 3546581 ± 3% softirqs.RCU 1764877 ± 2% +20.5% 2126626 ± 2% softirqs.SCHED 3831020 ± 12% +19.8% 4591281 ± 2% softirqs.TIMER 38.39 ± 4% -2.6 35.76 ± 3% perf-profile.calltrace.cycles-pp.write 37.65 ± 4% -2.5 35.17 ± 4% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 37.67 ± 4% -2.5 35.19 ± 4% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.77 ± 4% -2.5 35.30 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 37.77 ± 4% -2.5 35.30 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 37.75 ± 4% -2.5 35.28 ± 3% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 37.74 ± 4% -2.5 35.27 ± 4% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 36.01 ± 4% -2.4 33.62 ± 4% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 4.23 ± 8% -1.5 2.74 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 4.17 ± 8% -1.5 2.70 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node 6.75 ± 5% -1.1 5.60 ± 7% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper 6.75 ± 5% -1.1 5.60 ± 7% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io 2.65 ± 11% -1.0 1.60 ± 16% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 6.82 ± 5% -1.0 5.82 ± 4% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents 8.42 ± 5% -0.9 7.53 ± 6% perf-profile.calltrace.cycles-pp.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 2.46 ± 15% -0.9 1.58 ± 16% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node 8.12 ± 5% -0.9 7.27 ± 6% perf-profile.calltrace.cycles-pp.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper.process_one_work 4.06 ± 9% -0.9 3.21 ± 7% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.insert_reserved_file_extent 9.60 ± 4% -0.8 8.75 ± 4% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 9.58 ± 4% -0.8 8.74 ± 4% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread 1.25 ± 4% -0.8 0.43 ± 59% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 1.32 -0.8 0.56 ± 9% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 1.64 ± 6% -0.6 1.07 ± 28% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 2.04 ± 6% -0.5 1.50 ± 6% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent 5.54 ± 5% -0.5 5.08 ± 4% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 5.53 ± 5% -0.5 5.08 ± 4% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode 2.18 ± 4% -0.4 1.77 ± 6% perf-profile.calltrace.cycles-pp.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write 2.15 ± 4% -0.4 1.76 ± 6% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter 0.70 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.70 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file 2.16 ± 4% -0.4 1.77 ± 6% perf-profile.calltrace.cycles-pp.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.69 ± 7% -0.4 0.31 ±100% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_sync_log 2.12 ± 4% -0.4 1.74 ± 6% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops 2.13 ± 4% -0.4 1.75 ± 6% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range.start_ordered_ops.btrfs_sync_file 2.41 ± 7% -0.3 2.12 ± 6% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.__btrfs_drop_extents.btrfs_log_changed_extents 1.61 ± 4% -0.3 1.32 ± 11% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode.btrfs_commit_inode_delayed_items 1.59 ± 4% -0.3 1.31 ± 11% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode 1.29 ± 4% -0.3 1.02 ± 8% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range 0.70 ± 6% -0.3 0.43 ± 59% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 1.59 ± 8% -0.2 1.35 ± 5% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 2.14 ± 2% -0.2 1.90 ± 7% perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 2.14 ± 2% -0.2 1.91 ± 7% perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.btrfs_log_changed_extents 1.05 ± 7% -0.2 0.82 ± 8% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__filemap_fdatawrite_range 0.61 ± 8% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.btrfs_map_bio.btrfs_submit_bio_hook.submit_one_bio 1.12 ± 9% -0.2 0.94 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.80 ± 8% -0.2 0.62 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 1.08 ± 10% -0.2 0.91 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 0.84 ± 8% -0.2 0.68 ± 8% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.77 ± 9% -0.2 0.61 ± 10% perf-profile.calltrace.cycles-pp.cow_file_range.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 0.88 ± 4% -0.1 0.73 ± 4% perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_csum 0.81 ± 4% -0.1 0.71 ± 7% perf-profile.calltrace.cycles-pp.btrfs_submit_bio_hook.submit_one_bio.flush_write_bio.extent_writepages.do_writepages 0.82 ± 5% -0.1 0.72 ± 6% perf-profile.calltrace.cycles-pp.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range.btrfs_fdatawrite_range 0.81 ± 4% -0.1 0.72 ± 7% perf-profile.calltrace.cycles-pp.submit_one_bio.flush_write_bio.extent_writepages.do_writepages.__filemap_fdatawrite_range 2.63 ± 2% +0.2 2.85 ± 5% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 0.73 ± 6% +0.3 0.99 ± 6% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode.btrfs_log_inode_parent 2.79 ± 2% +0.3 3.07 ± 5% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.63 ± 2% +0.3 0.95 ± 8% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io 0.62 ± 2% +0.3 0.94 ± 8% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent 0.79 ± 3% +0.4 1.15 ± 9% perf-profile.calltrace.cycles-pp.setup_items_for_insert.__btrfs_drop_extents.insert_reserved_file_extent.btrfs_finish_ordered_io.normal_work_helper 2.34 ± 4% +0.4 2.71 ± 6% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 0.42 ± 58% +0.4 0.84 ± 6% perf-profile.calltrace.cycles-pp.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents.btrfs_log_inode 0.63 ± 8% +0.4 1.06 ± 6% perf-profile.calltrace.cycles-pp.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent 0.61 ± 8% +0.4 1.05 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode 0.40 ± 58% +0.4 0.84 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents.btrfs_log_changed_extents 2.74 ± 5% +0.5 3.28 ± 5% perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 0.28 ±100% +0.6 0.87 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up 0.30 ±100% +0.6 0.90 ± 7% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot 0.00 +0.6 0.59 ± 5% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function 0.31 ±100% +0.6 0.91 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.unlock_up.btrfs_search_slot.btrfs_insert_empty_items 0.00 +0.6 0.61 ± 6% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common 0.00 +0.6 0.62 ± 7% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock 7.53 ± 3% +0.8 8.32 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 0.00 +0.8 0.82 ± 10% perf-profile.calltrace.cycles-pp.unwind_next_frame.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity 7.34 ± 3% +0.8 8.17 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node 4.47 ± 3% +0.9 5.40 ± 9% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 4.33 ± 3% +0.9 5.28 ± 9% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node 0.00 +1.2 1.24 ± 6% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe 0.00 +1.3 1.31 ± 6% perf-profile.calltrace.cycles-pp.__save_stack_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 0.00 +1.3 1.31 ± 6% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert 0.00 +1.3 1.34 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.btrfs_unlock_up_safe.setup_items_for_insert.__btrfs_drop_extents 0.00 +1.4 1.37 ± 6% perf-profile.calltrace.cycles-pp.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate 0.80 ± 6% +1.7 2.55 ± 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +1.8 1.83 ± 4% perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending 0.27 ±100% +2.0 2.26 ± 5% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary 0.26 ±100% +2.0 2.25 ± 5% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry 0.00 +2.2 2.16 ± 6% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.do_idle 49.01 ± 3% +2.3 51.35 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 49.01 ± 3% +2.3 51.35 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 48.97 ± 3% +2.4 51.33 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 49.91 ± 2% +2.5 52.45 perf-profile.calltrace.cycles-pp.secondary_startup_64 38.39 ± 4% -2.6 35.77 ± 3% perf-profile.children.cycles-pp.write 37.67 ± 4% -2.5 35.19 ± 4% perf-profile.children.cycles-pp.new_sync_write 37.65 ± 4% -2.5 35.17 ± 4% perf-profile.children.cycles-pp.btrfs_file_write_iter 37.76 ± 4% -2.5 35.28 ± 3% perf-profile.children.cycles-pp.ksys_write 37.74 ± 4% -2.5 35.27 ± 4% perf-profile.children.cycles-pp.vfs_write 36.01 ± 4% -2.4 33.62 ± 4% perf-profile.children.cycles-pp.btrfs_sync_file 36.52 ± 3% -2.3 34.18 ± 2% perf-profile.children.cycles-pp.btrfs_search_slot 19.62 ± 2% -2.1 17.56 ± 2% perf-profile.children.cycles-pp.btrfs_read_lock_root_node 19.50 ± 2% -2.0 17.47 ± 2% perf-profile.children.cycles-pp.btrfs_tree_read_lock 8.71 ± 2% -1.9 6.79 ± 3% perf-profile.children.cycles-pp.finish_wait 38.42 ± 4% -1.9 36.52 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 38.41 ± 4% -1.9 36.52 ± 2% perf-profile.children.cycles-pp.do_syscall_64 12.40 ± 4% -1.5 10.86 ± 4% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 14.73 ± 3% -1.1 13.60 ± 3% perf-profile.children.cycles-pp.__btrfs_drop_extents 8.43 ± 5% -0.9 7.53 ± 6% perf-profile.children.cycles-pp.insert_reserved_file_extent 9.60 ± 4% -0.8 8.75 ± 4% perf-profile.children.cycles-pp.normal_work_helper 9.58 ± 4% -0.8 8.74 ± 4% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 2.97 ± 5% -0.8 2.14 ± 4% perf-profile.children.cycles-pp.queued_read_lock_slowpath 3.06 ± 4% -0.7 2.34 ± 3% perf-profile.children.cycles-pp.queued_write_lock_slowpath 2.92 ± 4% -0.5 2.38 ± 6% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 2.89 ± 5% -0.5 2.37 ± 6% perf-profile.children.cycles-pp.do_writepages 2.08 ± 8% -0.4 1.64 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_inode 2.18 ± 8% -0.4 1.75 ± 8% perf-profile.children.cycles-pp.__btrfs_update_delayed_inode 2.84 ± 4% -0.4 2.42 ± 2% perf-profile.children.cycles-pp.__schedule 2.18 ± 4% -0.4 1.77 ± 6% perf-profile.children.cycles-pp.start_ordered_ops 2.18 ± 4% -0.4 1.78 ± 6% perf-profile.children.cycles-pp.btrfs_fdatawrite_range 2.18 ± 5% -0.4 1.80 ± 5% perf-profile.children.cycles-pp.extent_writepages 1.33 ± 5% -0.3 1.06 ± 7% perf-profile.children.cycles-pp.extent_write_cache_pages 1.75 -0.3 1.49 ± 4% perf-profile.children.cycles-pp.schedule 1.06 ± 8% -0.2 0.83 ± 8% perf-profile.children.cycles-pp.__extent_writepage 0.66 ± 14% -0.2 0.43 ± 13% perf-profile.children.cycles-pp.btrfs_next_old_leaf 0.83 ± 2% -0.2 0.62 ± 7% perf-profile.children.cycles-pp.update_load_avg 1.14 ± 8% -0.2 0.95 perf-profile.children.cycles-pp.schedule_idle 0.80 ± 8% -0.2 0.62 ± 10% perf-profile.children.cycles-pp.btrfs_run_delalloc_range 0.30 ± 24% -0.2 0.12 ± 4% perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode 0.84 ± 8% -0.2 0.68 ± 8% perf-profile.children.cycles-pp.writepage_delalloc 1.24 ± 7% -0.2 1.08 ± 8% perf-profile.children.cycles-pp.flush_write_bio 0.77 ± 9% -0.2 0.61 ± 10% perf-profile.children.cycles-pp.cow_file_range 0.49 ± 6% -0.2 0.33 ± 10% perf-profile.children.cycles-pp.__list_del_entry_valid 1.04 ± 9% -0.1 0.89 ± 8% perf-profile.children.cycles-pp.generic_make_request 0.70 ± 7% -0.1 0.55 ± 12% perf-profile.children.cycles-pp.btree_write_cache_pages 0.70 ± 6% -0.1 0.56 ± 11% perf-profile.children.cycles-pp.btrfs_write_marked_extents 1.04 ± 9% -0.1 0.90 ± 8% perf-profile.children.cycles-pp.submit_bio 0.62 ± 4% -0.1 0.48 ± 6% perf-profile.children.cycles-pp.read_block_for_search 0.86 ± 11% -0.1 0.72 ± 8% perf-profile.children.cycles-pp.brd_make_request 0.51 ± 7% -0.1 0.40 ± 9% perf-profile.children.cycles-pp.select_task_rq_fair 0.84 ± 4% -0.1 0.74 ± 6% perf-profile.children.cycles-pp.btrfs_submit_bio_hook 0.49 ± 10% -0.1 0.39 ± 6% perf-profile.children.cycles-pp.brd_do_bvec 0.77 ± 6% -0.1 0.67 ± 4% perf-profile.children.cycles-pp.dequeue_entity 0.61 ± 8% -0.1 0.54 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 0.27 ± 10% -0.1 0.20 ± 16% perf-profile.children.cycles-pp.map_private_extent_buffer 0.40 ± 8% -0.1 0.33 ± 5% perf-profile.children.cycles-pp.find_extent_buffer 0.25 ± 4% -0.1 0.18 ± 13% perf-profile.children.cycles-pp.__update_load_avg_se 0.21 ± 2% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.write_one_eb 0.09 ± 13% -0.1 0.03 ±100% perf-profile.children.cycles-pp.create_io_em 0.21 ± 13% -0.1 0.15 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_unlock 0.35 ± 14% -0.1 0.30 ± 9% perf-profile.children.cycles-pp.__list_add_valid 0.21 ± 10% -0.1 0.15 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.10 ± 22% -0.0 0.06 ± 14% perf-profile.children.cycles-pp.xas_load 0.12 ± 19% -0.0 0.08 ± 16% perf-profile.children.cycles-pp.__test_set_page_writeback 0.16 ± 15% -0.0 0.12 ± 17% perf-profile.children.cycles-pp.block_group_cache_tree_search 0.29 ± 5% -0.0 0.25 ± 7% perf-profile.children.cycles-pp.memmove 0.29 ± 10% -0.0 0.25 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc 0.26 ± 7% -0.0 0.22 ± 8% perf-profile.children.cycles-pp._raw_read_lock 0.18 ± 8% -0.0 0.15 ± 7% perf-profile.children.cycles-pp.__switch_to 0.13 ± 12% -0.0 0.10 ± 7% perf-profile.children.cycles-pp.update_cfs_rq_h_load 0.16 ± 6% -0.0 0.13 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.11 ± 17% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64 0.11 ± 7% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.btrfs_set_extent_delalloc 0.25 ± 7% +0.0 0.28 ± 2% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.24 ± 9% +0.0 0.28 ± 2% perf-profile.children.cycles-pp.copy_page 0.13 ± 9% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.queue_work_on 0.13 ± 12% +0.1 0.20 ± 9% perf-profile.children.cycles-pp.__queue_work 0.07 ± 58% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.00 +0.1 0.08 ± 34% perf-profile.children.cycles-pp.in_sched_functions 0.04 ± 57% +0.1 0.12 ± 10% perf-profile.children.cycles-pp.reschedule_interrupt 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.stack_access_ok 0.09 ± 37% +0.1 0.19 ± 17% perf-profile.children.cycles-pp.tick_irq_enter 0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.scheduler_ipi 0.15 ± 22% +0.1 0.26 ± 15% perf-profile.children.cycles-pp.irq_enter 0.58 ± 3% +0.1 0.71 ± 15% perf-profile.children.cycles-pp.update_process_times 1.26 ± 2% +0.2 1.43 ± 8% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.43 ± 8% +0.2 0.61 ± 12% perf-profile.children.cycles-pp.btrfs_release_path 0.00 +0.2 0.22 ± 12% perf-profile.children.cycles-pp.save_stack_address 1.87 ± 2% +0.2 2.11 ± 7% perf-profile.children.cycles-pp.hrtimer_interrupt 0.00 +0.3 0.26 ± 16% perf-profile.children.cycles-pp.__module_text_address 0.00 +0.3 0.29 ± 13% perf-profile.children.cycles-pp.is_module_text_address 0.00 +0.3 0.29 ± 4% perf-profile.children.cycles-pp.__unwind_start 3.26 ± 2% +0.3 3.61 ± 4% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.00 +0.4 0.40 ± 10% perf-profile.children.cycles-pp.kernel_text_address 3.47 ± 3% +0.4 3.90 ± 3% perf-profile.children.cycles-pp.apic_timer_interrupt 0.00 +0.4 0.43 ± 9% perf-profile.children.cycles-pp.__kernel_text_address 0.00 +0.5 0.46 ± 9% perf-profile.children.cycles-pp.unwind_get_return_address 0.00 +0.5 0.47 ± 7% perf-profile.children.cycles-pp.__module_address 0.78 ± 5% +0.5 1.25 ± 5% perf-profile.children.cycles-pp.unlock_up 1.80 ± 3% +0.6 2.42 ± 5% perf-profile.children.cycles-pp.setup_items_for_insert 1.26 +0.7 1.94 ± 4% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.00 +0.8 0.85 ± 5% perf-profile.children.cycles-pp.__orc_find 0.00 +1.1 1.12 ± 4% perf-profile.children.cycles-pp.orc_find 1.87 ± 4% +1.4 3.30 ± 2% perf-profile.children.cycles-pp.__wake_up_common 1.80 ± 4% +1.4 3.23 ± 3% perf-profile.children.cycles-pp.autoremove_wake_function 1.83 ± 4% +1.6 3.41 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 2.43 ± 4% +1.6 4.06 perf-profile.children.cycles-pp.__wake_up_common_lock 14.82 ± 3% +1.7 16.56 ± 2% perf-profile.children.cycles-pp.prepare_to_wait_event 0.85 ± 4% +1.9 2.71 ± 2% perf-profile.children.cycles-pp.sched_ttwu_pending 0.00 +2.0 1.96 ± 5% perf-profile.children.cycles-pp.unwind_next_frame 49.01 ± 3% +2.3 51.35 ± 2% perf-profile.children.cycles-pp.start_secondary 49.91 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.secondary_startup_64 49.91 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.cpu_startup_entry 49.90 ± 2% +2.5 52.45 perf-profile.children.cycles-pp.do_idle 0.00 +2.8 2.78 ± 4% perf-profile.children.cycles-pp.__save_stack_trace 0.00 +2.9 2.90 ± 4% perf-profile.children.cycles-pp.save_stack_trace_tsk 1.02 ± 10% +3.7 4.73 ± 2% perf-profile.children.cycles-pp.ttwu_do_activate 0.82 ± 9% +3.7 4.54 ± 2% perf-profile.children.cycles-pp.enqueue_entity 0.97 ± 9% +3.7 4.69 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair 0.00 +3.8 3.84 ± 2% perf-profile.children.cycles-pp.__account_scheduler_latency 0.54 ± 15% -0.2 0.35 ± 5% perf-profile.self.cycles-pp.btrfs_tree_read_lock 0.48 ± 9% -0.2 0.30 ± 7% perf-profile.self.cycles-pp.prepare_to_wait_event 0.49 ± 5% -0.2 0.32 ± 11% perf-profile.self.cycles-pp.__list_del_entry_valid 0.74 ± 7% -0.2 0.58 perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.35 ± 8% -0.2 0.20 ± 7% perf-profile.self.cycles-pp.btrfs_tree_lock 0.48 ± 4% -0.1 0.37 ± 6% perf-profile.self.cycles-pp.__schedule 0.69 ± 7% -0.1 0.58 ± 4% perf-profile.self.cycles-pp._raw_spin_lock 0.44 ± 6% -0.1 0.33 ± 6% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.27 ± 6% -0.1 0.19 ± 16% perf-profile.self.cycles-pp.update_load_avg 0.23 ± 13% -0.1 0.17 ± 18% perf-profile.self.cycles-pp.brd_do_bvec 0.24 ± 6% -0.1 0.17 ± 11% perf-profile.self.cycles-pp.__update_load_avg_se 0.32 ± 2% -0.1 0.26 ± 7% perf-profile.self.cycles-pp.try_to_wake_up 0.29 ± 9% -0.1 0.23 ± 10% perf-profile.self.cycles-pp.select_task_rq_fair 0.24 ± 9% -0.1 0.19 ± 15% perf-profile.self.cycles-pp.map_private_extent_buffer 0.21 ± 14% -0.1 0.15 ± 16% perf-profile.self.cycles-pp.btrfs_tree_read_unlock 0.35 ± 14% -0.1 0.30 ± 9% perf-profile.self.cycles-pp.__list_add_valid 0.26 ± 7% -0.0 0.21 ± 8% perf-profile.self.cycles-pp._raw_read_lock 0.17 ± 9% -0.0 0.14 ± 8% perf-profile.self.cycles-pp.__switch_to 0.13 ± 12% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.update_cfs_rq_h_load 0.28 ± 6% -0.0 0.25 ± 7% perf-profile.self.cycles-pp.memmove 0.14 ± 8% -0.0 0.11 ± 9% perf-profile.self.cycles-pp.___perf_sw_event 0.19 ± 9% +0.0 0.22 ± 11% perf-profile.self.cycles-pp.enqueue_entity 0.00 +0.1 0.06 ± 16% perf-profile.self.cycles-pp.__unwind_start 0.00 +0.1 0.07 ± 17% perf-profile.self.cycles-pp.kernel_text_address 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.__save_stack_trace 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.stack_access_ok 0.00 +0.1 0.12 ± 13% perf-profile.self.cycles-pp.save_stack_trace_tsk 0.00 +0.1 0.14 ± 21% perf-profile.self.cycles-pp.orc_find 0.00 +0.1 0.14 ± 18% perf-profile.self.cycles-pp.save_stack_address 0.00 +0.5 0.46 ± 7% perf-profile.self.cycles-pp.__module_address 0.00 +0.6 0.61 ± 7% perf-profile.self.cycles-pp.__account_scheduler_latency 0.00 +0.6 0.62 ± 7% perf-profile.self.cycles-pp.unwind_next_frame 0.00 +0.8 0.84 ± 5% perf-profile.self.cycles-pp.__orc_find Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

2 years, 5 months

2
2
0 / 0

Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression

by Xing Zhengjun

On 5/30/2019 10:00 AM, Trond Myklebust wrote: > Hi Xing, > > On Thu, 2019-05-30 at 09:35 +0800, Xing Zhengjun wrote: >> Hi Trond, >> >> On 5/20/2019 1:54 PM, kernel test robot wrote: >>> Greeting, >>> >>> FYI, we noticed a 16.0% improvement of fsmark.app_overhead due to >>> commit: >>> >>> >>> commit: 0472e476604998c127f3c80d291113e77c5676ac ("SUNRPC: Convert >>> socket page send code to use iov_iter()") >>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git >>> master >>> >>> in testcase: fsmark >>> on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ >>> 3.00GHz with 384G memory >>> with following parameters: >>> >>> iterations: 1x >>> nr_threads: 64t >>> disk: 1BRD_48G >>> fs: xfs >>> fs2: nfsv4 >>> filesize: 4M >>> test_size: 40G >>> sync_method: fsyncBeforeClose >>> cpufreq_governor: performance >>> >>> test-description: The fsmark is a file system benchmark to test >>> synchronous write workloads, for example, mail servers workload. >>> test-url: https://sourceforge.net/projects/fsmark/ >>> >>> >>> >>> Details are as below: >>> ----------------------------------------------------------------- >>> ---------------------------------> >>> >>> >>> To reproduce: >>> >>> git clone https://github.com/intel/lkp-tests.git >>> cd lkp-tests >>> bin/lkp install job.yaml # job file is attached in this >>> email >>> bin/lkp run job.yaml >>> >>> =================================================================== >>> ====================== >>> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/n >>> r_threads/rootfs/sync_method/tbox_group/test_size/testcase: >>> gcc-7/performance/1BRD_48G/4M/nfsv4/xfs/1x/x86_64-rhel- >>> 7.6/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/lkp-ivb- >>> ep01/40G/fsmark >>> >>> commit: >>> e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use >>> iov_iter_kvec()") >>> 0472e47660 ("SUNRPC: Convert socket page send code to use >>> iov_iter()") >>> >>> e791f8e9380d945e 0472e476604998c127f3c80d291 >>> ---------------- --------------------------- >>> fail:runs %reproduction fail:runs >>> | | | >>> :4 50% 2:4 dmesg.WARNING:at#for >>> _ip_interrupt_entry/0x >>> %stddev %change %stddev >>> \ | \ >>> 15118573 ± 2% +16.0% 17538083 fsmark.app_overhead >>> 510.93 -22.7% 395.12 fsmark.files_per_sec >>> 24.90 +22.8% 30.57 fsmark.time.elapsed_ >>> time >>> 24.90 +22.8% 30.57 fsmark.time.elapsed_ >>> time.max >>> 288.00 ± 2% - >>> 27.8% 208.00 fsmark.time.percent_of_cpu_this_job_got >>> 70.03 ± 2% - >>> 11.3% 62.14 fsmark.time.system_time >>> >> >> Do you have time to take a look at this regression? > > From your stats, it looks to me as if the problem is increased NUMA > overhead. Pretty much everything else appears to be the same or > actually performing better than previously. Am I interpreting that > correctly? The real regression is the throughput(fsmark.files_per_sec) is decreased by 22.7%. > > If my interpretation above is correct, then I'm not seeing where this > patch would be introducing new NUMA regressions. It is just converting > from using one method of doing socket I/O to another. Could it perhaps > be a memory artefact due to your running the NFS client and server on > the same machine? > > Apologies for pushing back a little, but I just don't have the > hardware available to test NUMA configurations, so I'm relying on > external testing for the above kind of scenario. > Thanks for looking at this. If you need more information, please let me know. > Thanks > Trond > -- Zhengjun Xing

2 years, 5 months

1
0
0 / 0