July 2019 - LKP - Ml01.01.Org

[btrfs] 8d47a0d8f7: fio.write_bw_MBps -28.6% regression

by kernel test robot

Greeting, FYI, we noticed a -28.6% regression of fio.write_bw_MBps due to commit: commit: 8d47a0d8f7947422dd359ac8e462687f81a7a137 ("btrfs: Do mandatory tree block check before submitting bio") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory with following parameters: disk: 2pmem fs: btrfs runtime: 200s nr_task: 50% time_based: tb rw: randwrite bs: 4k ioengine: libaio test_size: 100G cpufreq_governor: performance ucode: 0x3d test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 4k/gcc-7/performance/2pmem/btrfs/libaio/x86_64-rhel-7.6/50%/debian-x86_64-2018-04-03.cgz/200s/randwrite/lkp-hsw-ep2/100G/fio-basic/tb/0x3d commit: ff2ac107fa ("btrfs: tree-checker: Remove comprehensive root owner check") 8d47a0d8f7 ("btrfs: Do mandatory tree block check before submitting bio") ff2ac107fae2440b 8d47a0d8f7947422dd359ac8e46 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.BUG:unable_to_handle_kernel :4 25% 1:4 dmesg.Kernel_panic-not_syncing:Fatal_exception_in_interrupt :4 25% 1:4 dmesg.Oops:#[##] :4 25% 1:4 dmesg.RIP:cpuidle_enter_state :4 25% 1:4 dmesg.RIP:native_write_msr :4 25% 1:4 dmesg.RIP:perf_prepare_sample 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 0.01 ± 21% +0.3 0.28 ± 20% fio.latency_1000ms% 4.14 ± 13% +1.8 5.95 ± 9% fio.latency_10ms% 8.73 ± 15% +3.0 11.68 ± 8% fio.latency_20ms% 1.91 ± 5% -0.4 1.52 ± 3% fio.latency_250ms% 66.81 -5.9 60.90 ± 2% fio.latency_4ms% 0.18 ± 15% +0.1 0.29 ± 16% fio.latency_750us% 1.017e+08 -28.5% 72701525 fio.time.file_system_outputs 109.75 -26.8% 80.33 ± 2% fio.time.percent_of_cpu_this_job_got 197.97 -26.4% 145.79 ± 2% fio.time.system_time 38.66 ± 4% -24.0% 29.39 ± 3% fio.time.user_time 12388015 -28.5% 8858878 fio.time.voluntary_context_switches 12711059 -28.5% 9083954 fio.workload 247.91 -28.6% 176.96 fio.write_bw_MBps 309248 ± 2% +71.7% 531114 ± 4% fio.write_clat_99%_us 17574 +40.0% 24609 fio.write_clat_mean_us 57989 ± 4% +82.2% 105661 fio.write_clat_stddev 63463 -28.6% 45302 fio.write_iops 564.61 +40.2% 791.54 fio.write_slat_mean_us 9375 ± 7% +96.3% 18406 fio.write_slat_stddev 21.25 +8.5% 23.06 ± 9% boot-time.dhcp 93.32 +1.4% 94.58 iostat.cpu.idle 6.43 -18.6% 5.23 ± 2% iostat.cpu.system 6.48 -1.2 5.27 ± 2% mpstat.cpu.all.sys% 0.24 ± 5% -0.1 0.18 ± 2% mpstat.cpu.all.usr% 93.00 +1.1% 94.00 vmstat.cpu.id 591200 -30.0% 413921 vmstat.io.bo 31653776 ± 2% -10.9% 28196588 ± 2% vmstat.memory.cache 11927341 ± 7% +29.8% 15486805 ± 4% vmstat.memory.free 5.00 -26.7% 3.67 ± 12% vmstat.procs.r 635910 ± 2% -27.6% 460384 vmstat.system.cs 7822384 ± 2% -25.0% 5870498 numa-numastat.node0.local_node 7839861 ± 2% -24.8% 5893785 numa-numastat.node0.numa_hit 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node0.numa_miss 242465 ± 44% -84.3% 37964 ± 18% numa-numastat.node0.other_node 7762623 ± 12% -26.8% 5681358 ± 6% numa-numastat.node1.local_node 224988 ± 47% -93.5% 14675 ± 48% numa-numastat.node1.numa_foreign 7768618 ± 12% -26.9% 5681573 ± 6% numa-numastat.node1.numa_hit 9.429e+08 ± 10% -35.4% 6.094e+08 cpuidle.C1.time 53504154 ± 6% -31.8% 36504731 cpuidle.C1.usage 1.093e+09 ± 22% -50.3% 5.435e+08 ± 3% cpuidle.C1E.time 16541925 ± 13% -45.0% 9103775 cpuidle.C1E.usage 5.366e+09 ± 36% +71.7% 9.214e+09 ± 18% cpuidle.C3.time 16163664 ± 33% +57.0% 25379738 ± 5% cpuidle.C3.usage 9697055 ± 16% -35.8% 6225168 ± 19% cpuidle.C6.usage 19313595 ± 17% -16.3% 16166123 cpuidle.POLL.time 10291838 ± 9% -26.8% 7537045 ± 7% meminfo.Active 10034782 ± 9% -27.4% 7280710 ± 7% meminfo.Active(file) 31524905 ± 2% -10.9% 28082249 ± 2% meminfo.Cached 115157 ± 4% +61.1% 185560 ± 3% meminfo.CmaFree 6430720 ± 8% +19.1% 7660885 ± 6% meminfo.DirectMap2M 534230 ± 5% -12.8% 465932 ± 6% meminfo.Dirty 11737322 ± 7% +30.2% 15281189 ± 4% meminfo.MemFree 37604344 ± 2% -9.4% 34060483 meminfo.Memused 806.50 ± 37% +137.4% 1914 ± 8% meminfo.Writeback 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.active_objs 370.50 ± 11% +36.8% 507.00 slabinfo.bdev_cache.num_objs 15147 ± 5% -8.3% 13897 ± 2% slabinfo.blkdev_ioc.active_objs 17839 ± 6% -11.7% 15745 ± 2% slabinfo.blkdev_ioc.num_objs 12178 ± 4% -7.4% 11275 ± 2% slabinfo.kmalloc-512.num_objs 7794 ± 3% +25.4% 9774 slabinfo.proc_inode_cache.active_objs 8017 ± 2% +23.4% 9895 slabinfo.proc_inode_cache.num_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.active_objs 2366 ± 23% -32.6% 1595 ± 5% slabinfo.task_group.num_objs 263.75 ± 2% -15.1% 224.00 ± 2% turbostat.Avg_MHz 9.46 ± 2% -1.4 8.07 ± 2% turbostat.Busy% 53502248 ± 6% -31.8% 36503898 turbostat.C1 5.86 ± 8% -2.0 3.82 turbostat.C1% 16540396 ± 13% -45.0% 9102733 turbostat.C1E 6.80 ± 22% -3.4 3.40 ± 4% turbostat.C1E% 16163332 ± 33% +57.0% 25379351 ± 5% turbostat.C3 33.55 ± 36% +24.1 57.61 ± 18% turbostat.C3% 9689445 ± 16% -35.8% 6221033 ± 19% turbostat.C6 18.04 ± 26% +49.1% 26.90 ± 9% turbostat.CPU%c3 53.25 ± 3% +10.8% 59.00 ± 4% turbostat.CoreTmp 4.67 ± 14% +62.7% 7.59 ± 26% turbostat.Pkg%pc2 56.50 +9.7% 62.00 ± 2% turbostat.PkgTmp 11.84 -4.8% 11.28 turbostat.RAMWatt 7232478 ± 10% -37.2% 4543037 ± 22% numa-meminfo.node0.Active 7072564 ± 10% -37.6% 4410135 ± 23% numa-meminfo.node0.Active(file) 308167 ± 6% -24.7% 232152 ± 10% numa-meminfo.node0.Dirty 8784028 ± 13% +21.7% 10686327 ± 2% numa-meminfo.node0.Inactive 8669548 ± 12% +20.7% 10465444 ± 2% numa-meminfo.node0.Inactive(file) 572.50 ± 22% +115.9% 1236 ± 25% numa-meminfo.node0.Writeback 15098140 ± 3% -17.7% 12425670 ± 3% numa-meminfo.node1.FilePages 11599549 ± 7% -22.3% 9018051 ± 4% numa-meminfo.node1.Inactive 105067 ± 92% -99.3% 725.33 ± 51% numa-meminfo.node1.Inactive(anon) 11494482 ± 7% -21.6% 9017325 ± 4% numa-meminfo.node1.Inactive(file) 5504 ± 9% -12.5% 4819 ± 2% numa-meminfo.node1.KernelStack 6709938 ± 11% +45.8% 9781880 ± 5% numa-meminfo.node1.MemFree 18029136 ± 4% -17.0% 14957200 ± 3% numa-meminfo.node1.MemUsed 4427 ± 69% -77.5% 996.00 ± 8% numa-meminfo.node1.PageTables 857660 ± 32% -37.2% 538646 ± 8% numa-meminfo.node1.SUnreclaim 109232 ± 88% -96.1% 4274 ± 10% numa-meminfo.node1.Shmem 986740 ± 35% -40.0% 592452 ± 10% numa-meminfo.node1.Slab 575.21 ± 15% +30.1% 748.58 ± 2% sched_debug.cfs_rq:/.runnable_load_avg.max 100.45 ± 15% +28.6% 129.19 ± 5% sched_debug.cfs_rq:/.runnable_load_avg.stddev 830.19 ± 9% +15.7% 960.25 ± 6% sched_debug.cfs_rq:/.util_avg.max 29.97 ± 16% -25.2% 22.43 ± 31% sched_debug.cfs_rq:/.util_est_enqueued.avg 591.17 ± 13% -22.2% 459.83 ± 18% sched_debug.cfs_rq:/.util_est_enqueued.max 107.52 ± 13% -23.0% 82.84 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock.stddev 2.06 ± 5% -9.3% 1.87 sched_debug.cpu.clock_task.stddev 24.51 ± 7% +21.5% 29.77 ± 8% sched_debug.cpu.cpu_load[0].avg 541.29 ± 15% +38.3% 748.58 ± 2% sched_debug.cpu.cpu_load[0].max 93.97 ± 7% +27.8% 120.09 ± 2% sched_debug.cpu.cpu_load[0].stddev 519.08 ± 6% +40.5% 729.42 ± 4% sched_debug.cpu.cpu_load[1].max 87.45 ± 5% +28.5% 112.39 ± 4% sched_debug.cpu.cpu_load[1].stddev 480.83 ± 7% +39.5% 670.92 ± 13% sched_debug.cpu.cpu_load[2].max 80.92 ± 5% +25.7% 101.75 ± 8% sched_debug.cpu.cpu_load[2].stddev 408.83 ± 4% +54.0% 629.58 ± 19% sched_debug.cpu.cpu_load[3].max 71.62 ± 2% +30.7% 93.61 ± 11% sched_debug.cpu.cpu_load[3].stddev 381.50 ± 8% +66.7% 636.00 ± 18% sched_debug.cpu.cpu_load[4].max 64.40 ± 6% +39.7% 89.96 ± 12% sched_debug.cpu.cpu_load[4].stddev 106.75 ± 20% -88.4% 12.33 ± 10% proc-vmstat.kswapd_high_wmark_hit_quickly 2508660 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_active_file 33315360 -30.4% 23192301 proc-vmstat.nr_dirtied 133447 ± 5% -12.6% 116580 ± 6% proc-vmstat.nr_dirty 7881043 ± 2% -10.9% 7019533 ± 2% proc-vmstat.nr_file_pages 28790 ± 4% +61.1% 46392 ± 3% proc-vmstat.nr_free_cma 2934503 ± 7% +30.2% 3821369 ± 4% proc-vmstat.nr_free_pages 5044168 -3.4% 4872965 proc-vmstat.nr_inactive_file 430282 -6.0% 404589 proc-vmstat.nr_slab_unreclaimable 197.75 ± 38% +136.5% 467.67 ± 10% proc-vmstat.nr_writeback 33243480 -30.4% 23143756 proc-vmstat.nr_written 2508665 ± 9% -27.5% 1819662 ± 7% proc-vmstat.nr_zone_active_file 5044399 -3.4% 4873026 proc-vmstat.nr_zone_inactive_file 134972 ± 5% -12.4% 118262 ± 5% proc-vmstat.nr_zone_write_pending 15632301 ± 5% -25.8% 11598535 ± 3% proc-vmstat.numa_hit 15608826 ± 5% -25.8% 11575029 ± 3% proc-vmstat.numa_local 2300530 ± 10% -31.1% 1585363 ± 10% proc-vmstat.pgactivate 16955745 -26.3% 12499611 ± 2% proc-vmstat.pgalloc_normal 17488000 -25.9% 12959812 ± 2% proc-vmstat.pgfree 1.33e+08 -30.4% 92575440 proc-vmstat.pgpgout 5728177 ± 6% -78.5% 1232433 ± 25% proc-vmstat.pgscan_kswapd 5727967 ± 6% -78.5% 1232244 ± 25% proc-vmstat.pgsteal_kswapd 32760 -7.6% 30285 proc-vmstat.slabs_scanned 1768211 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_active_file 10410188 ± 7% -32.5% 7023939 ± 7% numa-vmstat.node0.nr_dirtied 76995 ± 6% -24.6% 58035 ± 10% numa-vmstat.node0.nr_dirty 2167457 ± 12% +20.7% 2616147 ± 2% numa-vmstat.node0.nr_inactive_file 142.00 ± 24% +120.4% 313.00 ± 24% numa-vmstat.node0.nr_writeback 10308758 ± 7% -32.6% 6949497 ± 7% numa-vmstat.node0.nr_written 1768216 ± 10% -37.7% 1102275 ± 23% numa-vmstat.node0.nr_zone_active_file 2167563 ± 12% +20.7% 2616189 ± 2% numa-vmstat.node0.nr_zone_inactive_file 77786 ± 7% -24.3% 58899 ± 10% numa-vmstat.node0.nr_zone_write_pending 5889240 ± 3% -14.3% 5046874 ± 3% numa-vmstat.node0.numa_hit 5837851 ± 4% -14.0% 5023120 ± 3% numa-vmstat.node0.numa_local 68376 ± 48% -96.9% 2149 ± 46% numa-vmstat.node0.numa_miss 119766 ± 42% -78.4% 25904 ± 3% numa-vmstat.node0.numa_other 30.50 ± 38% -100.0% 0.00 numa-vmstat.node0.workingset_nodes 6829126 ± 6% -22.8% 5269898 ± 10% numa-vmstat.node1.nr_dirtied 3774645 ± 3% -17.7% 3106081 ± 3% numa-vmstat.node1.nr_file_pages 28796 ± 4% +61.1% 46405 ± 3% numa-vmstat.node1.nr_free_cma 1677332 ± 11% +45.8% 2445778 ± 5% numa-vmstat.node1.nr_free_pages 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_inactive_anon 2873702 ± 7% -21.6% 2254116 ± 4% numa-vmstat.node1.nr_inactive_file 5506 ± 9% -12.5% 4820 ± 2% numa-vmstat.node1.nr_kernel_stack 29031 ± 85% -88.2% 3436 numa-vmstat.node1.nr_mapped 1106 ± 69% -77.5% 249.00 ± 8% numa-vmstat.node1.nr_page_table_pages 27308 ± 88% -96.1% 1068 ± 10% numa-vmstat.node1.nr_shmem 214418 ± 32% -37.2% 134663 ± 8% numa-vmstat.node1.nr_slab_unreclaimable 75.00 ± 35% +145.3% 184.00 ± 18% numa-vmstat.node1.nr_writeback 6761224 ± 6% -23.1% 5202586 ± 10% numa-vmstat.node1.nr_written 26266 ± 92% -99.3% 180.67 ± 51% numa-vmstat.node1.nr_zone_inactive_anon 2873810 ± 7% -21.6% 2254135 ± 4% numa-vmstat.node1.nr_zone_inactive_file 68381 ± 48% -96.9% 2152 ± 46% numa-vmstat.node1.numa_foreign 5035283 ± 15% -26.9% 3679318 ± 7% numa-vmstat.node1.numa_hit 4905249 ± 15% -28.2% 3522229 ± 8% numa-vmstat.node1.numa_local 18.43 ± 6% -28.3% 13.21 ± 3% perf-stat.i.MPKI 2.205e+09 ± 3% +10.5% 2.437e+09 perf-stat.i.branch-instructions 1.83 ± 5% -0.6 1.24 ± 7% perf-stat.i.branch-miss-rate% 37019551 ± 3% -29.4% 26140844 ± 5% perf-stat.i.branch-misses 28932020 ± 6% -31.1% 19933819 ± 3% perf-stat.i.cache-misses 2.03e+08 ± 4% -24.8% 1.526e+08 perf-stat.i.cache-references 645248 ± 2% -27.4% 468301 perf-stat.i.context-switches 1.97 ± 7% -22.1% 1.53 ± 2% perf-stat.i.cpi 1.983e+10 ± 3% -15.1% 1.684e+10 perf-stat.i.cpu-cycles 1.03 ± 5% -0.4 0.64 ± 10% perf-stat.i.dTLB-load-miss-rate% 31879502 ± 6% -37.9% 19807598 ± 9% perf-stat.i.dTLB-load-misses 3.201e+09 ± 3% +18.8% 3.804e+09 perf-stat.i.dTLB-loads 0.13 ± 7% -0.0 0.09 ± 10% perf-stat.i.dTLB-store-miss-rate% 2516181 ± 10% -28.3% 1805272 ± 10% perf-stat.i.dTLB-store-misses 2.143e+09 ± 3% +19.8% 2.568e+09 ± 2% perf-stat.i.dTLB-stores 13781489 ± 3% -28.4% 9866239 ± 3% perf-stat.i.iTLB-loads 1.164e+10 ± 3% +21.6% 1.415e+10 perf-stat.i.instructions 8435 ± 17% +42.5% 12023 ± 14% perf-stat.i.instructions-per-iTLB-miss 0.57 ± 3% +43.6% 0.82 perf-stat.i.ipc 16050361 ± 9% -30.7% 11127721 ± 3% perf-stat.i.node-load-misses 8120610 ± 9% -31.3% 5580996 perf-stat.i.node-loads 2591649 ± 13% -31.1% 1785932 ± 6% perf-stat.i.node-store-misses 1809775 ± 12% -31.2% 1244360 ± 2% perf-stat.i.node-stores 17.45 ± 4% -38.2% 10.79 perf-stat.overall.MPKI 1.68 ± 3% -0.6 1.07 ± 5% perf-stat.overall.branch-miss-rate% 1.70 ± 3% -30.2% 1.19 perf-stat.overall.cpi 689.69 ± 9% +22.6% 845.48 ± 2% perf-stat.overall.cycles-between-cache-misses 0.99 ± 5% -0.5 0.52 ± 10% perf-stat.overall.dTLB-load-miss-rate% 0.12 ± 8% -0.0 0.07 ± 10% perf-stat.overall.dTLB-store-miss-rate% 10.81 ± 14% +2.3 13.09 ± 10% perf-stat.overall.iTLB-load-miss-rate% 7119 ± 14% +35.9% 9679 ± 13% perf-stat.overall.instructions-per-iTLB-miss 0.59 ± 3% +43.1% 0.84 perf-stat.overall.ipc 202912 ± 2% +68.8% 342615 ± 3% perf-stat.overall.path-length 2.195e+09 ± 3% +10.5% 2.425e+09 perf-stat.ps.branch-instructions 36847010 ± 3% -29.4% 26015686 ± 5% perf-stat.ps.branch-misses 28799092 ± 6% -31.1% 19837306 ± 3% perf-stat.ps.cache-misses 2.021e+08 ± 4% -24.8% 1.519e+08 perf-stat.ps.cache-references 642266 ± 2% -27.4% 465968 perf-stat.ps.context-switches 1.974e+10 ± 3% -15.1% 1.676e+10 perf-stat.ps.cpu-cycles 31729472 ± 6% -37.9% 19715594 ± 9% perf-stat.ps.dTLB-load-misses 3.186e+09 ± 3% +18.8% 3.785e+09 perf-stat.ps.dTLB-loads 2504443 ± 10% -28.2% 1797154 ± 10% perf-stat.ps.dTLB-store-misses 2.133e+09 ± 3% +19.8% 2.556e+09 ± 2% perf-stat.ps.dTLB-stores 13717699 ± 3% -28.4% 9816979 ± 3% perf-stat.ps.iTLB-loads 1.159e+10 ± 3% +21.5% 1.408e+10 perf-stat.ps.instructions 15979371 ± 9% -30.7% 11073468 ± 3% perf-stat.ps.node-load-misses 8081205 ± 9% -31.3% 5554114 perf-stat.ps.node-loads 2580198 ± 13% -31.1% 1777169 ± 6% perf-stat.ps.node-store-misses 1801590 ± 12% -31.3% 1238321 ± 2% perf-stat.ps.node-stores 2.579e+12 +20.6% 3.111e+12 ± 2% perf-stat.total.instructions 36149 ± 4% -26.2% 26686 ± 5% softirqs.CPU0.RCU 37255 ± 3% -26.8% 27256 ± 2% softirqs.CPU1.RCU 37834 ± 5% -26.5% 27801 ± 9% softirqs.CPU10.RCU 39075 ± 3% -24.5% 29494 ± 2% softirqs.CPU11.RCU 38137 ± 6% -23.1% 29340 ± 4% softirqs.CPU12.RCU 37480 ± 3% -20.5% 29797 ± 2% softirqs.CPU13.RCU 37690 ± 5% -28.7% 26861 ± 4% softirqs.CPU14.RCU 33863 ± 9% -29.5% 23885 ± 5% softirqs.CPU15.RCU 34306 ± 9% -31.1% 23641 ± 12% softirqs.CPU16.RCU 34374 ± 9% -29.0% 24402 ± 7% softirqs.CPU17.RCU 44013 ± 9% -31.7% 30059 softirqs.CPU18.RCU 43114 ± 8% -29.9% 30219 softirqs.CPU19.RCU 36276 ± 8% -23.3% 27825 ± 5% softirqs.CPU2.RCU 67049 ± 7% +39.1% 93267 ± 18% softirqs.CPU2.TIMER 39721 ± 13% -31.4% 27233 ± 3% softirqs.CPU20.RCU 42340 ± 5% -28.6% 30215 ± 6% softirqs.CPU21.RCU 40896 ± 7% -30.3% 28506 ± 2% softirqs.CPU22.RCU 42422 ± 7% -33.2% 28357 ± 2% softirqs.CPU23.RCU 42330 ± 9% -33.5% 28150 ± 5% softirqs.CPU24.RCU 41406 ± 9% -31.4% 28398 softirqs.CPU25.RCU 39754 ± 5% -29.4% 28063 softirqs.CPU26.RCU 40207 ± 6% -27.9% 28995 ± 12% softirqs.CPU27.RCU 40351 ± 9% -29.4% 28487 ± 3% softirqs.CPU28.RCU 41880 ± 5% -29.6% 29472 softirqs.CPU29.RCU 36293 ± 4% -17.7% 29864 ± 7% softirqs.CPU3.RCU 39341 ± 9% -34.3% 25847 softirqs.CPU30.RCU 39687 ± 9% -33.5% 26400 ± 2% softirqs.CPU31.RCU 38062 ± 14% -39.9% 22878 ± 10% softirqs.CPU32.RCU 39769 ± 13% -33.8% 26314 ± 5% softirqs.CPU33.RCU 39504 ± 10% -37.2% 24824 ± 6% softirqs.CPU34.RCU 39591 ± 11% -38.6% 24300 ± 5% softirqs.CPU35.RCU 40398 ± 8% -27.0% 29488 ± 12% softirqs.CPU36.RCU 39464 ± 12% -25.3% 29478 ± 10% softirqs.CPU37.RCU 41177 ± 9% -34.0% 27190 ± 19% softirqs.CPU38.RCU 38734 ± 13% -24.8% 29145 ± 5% softirqs.CPU39.RCU 36804 ± 5% -23.0% 28354 softirqs.CPU4.RCU 40421 ± 12% -28.8% 28776 ± 6% softirqs.CPU40.RCU 40371 ± 11% -31.1% 27834 ± 8% softirqs.CPU41.RCU 40633 ± 10% -29.9% 28469 ± 10% softirqs.CPU42.RCU 41018 ± 9% -33.6% 27252 ± 6% softirqs.CPU43.RCU 38181 ± 12% -27.0% 27856 ± 10% softirqs.CPU44.RCU 39605 ± 12% -32.3% 26803 ± 12% softirqs.CPU45.RCU 42686 ± 11% -29.7% 30016 ± 9% softirqs.CPU47.RCU 40433 ± 9% -31.8% 27586 ± 11% softirqs.CPU48.RCU 38371 ± 12% -23.0% 29528 ± 7% softirqs.CPU49.RCU 37172 ± 2% -25.9% 27562 ± 3% softirqs.CPU5.RCU 39660 ± 12% -27.5% 28752 ± 11% softirqs.CPU50.RCU 40375 ± 10% -26.7% 29582 ± 3% softirqs.CPU51.RCU 40568 ± 9% -28.8% 28886 ± 6% softirqs.CPU52.RCU 39967 ± 10% -27.3% 29073 ± 11% softirqs.CPU53.RCU 40874 ± 11% -37.1% 25725 ± 6% softirqs.CPU54.RCU 41227 ± 14% -37.9% 25606 ± 6% softirqs.CPU55.RCU 38856 ± 13% -34.7% 25380 ± 8% softirqs.CPU56.RCU 39618 ± 13% -39.1% 24119 ± 4% softirqs.CPU57.RCU 39412 ± 12% -36.6% 24970 ± 3% softirqs.CPU58.RCU 41572 ± 14% -41.3% 24399 ± 10% softirqs.CPU59.RCU 36927 ± 3% -24.4% 27923 ± 22% softirqs.CPU6.RCU 37057 ± 10% -31.8% 25261 ± 2% softirqs.CPU60.RCU 36584 ± 8% -27.8% 26419 ± 2% softirqs.CPU61.RCU 35859 ± 4% -28.2% 25729 ± 5% softirqs.CPU62.RCU 35849 ± 7% -33.2% 23933 ± 5% softirqs.CPU63.RCU 35379 ± 6% -26.4% 26041 ± 4% softirqs.CPU64.RCU 36908 ± 6% -26.5% 27110 ± 2% softirqs.CPU65.RCU 37595 ± 3% -26.1% 27783 ± 4% softirqs.CPU66.RCU 36327 ± 2% -27.0% 26533 ± 2% softirqs.CPU67.RCU 35136 ± 7% -31.7% 23998 ± 9% softirqs.CPU68.RCU 35638 ± 4% -28.6% 25435 ± 4% softirqs.CPU69.RCU 37268 ± 6% -25.4% 27786 ± 4% softirqs.CPU7.RCU 36693 ± 4% -29.2% 25986 ± 5% softirqs.CPU70.RCU 36859 ± 3% -28.8% 26260 ± 4% softirqs.CPU71.RCU 37539 ± 7% -27.7% 27144 ± 4% softirqs.CPU8.RCU 37584 ± 4% -26.0% 27802 ± 15% softirqs.CPU9.RCU 2795313 ± 7% -29.7% 1964363 ± 4% softirqs.RCU 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.41:PCI-MSI.1572870-edge.eth0-TxRx-6 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.42:PCI-MSI.1572871-edge.eth0-TxRx-7 150.50 ± 43% -28.0% 108.33 interrupts.84:PCI-MSI.1572913-edge.eth0-TxRx-49 10393 ± 10% -26.4% 7653 ± 10% interrupts.CPU1.RES:Rescheduling_interrupts 12518 ± 15% -31.0% 8642 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.NMI:Non-maskable_interrupts 945.25 ± 79% -75.4% 232.67 ±141% interrupts.CPU11.PMI:Performance_monitoring_interrupts 13068 ± 18% -30.6% 9075 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 11949 ± 11% -49.8% 6001 ± 29% interrupts.CPU12.RES:Rescheduling_interrupts 13711 ± 8% -19.4% 11052 ± 24% interrupts.CPU13.RES:Rescheduling_interrupts 13361 ± 6% -38.8% 8170 ± 25% interrupts.CPU14.RES:Rescheduling_interrupts 7628 ± 12% -29.2% 5401 ± 7% interrupts.CPU18.RES:Rescheduling_interrupts 7909 ± 25% -30.4% 5501 ± 37% interrupts.CPU19.RES:Rescheduling_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.NMI:Non-maskable_interrupts 1482 ± 39% -100.0% 0.00 interrupts.CPU2.PMI:Performance_monitoring_interrupts 2116 +23.2% 2606 ± 10% interrupts.CPU20.CAL:Function_call_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.NMI:Non-maskable_interrupts 194.75 ±173% +380.1% 935.00 ± 30% interrupts.CPU20.PMI:Performance_monitoring_interrupts 2096 ± 4% +18.0% 2474 ± 9% interrupts.CPU21.CAL:Function_call_interrupts 11313 ± 20% -56.4% 4935 ± 23% interrupts.CPU21.RES:Rescheduling_interrupts 2111 ± 3% +20.1% 2534 ± 11% interrupts.CPU22.CAL:Function_call_interrupts 2109 ± 4% +22.6% 2585 ± 10% interrupts.CPU23.CAL:Function_call_interrupts 10305 ± 16% -40.7% 6115 ± 17% interrupts.CPU23.RES:Rescheduling_interrupts 2128 ± 2% +17.8% 2507 ± 10% interrupts.CPU24.CAL:Function_call_interrupts 12271 ± 20% -39.0% 7488 ± 29% interrupts.CPU24.RES:Rescheduling_interrupts 2123 +19.7% 2541 ± 7% interrupts.CPU25.CAL:Function_call_interrupts 9780 ± 21% -32.9% 6563 ± 25% interrupts.CPU25.RES:Rescheduling_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.NMI:Non-maskable_interrupts 980.75 ± 89% -88.9% 109.00 ±141% interrupts.CPU26.PMI:Performance_monitoring_interrupts 11163 ± 28% -49.8% 5605 ± 13% interrupts.CPU27.RES:Rescheduling_interrupts 11071 ± 12% -56.3% 4835 ± 26% interrupts.CPU29.RES:Rescheduling_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.NMI:Non-maskable_interrupts 728.50 ± 66% -100.0% 0.00 interrupts.CPU3.PMI:Performance_monitoring_interrupts 10599 ± 10% -38.7% 6501 ± 16% interrupts.CPU31.RES:Rescheduling_interrupts 2213 ± 4% +14.2% 2527 ± 7% interrupts.CPU32.CAL:Function_call_interrupts 8862 ± 16% -39.6% 5354 ± 28% interrupts.CPU32.RES:Rescheduling_interrupts 2156 ± 6% +20.5% 2599 ± 9% interrupts.CPU33.CAL:Function_call_interrupts 2169 ± 5% +19.9% 2601 ± 9% interrupts.CPU34.CAL:Function_call_interrupts 10483 ± 18% -48.8% 5371 ± 33% interrupts.CPU34.RES:Rescheduling_interrupts 2049 +24.4% 2549 ± 8% interrupts.CPU35.CAL:Function_call_interrupts 10929 ± 15% -49.0% 5578 ± 14% interrupts.CPU35.RES:Rescheduling_interrupts 1993 ± 4% +28.5% 2561 ± 12% interrupts.CPU36.CAL:Function_call_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.NMI:Non-maskable_interrupts 161.75 ±161% +717.9% 1323 ± 43% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1987 ± 4% +30.5% 2594 ± 13% interrupts.CPU37.CAL:Function_call_interrupts 1863 ± 10% +35.7% 2529 ± 14% interrupts.CPU38.CAL:Function_call_interrupts 1984 ± 6% +26.5% 2509 ± 12% interrupts.CPU41.CAL:Function_call_interrupts 12830 ± 30% -26.4% 9445 ± 9% interrupts.CPU43.RES:Rescheduling_interrupts 2067 ± 2% +27.4% 2632 ± 9% interrupts.CPU44.CAL:Function_call_interrupts 2066 ± 3% +27.2% 2629 ± 9% interrupts.CPU45.CAL:Function_call_interrupts 2158 ± 3% +19.6% 2582 ± 9% interrupts.CPU46.CAL:Function_call_interrupts 2036 ± 6% +30.3% 2652 ± 10% interrupts.CPU47.CAL:Function_call_interrupts 150.50 ± 43% -28.0% 108.33 interrupts.CPU49.84:PCI-MSI.1572913-edge.eth0-TxRx-49 11823 ± 12% -36.9% 7460 ± 18% interrupts.CPU49.RES:Rescheduling_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.NMI:Non-maskable_interrupts 1139 ± 58% -75.5% 278.67 ±141% interrupts.CPU5.PMI:Performance_monitoring_interrupts 11748 ± 22% -40.0% 7048 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts 2102 ± 5% +19.6% 2515 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 2115 ± 6% +19.5% 2527 ± 7% interrupts.CPU53.CAL:Function_call_interrupts 11318 ± 17% -20.4% 9004 ± 26% interrupts.CPU53.RES:Rescheduling_interrupts 2177 ± 5% +17.4% 2556 ± 9% interrupts.CPU54.CAL:Function_call_interrupts 10968 ± 23% -62.3% 4134 ± 21% interrupts.CPU54.RES:Rescheduling_interrupts 2161 ± 5% +20.2% 2598 ± 9% interrupts.CPU55.CAL:Function_call_interrupts 9334 ± 19% -55.4% 4158 ± 34% interrupts.CPU55.RES:Rescheduling_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.NMI:Non-maskable_interrupts 1443 ± 22% -100.0% 0.00 interrupts.CPU56.PMI:Performance_monitoring_interrupts 11377 ± 31% -56.9% 4907 ± 43% interrupts.CPU56.RES:Rescheduling_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.NMI:Non-maskable_interrupts 601.00 ± 99% -79.6% 122.67 ±141% interrupts.CPU57.PMI:Performance_monitoring_interrupts 8295 ± 31% -47.7% 4340 ± 24% interrupts.CPU57.RES:Rescheduling_interrupts 9182 ± 37% -59.7% 3701 ± 46% interrupts.CPU58.RES:Rescheduling_interrupts 249.50 ± 76% -50.4% 123.67 ± 2% interrupts.CPU6.41:PCI-MSI.1572870-edge.eth0-TxRx-6 12473 ± 27% -44.4% 6940 ± 13% interrupts.CPU6.RES:Rescheduling_interrupts 7886 ± 12% -42.2% 4555 ± 31% interrupts.CPU60.RES:Rescheduling_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.NMI:Non-maskable_interrupts 971.50 ± 47% -86.8% 128.33 ±141% interrupts.CPU61.PMI:Performance_monitoring_interrupts 9253 ± 26% -56.4% 4033 ± 16% interrupts.CPU61.RES:Rescheduling_interrupts 2103 ± 5% +19.1% 2505 ± 11% interrupts.CPU63.CAL:Function_call_interrupts 2059 ± 2% +20.6% 2483 ± 10% interrupts.CPU64.CAL:Function_call_interrupts 8545 ± 17% -41.2% 5025 ± 8% interrupts.CPU65.RES:Rescheduling_interrupts 7378 ± 23% -51.0% 3614 ± 30% interrupts.CPU68.RES:Rescheduling_interrupts 8573 ± 28% -50.1% 4277 ± 6% interrupts.CPU69.RES:Rescheduling_interrupts 154.50 ± 13% -10.9% 137.67 ± 10% interrupts.CPU7.42:PCI-MSI.1572871-edge.eth0-TxRx-7 9910 ± 28% -51.2% 4839 ± 40% interrupts.CPU70.RES:Rescheduling_interrupts 8836 ± 26% -52.7% 4180 ± 6% interrupts.CPU71.RES:Rescheduling_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.NMI:Non-maskable_interrupts 911.50 ± 37% -93.2% 61.67 ±141% interrupts.CPU8.PMI:Performance_monitoring_interrupts 12326 ± 15% -32.3% 8350 ± 10% interrupts.CPU8.RES:Rescheduling_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.NMI:Non-maskable_interrupts 1392 ± 58% -100.0% 0.00 interrupts.CPU9.PMI:Performance_monitoring_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.NMI:Non-maskable_interrupts 60185 ± 19% -30.5% 41822 ± 11% interrupts.PMI:Performance_monitoring_interrupts 749419 -31.9% 510225 ± 3% interrupts.RES:Rescheduling_interrupts 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.calltrace.cycles-pp.normal_work_helper.process_one_work.worker_thread.kthread.ret_from_fork 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread.kthread 5.99 ± 8% -1.6 4.37 ± 6% perf-profile.calltrace.cycles-pp.io_submit 5.83 ± 8% -1.6 4.25 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.io_submit 5.82 ± 8% -1.6 4.24 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 5.65 ± 6% -1.6 4.08 ± 20% perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work.worker_thread 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 6.90 ± 8% -1.5 5.38 ± 10% perf-profile.calltrace.cycles-pp.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes 5.58 ± 8% -1.5 4.07 ± 6% perf-profile.calltrace.cycles-pp.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.50 ± 3% -1.5 2.99 ± 19% perf-profile.calltrace.cycles-pp.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper.process_one_work 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit.do_syscall_64 6.63 ± 8% -1.5 5.17 ± 10% perf-profile.calltrace.cycles-pp.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages.__writeback_single_inode 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one.__x64_sys_io_submit 5.55 ± 8% -1.2 4.32 ± 10% perf-profile.calltrace.cycles-pp.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 4.98 ± 8% -1.1 3.86 ± 10% perf-profile.calltrace.cycles-pp.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages 5.00 ± 8% -1.1 3.88 ± 10% perf-profile.calltrace.cycles-pp.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage.extent_write_cache_pages.extent_writepages 2.80 ± 4% -1.0 1.78 ± 22% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper.process_one_work 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_cow_block.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io 1.45 ± 3% -0.7 0.79 ± 18% perf-profile.calltrace.cycles-pp.btrfs_cow_block.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 2.32 ± 3% -0.6 1.72 ± 18% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.91 ± 8% -0.5 0.42 ± 73% perf-profile.calltrace.cycles-pp.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io.normal_work_helper 0.88 ± 8% -0.5 0.41 ± 74% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csum.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 0.83 ± 5% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums 1.90 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 1.89 ± 20% -0.4 1.48 ± 29% perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.82 ± 4% -0.4 0.42 ± 71% perf-profile.calltrace.cycles-pp.run_one_async_done.normal_work_helper.process_one_work.worker_thread.kthread 0.81 ± 4% -0.4 0.41 ± 71% perf-profile.calltrace.cycles-pp.btrfs_map_bio.run_one_async_done.normal_work_helper.process_one_work.worker_thread 0.74 ± 6% -0.4 0.36 ± 70% perf-profile.calltrace.cycles-pp.push_leaf_right.split_leaf.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks 1.19 ± 3% -0.4 0.83 ± 16% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_csum_file_blocks.add_pending_csums.btrfs_finish_ordered_io 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.calltrace.cycles-pp.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.25 ± 6% -0.3 0.90 ± 7% perf-profile.calltrace.cycles-pp.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.93 ± 8% -0.3 0.59 ± 7% perf-profile.calltrace.cycles-pp.__set_extent_bit.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter 0.94 ± 7% -0.3 0.60 ± 6% perf-profile.calltrace.cycles-pp.lock_extent_bits.lock_and_cleanup_extent_if_need.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.70 ± 8% -0.3 0.37 ± 70% perf-profile.calltrace.cycles-pp.__lookup_extent_mapping.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter 0.79 ± 6% -0.3 0.47 ± 71% perf-profile.calltrace.cycles-pp.setup_items_for_insert.btrfs_duplicate_item.btrfs_mark_extent_written.btrfs_finish_ordered_io.normal_work_helper 0.71 ± 21% -0.3 0.43 ± 74% perf-profile.calltrace.cycles-pp.__btrfs_free_extent.__btrfs_run_delayed_refs.btrfs_run_delayed_refs.flush_space.btrfs_async_reclaim_metadata_space 1.15 ± 11% -0.3 0.88 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.15 ± 11% -0.3 0.88 ± 8% perf-profile.calltrace.cycles-pp.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 1.43 ± 13% -0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write.io_submit_one 1.06 ± 5% -0.3 0.81 ± 7% perf-profile.calltrace.cycles-pp.btrfs_drop_extent_cache.create_io_em.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.30 ± 15% -0.2 1.06 ± 4% perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.aio_write 1.03 ± 13% -0.2 0.79 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_csums_range.csum_exist_in_range.run_delalloc_nocow.btrfs_run_delalloc_range 1.20 ± 4% -0.2 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary 1.24 ± 3% -0.2 1.01 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.05 ± 8% -0.2 0.82 ± 12% perf-profile.calltrace.cycles-pp.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages.do_writepages 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.calltrace.cycles-pp.wait_reserve_ticket.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 1.01 ± 9% -0.2 0.84 ± 9% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc 1.01 ± 10% -0.2 0.85 ± 9% perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.run_delalloc_nocow.btrfs_run_delalloc_range.writepage_delalloc.__extent_writepage 0.72 ± 7% -0.2 0.56 ± 6% perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.aio_write 0.75 ± 8% -0.1 0.60 ± 10% perf-profile.calltrace.cycles-pp.submit_extent_page.__extent_writepage_io.__extent_writepage.extent_write_cache_pages.extent_writepages 0.92 ± 15% +0.2 1.14 ± 3% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 1.01 ± 15% +0.2 1.25 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 1.15 ± 15% +0.3 1.44 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.27 ±100% +0.4 0.63 ± 4% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 1.74 ± 9% +0.4 2.11 ± 2% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 2.63 ± 10% +0.6 3.26 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +0.8 0.75 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread 0.00 +0.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.map_private_extent_buffer.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook 4.71 ± 11% +1.0 5.74 perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 5.04 ± 10% +1.2 6.19 perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.00 +1.3 1.30 ± 19% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range 0.00 +1.4 1.36 ± 19% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents 0.29 ±101% +1.4 1.67 ± 18% perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space.process_one_work 0.00 +1.4 1.39 ± 19% perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.flush_space.btrfs_async_reclaim_metadata_space 0.00 +1.5 1.52 ± 18% perf-profile.calltrace.cycles-pp.btrfs_get_token_64.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 0.00 +2.2 2.20 ± 14% perf-profile.calltrace.cycles-pp.btrfs_get_token_32.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work.worker_thread 7.97 ± 7% +3.5 11.44 ± 15% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.wb_writeback.wb_workfn.process_one_work 49.45 ± 2% +4.3 53.79 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 1.07 ± 18% +5.0 6.06 ± 20% perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes.wb_writeback 0.93 ± 18% +5.0 5.95 ± 20% perf-profile.calltrace.cycles-pp.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode.writeback_sb_inodes 61.26 ± 2% +5.1 66.36 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 61.21 ± 2% +5.1 66.32 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.33 ±100% +5.3 5.68 ± 20% perf-profile.calltrace.cycles-pp.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages.__writeback_single_inode 55.45 ± 2% +5.4 60.88 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.00 +6.4 6.39 ± 13% perf-profile.calltrace.cycles-pp.check_leaf.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page 0.00 +6.5 6.50 ± 13% perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages.do_writepages 0.29 ±100% +6.6 6.92 ± 13% perf-profile.calltrace.cycles-pp.btree_submit_bio_hook.submit_one_bio.submit_extent_page.write_one_eb.btree_write_cache_pages 15.05 ± 6% -4.1 10.97 ± 20% perf-profile.children.cycles-pp.normal_work_helper 12.66 ± 5% -4.0 8.68 ± 20% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 7.84 ± 5% -2.4 5.39 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_writepages 7.30 ± 7% -1.7 5.59 ± 11% perf-profile.children.cycles-pp.extent_write_cache_pages 7.03 ± 7% -1.6 5.38 ± 12% perf-profile.children.cycles-pp.__extent_writepage 6.19 ± 8% -1.6 4.56 ± 7% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.17 ± 8% -1.6 4.55 ± 7% perf-profile.children.cycles-pp.do_syscall_64 6.01 ± 8% -1.6 4.39 ± 7% perf-profile.children.cycles-pp.io_submit 5.81 ± 8% -1.6 4.23 ± 6% perf-profile.children.cycles-pp.__x64_sys_io_submit 5.76 ± 8% -1.6 4.19 ± 6% perf-profile.children.cycles-pp.io_submit_one 5.66 ± 5% -1.6 4.09 ± 20% perf-profile.children.cycles-pp.btrfs_mark_extent_written 4.52 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.add_pending_csums 5.59 ± 8% -1.5 4.07 ± 6% perf-profile.children.cycles-pp.aio_write 4.51 ± 3% -1.5 2.99 ± 19% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 5.44 ± 8% -1.5 3.96 ± 7% perf-profile.children.cycles-pp.btrfs_file_write_iter 5.34 ± 9% -1.4 3.90 ± 6% perf-profile.children.cycles-pp.btrfs_buffered_write 5.88 ± 7% -1.4 4.47 ± 11% perf-profile.children.cycles-pp.writepage_delalloc 5.30 ± 8% -1.3 4.01 ± 11% perf-profile.children.cycles-pp.btrfs_run_delalloc_range 5.28 ± 8% -1.3 3.99 ± 11% perf-profile.children.cycles-pp.run_delalloc_nocow 2.55 ± 2% -0.9 1.67 ± 16% perf-profile.children.cycles-pp.btrfs_cow_block 2.54 ± 2% -0.9 1.66 ± 16% perf-profile.children.cycles-pp.__btrfs_cow_block 2.15 ± 13% -0.8 1.31 ± 8% perf-profile.children.cycles-pp.__lookup_extent_mapping 1.87 ± 12% -0.8 1.04 ± 12% perf-profile.children.cycles-pp.__etree_search 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.printk 0.94 ± 51% -0.7 0.22 ± 34% perf-profile.children.cycles-pp.vprintk_emit 2.60 ± 2% -0.7 1.92 ± 14% perf-profile.children.cycles-pp.btrfs_insert_empty_items 3.00 ± 5% -0.7 2.33 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 1.89 ± 2% -0.7 1.23 ± 7% perf-profile.children.cycles-pp.__set_extent_bit 2.78 ± 4% -0.6 2.14 ± 14% perf-profile.children.cycles-pp.__schedule 1.57 ± 2% -0.5 1.02 ± 9% perf-profile.children.cycles-pp.lock_extent_bits 1.19 ± 14% -0.5 0.70 ± 11% perf-profile.children.cycles-pp.__clear_extent_bit 1.51 ± 7% -0.5 1.04 ± 9% perf-profile.children.cycles-pp.read_block_for_search 2.08 ± 5% -0.5 1.62 ± 18% perf-profile.children.cycles-pp.setup_items_for_insert 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.btrfs_run_delayed_refs 2.08 ± 17% -0.4 1.67 ± 24% perf-profile.children.cycles-pp.__btrfs_run_delayed_refs 1.60 ± 6% -0.4 1.20 ± 12% perf-profile.children.cycles-pp.schedule 1.95 ± 11% -0.4 1.58 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 1.31 ± 6% -0.4 0.94 ± 9% perf-profile.children.cycles-pp.create_io_em 0.99 ± 4% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 1.00 ± 6% -0.4 0.65 ± 7% perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need 1.27 ± 9% -0.4 0.92 ± 11% perf-profile.children.cycles-pp.__wake_up_common_lock 0.98 ± 3% -0.4 0.63 ± 13% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 1.27 ± 5% -0.4 0.92 ± 7% perf-profile.children.cycles-pp.btrfs_dirty_pages 1.34 ± 7% -0.3 0.99 ± 10% perf-profile.children.cycles-pp.btrfs_map_bio 1.11 ± 9% -0.3 0.77 ± 11% perf-profile.children.cycles-pp.generic_bin_search 1.16 ± 6% -0.3 0.82 ± 9% perf-profile.children.cycles-pp.find_extent_buffer 0.91 ± 7% -0.3 0.57 ± 21% perf-profile.children.cycles-pp.btrfs_lookup_csum 1.21 ± 11% -0.3 0.90 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_csums_range 1.21 ± 11% -0.3 0.91 ± 9% perf-profile.children.cycles-pp.csum_exist_in_range 1.10 ± 10% -0.3 0.80 ± 9% perf-profile.children.cycles-pp.__wake_up_common 0.37 ± 70% -0.3 0.07 ± 18% perf-profile.children.cycles-pp.test_range_bit 1.61 ± 4% -0.3 1.32 ± 15% perf-profile.children.cycles-pp.push_leaf_right 0.60 ± 4% -0.3 0.31 ± 25% perf-profile.children.cycles-pp.btrfs_extend_item 1.02 ± 9% -0.3 0.73 ± 10% perf-profile.children.cycles-pp.autoremove_wake_function 1.12 ± 4% -0.3 0.84 ± 9% perf-profile.children.cycles-pp.btrfs_drop_extent_cache 0.82 ± 6% -0.3 0.55 ± 4% perf-profile.children.cycles-pp.btrfs_release_path 0.85 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_extent_buffer_full 0.99 ± 4% -0.3 0.73 ± 13% perf-profile.children.cycles-pp.btrfs_set_token_32 1.44 ± 13% -0.3 1.18 ± 5% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 0.84 ± 8% -0.3 0.59 ± 17% perf-profile.children.cycles-pp.copy_page 0.87 ± 11% -0.3 0.62 ± 11% perf-profile.children.cycles-pp.pagecache_get_page 0.84 ± 7% -0.2 0.60 ± 11% perf-profile.children.cycles-pp.kmem_cache_alloc 1.12 ± 7% -0.2 0.88 ± 17% perf-profile.children.cycles-pp.__extent_writepage_io 1.30 ± 15% -0.2 1.06 ± 5% perf-profile.children.cycles-pp.reserve_metadata_bytes 0.82 ± 4% -0.2 0.58 ± 13% perf-profile.children.cycles-pp.run_one_async_done 1.25 ± 3% -0.2 1.01 ± 15% perf-profile.children.cycles-pp.schedule_idle 0.34 ± 53% -0.2 0.10 ± 28% perf-profile.children.cycles-pp.delay_tsc 0.70 ± 13% -0.2 0.47 ± 11% perf-profile.children.cycles-pp.prepare_pages 0.50 ± 22% -0.2 0.28 ± 13% perf-profile.children.cycles-pp.clear_state_bit 0.84 ± 5% -0.2 0.63 ± 6% perf-profile.children.cycles-pp.btrfs_get_extent 0.54 ± 11% -0.2 0.34 ± 4% perf-profile.children.cycles-pp.btrfs_free_path 1.16 ± 11% -0.2 0.96 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.53 ± 9% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.unpin_extent_cache 1.07 ± 10% -0.2 0.88 ± 10% perf-profile.children.cycles-pp.btrfs_lookup_file_extent 0.50 ± 4% -0.2 0.31 ± 15% perf-profile.children.cycles-pp.clear_extent_bit 0.45 ± 13% -0.2 0.26 ± 11% perf-profile.children.cycles-pp.kmem_cache_free 0.40 ± 14% -0.2 0.21 ± 50% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.64 ± 7% -0.2 0.45 ± 10% perf-profile.children.cycles-pp.dequeue_task_fair 0.50 ± 8% -0.2 0.33 ± 23% perf-profile.children.cycles-pp.btrfs_add_delayed_tree_ref 0.88 ± 13% -0.2 0.71 ± 2% perf-profile.children.cycles-pp.wait_reserve_ticket 0.52 ± 9% -0.2 0.35 ± 7% perf-profile.children.cycles-pp.__radix_tree_lookup 0.52 ± 8% -0.2 0.36 ± 8% perf-profile.children.cycles-pp.xas_load 0.78 ± 19% -0.2 0.62 ± 23% perf-profile.children.cycles-pp.__btrfs_free_extent 0.56 ± 17% -0.1 0.42 ± 15% perf-profile.children.cycles-pp.btrfs_del_items 0.51 ± 5% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.dequeue_entity 0.39 ± 7% -0.1 0.24 ± 6% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.82 ± 14% -0.1 0.68 ± 18% perf-profile.children.cycles-pp.ttwu_do_activate 0.55 ± 8% -0.1 0.41 ± 8% perf-profile.children.cycles-pp.btrfs_submit_bio_hook 0.53 ± 5% -0.1 0.40 ± 13% perf-profile.children.cycles-pp.update_load_avg 0.40 ± 8% -0.1 0.27 ± 27% perf-profile.children.cycles-pp.add_delayed_ref_head 0.35 ± 13% -0.1 0.22 ± 3% perf-profile.children.cycles-pp.set_extent_bit 0.52 ± 9% -0.1 0.39 ± 7% perf-profile.children.cycles-pp.btrfs_wq_submit_bio 0.35 ± 3% -0.1 0.23 ± 32% perf-profile.children.cycles-pp.queued_write_lock_slowpath 0.33 ± 6% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.__slab_alloc 0.63 ± 19% -0.1 0.51 ± 13% perf-profile.children.cycles-pp.sched_ttwu_pending 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode 0.31 ± 11% -0.1 0.19 ± 28% perf-profile.children.cycles-pp.btrfs_update_inode_fallback 0.57 ± 5% -0.1 0.46 ± 10% perf-profile.children.cycles-pp.find_lock_delalloc_range 0.45 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.btrfs_cross_ref_exist 0.29 ± 3% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.free_extent_buffer 0.38 ± 3% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.btrfs_mark_buffer_dirty 0.46 ± 5% -0.1 0.35 ± 13% perf-profile.children.cycles-pp.find_delalloc_range 0.41 ± 9% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.btrfs_copy_from_user 0.40 ± 9% -0.1 0.29 ± 2% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.38 ± 10% -0.1 0.27 perf-profile.children.cycles-pp.find_get_entry 0.31 ± 12% -0.1 0.20 ± 31% perf-profile.children.cycles-pp.btrfs_remove_ordered_extent 0.30 ± 6% -0.1 0.20 ± 14% perf-profile.children.cycles-pp.___slab_alloc 0.26 ± 8% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.merge_state 0.38 ± 7% -0.1 0.28 ± 3% perf-profile.children.cycles-pp.copyin 0.26 ± 17% -0.1 0.15 ± 26% perf-profile.children.cycles-pp.btrfs_delayed_update_inode 0.25 ± 4% -0.1 0.15 ± 5% perf-profile.children.cycles-pp.insert_state 0.40 ± 13% -0.1 0.30 ± 13% perf-profile.children.cycles-pp.extent_clear_unlock_delalloc 0.36 ± 9% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.alloc_extent_state 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.39 ± 3% -0.1 0.29 ± 14% perf-profile.children.cycles-pp.alloc_extent_buffer 0.31 ± 4% -0.1 0.22 ± 11% perf-profile.children.cycles-pp.set_extent_buffer_dirty 0.26 ± 11% -0.1 0.17 ± 23% perf-profile.children.cycles-pp.unlock_up 0.32 ± 3% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.__btrfs_map_block 0.23 ± 13% -0.1 0.15 ± 28% perf-profile.children.cycles-pp.btrfs_add_delayed_data_ref 0.25 ± 12% -0.1 0.17 ± 2% perf-profile.children.cycles-pp.__slab_free 0.47 ± 6% -0.1 0.39 ± 15% perf-profile.children.cycles-pp.__push_leaf_right 0.26 ± 9% -0.1 0.18 ± 19% perf-profile.children.cycles-pp.__test_set_page_writeback 0.32 ± 11% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.btrfs_add_ordered_extent 0.15 ± 8% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__xa_set_mark 0.23 ± 7% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.release_extent_buffer 0.18 ± 8% -0.1 0.11 ± 26% perf-profile.children.cycles-pp.alloc_extent_map 0.32 ± 10% -0.1 0.24 ± 27% perf-profile.children.cycles-pp.__btrfs_add_ordered_extent 0.32 ± 17% -0.1 0.24 ± 10% perf-profile.children.cycles-pp.btrfs_tree_read_lock_atomic 0.20 ± 13% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.btrfs_inc_extent_ref 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.find_ref_head 0.13 ± 32% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.rb_erase_cached 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_submit_bio_start 0.53 ± 12% -0.1 0.46 ± 16% perf-profile.children.cycles-pp.btrfs_tree_read_lock 0.21 ± 7% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.btrfs_csum_one_bio 0.17 ± 16% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.rb_erase 0.21 ± 7% -0.1 0.14 ± 16% perf-profile.children.cycles-pp.run_one_async_start 0.22 ± 3% -0.1 0.15 ± 22% perf-profile.children.cycles-pp.btrfs_try_tree_write_lock 0.21 ± 9% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.btrfs_leaf_free_space 0.15 ± 23% -0.1 0.09 ± 35% perf-profile.children.cycles-pp.free_extent_map 0.20 ± 9% -0.1 0.14 ± 20% perf-profile.children.cycles-pp.leaf_space_used 0.19 ± 8% -0.1 0.13 ± 13% perf-profile.children.cycles-pp.check_delayed_ref 0.17 ± 13% -0.1 0.12 ± 31% perf-profile.children.cycles-pp.get_page_from_freelist 0.13 ± 8% -0.1 0.08 ± 26% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 2% -0.1 0.12 ± 18% perf-profile.children.cycles-pp.btrfs_set_range_writeback 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.children.cycles-pp.___might_sleep 0.20 ± 11% -0.1 0.15 ± 35% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_unlock 0.14 ± 11% -0.1 0.09 ± 28% perf-profile.children.cycles-pp.btrfs_reserve_extent 0.10 ± 33% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.__btrfs_release_delayed_node 0.22 ± 9% -0.0 0.17 ± 25% perf-profile.children.cycles-pp.mark_extent_buffer_accessed 0.09 ± 13% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_tree_read_unlock_blocking 0.18 ± 8% -0.0 0.14 ± 24% perf-profile.children.cycles-pp.__list_del_entry_valid 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.children.cycles-pp.verify_parent_transid 0.11 ± 19% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.22 ± 5% -0.0 0.18 ± 12% perf-profile.children.cycles-pp.check_committed_ref 0.11 ± 19% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.select_idle_sibling 0.14 ± 11% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_io_getevents 0.15 ± 7% -0.0 0.11 ± 4% perf-profile.children.cycles-pp.__x64_sys_io_getevents 0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.btrfs_release_extent_buffer_pages 0.17 ± 18% -0.0 0.13 ± 9% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.14 ± 5% -0.0 0.10 ± 25% perf-profile.children.cycles-pp.block_group_cache_tree_search 0.18 ± 4% -0.0 0.14 ± 11% perf-profile.children.cycles-pp.btrfs_get_chunk_map 0.11 ± 13% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.btrfs_buffer_uptodate 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.children.cycles-pp.xas_find_marked 0.12 ± 15% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.btrfs_unlock_up_safe 0.12 ± 14% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.btrfs_set_path_blocking 0.11 ± 11% -0.0 0.07 ± 28% perf-profile.children.cycles-pp.find_free_extent 0.24 ± 5% -0.0 0.20 ± 6% perf-profile.children.cycles-pp.btrfs_root_node 0.07 ± 6% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.extent_mergeable 0.13 ± 12% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.__switch_to 0.10 ± 18% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.rb_insert_color_cached 0.07 ± 14% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.rb_prev 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.get_io_u 0.10 ± 14% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.available_idle_cpu 0.17 ± 10% -0.0 0.13 ± 3% perf-profile.children.cycles-pp._raw_write_lock 0.10 ± 11% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.17 ± 7% -0.0 0.14 ± 12% perf-profile.children.cycles-pp._raw_read_lock 0.17 ± 10% -0.0 0.14 ± 12% perf-profile.children.cycles-pp.can_overcommit 0.10 ± 11% -0.0 0.07 perf-profile.children.cycles-pp.__list_add_valid 0.09 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.aio_read_events 0.11 ± 10% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.11 ± 9% -0.0 0.09 ± 18% perf-profile.children.cycles-pp.get_alloc_profile 0.11 ± 12% -0.0 0.08 ± 14% perf-profile.children.cycles-pp.pick_next_task_idle 0.10 ± 13% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.read_events 0.08 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.replace_extent_mapping 0.10 ± 5% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.menu_reflect 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.05 ± 58% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.rcu_irq_enter 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.children.cycles-pp.native_irq_return_iret 0.20 ± 14% +0.1 0.27 ± 8% perf-profile.children.cycles-pp.btrfs_comp_cpu_keys 0.38 ± 9% +0.1 0.46 perf-profile.children.cycles-pp.load_balance 0.30 ± 8% +0.1 0.38 ± 15% perf-profile.children.cycles-pp.irq_enter 0.21 ± 24% +0.1 0.30 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.22 ± 3% +0.1 0.32 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.52 ± 12% +0.1 0.63 ± 3% perf-profile.children.cycles-pp.rebalance_domains 1.90 ± 10% +0.3 2.23 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.23 ± 12% +0.4 0.67 ± 15% perf-profile.children.cycles-pp.read_extent_buffer 2.88 ± 11% +0.5 3.41 perf-profile.children.cycles-pp.hrtimer_interrupt 0.09 ± 12% +0.7 0.83 ± 15% perf-profile.children.cycles-pp.btrfs_get_token_8 2.37 ± 19% +0.8 3.19 ± 5% perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space 2.33 ± 19% +0.8 3.17 ± 5% perf-profile.children.cycles-pp.flush_space 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.14 ± 34% +1.3 1.39 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 0.42 ± 43% +1.3 1.67 ± 18% perf-profile.children.cycles-pp.btrfs_commit_transaction 0.25 ± 5% +1.9 2.17 ± 10% perf-profile.children.cycles-pp.btrfs_get_token_64 1.71 ± 5% +2.1 3.79 ± 5% perf-profile.children.cycles-pp.btrfs_get_token_32 0.43 ± 8% +2.4 2.84 ± 10% perf-profile.children.cycles-pp.map_private_extent_buffer 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_writeback 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.writeback_sb_inodes 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.__writeback_single_inode 8.37 ± 7% +3.3 11.65 ± 16% perf-profile.children.cycles-pp.wb_workfn 50.06 ± 2% +3.8 53.82 ± 4% perf-profile.children.cycles-pp.intel_idle 61.99 ± 2% +4.4 66.39 ± 3% perf-profile.children.cycles-pp.do_idle 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 61.98 ± 2% +4.4 66.38 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 8.51 ± 7% +4.5 13.04 ± 12% perf-profile.children.cycles-pp.do_writepages 56.43 ± 3% +4.9 61.30 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 61.27 ± 2% +5.1 66.37 ± 3% perf-profile.children.cycles-pp.start_secondary 1.43 ± 7% +6.2 7.64 ± 13% perf-profile.children.cycles-pp.submit_extent_page 1.20 ± 17% +6.2 7.45 ± 13% perf-profile.children.cycles-pp.btree_write_cache_pages 1.05 ± 17% +6.3 7.31 ± 13% perf-profile.children.cycles-pp.write_one_eb 1.11 ± 7% +6.3 7.37 ± 13% perf-profile.children.cycles-pp.submit_one_bio 0.55 ± 16% +6.4 6.95 ± 13% perf-profile.children.cycles-pp.btree_submit_bio_hook 0.00 +6.5 6.48 ± 13% perf-profile.children.cycles-pp.check_leaf 0.00 +6.5 6.53 ± 13% perf-profile.children.cycles-pp.btree_csum_one_bio 2.12 ± 11% -0.8 1.29 ± 7% perf-profile.self.cycles-pp.__lookup_extent_mapping 1.83 ± 12% -0.8 1.03 ± 13% perf-profile.self.cycles-pp.__etree_search 2.06 ± 5% -0.5 1.59 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 9% -0.3 0.52 ± 10% perf-profile.self.cycles-pp.generic_bin_search 0.84 ± 7% -0.3 0.58 ± 17% perf-profile.self.cycles-pp.copy_page 0.33 ± 54% -0.2 0.10 ± 28% perf-profile.self.cycles-pp.delay_tsc 0.89 ± 9% -0.2 0.69 ± 14% perf-profile.self.cycles-pp.btrfs_set_token_32 0.81 ± 4% -0.2 0.63 ± 14% perf-profile.self.cycles-pp.__schedule 0.51 ± 9% -0.2 0.35 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup 0.45 ± 8% -0.1 0.32 ± 10% perf-profile.self.cycles-pp.xas_load 0.43 ± 4% -0.1 0.31 ± 6% perf-profile.self.cycles-pp.find_extent_buffer 0.77 ± 5% -0.1 0.66 ± 11% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.31 ± 10% -0.1 0.20 ± 28% perf-profile.self.cycles-pp.add_delayed_ref_head 0.39 ± 16% -0.1 0.29 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.31 ± 4% -0.1 0.20 ± 26% perf-profile.self.cycles-pp.queued_write_lock_slowpath 0.38 ± 7% -0.1 0.29 ± 3% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.20 ± 17% -0.1 0.11 ± 14% perf-profile.self.cycles-pp.kmem_cache_free 0.25 ± 11% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.__slab_free 0.16 ± 9% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.free_extent_buffer 0.28 ± 17% -0.1 0.20 ± 11% perf-profile.self.cycles-pp.btrfs_search_slot 0.31 ± 4% -0.1 0.24 ± 20% perf-profile.self.cycles-pp.kmem_cache_alloc 0.14 ± 21% -0.1 0.06 ± 19% perf-profile.self.cycles-pp.find_ref_head 0.16 ± 10% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.insert_state 0.12 ± 8% -0.1 0.06 ± 72% perf-profile.self.cycles-pp.__update_load_avg_se 0.16 ± 15% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.rb_erase 0.10 ± 9% -0.1 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_unlock 0.15 ± 13% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.___slab_alloc 0.28 ± 13% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.queued_read_lock_slowpath 0.15 ± 21% -0.1 0.09 ± 35% perf-profile.self.cycles-pp.free_extent_map 0.18 ± 6% -0.1 0.12 ± 11% perf-profile.self.cycles-pp.update_load_avg 0.10 ± 11% -0.1 0.04 ± 70% perf-profile.self.cycles-pp.btrfs_extend_item 0.15 ± 23% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.add_extent_mapping 0.15 ± 8% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_set_mark 0.18 ± 6% -0.1 0.13 ± 16% perf-profile.self.cycles-pp.___might_sleep 0.18 ± 6% -0.0 0.13 ± 22% perf-profile.self.cycles-pp.__list_del_entry_valid 0.18 ± 4% -0.0 0.14 ± 22% perf-profile.self.cycles-pp.pick_next_task_fair 0.09 ± 13% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.verify_parent_transid 0.09 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.btrfs_tree_read_unlock_blocking 0.16 ± 13% -0.0 0.12 ± 24% perf-profile.self.cycles-pp.mark_page_accessed 0.12 ± 14% -0.0 0.08 ± 20% perf-profile.self.cycles-pp.dequeue_task_fair 0.10 ± 19% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.08 ± 14% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.unlock_up 0.15 ± 15% -0.0 0.11 ± 19% perf-profile.self.cycles-pp.xas_find_marked 0.24 ± 6% -0.0 0.20 ± 8% perf-profile.self.cycles-pp.btrfs_root_node 0.17 ± 6% -0.0 0.13 ± 10% perf-profile.self.cycles-pp._raw_read_lock 0.10 ± 13% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.btrfs_csum_one_bio 0.16 ± 7% -0.0 0.13 ± 9% perf-profile.self.cycles-pp.dequeue_entity 0.07 -0.0 0.04 ± 71% perf-profile.self.cycles-pp.io_u_queued_complete 0.08 -0.0 0.05 ± 70% perf-profile.self.cycles-pp.mutex_lock 0.11 ± 11% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.get_io_u 0.16 ± 9% -0.0 0.13 perf-profile.self.cycles-pp._raw_write_lock 0.07 ± 17% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.btrfs_tree_read_lock 0.12 ± 8% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.__switch_to 0.10 ± 11% -0.0 0.06 ± 14% perf-profile.self.cycles-pp.__clear_extent_bit 0.07 ± 12% -0.0 0.04 ± 71% perf-profile.self.cycles-pp.rb_prev 0.10 ± 15% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.10 ± 12% -0.0 0.07 ± 23% perf-profile.self.cycles-pp.available_idle_cpu 0.10 ± 11% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__list_add_valid 0.08 ± 13% -0.0 0.06 ± 8% perf-profile.self.cycles-pp.__might_sleep 0.12 ± 17% -0.0 0.09 ± 15% perf-profile.self.cycles-pp.release_extent_buffer 0.09 ± 17% -0.0 0.07 ± 14% perf-profile.self.cycles-pp.___perf_sw_event 0.08 ± 5% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.__push_leaf_right 0.11 ± 7% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.set_extent_buffer_dirty 0.11 ± 4% -0.0 0.08 ± 11% perf-profile.self.cycles-pp.__set_extent_bit 0.09 ± 7% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.hrtimer_interrupt 0.07 ± 17% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.07 ± 7% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.load_balance 0.01 ±173% +0.1 0.07 ± 17% perf-profile.self.cycles-pp.lapic_next_deadline 0.00 +0.1 0.06 ± 23% perf-profile.self.cycles-pp.find_next_and_bit 0.19 ± 11% +0.1 0.26 ± 17% perf-profile.self.cycles-pp.native_irq_return_iret 0.19 ± 24% +0.1 0.28 ± 7% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.50 ± 8% +0.2 0.69 ± 17% perf-profile.self.cycles-pp.cpuidle_enter_state 0.07 ± 11% +0.4 0.46 ± 17% perf-profile.self.cycles-pp.btrfs_get_token_8 0.22 ± 13% +0.4 0.66 ± 16% perf-profile.self.cycles-pp.read_extent_buffer 0.00 +0.6 0.65 ± 14% perf-profile.self.cycles-pp.check_leaf 0.15 ± 11% +1.0 1.13 ± 11% perf-profile.self.cycles-pp.btrfs_get_token_64 1.48 ± 7% +1.0 2.51 ± 5% perf-profile.self.cycles-pp.btrfs_get_token_32 0.41 ± 8% +2.2 2.61 ± 9% perf-profile.self.cycles-pp.map_private_extent_buffer 49.99 ± 2% +3.7 53.74 ± 4% perf-profile.self.cycles-pp.intel_idle fio.write_bw_MBps 300 +-+-------------------------------------------------------------------+ | | 250 +-++.+ +. .+.. .+..+..+.+.. .+..+.. .+ | | : : +. +..+ + +..+ +. + | | : : : : : : | 200 +-+ : : : : : : O O | O O O: O :O O O O O O O O O O: O:O O: O: O O O O O 150 +-+ : : : : : : | | : : : : : : | 100 +-+ : : : : : : | | : : : : : : | | : : : : : : | 50 +-+ :: :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_iops 70000 +-+-----------------------------------------------------------------+ |..+.+ +..+..+.+.. .+.+..+.+..+ +.. +..+.+..+..+ | 60000 +-+ : : +. : : + : | | : : : : : : | 50000 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 40000 +-+ : O: : : : : | | : : : : : : | 30000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 10000 +-+ : : : | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_mean_us 30000 +-+-----------------------------------------------------------------+ | | 25000 O-+O O O O O O O O O O O O O O O O O O O | | O O O O O | | 20000 +-+ | |..+.+ +..+..+.+..+..+.+..+.+..+ +..+ +..+.+..+..+ | 15000 +-+ : : : : : : | | : : : : : : | 10000 +-+ : : : : : : | | : : : : : : | | : : : : : : | 5000 +-+ : : : : :: | | : : : | 0 +-+--------------------------------------------O------------O-------+ fio.write_clat_stddev 160000 +-+----------------------------------------------------------------+ | | 140000 O-+O O O O O O O O O O O O O O | 120000 +-+ O O | | | 100000 +-+ O O O O O O O O | | 80000 +-+ | | .+.. | 60000 +-++.+ +..+.+. +.+..+.+..+..+ +.+ +..+..+.+..+ | 40000 +-+ : : : : : : | | : : : : : : | 20000 +-+ : : : : : : | | :: :: :: | 0 +-+-------------------------------------------O------------O-------+ fio.write_slat_mean_us 900 +-+-------------------------------------------------------------------+ O O O O O O O O O O O O O O O O O O | 800 +-+O O O O O O O 700 +-+ | | | 600 +-++.+ +.+..+..+..+.+..+..+.+..+ +..+ +..+..+..+.+ | 500 +-+ : : : : : : | | : : : : : : | 400 +-+ : : : : : : | 300 +-+ : : : : : : | | : : : : : : | 200 +-+ : : : : : : | 100 +-+ : : :: :: | | : : : | 0 +-+---------------------------------------------O-------------O-------+ fio.write_slat_stddev 25000 +-+-----O---------O-------------------------------------------------+ O O O O O O O O O O O O O O O | | | 20000 +-+ | | O O O O O O O O | | 15000 +-+ | | | 10000 +-++. .+.. .+.. .+..+ .+ .+..+ | |. + +..+..+ +..+ + : +. : +..+.+. | | : : : : : : | 5000 +-+ : : : : : : | | : : : : : : | | :: : :: | 0 +-+--------------------------------------------O------------O-------+ fio.latency_1000ms_ 0.4 +-+------------------------------------------------------------------+ | | 0.35 +-+ O | 0.3 +-+ | | O O | 0.25 +-+ O O | | O O O 0.2 O-+ O O O O O O O O O O | | O O | 0.15 +-+O O O O | 0.1 +-+ | | | 0.05 +-+ | | .+. .+..+..+.+..+.. .+. .+.+..+.. | 0 +-+---------------------------------------------O------------O-------+ fio.workload 1.4e+07 +-+---------------------------------------------------------------+ |..+.+ +..+.+..+.+..+.+..+.+..+ +.. +.+..+.+..+ | 1.2e+07 +-+ : : : : + : | | : : : : : : | 1e+07 +-+ : : : : : : | O O O: :O O O O O O O O O O: O:O O:O : O O O O O O O 8e+06 +-+ : O: : : : : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | :: :: :: | 2e+06 +-+ : : : | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.voluntary_context_switches 1.4e+07 +-+---------------------------------------------------------------+ | | 1.2e+07 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | 1e+07 +-+ : : : : : : | | O O: : O O O : : :O : O O O O O O O 8e+06 O-+ : O:O O O O O O O: O:O O: : | | : : : : : : | 6e+06 +-+ : : : : : : | | : : : : : : | 4e+06 +-+ : : : : : : | | : : : : :: | 2e+06 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ fio.time.file_system_outputs 1.2e+08 +-+---------------------------------------------------------------+ | | 1e+08 +-++.+ +..+.+..+.+..+.+..+.+..+ +..+ +.+..+.+..+ | | : : : : : : | | : : : : : : | 8e+07 +-+ : : : : : : O O O O O O O O: O:O O O O O O O O O O: O:O O:O : O O | 6e+07 +-+ : : : : : : | | : : : : : : | 4e+07 +-+ : : : : : : | | : : : : : : | | : : : : :: | 2e+07 +-+ : : :: | | : : : | 0 +-+-------------------------------------------O-----------O-------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

3 months, 3 weeks

4
4
0 / 0

[drm/mgag200] 90f479ae51: vm-scalability.median -18.8% regression

by kernel test robot

Greeting, FYI, we noticed a -18.8% regression of vm-scalability.median due to commit: commit: 90f479ae51afa45efab97afdde9b94b9660dd3e4 ("drm/mgag200: Replace struct mga_fbdev with generic framebuffer emulation") https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git master in testcase: vm-scalability on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: runtime: 300s size: 8T test: anon-cow-seq-hugetlb cpufreq_governor: performance test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/8T/lkp-knm01/anon-cow-seq-hugetlb/vm-scalability commit: f1f8555dfb ("drm/bochs: Use shadow buffer for bochs framebuffer console") 90f479ae51 ("drm/mgag200: Replace struct mga_fbdev with generic framebuffer emulation") f1f8555dfb9a70a2 90f479ae51afa45efab97afdde9 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 2:4 -50% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x :4 25% 1:4 dmesg.WARNING:at_ip___perf_sw_event/0x :4 25% 1:4 dmesg.WARNING:at_ip__fsnotify_parent/0x %stddev %change %stddev \ | \ 43955 ± 2% -18.8% 35691 vm-scalability.median 0.06 ± 7% +193.0% 0.16 ± 2% vm-scalability.median_stddev 14906559 ± 2% -17.9% 12237079 vm-scalability.throughput 87651 ± 2% -17.4% 72374 vm-scalability.time.involuntary_context_switches 2086168 -23.6% 1594224 vm-scalability.time.minor_page_faults 15082 ± 2% -10.4% 13517 vm-scalability.time.percent_of_cpu_this_job_got 29987 -8.9% 27327 vm-scalability.time.system_time 15755 -12.4% 13795 vm-scalability.time.user_time 122011 -19.3% 98418 vm-scalability.time.voluntary_context_switches 3.034e+09 -23.6% 2.318e+09 vm-scalability.workload 242478 ± 12% +68.5% 408518 ± 23% cpuidle.POLL.time 2788 ± 21% +117.4% 6062 ± 26% cpuidle.POLL.usage 56653 ± 10% +64.4% 93144 ± 20% meminfo.Mapped 120392 ± 7% +14.0% 137212 ± 4% meminfo.Shmem 47221 ± 11% +77.1% 83634 ± 22% numa-meminfo.node0.Mapped 120465 ± 7% +13.9% 137205 ± 4% numa-meminfo.node0.Shmem 2885513 -16.5% 2409384 numa-numastat.node0.local_node 2885471 -16.5% 2409354 numa-numastat.node0.numa_hit 11813 ± 11% +76.3% 20824 ± 22% numa-vmstat.node0.nr_mapped 30096 ± 7% +13.8% 34238 ± 4% numa-vmstat.node0.nr_shmem 43.72 ± 2% +5.5 49.20 mpstat.cpu.all.idle% 0.03 ± 4% +0.0 0.05 ± 6% mpstat.cpu.all.soft% 19.51 -2.4 17.08 mpstat.cpu.all.usr% 1012 -7.9% 932.75 turbostat.Avg_MHz 32.38 ± 10% +25.8% 40.73 turbostat.CPU%c1 145.51 -3.1% 141.01 turbostat.PkgWatt 15.09 -19.2% 12.19 turbostat.RAMWatt 43.50 ± 2% +13.2% 49.25 vmstat.cpu.id 18.75 ± 2% -13.3% 16.25 ± 2% vmstat.cpu.us 152.00 ± 2% -9.5% 137.50 vmstat.procs.r 4800 -13.1% 4173 vmstat.system.cs 156170 -11.9% 137594 slabinfo.anon_vma.active_objs 3395 -11.9% 2991 slabinfo.anon_vma.active_slabs 156190 -11.9% 137606 slabinfo.anon_vma.num_objs 3395 -11.9% 2991 slabinfo.anon_vma.num_slabs 1716 ± 5% +11.5% 1913 ± 8% slabinfo.dmaengine-unmap-16.active_objs 1716 ± 5% +11.5% 1913 ± 8% slabinfo.dmaengine-unmap-16.num_objs 1767 ± 2% -19.0% 1431 ± 2% slabinfo.hugetlbfs_inode_cache.active_objs 1767 ± 2% -19.0% 1431 ± 2% slabinfo.hugetlbfs_inode_cache.num_objs 3597 ± 5% -16.4% 3006 ± 3% slabinfo.skbuff_ext_cache.active_objs 3597 ± 5% -16.4% 3006 ± 3% slabinfo.skbuff_ext_cache.num_objs 1330122 -23.6% 1016557 proc-vmstat.htlb_buddy_alloc_success 77214 ± 3% +6.4% 82128 ± 2% proc-vmstat.nr_active_anon 67277 +2.9% 69246 proc-vmstat.nr_anon_pages 218.50 ± 3% -10.6% 195.25 proc-vmstat.nr_dirtied 288628 +1.4% 292755 proc-vmstat.nr_file_pages 360.50 -2.7% 350.75 proc-vmstat.nr_inactive_file 14225 ± 9% +63.8% 23304 ± 20% proc-vmstat.nr_mapped 30109 ± 7% +13.8% 34259 ± 4% proc-vmstat.nr_shmem 99870 -1.3% 98597 proc-vmstat.nr_slab_unreclaimable 204.00 ± 4% -12.1% 179.25 proc-vmstat.nr_written 77214 ± 3% +6.4% 82128 ± 2% proc-vmstat.nr_zone_active_anon 360.50 -2.7% 350.75 proc-vmstat.nr_zone_inactive_file 8810 ± 19% -66.1% 2987 ± 42% proc-vmstat.numa_hint_faults 8810 ± 19% -66.1% 2987 ± 42% proc-vmstat.numa_hint_faults_local 2904082 -16.4% 2427026 proc-vmstat.numa_hit 2904081 -16.4% 2427025 proc-vmstat.numa_local 6.828e+08 -23.5% 5.221e+08 proc-vmstat.pgalloc_normal 2900008 -17.2% 2400195 proc-vmstat.pgfault 6.827e+08 -23.5% 5.22e+08 proc-vmstat.pgfree 1.635e+10 -17.0% 1.357e+10 perf-stat.i.branch-instructions 1.53 ± 4% -0.1 1.45 ± 3% perf-stat.i.branch-miss-rate% 2.581e+08 ± 3% -20.5% 2.051e+08 ± 2% perf-stat.i.branch-misses 12.66 +1.1 13.78 perf-stat.i.cache-miss-rate% 72720849 -12.0% 63958986 perf-stat.i.cache-misses 5.766e+08 -18.6% 4.691e+08 perf-stat.i.cache-references 4674 ± 2% -13.0% 4064 perf-stat.i.context-switches 4.29 +12.5% 4.83 perf-stat.i.cpi 2.573e+11 -7.4% 2.383e+11 perf-stat.i.cpu-cycles 231.35 -21.5% 181.56 perf-stat.i.cpu-migrations 3522 +4.4% 3677 perf-stat.i.cycles-between-cache-misses 0.09 ± 13% +0.0 0.12 ± 5% perf-stat.i.iTLB-load-miss-rate% 5.894e+10 -15.8% 4.961e+10 perf-stat.i.iTLB-loads 5.901e+10 -15.8% 4.967e+10 perf-stat.i.instructions 1291 ± 14% -21.8% 1010 perf-stat.i.instructions-per-iTLB-miss 0.24 -11.0% 0.21 perf-stat.i.ipc 9476 -17.5% 7821 perf-stat.i.minor-faults 9478 -17.5% 7821 perf-stat.i.page-faults 9.76 -3.6% 9.41 perf-stat.overall.MPKI 1.59 ± 4% -0.1 1.52 perf-stat.overall.branch-miss-rate% 12.61 +1.1 13.71 perf-stat.overall.cache-miss-rate% 4.38 +10.5% 4.83 perf-stat.overall.cpi 3557 +5.3% 3747 perf-stat.overall.cycles-between-cache-misses 0.08 ± 12% +0.0 0.10 perf-stat.overall.iTLB-load-miss-rate% 1268 ± 15% -23.0% 976.22 perf-stat.overall.instructions-per-iTLB-miss 0.23 -9.5% 0.21 perf-stat.overall.ipc 5815 +9.7% 6378 perf-stat.overall.path-length 1.634e+10 -17.5% 1.348e+10 perf-stat.ps.branch-instructions 2.595e+08 ± 3% -21.2% 2.043e+08 ± 2% perf-stat.ps.branch-misses 72565205 -12.2% 63706339 perf-stat.ps.cache-misses 5.754e+08 -19.2% 4.646e+08 perf-stat.ps.cache-references 4640 ± 2% -12.5% 4060 perf-stat.ps.context-switches 2.581e+11 -7.5% 2.387e+11 perf-stat.ps.cpu-cycles 229.91 -22.0% 179.42 perf-stat.ps.cpu-migrations 5.889e+10 -16.3% 4.927e+10 perf-stat.ps.iTLB-loads 5.899e+10 -16.3% 4.938e+10 perf-stat.ps.instructions 9388 -18.2% 7677 perf-stat.ps.minor-faults 9389 -18.2% 7677 perf-stat.ps.page-faults 1.764e+13 -16.2% 1.479e+13 perf-stat.total.instructions 46803 ± 3% -18.8% 37982 ± 6% sched_debug.cfs_rq:/.exec_clock.min 5320 ± 3% +23.7% 6581 ± 3% sched_debug.cfs_rq:/.exec_clock.stddev 6737 ± 14% +58.1% 10649 ± 10% sched_debug.cfs_rq:/.load.avg 587978 ± 17% +58.2% 930382 ± 9% sched_debug.cfs_rq:/.load.max 46952 ± 16% +64.8% 77388 ± 11% sched_debug.cfs_rq:/.load.stddev 7.12 ± 4% +49.1% 10.62 ± 6% sched_debug.cfs_rq:/.load_avg.avg 474.40 ± 23% +67.5% 794.60 ± 10% sched_debug.cfs_rq:/.load_avg.max 37.70 ± 11% +74.8% 65.90 ± 9% sched_debug.cfs_rq:/.load_avg.stddev 13424269 ± 4% -15.6% 11328098 ± 2% sched_debug.cfs_rq:/.min_vruntime.avg 15411275 ± 3% -12.4% 13505072 ± 2% sched_debug.cfs_rq:/.min_vruntime.max 7939295 ± 6% -17.5% 6551322 ± 7% sched_debug.cfs_rq:/.min_vruntime.min 21.44 ± 7% -56.1% 9.42 ± 4% sched_debug.cfs_rq:/.nr_spread_over.avg 117.45 ± 11% -60.6% 46.30 ± 14% sched_debug.cfs_rq:/.nr_spread_over.max 19.33 ± 8% -66.4% 6.49 ± 9% sched_debug.cfs_rq:/.nr_spread_over.stddev 4.32 ± 15% +84.4% 7.97 ± 3% sched_debug.cfs_rq:/.runnable_load_avg.avg 353.85 ± 29% +118.8% 774.35 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.max 27.30 ± 24% +118.5% 59.64 ± 9% sched_debug.cfs_rq:/.runnable_load_avg.stddev 6729 ± 14% +58.2% 10644 ± 10% sched_debug.cfs_rq:/.runnable_weight.avg 587978 ± 17% +58.2% 930382 ± 9% sched_debug.cfs_rq:/.runnable_weight.max 46950 ± 16% +64.8% 77387 ± 11% sched_debug.cfs_rq:/.runnable_weight.stddev 5305069 ± 4% -17.4% 4380376 ± 7% sched_debug.cfs_rq:/.spread0.avg 7328745 ± 3% -9.9% 6600897 ± 3% sched_debug.cfs_rq:/.spread0.max 2220837 ± 4% +55.8% 3460596 ± 5% sched_debug.cpu.avg_idle.avg 4590666 ± 9% +76.8% 8117037 ± 15% sched_debug.cpu.avg_idle.max 485052 ± 7% +80.3% 874679 ± 10% sched_debug.cpu.avg_idle.stddev 561.50 ± 26% +37.7% 773.30 ± 15% sched_debug.cpu.clock.stddev 561.50 ± 26% +37.7% 773.30 ± 15% sched_debug.cpu.clock_task.stddev 3.20 ± 10% +109.6% 6.70 ± 3% sched_debug.cpu.cpu_load[0].avg 309.10 ± 20% +150.3% 773.75 ± 12% sched_debug.cpu.cpu_load[0].max 21.02 ± 14% +160.8% 54.80 ± 9% sched_debug.cpu.cpu_load[0].stddev 3.19 ± 8% +109.8% 6.70 ± 3% sched_debug.cpu.cpu_load[1].avg 299.75 ± 19% +158.0% 773.30 ± 12% sched_debug.cpu.cpu_load[1].max 20.32 ± 12% +168.7% 54.62 ± 9% sched_debug.cpu.cpu_load[1].stddev 3.20 ± 8% +109.1% 6.69 ± 4% sched_debug.cpu.cpu_load[2].avg 288.90 ± 20% +167.0% 771.40 ± 12% sched_debug.cpu.cpu_load[2].max 19.70 ± 12% +175.4% 54.27 ± 9% sched_debug.cpu.cpu_load[2].stddev 3.16 ± 8% +110.9% 6.66 ± 6% sched_debug.cpu.cpu_load[3].avg 275.50 ± 24% +178.4% 766.95 ± 12% sched_debug.cpu.cpu_load[3].max 18.92 ± 15% +184.2% 53.77 ± 10% sched_debug.cpu.cpu_load[3].stddev 3.08 ± 8% +115.7% 6.65 ± 7% sched_debug.cpu.cpu_load[4].avg 263.55 ± 28% +188.7% 760.85 ± 12% sched_debug.cpu.cpu_load[4].max 18.03 ± 18% +196.6% 53.46 ± 11% sched_debug.cpu.cpu_load[4].stddev 14543 -9.6% 13150 sched_debug.cpu.curr->pid.max 5293 ± 16% +74.7% 9248 ± 11% sched_debug.cpu.load.avg 587978 ± 17% +58.2% 930382 ± 9% sched_debug.cpu.load.max 40887 ± 19% +78.3% 72891 ± 9% sched_debug.cpu.load.stddev 1141679 ± 4% +56.9% 1790907 ± 5% sched_debug.cpu.max_idle_balance_cost.avg 2432100 ± 9% +72.6% 4196779 ± 13% sched_debug.cpu.max_idle_balance_cost.max 745656 +29.3% 964170 ± 5% sched_debug.cpu.max_idle_balance_cost.min 239032 ± 9% +81.9% 434806 ± 10% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 27% +92.1% 0.00 ± 31% sched_debug.cpu.next_balance.stddev 1030 ± 4% -10.4% 924.00 ± 2% sched_debug.cpu.nr_switches.min 0.04 ± 26% +139.0% 0.09 ± 41% sched_debug.cpu.nr_uninterruptible.avg 830.35 ± 6% -12.0% 730.50 ± 2% sched_debug.cpu.sched_count.min 912.00 ± 2% -9.5% 825.38 sched_debug.cpu.ttwu_count.avg 433.05 ± 3% -19.2% 350.05 ± 3% sched_debug.cpu.ttwu_count.min 160.70 ± 3% -12.5% 140.60 ± 4% sched_debug.cpu.ttwu_local.min 9072 ± 11% -36.4% 5767 ± 8% softirqs.CPU1.RCU 12769 ± 5% +15.3% 14718 ± 3% softirqs.CPU101.SCHED 13198 +11.5% 14717 ± 3% softirqs.CPU102.SCHED 12981 ± 4% +13.9% 14788 ± 3% softirqs.CPU105.SCHED 13486 ± 3% +11.8% 15071 ± 4% softirqs.CPU111.SCHED 12794 ± 4% +14.1% 14601 ± 9% softirqs.CPU112.SCHED 12999 ± 4% +10.1% 14314 ± 4% softirqs.CPU115.SCHED 12844 ± 4% +10.6% 14202 ± 2% softirqs.CPU120.SCHED 13336 ± 3% +9.4% 14585 ± 3% softirqs.CPU122.SCHED 12639 ± 4% +20.2% 15195 softirqs.CPU123.SCHED 13040 ± 5% +15.2% 15024 ± 5% softirqs.CPU126.SCHED 13123 +15.1% 15106 ± 5% softirqs.CPU127.SCHED 9188 ± 6% -35.7% 5911 ± 2% softirqs.CPU13.RCU 13054 ± 3% +13.1% 14761 ± 5% softirqs.CPU130.SCHED 13158 ± 2% +13.9% 14985 ± 5% softirqs.CPU131.SCHED 12797 ± 6% +13.5% 14524 ± 3% softirqs.CPU133.SCHED 12452 ± 5% +14.8% 14297 softirqs.CPU134.SCHED 13078 ± 3% +10.4% 14439 ± 3% softirqs.CPU138.SCHED 12617 ± 2% +14.5% 14442 ± 5% softirqs.CPU139.SCHED 12974 ± 3% +13.7% 14752 ± 4% softirqs.CPU142.SCHED 12579 ± 4% +19.1% 14983 ± 3% softirqs.CPU143.SCHED 9122 ± 24% -44.6% 5053 ± 5% softirqs.CPU144.RCU 13366 ± 2% +11.1% 14848 ± 3% softirqs.CPU149.SCHED 13246 ± 2% +22.0% 16162 ± 7% softirqs.CPU150.SCHED 13452 ± 3% +20.5% 16210 ± 7% softirqs.CPU151.SCHED 13507 +10.1% 14869 softirqs.CPU156.SCHED 13808 ± 3% +9.2% 15079 ± 4% softirqs.CPU157.SCHED 13442 ± 2% +13.4% 15248 ± 4% softirqs.CPU160.SCHED 13311 +12.1% 14920 ± 2% softirqs.CPU162.SCHED 13544 ± 3% +8.5% 14695 ± 4% softirqs.CPU163.SCHED 13648 ± 3% +11.2% 15179 ± 2% softirqs.CPU166.SCHED 13404 ± 4% +12.5% 15079 ± 3% softirqs.CPU168.SCHED 13421 ± 6% +16.0% 15568 ± 8% softirqs.CPU169.SCHED 13115 ± 3% +23.1% 16139 ± 10% softirqs.CPU171.SCHED 13424 ± 6% +10.4% 14822 ± 3% softirqs.CPU175.SCHED 13274 ± 3% +13.7% 15087 ± 9% softirqs.CPU185.SCHED 13409 ± 3% +12.3% 15063 ± 3% softirqs.CPU190.SCHED 13181 ± 7% +13.4% 14946 ± 3% softirqs.CPU196.SCHED 13578 ± 3% +10.9% 15061 softirqs.CPU197.SCHED 13323 ± 5% +24.8% 16627 ± 6% softirqs.CPU198.SCHED 14072 ± 2% +12.3% 15798 ± 7% softirqs.CPU199.SCHED 12604 ± 13% +17.9% 14865 softirqs.CPU201.SCHED 13380 ± 4% +14.8% 15356 ± 3% softirqs.CPU203.SCHED 13481 ± 8% +14.2% 15390 ± 3% softirqs.CPU204.SCHED 12921 ± 2% +13.8% 14710 ± 3% softirqs.CPU206.SCHED 13468 +13.0% 15218 ± 2% softirqs.CPU208.SCHED 13253 ± 2% +13.1% 14992 softirqs.CPU209.SCHED 13319 ± 2% +14.3% 15225 ± 7% softirqs.CPU210.SCHED 13673 ± 5% +16.3% 15895 ± 3% softirqs.CPU211.SCHED 13290 +17.0% 15556 ± 5% softirqs.CPU212.SCHED 13455 ± 4% +14.4% 15392 ± 3% softirqs.CPU213.SCHED 13454 ± 4% +14.3% 15377 ± 3% softirqs.CPU215.SCHED 13872 ± 7% +9.7% 15221 ± 5% softirqs.CPU220.SCHED 13555 ± 4% +17.3% 15896 ± 5% softirqs.CPU222.SCHED 13411 ± 4% +20.8% 16197 ± 6% softirqs.CPU223.SCHED 8472 ± 21% -44.8% 4680 ± 3% softirqs.CPU224.RCU 13141 ± 3% +16.2% 15265 ± 7% softirqs.CPU225.SCHED 14084 ± 3% +8.2% 15242 ± 2% softirqs.CPU226.SCHED 13528 ± 4% +11.3% 15063 ± 4% softirqs.CPU228.SCHED 13218 ± 3% +16.3% 15377 ± 4% softirqs.CPU229.SCHED 14031 ± 4% +10.2% 15467 ± 2% softirqs.CPU231.SCHED 13770 ± 3% +14.0% 15700 ± 3% softirqs.CPU232.SCHED 13456 ± 3% +12.3% 15105 ± 3% softirqs.CPU233.SCHED 13137 ± 4% +13.5% 14909 ± 3% softirqs.CPU234.SCHED 13318 ± 2% +14.7% 15280 ± 2% softirqs.CPU235.SCHED 13690 ± 2% +13.7% 15563 ± 7% softirqs.CPU238.SCHED 13771 ± 5% +20.8% 16634 ± 7% softirqs.CPU241.SCHED 13317 ± 7% +19.5% 15919 ± 9% softirqs.CPU243.SCHED 8234 ± 16% -43.9% 4616 ± 5% softirqs.CPU244.RCU 13845 ± 6% +13.0% 15643 ± 3% softirqs.CPU244.SCHED 13179 ± 3% +16.3% 15323 softirqs.CPU246.SCHED 13754 +12.2% 15438 ± 3% softirqs.CPU248.SCHED 13769 ± 4% +10.9% 15276 ± 2% softirqs.CPU252.SCHED 13702 +10.5% 15147 ± 2% softirqs.CPU254.SCHED 13315 ± 2% +12.5% 14980 ± 3% softirqs.CPU255.SCHED 13785 ± 3% +12.9% 15568 ± 5% softirqs.CPU256.SCHED 13307 ± 3% +15.0% 15298 ± 3% softirqs.CPU257.SCHED 13864 ± 3% +10.5% 15313 ± 2% softirqs.CPU259.SCHED 13879 ± 2% +11.4% 15465 softirqs.CPU261.SCHED 13815 +13.6% 15687 ± 5% softirqs.CPU264.SCHED 119574 ± 2% +11.8% 133693 ± 11% softirqs.CPU266.TIMER 13688 +10.9% 15180 ± 6% softirqs.CPU267.SCHED 11716 ± 4% +19.3% 13974 ± 8% softirqs.CPU27.SCHED 13866 ± 3% +13.7% 15765 ± 4% softirqs.CPU271.SCHED 13887 ± 5% +12.5% 15621 softirqs.CPU272.SCHED 13383 ± 3% +19.8% 16031 ± 2% softirqs.CPU274.SCHED 13347 +14.1% 15232 ± 3% softirqs.CPU275.SCHED 12884 ± 2% +21.0% 15593 ± 4% softirqs.CPU276.SCHED 13131 ± 5% +13.4% 14891 ± 5% softirqs.CPU277.SCHED 12891 ± 2% +19.2% 15371 ± 4% softirqs.CPU278.SCHED 13313 ± 4% +13.0% 15049 ± 2% softirqs.CPU279.SCHED 13514 ± 3% +10.2% 14897 ± 2% softirqs.CPU280.SCHED 13501 ± 3% +13.7% 15346 softirqs.CPU281.SCHED 13261 +17.5% 15577 softirqs.CPU282.SCHED 8076 ± 15% -43.7% 4546 ± 5% softirqs.CPU283.RCU 13686 ± 3% +12.6% 15413 ± 2% softirqs.CPU284.SCHED 13439 ± 2% +9.2% 14670 ± 4% softirqs.CPU285.SCHED 8878 ± 9% -35.4% 5735 ± 4% softirqs.CPU35.RCU 11690 ± 2% +13.6% 13274 ± 5% softirqs.CPU40.SCHED 11714 ± 2% +19.3% 13975 ± 13% softirqs.CPU41.SCHED 11763 +12.5% 13239 ± 4% softirqs.CPU45.SCHED 11662 ± 2% +9.4% 12757 ± 3% softirqs.CPU46.SCHED 11805 ± 2% +9.3% 12902 ± 2% softirqs.CPU50.SCHED 12158 ± 3% +12.3% 13655 ± 8% softirqs.CPU55.SCHED 11716 ± 4% +8.8% 12751 ± 3% softirqs.CPU58.SCHED 11922 ± 2% +9.9% 13100 ± 4% softirqs.CPU64.SCHED 9674 ± 17% -41.8% 5625 ± 6% softirqs.CPU66.RCU 11818 +12.0% 13237 softirqs.CPU66.SCHED 124682 ± 7% -6.1% 117088 ± 5% softirqs.CPU66.TIMER 8637 ± 9% -34.0% 5700 ± 7% softirqs.CPU70.RCU 11624 ± 2% +11.0% 12901 ± 2% softirqs.CPU70.SCHED 12372 ± 2% +13.2% 14003 ± 3% softirqs.CPU71.SCHED 9949 ± 25% -33.9% 6574 ± 31% softirqs.CPU72.RCU 10392 ± 26% -35.1% 6745 ± 35% softirqs.CPU73.RCU 12766 ± 3% +11.1% 14188 ± 3% softirqs.CPU76.SCHED 12611 ± 2% +18.8% 14984 ± 5% softirqs.CPU78.SCHED 12786 ± 3% +17.9% 15079 ± 7% softirqs.CPU79.SCHED 11947 ± 4% +9.7% 13103 ± 4% softirqs.CPU8.SCHED 13379 ± 7% +11.8% 14962 ± 4% softirqs.CPU83.SCHED 13438 ± 5% +9.7% 14738 ± 2% softirqs.CPU84.SCHED 12768 +19.4% 15241 ± 6% softirqs.CPU88.SCHED 8604 ± 13% -39.3% 5222 ± 3% softirqs.CPU89.RCU 13077 ± 2% +17.1% 15308 ± 7% softirqs.CPU89.SCHED 11887 ± 3% +20.1% 14272 ± 5% softirqs.CPU9.SCHED 12723 ± 3% +11.3% 14165 ± 4% softirqs.CPU90.SCHED 8439 ± 12% -38.9% 5153 ± 4% softirqs.CPU91.RCU 13429 ± 3% +10.3% 14806 ± 2% softirqs.CPU95.SCHED 12852 ± 4% +10.3% 14174 ± 5% softirqs.CPU96.SCHED 13010 ± 2% +14.4% 14888 ± 5% softirqs.CPU97.SCHED 2315644 ± 4% -36.2% 1477200 ± 4% softirqs.RCU 1572 ± 10% +63.9% 2578 ± 39% interrupts.CPU0.NMI:Non-maskable_interrupts 1572 ± 10% +63.9% 2578 ± 39% interrupts.CPU0.PMI:Performance_monitoring_interrupts 252.00 ± 11% -35.2% 163.25 ± 13% interrupts.CPU104.RES:Rescheduling_interrupts 2738 ± 24% +52.4% 4173 ± 19% interrupts.CPU105.NMI:Non-maskable_interrupts 2738 ± 24% +52.4% 4173 ± 19% interrupts.CPU105.PMI:Performance_monitoring_interrupts 245.75 ± 19% -31.0% 169.50 ± 7% interrupts.CPU105.RES:Rescheduling_interrupts 228.75 ± 13% -24.7% 172.25 ± 19% interrupts.CPU106.RES:Rescheduling_interrupts 2243 ± 15% +66.3% 3730 ± 35% interrupts.CPU113.NMI:Non-maskable_interrupts 2243 ± 15% +66.3% 3730 ± 35% interrupts.CPU113.PMI:Performance_monitoring_interrupts 2703 ± 31% +67.0% 4514 ± 33% interrupts.CPU118.NMI:Non-maskable_interrupts 2703 ± 31% +67.0% 4514 ± 33% interrupts.CPU118.PMI:Performance_monitoring_interrupts 2613 ± 25% +42.2% 3715 ± 24% interrupts.CPU121.NMI:Non-maskable_interrupts 2613 ± 25% +42.2% 3715 ± 24% interrupts.CPU121.PMI:Performance_monitoring_interrupts 311.50 ± 23% -47.7% 163.00 ± 9% interrupts.CPU122.RES:Rescheduling_interrupts 266.75 ± 19% -31.6% 182.50 ± 15% interrupts.CPU124.RES:Rescheduling_interrupts 293.75 ± 33% -32.3% 198.75 ± 19% interrupts.CPU125.RES:Rescheduling_interrupts 2601 ± 36% +43.2% 3724 ± 29% interrupts.CPU127.NMI:Non-maskable_interrupts 2601 ± 36% +43.2% 3724 ± 29% interrupts.CPU127.PMI:Performance_monitoring_interrupts 2258 ± 21% +68.2% 3797 ± 29% interrupts.CPU13.NMI:Non-maskable_interrupts 2258 ± 21% +68.2% 3797 ± 29% interrupts.CPU13.PMI:Performance_monitoring_interrupts 3338 ± 29% +54.6% 5160 ± 9% interrupts.CPU139.NMI:Non-maskable_interrupts 3338 ± 29% +54.6% 5160 ± 9% interrupts.CPU139.PMI:Performance_monitoring_interrupts 219.50 ± 27% -23.0% 169.00 ± 21% interrupts.CPU139.RES:Rescheduling_interrupts 290.25 ± 25% -32.5% 196.00 ± 11% interrupts.CPU14.RES:Rescheduling_interrupts 243.50 ± 4% -16.0% 204.50 ± 12% interrupts.CPU140.RES:Rescheduling_interrupts 1797 ± 15% +135.0% 4223 ± 46% interrupts.CPU147.NMI:Non-maskable_interrupts 1797 ± 15% +135.0% 4223 ± 46% interrupts.CPU147.PMI:Performance_monitoring_interrupts 2537 ± 22% +89.6% 4812 ± 28% interrupts.CPU15.NMI:Non-maskable_interrupts 2537 ± 22% +89.6% 4812 ± 28% interrupts.CPU15.PMI:Performance_monitoring_interrupts 292.25 ± 34% -33.9% 193.25 ± 6% interrupts.CPU15.RES:Rescheduling_interrupts 424.25 ± 37% -58.5% 176.25 ± 14% interrupts.CPU158.RES:Rescheduling_interrupts 312.50 ± 42% -54.2% 143.00 ± 18% interrupts.CPU159.RES:Rescheduling_interrupts 725.00 ±118% -75.7% 176.25 ± 14% interrupts.CPU163.RES:Rescheduling_interrupts 2367 ± 6% +59.9% 3786 ± 24% interrupts.CPU177.NMI:Non-maskable_interrupts 2367 ± 6% +59.9% 3786 ± 24% interrupts.CPU177.PMI:Performance_monitoring_interrupts 239.50 ± 30% -46.6% 128.00 ± 14% interrupts.CPU179.RES:Rescheduling_interrupts 320.75 ± 15% -24.0% 243.75 ± 20% interrupts.CPU20.RES:Rescheduling_interrupts 302.50 ± 17% -47.2% 159.75 ± 8% interrupts.CPU200.RES:Rescheduling_interrupts 2166 ± 5% +92.0% 4157 ± 40% interrupts.CPU207.NMI:Non-maskable_interrupts 2166 ± 5% +92.0% 4157 ± 40% interrupts.CPU207.PMI:Performance_monitoring_interrupts 217.00 ± 11% -34.6% 142.00 ± 12% interrupts.CPU214.RES:Rescheduling_interrupts 2610 ± 36% +47.4% 3848 ± 35% interrupts.CPU215.NMI:Non-maskable_interrupts 2610 ± 36% +47.4% 3848 ± 35% interrupts.CPU215.PMI:Performance_monitoring_interrupts 2046 ± 13% +118.6% 4475 ± 43% interrupts.CPU22.NMI:Non-maskable_interrupts 2046 ± 13% +118.6% 4475 ± 43% interrupts.CPU22.PMI:Performance_monitoring_interrupts 289.50 ± 28% -41.1% 170.50 ± 8% interrupts.CPU22.RES:Rescheduling_interrupts 2232 ± 6% +33.0% 2970 ± 24% interrupts.CPU221.NMI:Non-maskable_interrupts 2232 ± 6% +33.0% 2970 ± 24% interrupts.CPU221.PMI:Performance_monitoring_interrupts 4552 ± 12% -27.6% 3295 ± 15% interrupts.CPU222.NMI:Non-maskable_interrupts 4552 ± 12% -27.6% 3295 ± 15% interrupts.CPU222.PMI:Performance_monitoring_interrupts 2013 ± 15% +80.9% 3641 ± 27% interrupts.CPU226.NMI:Non-maskable_interrupts 2013 ± 15% +80.9% 3641 ± 27% interrupts.CPU226.PMI:Performance_monitoring_interrupts 2575 ± 49% +67.1% 4302 ± 34% interrupts.CPU227.NMI:Non-maskable_interrupts 2575 ± 49% +67.1% 4302 ± 34% interrupts.CPU227.PMI:Performance_monitoring_interrupts 248.00 ± 36% -36.3% 158.00 ± 19% interrupts.CPU228.RES:Rescheduling_interrupts 2441 ± 24% +43.0% 3490 ± 30% interrupts.CPU23.NMI:Non-maskable_interrupts 2441 ± 24% +43.0% 3490 ± 30% interrupts.CPU23.PMI:Performance_monitoring_interrupts 404.25 ± 69% -65.5% 139.50 ± 17% interrupts.CPU236.RES:Rescheduling_interrupts 566.50 ± 40% -73.6% 149.50 ± 31% interrupts.CPU237.RES:Rescheduling_interrupts 243.50 ± 26% -37.1% 153.25 ± 21% interrupts.CPU248.RES:Rescheduling_interrupts 258.25 ± 12% -53.5% 120.00 ± 18% interrupts.CPU249.RES:Rescheduling_interrupts 2888 ± 27% +49.4% 4313 ± 30% interrupts.CPU253.NMI:Non-maskable_interrupts 2888 ± 27% +49.4% 4313 ± 30% interrupts.CPU253.PMI:Performance_monitoring_interrupts 2468 ± 44% +67.3% 4131 ± 37% interrupts.CPU256.NMI:Non-maskable_interrupts 2468 ± 44% +67.3% 4131 ± 37% interrupts.CPU256.PMI:Performance_monitoring_interrupts 425.00 ± 59% -60.3% 168.75 ± 34% interrupts.CPU258.RES:Rescheduling_interrupts 1859 ± 16% +106.3% 3834 ± 44% interrupts.CPU268.NMI:Non-maskable_interrupts 1859 ± 16% +106.3% 3834 ± 44% interrupts.CPU268.PMI:Performance_monitoring_interrupts 2684 ± 28% +61.2% 4326 ± 36% interrupts.CPU269.NMI:Non-maskable_interrupts 2684 ± 28% +61.2% 4326 ± 36% interrupts.CPU269.PMI:Performance_monitoring_interrupts 2171 ± 6% +108.8% 4533 ± 20% interrupts.CPU270.NMI:Non-maskable_interrupts 2171 ± 6% +108.8% 4533 ± 20% interrupts.CPU270.PMI:Performance_monitoring_interrupts 2262 ± 14% +61.8% 3659 ± 37% interrupts.CPU273.NMI:Non-maskable_interrupts 2262 ± 14% +61.8% 3659 ± 37% interrupts.CPU273.PMI:Performance_monitoring_interrupts 2203 ± 11% +50.7% 3320 ± 38% interrupts.CPU279.NMI:Non-maskable_interrupts 2203 ± 11% +50.7% 3320 ± 38% interrupts.CPU279.PMI:Performance_monitoring_interrupts 2433 ± 17% +52.9% 3721 ± 25% interrupts.CPU280.NMI:Non-maskable_interrupts 2433 ± 17% +52.9% 3721 ± 25% interrupts.CPU280.PMI:Performance_monitoring_interrupts 2778 ± 33% +63.1% 4531 ± 36% interrupts.CPU283.NMI:Non-maskable_interrupts 2778 ± 33% +63.1% 4531 ± 36% interrupts.CPU283.PMI:Performance_monitoring_interrupts 331.75 ± 32% -39.8% 199.75 ± 17% interrupts.CPU29.RES:Rescheduling_interrupts 2178 ± 22% +53.9% 3353 ± 31% interrupts.CPU3.NMI:Non-maskable_interrupts 2178 ± 22% +53.9% 3353 ± 31% interrupts.CPU3.PMI:Performance_monitoring_interrupts 298.50 ± 30% -39.7% 180.00 ± 6% interrupts.CPU34.RES:Rescheduling_interrupts 2490 ± 3% +58.7% 3953 ± 28% interrupts.CPU35.NMI:Non-maskable_interrupts 2490 ± 3% +58.7% 3953 ± 28% interrupts.CPU35.PMI:Performance_monitoring_interrupts 270.50 ± 24% -31.1% 186.25 ± 3% interrupts.CPU36.RES:Rescheduling_interrupts 2493 ± 7% +57.0% 3915 ± 27% interrupts.CPU43.NMI:Non-maskable_interrupts 2493 ± 7% +57.0% 3915 ± 27% interrupts.CPU43.PMI:Performance_monitoring_interrupts 286.75 ± 36% -32.4% 193.75 ± 7% interrupts.CPU45.RES:Rescheduling_interrupts 259.00 ± 12% -23.6% 197.75 ± 13% interrupts.CPU46.RES:Rescheduling_interrupts 244.00 ± 21% -35.6% 157.25 ± 11% interrupts.CPU47.RES:Rescheduling_interrupts 230.00 ± 7% -21.3% 181.00 ± 11% interrupts.CPU48.RES:Rescheduling_interrupts 281.00 ± 13% -27.4% 204.00 ± 15% interrupts.CPU53.RES:Rescheduling_interrupts 256.75 ± 5% -18.4% 209.50 ± 12% interrupts.CPU54.RES:Rescheduling_interrupts 2433 ± 9% +68.4% 4098 ± 35% interrupts.CPU58.NMI:Non-maskable_interrupts 2433 ± 9% +68.4% 4098 ± 35% interrupts.CPU58.PMI:Performance_monitoring_interrupts 316.00 ± 25% -41.4% 185.25 ± 13% interrupts.CPU59.RES:Rescheduling_interrupts 2703 ± 38% +56.0% 4217 ± 31% interrupts.CPU60.NMI:Non-maskable_interrupts 2703 ± 38% +56.0% 4217 ± 31% interrupts.CPU60.PMI:Performance_monitoring_interrupts 2425 ± 16% +39.9% 3394 ± 27% interrupts.CPU61.NMI:Non-maskable_interrupts 2425 ± 16% +39.9% 3394 ± 27% interrupts.CPU61.PMI:Performance_monitoring_interrupts 2388 ± 18% +69.5% 4047 ± 29% interrupts.CPU66.NMI:Non-maskable_interrupts 2388 ± 18% +69.5% 4047 ± 29% interrupts.CPU66.PMI:Performance_monitoring_interrupts 2322 ± 11% +93.4% 4491 ± 35% interrupts.CPU67.NMI:Non-maskable_interrupts 2322 ± 11% +93.4% 4491 ± 35% interrupts.CPU67.PMI:Performance_monitoring_interrupts 319.00 ± 40% -44.7% 176.25 ± 9% interrupts.CPU67.RES:Rescheduling_interrupts 2512 ± 8% +28.1% 3219 ± 25% interrupts.CPU70.NMI:Non-maskable_interrupts 2512 ± 8% +28.1% 3219 ± 25% interrupts.CPU70.PMI:Performance_monitoring_interrupts 2290 ± 39% +78.7% 4094 ± 28% interrupts.CPU74.NMI:Non-maskable_interrupts 2290 ± 39% +78.7% 4094 ± 28% interrupts.CPU74.PMI:Performance_monitoring_interrupts 2446 ± 40% +94.8% 4764 ± 23% interrupts.CPU75.NMI:Non-maskable_interrupts 2446 ± 40% +94.8% 4764 ± 23% interrupts.CPU75.PMI:Performance_monitoring_interrupts 426.75 ± 61% -67.7% 138.00 ± 8% interrupts.CPU75.RES:Rescheduling_interrupts 192.50 ± 13% +45.6% 280.25 ± 45% interrupts.CPU76.RES:Rescheduling_interrupts 274.25 ± 34% -42.2% 158.50 ± 34% interrupts.CPU77.RES:Rescheduling_interrupts 2357 ± 9% +73.0% 4078 ± 23% interrupts.CPU78.NMI:Non-maskable_interrupts 2357 ± 9% +73.0% 4078 ± 23% interrupts.CPU78.PMI:Performance_monitoring_interrupts 348.50 ± 53% -47.3% 183.75 ± 29% interrupts.CPU80.RES:Rescheduling_interrupts 2650 ± 43% +46.2% 3874 ± 36% interrupts.CPU84.NMI:Non-maskable_interrupts 2650 ± 43% +46.2% 3874 ± 36% interrupts.CPU84.PMI:Performance_monitoring_interrupts 2235 ± 10% +117.8% 4867 ± 10% interrupts.CPU90.NMI:Non-maskable_interrupts 2235 ± 10% +117.8% 4867 ± 10% interrupts.CPU90.PMI:Performance_monitoring_interrupts 2606 ± 33% +38.1% 3598 ± 21% interrupts.CPU92.NMI:Non-maskable_interrupts 2606 ± 33% +38.1% 3598 ± 21% interrupts.CPU92.PMI:Performance_monitoring_interrupts 408.75 ± 58% -56.8% 176.75 ± 25% interrupts.CPU92.RES:Rescheduling_interrupts 399.00 ± 64% -63.6% 145.25 ± 16% interrupts.CPU93.RES:Rescheduling_interrupts 314.75 ± 36% -44.2% 175.75 ± 13% interrupts.CPU94.RES:Rescheduling_interrupts 191.00 ± 15% -29.1% 135.50 ± 9% interrupts.CPU97.RES:Rescheduling_interrupts 94.00 ± 8% +50.0% 141.00 ± 12% interrupts.IWI:IRQ_work_interrupts 841457 ± 7% +16.6% 980751 ± 3% interrupts.NMI:Non-maskable_interrupts 841457 ± 7% +16.6% 980751 ± 3% interrupts.PMI:Performance_monitoring_interrupts 12.75 ± 11% -4.1 8.67 ± 31% perf-profile.calltrace.cycles-pp.do_rw_once 1.02 ± 16% -0.6 0.47 ± 59% perf-profile.calltrace.cycles-pp.sched_clock.sched_clock_cpu.cpuidle_enter_state.cpuidle_enter.do_idle 1.10 ± 15% -0.4 0.66 ± 14% perf-profile.calltrace.cycles-pp.sched_clock_cpu.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 1.05 ± 16% -0.4 0.61 ± 14% perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock.sched_clock_cpu.cpuidle_enter_state.cpuidle_enter 1.58 ± 4% +0.3 1.91 ± 7% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.copy_page 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.11 ± 4% +0.5 2.60 ± 7% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.osq_lock.__mutex_lock.hugetlb_fault.handle_mm_fault 0.83 ± 26% +0.5 1.32 ± 18% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 0.83 ± 26% +0.5 1.32 ± 18% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.90 ± 5% +0.6 2.45 ± 7% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.copy_page.copy_subpage 0.65 ± 62% +0.6 1.20 ± 15% perf-profile.calltrace.cycles-pp.alloc_fresh_huge_page.alloc_surplus_huge_page.alloc_huge_page.hugetlb_cow.hugetlb_fault 0.60 ± 62% +0.6 1.16 ± 18% perf-profile.calltrace.cycles-pp.free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap 0.95 ± 17% +0.6 1.52 ± 8% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.mutex_spin_on_owner 0.61 ± 62% +0.6 1.18 ± 18% perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput 0.61 ± 62% +0.6 1.19 ± 19% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit 0.61 ± 62% +0.6 1.19 ± 19% perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit 0.64 ± 61% +0.6 1.23 ± 18% perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.64 ± 61% +0.6 1.23 ± 18% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group 1.30 ± 9% +0.6 1.92 ± 8% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.mutex_spin_on_owner.__mutex_lock 0.19 ±173% +0.7 0.89 ± 20% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.free_huge_page.release_pages.tlb_flush_mmu 0.19 ±173% +0.7 0.90 ± 20% perf-profile.calltrace.cycles-pp._raw_spin_lock.free_huge_page.release_pages.tlb_flush_mmu.tlb_finish_mmu 0.00 +0.8 0.77 ± 30% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.prep_new_huge_page.alloc_fresh_huge_page.alloc_surplus_huge_page 0.00 +0.8 0.78 ± 30% perf-profile.calltrace.cycles-pp._raw_spin_lock.prep_new_huge_page.alloc_fresh_huge_page.alloc_surplus_huge_page.alloc_huge_page 0.00 +0.8 0.79 ± 29% perf-profile.calltrace.cycles-pp.prep_new_huge_page.alloc_fresh_huge_page.alloc_surplus_huge_page.alloc_huge_page.hugetlb_cow 0.82 ± 67% +0.9 1.72 ± 22% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_huge_page.hugetlb_cow.hugetlb_fault 0.84 ± 66% +0.9 1.74 ± 20% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_surplus_huge_page.alloc_huge_page.hugetlb_cow 2.52 ± 6% +0.9 3.44 ± 9% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.copy_page.copy_subpage.copy_user_huge_page 0.83 ± 67% +0.9 1.75 ± 21% perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault 0.84 ± 66% +0.9 1.77 ± 20% perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_surplus_huge_page.alloc_huge_page.hugetlb_cow.hugetlb_fault 1.64 ± 12% +1.0 2.67 ± 7% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.mutex_spin_on_owner.__mutex_lock.hugetlb_fault 1.65 ± 45% +1.3 2.99 ± 18% perf-profile.calltrace.cycles-pp.alloc_surplus_huge_page.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault 1.74 ± 13% +1.4 3.16 ± 6% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.mutex_spin_on_owner.__mutex_lock.hugetlb_fault.handle_mm_fault 2.56 ± 48% +2.2 4.81 ± 19% perf-profile.calltrace.cycles-pp.alloc_huge_page.hugetlb_cow.hugetlb_fault.handle_mm_fault.__do_page_fault 12.64 ± 14% +3.6 16.20 ± 8% perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.hugetlb_fault.handle_mm_fault.__do_page_fault 2.97 ± 7% +3.8 6.74 ± 9% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.copy_page.copy_subpage.copy_user_huge_page.hugetlb_cow 19.99 ± 9% +4.1 24.05 ± 6% perf-profile.calltrace.cycles-pp.hugetlb_cow.hugetlb_fault.handle_mm_fault.__do_page_fault.do_page_fault 1.37 ± 15% -0.5 0.83 ± 13% perf-profile.children.cycles-pp.sched_clock_cpu 1.31 ± 16% -0.5 0.78 ± 13% perf-profile.children.cycles-pp.sched_clock 1.29 ± 16% -0.5 0.77 ± 13% perf-profile.children.cycles-pp.native_sched_clock 1.80 ± 2% -0.3 1.47 ± 10% perf-profile.children.cycles-pp.task_tick_fair 0.73 ± 2% -0.2 0.54 ± 11% perf-profile.children.cycles-pp.update_curr 0.42 ± 17% -0.2 0.27 ± 16% perf-profile.children.cycles-pp.account_process_tick 0.73 ± 10% -0.2 0.58 ± 9% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.27 ± 6% -0.1 0.14 ± 14% perf-profile.children.cycles-pp.__acct_update_integrals 0.27 ± 18% -0.1 0.16 ± 13% perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs 0.40 ± 12% -0.1 0.30 ± 14% perf-profile.children.cycles-pp.__next_timer_interrupt 0.47 ± 7% -0.1 0.39 ± 13% perf-profile.children.cycles-pp.update_rq_clock 0.29 ± 12% -0.1 0.21 ± 15% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.21 ± 7% -0.1 0.14 ± 12% perf-profile.children.cycles-pp.account_system_index_time 0.38 ± 2% -0.1 0.31 ± 12% perf-profile.children.cycles-pp.timerqueue_add 0.26 ± 11% -0.1 0.20 ± 13% perf-profile.children.cycles-pp.find_next_bit 0.23 ± 15% -0.1 0.17 ± 15% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.14 ± 8% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.account_user_time 0.17 ± 6% -0.0 0.12 ± 10% perf-profile.children.cycles-pp.cpuacct_charge 0.18 ± 20% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.irq_work_tick 0.11 ± 13% -0.0 0.07 ± 25% perf-profile.children.cycles-pp.tick_sched_do_timer 0.12 ± 10% -0.0 0.08 ± 15% perf-profile.children.cycles-pp.get_cpu_device 0.07 ± 11% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.raise_softirq 0.12 ± 3% -0.0 0.09 ± 8% perf-profile.children.cycles-pp.write 0.11 ± 13% +0.0 0.14 ± 8% perf-profile.children.cycles-pp.native_write_msr 0.09 ± 9% +0.0 0.11 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.10 ± 10% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.schedule_idle 0.07 ± 6% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.__read_nocancel 0.04 ± 58% +0.0 0.07 ± 15% perf-profile.children.cycles-pp.__free_pages_ok 0.06 ± 7% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.perf_read 0.07 +0.0 0.11 ± 14% perf-profile.children.cycles-pp.perf_evsel__read_counter 0.07 +0.0 0.11 ± 13% perf-profile.children.cycles-pp.cmd_stat 0.07 +0.0 0.11 ± 13% perf-profile.children.cycles-pp.__run_perf_stat 0.07 +0.0 0.11 ± 13% perf-profile.children.cycles-pp.process_interval 0.07 +0.0 0.11 ± 13% perf-profile.children.cycles-pp.read_counters 0.07 ± 22% +0.0 0.11 ± 19% perf-profile.children.cycles-pp.__handle_mm_fault 0.07 ± 19% +0.1 0.13 ± 8% perf-profile.children.cycles-pp.rb_erase 0.03 ±100% +0.1 0.09 ± 9% perf-profile.children.cycles-pp.smp_call_function_single 0.01 ±173% +0.1 0.08 ± 11% perf-profile.children.cycles-pp.perf_event_read 0.00 +0.1 0.07 ± 13% perf-profile.children.cycles-pp.__perf_event_read_value 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.08 ± 17% +0.1 0.15 ± 8% perf-profile.children.cycles-pp.native_apic_msr_eoi_write 0.04 ±103% +0.1 0.13 ± 58% perf-profile.children.cycles-pp.shmem_getpage_gfp 0.38 ± 14% +0.1 0.51 ± 6% perf-profile.children.cycles-pp.run_timer_softirq 0.11 ± 4% +0.3 0.37 ± 32% perf-profile.children.cycles-pp.worker_thread 0.20 ± 5% +0.3 0.48 ± 25% perf-profile.children.cycles-pp.ret_from_fork 0.20 ± 4% +0.3 0.48 ± 25% perf-profile.children.cycles-pp.kthread 0.00 +0.3 0.29 ± 38% perf-profile.children.cycles-pp.memcpy_erms 0.00 +0.3 0.29 ± 38% perf-profile.children.cycles-pp.drm_fb_helper_dirty_work 0.00 +0.3 0.31 ± 37% perf-profile.children.cycles-pp.process_one_work 0.47 ± 48% +0.4 0.91 ± 19% perf-profile.children.cycles-pp.prep_new_huge_page 0.70 ± 29% +0.5 1.16 ± 18% perf-profile.children.cycles-pp.free_huge_page 0.73 ± 29% +0.5 1.19 ± 18% perf-profile.children.cycles-pp.tlb_flush_mmu 0.72 ± 29% +0.5 1.18 ± 18% perf-profile.children.cycles-pp.release_pages 0.73 ± 29% +0.5 1.19 ± 18% perf-profile.children.cycles-pp.tlb_finish_mmu 0.76 ± 27% +0.5 1.23 ± 18% perf-profile.children.cycles-pp.exit_mmap 0.77 ± 27% +0.5 1.24 ± 18% perf-profile.children.cycles-pp.mmput 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.children.cycles-pp.__x64_sys_exit_group 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.children.cycles-pp.do_group_exit 0.79 ± 26% +0.5 1.27 ± 18% perf-profile.children.cycles-pp.do_exit 1.28 ± 29% +0.5 1.76 ± 9% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.77 ± 28% +0.5 1.26 ± 13% perf-profile.children.cycles-pp.alloc_fresh_huge_page 1.53 ± 15% +0.7 2.26 ± 14% perf-profile.children.cycles-pp.do_syscall_64 1.53 ± 15% +0.7 2.27 ± 14% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 1.13 ± 3% +0.9 2.07 ± 14% perf-profile.children.cycles-pp.interrupt_entry 0.79 ± 9% +1.0 1.76 ± 5% perf-profile.children.cycles-pp.perf_event_task_tick 1.71 ± 39% +1.4 3.08 ± 16% perf-profile.children.cycles-pp.alloc_surplus_huge_page 2.66 ± 42% +2.3 4.94 ± 17% perf-profile.children.cycles-pp.alloc_huge_page 2.89 ± 45% +2.7 5.54 ± 18% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 3.34 ± 35% +2.7 6.02 ± 17% perf-profile.children.cycles-pp._raw_spin_lock 12.77 ± 14% +3.9 16.63 ± 7% perf-profile.children.cycles-pp.mutex_spin_on_owner 20.12 ± 9% +4.0 24.16 ± 6% perf-profile.children.cycles-pp.hugetlb_cow 15.40 ± 10% -3.6 11.84 ± 28% perf-profile.self.cycles-pp.do_rw_once 4.02 ± 9% -1.3 2.73 ± 30% perf-profile.self.cycles-pp.do_access 2.00 ± 14% -0.6 1.41 ± 13% perf-profile.self.cycles-pp.cpuidle_enter_state 1.26 ± 16% -0.5 0.74 ± 13% perf-profile.self.cycles-pp.native_sched_clock 0.42 ± 17% -0.2 0.27 ± 16% perf-profile.self.cycles-pp.account_process_tick 0.27 ± 19% -0.2 0.12 ± 17% perf-profile.self.cycles-pp.timerqueue_del 0.53 ± 3% -0.1 0.38 ± 11% perf-profile.self.cycles-pp.update_curr 0.27 ± 6% -0.1 0.14 ± 14% perf-profile.self.cycles-pp.__acct_update_integrals 0.27 ± 18% -0.1 0.16 ± 13% perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs 0.61 ± 4% -0.1 0.51 ± 8% perf-profile.self.cycles-pp.task_tick_fair 0.20 ± 8% -0.1 0.12 ± 14% perf-profile.self.cycles-pp.account_system_index_time 0.23 ± 15% -0.1 0.16 ± 17% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.25 ± 11% -0.1 0.18 ± 14% perf-profile.self.cycles-pp.find_next_bit 0.10 ± 11% -0.1 0.03 ±100% perf-profile.self.cycles-pp.tick_sched_do_timer 0.29 -0.1 0.23 ± 11% perf-profile.self.cycles-pp.timerqueue_add 0.12 ± 10% -0.1 0.06 ± 17% perf-profile.self.cycles-pp.account_user_time 0.22 ± 15% -0.1 0.16 ± 6% perf-profile.self.cycles-pp.scheduler_tick 0.17 ± 6% -0.0 0.12 ± 10% perf-profile.self.cycles-pp.cpuacct_charge 0.18 ± 20% -0.0 0.13 ± 3% perf-profile.self.cycles-pp.irq_work_tick 0.07 ± 13% -0.0 0.03 ±100% perf-profile.self.cycles-pp.update_process_times 0.12 ± 7% -0.0 0.08 ± 15% perf-profile.self.cycles-pp.get_cpu_device 0.07 ± 11% -0.0 0.04 ± 58% perf-profile.self.cycles-pp.raise_softirq 0.12 ± 11% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length 0.11 ± 11% +0.0 0.14 ± 6% perf-profile.self.cycles-pp.native_write_msr 0.10 ± 5% +0.1 0.15 ± 8% perf-profile.self.cycles-pp.__remove_hrtimer 0.07 ± 23% +0.1 0.13 ± 8% perf-profile.self.cycles-pp.rb_erase 0.08 ± 17% +0.1 0.15 ± 7% perf-profile.self.cycles-pp.native_apic_msr_eoi_write 0.00 +0.1 0.08 ± 10% perf-profile.self.cycles-pp.smp_call_function_single 0.32 ± 17% +0.1 0.42 ± 7% perf-profile.self.cycles-pp.run_timer_softirq 0.22 ± 5% +0.1 0.34 ± 4% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.45 ± 15% +0.2 0.60 ± 12% perf-profile.self.cycles-pp.rcu_irq_enter 0.31 ± 8% +0.2 0.46 ± 16% perf-profile.self.cycles-pp.irq_enter 0.29 ± 10% +0.2 0.44 ± 16% perf-profile.self.cycles-pp.apic_timer_interrupt 0.71 ± 30% +0.2 0.92 ± 8% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.00 +0.3 0.28 ± 37% perf-profile.self.cycles-pp.memcpy_erms 1.12 ± 3% +0.9 2.02 ± 15% perf-profile.self.cycles-pp.interrupt_entry 0.79 ± 9% +0.9 1.73 ± 5% perf-profile.self.cycles-pp.perf_event_task_tick 2.49 ± 45% +2.1 4.55 ± 20% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 10.95 ± 15% +2.7 13.61 ± 8% perf-profile.self.cycles-pp.mutex_spin_on_owner vm-scalability.throughput 1.6e+07 +-+---------------------------------------------------------------+ |..+.+ +..+.+..+.+. +. +..+.+..+.+..+.+..+.+..+ + | 1.4e+07 +-+ : : O O O O | 1.2e+07 O-+O O O O O O O O O O O O O O O O O O | : : O O O O | 1e+07 +-+ : : | | : : | 8e+06 +-+ : : | | : : | 6e+06 +-+ : : | 4e+06 +-+ : : | | :: | 2e+06 +-+ : | | : | 0 +-+---------------------------------------------------------------+ vm-scalability.time.minor_page_faults 2.5e+06 +-+---------------------------------------------------------------+ | | |..+.+ +..+.+..+.+..+.+..+.+.. .+. .+.+..+.+..+.+..+.+..+ | 2e+06 +-+ : : +. +. | O O O: O O O O O O O O O O | | : : O O O O O O O O O O O O O O 1.5e+06 +-+ : : | | : : | 1e+06 +-+ : : | | : : | | : : | 500000 +-+ : : | | : | | : | 0 +-+---------------------------------------------------------------+ vm-scalability.workload 3.5e+09 +-+---------------------------------------------------------------+ | .+. .+.+.. .+.. | 3e+09 +-+ + +..+.+..+.+..+.+. +..+.+..+.+..+.+..+.+..+ + | | : : O O O | 2.5e+09 O-+O O: O O O O O O O O O | | : : O O O O O O O O O O O O 2e+09 +-+ : : | | : : | 1.5e+09 +-+ : : | | : : | 1e+09 +-+ : : | | : : | 5e+08 +-+ : | | : | 0 +-+---------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

7 months

8
50
0 / 0

Re: [LKP] [SUNRPC] 0472e47660: fsmark.app_overhead 16.0% regression

by Xing Zhengjun

Hi Trond, On 7/8/2019 7:44 PM, Trond Myklebust wrote: > I've asked several times now about how to interpret your results. As far > as I can tell from your numbers, the overhead appears to be entirely > contained in the NUMA section of your results. > IOW: it would appear to be a scheduling overhead due to NUMA. I've been > asking whether or not that is a correct interpretation of the numbers > you published. Thanks for your feedback. I used the same hardware and the same test parameters to test the two commits: e791f8e938 ("SUNRPC: Convert xs_send_kvec() to use iov_iter_kvec()") 0472e47660 ("SUNRPC: Convert socket page send code to use iov_iter()") If it is caused by NUMA, why only commit 0472e47660 throughput is decreased? The filesystem we test is NFS, commit 0472e47660 is related with the network, could you help to check if have any other clues for the regression. Thanks. -- Zhengjun Xing

10 months, 2 weeks

1
5
0 / 0

[ptrace] 201766a20e: kernel_selftests.seccomp.make_fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 201766a20e30f982ccfe36bebfad9602c3ff574a ("ptrace: add PTRACE_GET_SYSCALL_INFO request") https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: kernel_selftests with following parameters: group: kselftests-02 test-description: The kernel contains a set of "self tests" under the tools/testing/selftests/ directory. These are intended to be small unit tests to exercise individual code paths in the kernel. test-url: https://www.kernel.org/doc/Documentation/kselftest.txt on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> 2019-07-26 16:52:03 make run_tests -C seccomp make: Entering directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' gcc -Wl,-no-as-needed -Wall seccomp_bpf.c -lpthread -o seccomp_bpf In file included from seccomp_bpf.c:51:0: seccomp_bpf.c: In function ‘tracer_ptrace’: seccomp_bpf.c:1787:20: error: ‘PTRACE_EVENTMSG_SYSCALL_ENTRY’ undeclared (first use in this function) EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1787:20: note: each undeclared identifier is reported only once for each function it appears in EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ seccomp_bpf.c:1788:6: error: ‘PTRACE_EVENTMSG_SYSCALL_EXIT’ undeclared (first use in this function) : PTRACE_EVENTMSG_SYSCALL_EXIT, msg); ^ ../kselftest_harness.h:608:13: note: in definition of macro ‘__EXPECT’ __typeof__(_expected) __exp = (_expected); \ ^~~~~~~~~ seccomp_bpf.c:1787:2: note: in expansion of macro ‘EXPECT_EQ’ EXPECT_EQ(entry ? PTRACE_EVENTMSG_SYSCALL_ENTRY ^~~~~~~~~ Makefile:12: recipe for target 'seccomp_bpf' failed make: *** [seccomp_bpf] Error 1 make: Leaving directory '/usr/src/perf_selftests-x86_64-rhel-7.6-201766a20e30f982ccfe36bebfad9602c3ff574a/tools/testing/selftests/seccomp' To reproduce: # build kernel cd linux cp config-5.2.0-10889-g201766a20e30f9 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

11 months, 2 weeks

3
2
0 / 0

[compat_ioctl] 97ca70e16d: mdadm-selftests.enchmarks/mdadm-selftests/tests/06name.fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 97ca70e16dc76bad69ad774cf0bd0286c97d63b4 ("compat_ioctl: scsi: move ioctl handling into drivers") https://kernel.googlesource.com/pub/scm/linux/kernel/git/arnd/playground.git to-build in testcase: mdadm-selftests with following parameters: disk: 1HDD test_prefix: 06 on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 4G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Welcome to fdisk (util-linux 2.29.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0x8b600b10. Command (m for help): Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): Partition number (1-4, default 1): First sector (2048-536870911, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-536870911, default 536870911): Created a new partition 1 of type 'Linux' and of size 5 GiB. Command (m for help): The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. 2019-07-28 07:22:06 mkdir -p /var/tmp 2019-07-28 07:22:06 mke2fs -t ext3 -b 1024 -J size=1 -q /dev/vda1 2019-07-28 07:22:08 mount -t ext3 /dev/vda1 /var/tmp sed -e 's/{DEFAULT_METADATA}/1.2/g' \ -e 's,{MAP_PATH},/run/mdadm/map,g' mdadm.8.in > mdadm.8 /usr/bin/install -D -m 644 mdadm.8 /usr/share/man/man8/mdadm.8 /usr/bin/install -D -m 644 mdmon.8 /usr/share/man/man8/mdmon.8 /usr/bin/install -D -m 644 md.4 /usr/share/man/man4/md.4 /usr/bin/install -D -m 644 mdadm.conf.5 /usr/share/man/man5/mdadm.conf.5 /usr/bin/install -D -m 644 udev-md-raid-creating.rules /lib/udev/rules.d/01-md-raid-creating.rules /usr/bin/install -D -m 644 udev-md-raid-arrays.rules /lib/udev/rules.d/63-md-raid-arrays.rules /usr/bin/install -D -m 644 udev-md-raid-assembly.rules /lib/udev/rules.d/64-md-raid-assembly.rules /usr/bin/install -D -m 644 udev-md-clustered-confirm-device.rules /lib/udev/rules.d/69-md-clustered-confirm-device.rules /usr/bin/install -D -m 755 mdadm /sbin/mdadm /usr/bin/install -D -m 755 mdmon /sbin/mdmon Testing on linux-5.3.0-rc1-g97ca70e16dc76b kernel /lkp/benchmarks/mdadm-selftests/tests/06name... FAILED - see /var/tmp/06name.log and /var/tmp/fail06name.log for details 06name TIMEOUT Testing on linux-5.3.0-rc1-g97ca70e16dc76b kernel /lkp/benchmarks/mdadm-selftests/tests/06sysfs... succeeded 06sysfs TIMEOUT Testing on linux-5.3.0-rc1-g97ca70e16dc76b kernel /lkp/benchmarks/mdadm-selftests/tests/06wrmostly... FAILED - see /var/tmp/06wrmostly.log and /var/tmp/fail06wrmostly.log for details 06wrmostly TIMEOUT To reproduce: # build kernel cd linux cp config-5.3.0-rc1-g97ca70e16dc76b .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install cd <mod-install-dir> find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email Thanks, Rong Chen

1 year

4
3
0 / 0

[mm] 6471384af2: kernel_BUG_at_mm/slub.c

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 6471384af2a6530696fc0203bafe4de41a23c9ef ("mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options") https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master in testcase: trinity with following parameters: runtime: 300s test-description: Trinity is a linux system call fuzz tester. test-url: http://codemonkey.org.uk/projects/trinity/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------+------------+------------+ | | ba5c5e4a5d | 6471384af2 | +------------------------------------------+------------+------------+ | boot_successes | 8 | 0 | | boot_failures | 2 | 15 | | invoked_oom-killer:gfp_mask=0x | 1 | | | Mem-Info | 1 | | | kernel_BUG_at_security/keys/keyring.c | 1 | | | invalid_opcode:#[##] | 1 | 15 | | RIP:__key_link_begin | 1 | | | Kernel_panic-not_syncing:Fatal_exception | 1 | 15 | | kernel_BUG_at_mm/slub.c | 0 | 15 | | RIP:kfree | 0 | 15 | +------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 4.478342] kernel BUG at mm/slub.c:306! [ 4.482437] invalid opcode: 0000 [#1] PREEMPT PTI [ 4.485750] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-05754-g6471384a #4 [ 4.490635] RIP: 0010:kfree+0x58a/0x5c0 [ 4.493679] Code: 48 83 05 78 37 51 02 01 0f 0b 48 83 05 7e 37 51 02 01 48 83 05 7e 37 51 02 01 48 83 05 7e 37 51 02 01 48 83 05 d6 37 51 02 01 <0f> 0b 48 83 05 d4 37 51 02 01 48 83 05 d4 37 51 02 01 48 83 05 d4 [ 4.506827] RSP: 0000:ffffffff82603d90 EFLAGS: 00010002 [ 4.510475] RAX: ffff8c3976c04320 RBX: ffff8c3976c04300 RCX: 0000000000000000 [ 4.515420] RDX: ffff8c3976c04300 RSI: 0000000000000000 RDI: ffff8c3976c04320 [ 4.520331] RBP: ffffffff82603db8 R08: 0000000000000000 R09: 0000000000000000 [ 4.525288] R10: ffff8c3976c04320 R11: ffffffff8289e1e0 R12: ffffd52cc8db0100 [ 4.530180] R13: ffff8c3976c01a00 R14: ffffffff810f10d4 R15: ffff8c3976c04300 [ 4.535079] FS: 0000000000000000(0000) GS:ffffffff8266b000(0000) knlGS:0000000000000000 [ 4.540628] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4.544593] CR2: ffff8c397ffff000 CR3: 0000000125020000 CR4: 00000000000406b0 [ 4.549558] Call Trace: [ 4.551266] apply_wqattrs_prepare+0x154/0x280 [ 4.554357] apply_workqueue_attrs_locked+0x4e/0xe0 [ 4.557728] apply_workqueue_attrs+0x36/0x60 [ 4.560654] alloc_workqueue+0x25a/0x6d0 [ 4.563381] ? kmem_cache_alloc_trace+0x1e3/0x500 [ 4.566628] ? __mutex_unlock_slowpath+0x44/0x3f0 [ 4.569875] workqueue_init_early+0x246/0x348 [ 4.573025] start_kernel+0x3c7/0x7ec [ 4.575558] x86_64_start_reservations+0x40/0x49 [ 4.578738] x86_64_start_kernel+0xda/0xe4 [ 4.581600] secondary_startup_64+0xb6/0xc0 [ 4.584473] Modules linked in: [ 4.586620] ---[ end trace f67eb9af4d8d492b ]--- To reproduce: # build kernel cd linux cp config-5.2.0-05754-g6471384a .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

1 year

6
7
0 / 0

[can] 60649d4e0a: ltp.can_rcv_own_msgs.fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 60649d4e0af6c26b6c423dea9c57f39e823fc0c5 ("can: remove obsolete empty ioctl() handler") https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next.git master in testcase: ltp with following parameters: test: can test-description: The LTP testsuite contains a collection of tools for testing the Linux kernel and related features. test-url: http://linux-test-project.github.io/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> <<<test_start>>> tag=can_rcv_own_msgs stime=1564179096 cmdline="can_run_tests.sh can_rcv_own_msgs" contacts="" analysis=exit <<<test_output>>> SIOCGIFINDEX: Operation not supported can_rcv_own_msgs 1 TFAIL: Test can_rcv_own_msgs FAIL incrementing stop <<<execution_status>>> initiation_status="ok" duration=0 termination_type=exited termination_id=1 corefile=no cutime=1 cstime=3 <<<test_end>>> To reproduce: # build kernel cd linux cp config-5.3.0-rc1-00114-g60649d4 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install cd <mod-install-dir> find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email Thanks, Rong Chen

1 year

2
1
0 / 0

[SUNRPC] 661ccd4df3: ltp.proc01.fail

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 661ccd4df38fdcddfeb1f314ac1a1de6c947c3f9 ("[RFC PATCH] SUNRPC: Harden the cache 'channel' interface to only allow legitimate daemons") url: https://github.com/0day-ci/linux/commits/Dave-Wysochanski/SUNRPC-Harden-t... base: git://linux-nfs.org/~bfields/linux.git nfsd-next in testcase: ltp with following parameters: disk: 1HDD fs: btrfs test: fs-02 test-description: The LTP testsuite contains a collection of tools for testing the Linux kernel and related features. test-url: http://linux-test-project.github.io/ on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> <<<test_start>>> tag=proc01 stime=1564252108 cmdline="proc01 -m 128" contacts="" analysis=exit <<<test_output>>> proc01 0 TINFO : /proc/sys/fs/binfmt_misc/register: is write-only. proc01 0 TINFO : /proc/sys/net/ipv6/conf/all/stable_secret: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/sys/net/ipv6/conf/default/stable_secret: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/sys/net/ipv6/conf/eth0/stable_secret: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/sys/net/ipv6/conf/lo/stable_secret: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/kmsg: known issue: errno=EAGAIN/EWOULDBLOCK(11): Resource temporarily unavailable proc01 0 TINFO : /proc/sysrq-trigger: is write-only. proc01 1 TFAIL : proc01.c:397: read failed: /proc/self/task/2597/net/rpc/auth.unix.ip/channel: errno=EIO(5): Input/output error proc01 2 TFAIL : proc01.c:397: read failed: /proc/self/task/2597/net/rpc/auth.unix.gid/channel: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/self/task/2597/mem: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/self/task/2597/clear_refs: is write-only. proc01 0 TINFO : /proc/self/task/2597/pagemap: reached maxmbytes (-m) proc01 3 TFAIL : proc01.c:397: read failed: /proc/self/net/rpc/auth.unix.ip/channel: errno=EIO(5): Input/output error proc01 4 TFAIL : proc01.c:397: read failed: /proc/self/net/rpc/auth.unix.gid/channel: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/self/mem: known issue: errno=EIO(5): Input/output error proc01 0 TINFO : /proc/self/clear_refs: is write-only. proc01 0 TINFO : /proc/self/pagemap: reached maxmbytes (-m) proc01 5 TFAIL : proc01.c:466: readproc() failed with 4 errors. <<<execution_status>>> initiation_status="ok" duration=0 termination_type=exited termination_id=1 corefile=no cutime=7 cstime=73 <<<test_end>>> To reproduce: # build kernel cd linux cp config-5.2.0-rc2-00038-g661ccd4df38fdc .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 olddefconfig prepare modules_prepare bzImage modules make HOSTCC=gcc-7 CC=gcc-7 ARCH=x86_64 INSTALL_MOD_PATH=<mod-install-dir> modules_install cd <mod-install-dir> find lib/ | cpio -o -H newc --quiet | gzip > modules.cgz git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> -m modules.cgz job-script # job-script is attached in this email Thanks, Rong Chen

1 year

2
1
0 / 0

[sched/fair] df703e3e7a: unixbench.score -9.1% regression

by kernel test robot

Greeting, FYI, we noticed a -9.1% regression of unixbench.score due to commit: commit: df703e3e7adf42061ec0f44d36ab4a3bab0d264f ("sched/fair: evenly spread tasks when not overloaded") https://git.linaro.org/people/vincent.guittot/kernel.git sched-rework-load_balance in testcase: unixbench on test machine: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory with following parameters: runtime: 300s nr_task: 30% test: shell1 cpufreq_governor: performance ucode: 0x7000017 test-description: UnixBench is the original BYTE UNIX benchmark suite aims to test performance of Unix-like system. test-url: https://github.com/kdlucas/byte-unixbench In addition to that, the commit also has significant impact on the following tests: Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode: gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-de1/shell1/unixbench/0x7000017 commit: 6768d1b110 ("sched/fair: use load instead of runnable load") df703e3e7a ("sched/fair: evenly spread tasks when not overloaded") 6768d1b11013709b df703e3e7adf42061ec0f44d36a ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 26% 1:4 perf-profile.calltrace.cycles-pp.error_entry 3:4 2% 3:4 perf-profile.children.cycles-pp.error_entry %stddev %change %stddev \ | \ 7247 -9.1% 6588 unixbench.score 1251828 +58.9% 1989223 unixbench.time.involuntary_context_switches 3.08e+08 -8.8% 2.81e+08 unixbench.time.minor_page_faults 535.25 -9.1% 486.75 unixbench.time.percent_of_cpu_this_job_got 1534 -6.9% 1428 unixbench.time.system_time 1863 -10.3% 1670 unixbench.time.user_time 8830673 -12.6% 7720971 unixbench.time.voluntary_context_switches 19420772 -8.5% 17767588 unixbench.workload 37.74 +2.6% 38.72 boot-time.boot 543.75 +2.8% 559.16 boot-time.idle 0.01 ± 58% -0.0 0.00 ± 90% mpstat.cpu.all.soft% 14.91 -1.5 13.40 mpstat.cpu.all.usr% 21698 ± 2% -10.7% 19380 ± 7% slabinfo.kmalloc-32.active_objs 21836 ± 2% -9.4% 19781 ± 5% slabinfo.kmalloc-32.num_objs 2056 ± 5% -9.6% 1858 ± 7% slabinfo.signal_cache.active_objs 66.25 +4.2% 69.00 vmstat.cpu.id 14.50 ± 3% -10.3% 13.00 vmstat.cpu.us 39338 -6.6% 36729 vmstat.system.cs 14169148 +58.2% 22420947 ± 12% cpuidle.C1.time 3.173e+08 ± 2% +155.1% 8.094e+08 ± 10% cpuidle.C1E.time 3362901 +155.3% 8586261 ± 16% cpuidle.C1E.usage 5.039e+09 ± 10% -68.7% 1.575e+09 ± 13% cpuidle.C3.time 16636190 ± 6% -63.3% 6105608 ± 26% cpuidle.C3.usage 1.219e+09 ± 41% +268.1% 4.487e+09 ± 3% cpuidle.C6.time 2627144 ± 40% +172.4% 7156187 ± 2% cpuidle.C6.usage 5175 -0.8% 5133 proc-vmstat.nr_kernel_stack 13238 -6.9% 12325 proc-vmstat.nr_shmem 2.404e+08 -8.9% 2.19e+08 proc-vmstat.numa_hit 2.404e+08 -8.9% 2.19e+08 proc-vmstat.numa_local 11012 -10.8% 9826 ± 2% proc-vmstat.pgactivate 2.522e+08 -8.9% 2.298e+08 proc-vmstat.pgalloc_normal 3.09e+08 -8.8% 2.819e+08 proc-vmstat.pgfault 2.522e+08 -8.9% 2.298e+08 proc-vmstat.pgfree 10260 -9.2% 9313 proc-vmstat.thp_deferred_split_page 10260 -9.2% 9313 proc-vmstat.thp_fault_alloc 4283304 -9.0% 3896547 proc-vmstat.unevictable_pgs_culled 880.25 -8.7% 803.25 turbostat.Avg_MHz 0.14 +0.1 0.22 ± 11% turbostat.C1% 3362656 +155.3% 8586090 ± 16% turbostat.C1E 3.12 ± 2% +4.8 7.93 ± 10% turbostat.C1E% 16635629 ± 6% -63.3% 6104998 ± 26% turbostat.C3 49.46 ± 10% -34.0 15.44 ± 13% turbostat.C3% 2626288 ± 40% +172.5% 7155331 ± 2% turbostat.C6 11.96 ± 41% +32.0 43.98 ± 3% turbostat.C6% 52.78 -8.4% 48.33 turbostat.CPU%c1 10.12 ± 7% -51.3% 4.93 ± 12% turbostat.CPU%c3 1.77 ± 43% +718.4% 14.48 ± 4% turbostat.CPU%c6 0.17 ±140% +1064.3% 2.04 ± 9% turbostat.Pkg%pc6 32.48 -4.4% 31.04 turbostat.PkgWatt 2.96 -1.0% 2.93 turbostat.RAMWatt 113978 -68.5% 35880 ± 3% softirqs.CPU0.SCHED 110736 -71.5% 31597 ± 3% softirqs.CPU1.SCHED 109767 -71.8% 30908 ± 3% softirqs.CPU10.SCHED 109411 -71.7% 30921 softirqs.CPU11.SCHED 109446 -71.9% 30770 ± 2% softirqs.CPU12.SCHED 111145 -71.6% 31571 softirqs.CPU13.SCHED 242944 +12.0% 272158 ± 11% softirqs.CPU13.TIMER 110147 -71.4% 31536 ± 2% softirqs.CPU14.SCHED 108791 -71.5% 31029 ± 3% softirqs.CPU15.SCHED 110362 -69.8% 33318 ± 3% softirqs.CPU2.SCHED 109552 -70.8% 31953 ± 3% softirqs.CPU3.SCHED 110342 -71.6% 31390 softirqs.CPU4.SCHED 111136 -71.2% 32041 ± 3% softirqs.CPU5.SCHED 111095 -69.7% 33631 ± 4% softirqs.CPU6.SCHED 108435 ± 4% -71.2% 31186 softirqs.CPU7.SCHED 108589 -72.0% 30358 ± 2% softirqs.CPU8.SCHED 109600 -71.5% 31203 softirqs.CPU9.SCHED 1762540 -71.1% 509301 softirqs.SCHED 98982 -9.0% 90123 sched_debug.cfs_rq:/.exec_clock.min 203.76 ± 6% +48.8% 303.14 ± 5% sched_debug.cfs_rq:/.load_avg.avg 301.13 ± 7% +22.4% 368.57 ± 4% sched_debug.cfs_rq:/.load_avg.stddev 1234338 -18.9% 1000533 sched_debug.cfs_rq:/.min_vruntime.avg 1272643 -18.7% 1035067 sched_debug.cfs_rq:/.min_vruntime.max 1204874 ± 2% -18.6% 981312 sched_debug.cfs_rq:/.min_vruntime.min 129.91 ± 9% +76.1% 228.75 ± 7% sched_debug.cfs_rq:/.removed.load_avg.avg 293.63 ± 10% +27.1% 373.26 ± 5% sched_debug.cfs_rq:/.removed.load_avg.stddev 5990 ± 9% +76.1% 10549 ± 7% sched_debug.cfs_rq:/.removed.runnable_sum.avg 13541 ± 10% +27.1% 17214 ± 5% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 54.48 ± 9% +66.3% 90.62 ± 8% sched_debug.cfs_rq:/.removed.util_avg.avg 126.25 ± 11% +21.5% 153.35 ± 5% sched_debug.cfs_rq:/.removed.util_avg.stddev 1282 ±241% -1131.5% -13231 sched_debug.cfs_rq:/.spread0.avg 39588 ± 4% -46.2% 21304 ± 25% sched_debug.cfs_rq:/.spread0.max 512437 +14.6% 587353 sched_debug.cpu.avg_idle.avg 147096 ± 11% +44.0% 211769 ± 4% sched_debug.cpu.avg_idle.min 0.63 ± 10% -14.3% 0.54 ± 2% sched_debug.cpu.clock.stddev 0.63 ± 11% -14.4% 0.54 ± 2% sched_debug.cpu.clock_task.stddev 6738 ± 7% -34.0% 4444 ± 10% sched_debug.cpu.curr->pid.avg 18871 ± 3% -31.4% 12939 ± 12% sched_debug.cpu.curr->pid.max 8725 ± 2% -31.8% 5952 ± 12% sched_debug.cpu.curr->pid.stddev 1.75 ± 2% +11.7% 1.95 ± 7% sched_debug.cpu.nr_running.max 0.61 ± 2% +12.4% 0.68 ± 4% sched_debug.cpu.nr_running.stddev 20.91 ± 11% +44.5% 30.20 ± 15% sched_debug.cpu.nr_uninterruptible.max 335489 -14.4% 287208 sched_debug.cpu.sched_goidle.avg 339306 -13.6% 293002 sched_debug.cpu.sched_goidle.max 328517 ± 2% -13.9% 282910 sched_debug.cpu.sched_goidle.min 2.641e+09 -9.0% 2.403e+09 perf-stat.i.branch-instructions 2.51 ± 2% +0.3 2.78 perf-stat.i.branch-miss-rate% 62579199 -4.9% 59521032 perf-stat.i.branch-misses 2.353e+08 -7.3% 2.182e+08 perf-stat.i.cache-misses 2.353e+08 -7.3% 2.182e+08 perf-stat.i.cache-references 39504 -6.7% 36859 perf-stat.i.context-switches 1.384e+10 -9.0% 1.26e+10 perf-stat.i.cpu-cycles 2748 +43.4% 3942 perf-stat.i.cpu-migrations 60.35 -2.6% 58.76 perf-stat.i.cycles-between-cache-misses 10262457 ± 2% -8.0% 9438387 perf-stat.i.dTLB-load-misses 3.407e+09 -8.8% 3.108e+09 perf-stat.i.dTLB-loads 2268647 -8.4% 2078911 perf-stat.i.dTLB-store-misses 2.16e+09 -8.9% 1.968e+09 perf-stat.i.dTLB-stores 63.12 -1.0 62.13 perf-stat.i.iTLB-load-miss-rate% 4880236 -8.8% 4452641 perf-stat.i.iTLB-load-misses 2964633 -8.8% 2704009 perf-stat.i.iTLB-loads 1.32e+10 -9.0% 1.201e+10 perf-stat.i.instructions 476481 -8.9% 434097 perf-stat.i.minor-faults 476481 -8.9% 434097 perf-stat.i.page-faults 17.83 +1.9% 18.16 perf-stat.overall.MPKI 2.37 +0.1 2.48 perf-stat.overall.branch-miss-rate% 58.84 -1.9% 57.74 perf-stat.overall.cycles-between-cache-misses 2.636e+09 -9.0% 2.399e+09 perf-stat.ps.branch-instructions 62480492 -4.9% 59426662 perf-stat.ps.branch-misses 2.349e+08 -7.3% 2.179e+08 perf-stat.ps.cache-misses 2.349e+08 -7.3% 2.179e+08 perf-stat.ps.cache-references 39441 -6.7% 36800 perf-stat.ps.context-switches 1.382e+10 -9.0% 1.258e+10 perf-stat.ps.cpu-cycles 2744 +43.4% 3936 perf-stat.ps.cpu-migrations 10246227 ± 2% -8.0% 9423402 perf-stat.ps.dTLB-load-misses 3.402e+09 -8.8% 3.103e+09 perf-stat.ps.dTLB-loads 2265060 -8.4% 2075611 perf-stat.ps.dTLB-store-misses 2.156e+09 -8.9% 1.965e+09 perf-stat.ps.dTLB-stores 4872516 -8.8% 4445572 perf-stat.ps.iTLB-load-misses 2959944 -8.8% 2699716 perf-stat.ps.iTLB-loads 1.318e+10 -9.0% 1.199e+10 perf-stat.ps.instructions 475727 -8.9% 433407 perf-stat.ps.minor-faults 475727 -8.9% 433407 perf-stat.ps.page-faults 8.383e+12 -8.9% 7.637e+12 perf-stat.total.instructions 271375 ± 3% +41.3% 383460 ± 2% interrupts.CAL:Function_call_interrupts 16904 ± 3% +42.2% 24045 ± 2% interrupts.CPU0.CAL:Function_call_interrupts 8546 ± 3% -18.1% 6995 ± 2% interrupts.CPU0.RES:Rescheduling_interrupts 11034 ± 4% +65.7% 18287 ± 3% interrupts.CPU0.TLB:TLB_shootdowns 17057 ± 3% +40.5% 23964 ± 2% interrupts.CPU1.CAL:Function_call_interrupts 3847 -24.1% 2921 ± 23% interrupts.CPU1.NMI:Non-maskable_interrupts 3847 -24.1% 2921 ± 23% interrupts.CPU1.PMI:Performance_monitoring_interrupts 8181 ± 2% -16.3% 6850 ± 2% interrupts.CPU1.RES:Rescheduling_interrupts 11214 ± 4% +62.8% 18259 ± 3% interrupts.CPU1.TLB:TLB_shootdowns 16877 ± 2% +42.0% 23961 ± 2% interrupts.CPU10.CAL:Function_call_interrupts 3810 ± 2% -23.6% 2911 ± 25% interrupts.CPU10.NMI:Non-maskable_interrupts 3810 ± 2% -23.6% 2911 ± 25% interrupts.CPU10.PMI:Performance_monitoring_interrupts 8428 ± 7% -23.5% 6450 interrupts.CPU10.RES:Rescheduling_interrupts 11090 ± 3% +65.1% 18308 ± 3% interrupts.CPU10.TLB:TLB_shootdowns 16786 ± 3% +41.2% 23704 ± 4% interrupts.CPU11.CAL:Function_call_interrupts 3821 ± 2% -20.7% 3030 ± 25% interrupts.CPU11.NMI:Non-maskable_interrupts 3821 ± 2% -20.7% 3030 ± 25% interrupts.CPU11.PMI:Performance_monitoring_interrupts 8134 ± 2% -20.6% 6457 interrupts.CPU11.RES:Rescheduling_interrupts 11081 ± 5% +63.4% 18105 ± 4% interrupts.CPU11.TLB:TLB_shootdowns 16954 ± 2% +40.3% 23786 ± 2% interrupts.CPU12.CAL:Function_call_interrupts 3829 ± 2% -24.2% 2903 ± 25% interrupts.CPU12.NMI:Non-maskable_interrupts 3829 ± 2% -24.2% 2903 ± 25% interrupts.CPU12.PMI:Performance_monitoring_interrupts 8087 -20.5% 6427 interrupts.CPU12.RES:Rescheduling_interrupts 11183 ± 3% +61.9% 18102 ± 3% interrupts.CPU12.TLB:TLB_shootdowns 17016 ± 2% +39.9% 23803 ± 2% interrupts.CPU13.CAL:Function_call_interrupts 3818 ± 2% -21.5% 2997 ± 25% interrupts.CPU13.NMI:Non-maskable_interrupts 3818 ± 2% -21.5% 2997 ± 25% interrupts.CPU13.PMI:Performance_monitoring_interrupts 8189 ± 2% -19.6% 6586 ± 2% interrupts.CPU13.RES:Rescheduling_interrupts 11176 ± 3% +62.9% 18201 ± 3% interrupts.CPU13.TLB:TLB_shootdowns 17027 ± 3% +40.3% 23887 ± 3% interrupts.CPU14.CAL:Function_call_interrupts 3795 ± 2% -22.3% 2950 ± 25% interrupts.CPU14.NMI:Non-maskable_interrupts 3795 ± 2% -22.3% 2950 ± 25% interrupts.CPU14.PMI:Performance_monitoring_interrupts 8099 ± 2% -18.2% 6625 interrupts.CPU14.RES:Rescheduling_interrupts 11250 ± 4% +61.9% 18219 ± 3% interrupts.CPU14.TLB:TLB_shootdowns 16814 ± 2% +43.5% 24125 ± 3% interrupts.CPU15.CAL:Function_call_interrupts 3848 ± 2% -22.8% 2971 ± 26% interrupts.CPU15.NMI:Non-maskable_interrupts 3848 ± 2% -22.8% 2971 ± 26% interrupts.CPU15.PMI:Performance_monitoring_interrupts 8035 -19.0% 6512 interrupts.CPU15.RES:Rescheduling_interrupts 10929 ± 3% +68.7% 18439 ± 4% interrupts.CPU15.TLB:TLB_shootdowns 16981 ± 3% +42.3% 24161 ± 2% interrupts.CPU2.CAL:Function_call_interrupts 3796 ± 2% -23.4% 2908 ± 24% interrupts.CPU2.NMI:Non-maskable_interrupts 3796 ± 2% -23.4% 2908 ± 24% interrupts.CPU2.PMI:Performance_monitoring_interrupts 8239 ± 3% -19.3% 6646 interrupts.CPU2.RES:Rescheduling_interrupts 11177 ± 5% +65.3% 18476 ± 3% interrupts.CPU2.TLB:TLB_shootdowns 17045 ± 3% +40.9% 24009 ± 2% interrupts.CPU3.CAL:Function_call_interrupts 3848 -23.6% 2939 ± 25% interrupts.CPU3.NMI:Non-maskable_interrupts 3848 -23.6% 2939 ± 25% interrupts.CPU3.PMI:Performance_monitoring_interrupts 8056 -17.3% 6665 ± 2% interrupts.CPU3.RES:Rescheduling_interrupts 11182 ± 4% +63.7% 18301 ± 3% interrupts.CPU3.TLB:TLB_shootdowns 16862 ± 3% +42.2% 23982 ± 2% interrupts.CPU4.CAL:Function_call_interrupts 3813 ± 3% -23.3% 2924 ± 25% interrupts.CPU4.NMI:Non-maskable_interrupts 3813 ± 3% -23.3% 2924 ± 25% interrupts.CPU4.PMI:Performance_monitoring_interrupts 8116 -18.9% 6584 ± 2% interrupts.CPU4.RES:Rescheduling_interrupts 11229 ± 4% +62.5% 18248 ± 3% interrupts.CPU4.TLB:TLB_shootdowns 17071 ± 3% +41.0% 24068 ± 2% interrupts.CPU5.CAL:Function_call_interrupts 3834 ± 2% -11.4% 3398 interrupts.CPU5.NMI:Non-maskable_interrupts 3834 ± 2% -11.4% 3398 interrupts.CPU5.PMI:Performance_monitoring_interrupts 8302 -19.9% 6650 ± 2% interrupts.CPU5.RES:Rescheduling_interrupts 11258 ± 4% +63.1% 18368 ± 3% interrupts.CPU5.TLB:TLB_shootdowns 17028 ± 3% +42.4% 24243 ± 2% interrupts.CPU6.CAL:Function_call_interrupts 3793 ± 2% -21.6% 2976 ± 25% interrupts.CPU6.NMI:Non-maskable_interrupts 3793 ± 2% -21.6% 2976 ± 25% interrupts.CPU6.PMI:Performance_monitoring_interrupts 8099 ± 2% -18.1% 6633 interrupts.CPU6.RES:Rescheduling_interrupts 11197 ± 5% +65.8% 18564 ± 3% interrupts.CPU6.TLB:TLB_shootdowns 16668 ± 2% +43.4% 23899 ± 3% interrupts.CPU7.CAL:Function_call_interrupts 3845 -20.3% 3063 ± 24% interrupts.CPU7.NMI:Non-maskable_interrupts 3845 -20.3% 3063 ± 24% interrupts.CPU7.PMI:Performance_monitoring_interrupts 7819 ± 5% -13.7% 6745 interrupts.CPU7.RES:Rescheduling_interrupts 11063 +65.4% 18303 ± 3% interrupts.CPU7.TLB:TLB_shootdowns 17177 ± 4% +40.3% 24100 ± 2% interrupts.CPU8.CAL:Function_call_interrupts 3773 ± 3% -22.7% 2915 ± 24% interrupts.CPU8.NMI:Non-maskable_interrupts 3773 ± 3% -22.7% 2915 ± 24% interrupts.CPU8.PMI:Performance_monitoring_interrupts 7947 -18.9% 6442 interrupts.CPU8.RES:Rescheduling_interrupts 11288 ± 5% +63.9% 18499 ± 3% interrupts.CPU8.TLB:TLB_shootdowns 17100 ± 4% +38.7% 23715 ± 2% interrupts.CPU9.CAL:Function_call_interrupts 3845 -24.5% 2903 ± 25% interrupts.CPU9.NMI:Non-maskable_interrupts 3845 -24.5% 2903 ± 25% interrupts.CPU9.PMI:Performance_monitoring_interrupts 8145 ± 2% -20.1% 6506 interrupts.CPU9.RES:Rescheduling_interrupts 11189 ± 5% +62.0% 18126 ± 3% interrupts.CPU9.TLB:TLB_shootdowns 60656 -21.5% 47635 ± 23% interrupts.NMI:Non-maskable_interrupts 60656 -21.5% 47635 ± 23% interrupts.PMI:Performance_monitoring_interrupts 130427 -18.9% 105780 interrupts.RES:Rescheduling_interrupts 178546 ± 4% +64.0% 292811 ± 3% interrupts.TLB:TLB_shootdowns 39.70 -2.0 37.72 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 38.83 ± 2% -1.9 36.95 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 38.75 ± 2% -1.9 36.87 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 39.79 ± 2% -1.7 38.05 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 39.79 ± 2% -1.7 38.05 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 39.78 ± 2% -1.7 38.03 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 42.09 -1.7 40.42 perf-profile.calltrace.cycles-pp.secondary_startup_64 0.59 ± 2% -0.2 0.43 ± 57% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 0.55 ± 2% -0.2 0.40 ± 57% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.21 ± 3% +0.1 1.31 ± 5% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.83 ± 4% +0.1 0.92 ± 4% perf-profile.calltrace.cycles-pp.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.83 ± 4% +0.1 0.93 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.10 +0.2 4.26 perf-profile.calltrace.cycles-pp.setlocale 0.40 ± 58% +0.2 0.58 ± 7% perf-profile.calltrace.cycles-pp.create_elf_tables.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve 0.43 ± 57% +0.2 0.61 ± 7% perf-profile.calltrace.cycles-pp.__fxstat64.setlocale 5.18 +0.3 5.51 perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64 5.20 +0.3 5.54 perf-profile.calltrace.cycles-pp.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.16 +0.3 11.50 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 11.12 +0.3 11.47 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.29 ±101% +0.4 0.65 ± 4% perf-profile.calltrace.cycles-pp.ret_from_fork 0.27 ±100% +0.4 0.65 ± 4% perf-profile.calltrace.cycles-pp.wait4 0.14 ±173% +0.4 0.56 ± 5% perf-profile.calltrace.cycles-pp.search_binary_handler.load_script.search_binary_handler.__do_execve_file.__x64_sys_execve 0.14 ±173% +0.4 0.56 ± 5% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.load_script.search_binary_handler.__do_execve_file 0.15 ±173% +0.5 0.61 ± 5% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 0.12 ±173% +0.5 0.62 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4 0.12 ±173% +0.5 0.62 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait4 7.60 +0.6 8.15 perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 7.58 +0.6 8.13 perf-profile.calltrace.cycles-pp.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 7.60 +0.6 8.16 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 7.60 +0.6 8.16 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 7.63 +0.6 8.19 perf-profile.calltrace.cycles-pp.execve 39.70 -2.0 37.72 perf-profile.children.cycles-pp.intel_idle 41.08 -1.8 39.26 perf-profile.children.cycles-pp.cpuidle_enter_state 41.08 -1.8 39.26 perf-profile.children.cycles-pp.cpuidle_enter 39.79 ± 2% -1.7 38.05 perf-profile.children.cycles-pp.start_secondary 42.09 -1.7 40.42 perf-profile.children.cycles-pp.do_idle 42.09 -1.7 40.42 perf-profile.children.cycles-pp.secondary_startup_64 42.09 -1.7 40.42 perf-profile.children.cycles-pp.cpu_startup_entry 1.06 ± 4% -0.1 0.99 ± 5% perf-profile.children.cycles-pp.page_remove_rmap 0.23 ± 5% -0.0 0.20 ± 4% perf-profile.children.cycles-pp.__vmalloc_node_range 0.11 ± 6% -0.0 0.09 ± 12% perf-profile.children.cycles-pp.__list_add_valid 0.08 ± 15% -0.0 0.06 ± 20% perf-profile.children.cycles-pp.file_has_perm 0.11 ± 4% -0.0 0.09 ± 7% perf-profile.children.cycles-pp.getopt_long 0.10 ± 4% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.setup_new_exec 0.10 ± 12% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.native_write_msr 0.10 ± 9% +0.0 0.11 ± 9% perf-profile.children.cycles-pp.kfree 0.07 ± 7% +0.0 0.09 ± 10% perf-profile.children.cycles-pp.lapic_next_deadline 0.06 ± 15% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.perf_event_comm 0.09 ± 9% +0.0 0.11 ± 6% perf-profile.children.cycles-pp.move_page_tables 0.09 ± 13% +0.0 0.11 ± 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.20 ± 3% +0.0 0.22 ± 7% perf-profile.children.cycles-pp.d_path 0.12 ± 5% +0.0 0.15 ± 14% perf-profile.children.cycles-pp.may_open 0.38 ± 6% +0.0 0.41 ± 5% perf-profile.children.cycles-pp.unlink_file_vma 0.17 ± 6% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.__get_user_8 0.12 ± 11% +0.0 0.15 ± 2% perf-profile.children.cycles-pp.vm_area_dup 0.04 ± 57% +0.0 0.07 ± 6% perf-profile.children.cycles-pp.vma_merge 0.09 ± 4% +0.0 0.12 ± 15% perf-profile.children.cycles-pp.free_ldt_pgtables 0.08 ± 10% +0.0 0.11 ± 12% perf-profile.children.cycles-pp.__errno_location 0.21 ± 4% +0.0 0.25 ± 3% perf-profile.children.cycles-pp.signal_wake_up_state 0.10 ± 14% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.do_task_dead 0.07 ± 17% +0.0 0.10 ± 13% perf-profile.children.cycles-pp.vm_brk_flags 0.30 ± 7% +0.0 0.34 perf-profile.children.cycles-pp.do_notify_parent 0.25 ± 5% +0.0 0.29 perf-profile.children.cycles-pp.__send_signal 0.03 ±100% +0.0 0.07 ± 13% perf-profile.children.cycles-pp.security_bprm_set_creds 0.17 ± 2% +0.0 0.21 ± 7% perf-profile.children.cycles-pp.switch_mm_irqs_off 0.20 ± 4% +0.0 0.24 ± 5% perf-profile.children.cycles-pp.shift_arg_pages 0.37 ± 5% +0.0 0.41 ± 3% perf-profile.children.cycles-pp.setup_arg_pages 0.03 ±100% +0.0 0.07 ± 10% perf-profile.children.cycles-pp.__d_lookup_done 0.54 ± 4% +0.0 0.59 ± 4% perf-profile.children.cycles-pp.selinux_inode_permission 0.45 ± 3% +0.0 0.50 ± 4% perf-profile.children.cycles-pp.vma_interval_tree_insert 0.15 ± 5% +0.1 0.21 ± 2% perf-profile.children.cycles-pp.__lru_cache_add 0.39 ± 7% +0.1 0.44 ± 5% perf-profile.children.cycles-pp.pipe_read 0.04 ± 60% +0.1 0.10 ± 12% perf-profile.children.cycles-pp.preempt_schedule_common 0.24 ± 2% +0.1 0.30 ± 6% perf-profile.children.cycles-pp.open_exec 0.03 ±100% +0.1 0.08 ± 17% perf-profile.children.cycles-pp.move_queued_task 0.54 ± 6% +0.1 0.61 ± 4% perf-profile.children.cycles-pp.load_script 0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp._get_random_bytes 0.40 ± 2% +0.1 0.46 ± 5% perf-profile.children.cycles-pp.do_open_execat 0.30 ± 3% +0.1 0.37 ± 6% perf-profile.children.cycles-pp.menu_select 0.12 ± 15% +0.1 0.19 ± 8% perf-profile.children.cycles-pp.find_busiest_group 0.17 ± 13% +0.1 0.23 ± 4% perf-profile.children.cycles-pp.load_balance 0.03 ±100% +0.1 0.10 ± 39% perf-profile.children.cycles-pp.delay_tsc 0.72 ± 3% +0.1 0.80 ± 4% perf-profile.children.cycles-pp.vma_link 0.59 ± 6% +0.1 0.67 ± 5% perf-profile.children.cycles-pp.create_elf_tables 0.98 ± 5% +0.1 1.07 perf-profile.children.cycles-pp.perf_event_mmap 0.02 ±173% +0.1 0.12 ± 31% perf-profile.children.cycles-pp.wake_up_klogd_work_func 0.37 ± 6% +0.1 0.47 ± 3% perf-profile.children.cycles-pp.do_wait 0.38 ± 7% +0.1 0.49 ± 4% perf-profile.children.cycles-pp.kernel_wait4 0.39 ± 7% +0.1 0.50 ± 6% perf-profile.children.cycles-pp.__do_sys_wait4 0.15 ± 8% +0.1 0.27 ± 6% perf-profile.children.cycles-pp.smpboot_thread_fn 0.29 ± 21% +0.1 0.42 ± 14% perf-profile.children.cycles-pp.update_process_times 0.31 ± 20% +0.1 0.44 ± 13% perf-profile.children.cycles-pp.tick_sched_handle 0.48 ± 16% +0.1 0.61 ± 5% perf-profile.children.cycles-pp.kthread 0.52 ± 4% +0.1 0.66 ± 4% perf-profile.children.cycles-pp.wait4 0.81 ± 10% +0.1 0.95 ± 2% perf-profile.children.cycles-pp.ret_from_fork 0.12 ± 15% +0.1 0.27 ± 44% perf-profile.children.cycles-pp.uart_console_write 4.62 +0.2 4.77 perf-profile.children.cycles-pp.setlocale 0.42 ± 6% +0.2 0.58 ± 6% perf-profile.children.cycles-pp.schedule 0.36 ± 17% +0.2 0.51 ± 12% perf-profile.children.cycles-pp.tick_sched_timer 0.30 ± 10% +0.2 0.45 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 0.13 ± 14% +0.2 0.30 ± 45% perf-profile.children.cycles-pp.console_unlock 2.20 ± 3% +0.2 2.37 ± 3% perf-profile.children.cycles-pp.mmap_region 0.09 ± 48% +0.2 0.27 ± 50% perf-profile.children.cycles-pp.irq_work_run_list 0.62 ± 12% +0.2 0.83 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.85 ± 9% +0.2 1.09 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt 0.99 +0.3 1.25 ± 2% perf-profile.children.cycles-pp.__schedule 5.68 +0.4 6.08 perf-profile.children.cycles-pp.load_elf_binary 5.75 +0.4 6.15 perf-profile.children.cycles-pp.search_binary_handler 7.63 +0.6 8.19 perf-profile.children.cycles-pp.execve 8.41 +0.6 9.05 perf-profile.children.cycles-pp.__do_execve_file 8.43 +0.7 9.08 perf-profile.children.cycles-pp.__x64_sys_execve 29.84 +1.3 31.14 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 29.74 +1.3 31.05 perf-profile.children.cycles-pp.do_syscall_64 39.65 -2.0 37.66 perf-profile.self.cycles-pp.intel_idle 0.59 -0.0 0.57 ± 4% perf-profile.self.cycles-pp.__handle_mm_fault 0.08 ± 5% -0.0 0.06 ± 20% perf-profile.self.cycles-pp.__call_rcu 0.17 ± 4% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.mmap_region 0.05 ± 8% +0.0 0.07 ± 10% perf-profile.self.cycles-pp.time 0.10 ± 12% +0.0 0.12 ± 8% perf-profile.self.cycles-pp.native_write_msr 0.17 ± 6% +0.0 0.20 ± 5% perf-profile.self.cycles-pp.__get_user_8 0.10 ± 4% +0.0 0.13 ± 8% perf-profile.self.cycles-pp.create_elf_tables 0.04 ± 57% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.vma_merge 0.18 ± 6% +0.0 0.22 ± 6% perf-profile.self.cycles-pp.perf_iterate_sb 0.10 ± 13% +0.0 0.13 ± 5% perf-profile.self.cycles-pp.__check_object_size 0.08 ± 5% +0.0 0.12 ± 15% perf-profile.self.cycles-pp.update_load_avg 0.11 ± 11% +0.0 0.15 ± 11% perf-profile.self.cycles-pp.__schedule 0.45 ± 3% +0.0 0.49 ± 4% perf-profile.self.cycles-pp.vma_interval_tree_insert 0.09 ± 20% +0.1 0.14 ± 10% perf-profile.self.cycles-pp.find_busiest_group 0.21 ± 7% +0.1 0.27 ± 5% perf-profile.self.cycles-pp.menu_select 0.03 ±100% +0.1 0.10 ± 39% perf-profile.self.cycles-pp.delay_tsc 2.13 +0.1 2.27 perf-profile.self.cycles-pp.do_syscall_64 unixbench.time.user_time 2000 +-+------------------------------------------------------------------+ 1800 +-+ : : +++.+ + ++ +.+ +.++.+ +.+++ +.+++ ++.+ + ++.++.+| OO O OOO OOO OO OOO OOO OO OOO | 1600 +-+ : : : | 1400 +-+ : : : | | : : : | 1200 +-+ : : : | 1000 +-+ : :: | 800 +-+: :: | | : :: | 600 +-+: :: | 400 +-+: : | | : : | 200 +-+: : | 0 +-+O-----------------------------------------------------------------+ unixbench.time.percent_of_cpu_this_job_got 600 +-+-------------------------------------------------------------------+ | +.++ ++.+++.++.+++.++.+++.++.+++.++.+++.++.+++.++.+++.++.+++. +. | 500 +-+ : : : + +| OO O OOO OO OOO OO OOO OO OOO O | | : : : | 400 +-+ : : : | | : : : | 300 +-+ : : : | | : :: | 200 +-+: :: | | : :: | | : : | 100 +-+: : | | : : | 0 +-+O------------------------------------------------------------------+ unixbench.time.minor_page_faults 3.5e+08 +-+---------------------------------------------------------------+ | ++.+ +.+++.++++.+++.+++.+++.+++.+++.+++.+++.++++.+++.+++.+ | 3e+08 +-+ : : : ++.+| OO OO OOO OOO OOOO OOO OOO OO | 2.5e+08 +-+ : : : | | : : : | 2e+08 +-+ : : : | | : : : | 1.5e+08 +-+: : | | : : | 1e+08 +-+: : | | : : | 5e+07 +-+: : | | : : | 0 +-+O--------------------------------------------------------------+ unixbench.time.voluntary_context_switches 1e+07 +-+-----------------------------------------------------------------+ 9e+06 +-+ ++.+ +++.+++.+++.+++.++.+++.+++.+++.++.+++.+++.+++.+++.++.+ | | : : : ++.+| 8e+06 OO+ OO OO OOO OOO OOO OOO OO OO | 7e+06 +-+ : : : | | : : : | 6e+06 +-+ : : : | 5e+06 +-+ : : : | 4e+06 +-+: :: | | : :: | 3e+06 +-+: :: | 2e+06 +-+: : | | : : | 1e+06 +-+: : | 0 +-+O----------------------------------------------------------------+ unixbench.time.involuntary_context_switches 2.5e+06 +-+---------------------------------------------------------------+ | | | | 2e+06 OO+ OO OOO OOO OOOO OOO OOO OO | | | | | 1.5e+06 +-+ | | ++.+ +.+++.++++.+++.+++.+++.+++.+++.+++.+++.++++.+++.+++.+++.+| 1e+06 +-+ : : : | | : : : | | : : : | 500000 +-+: : | | : : | | : : | 0 +-+O--------------------------------------------------------------+ unixbench.score 8000 +-+------------------------------------------------------------------+ | +.++ +++.++.+++.+++.++.+++.++.+++.+++.++.+++.+++.++.+++.+++.++.+| 7000 OO+ O OOO OOO OO OOO OOO OO OOO | 6000 +-+ : : : | | : : : | 5000 +-+ : : : | | : : : | 4000 +-+ : : : | | : :: | 3000 +-+: :: | 2000 +-+: :: | | : : | 1000 +-+: : | | : : | 0 +-+O-----------------------------------------------------------------+ unixbench.workload 2.5e+07 +-+---------------------------------------------------------------+ | | | +. | 2e+07 +-+ ++.+ +.+++.+++ +++.+++.+++.+++.+++.+++.+++.++++.+++.+++.+++.+| OO OO OOO OOO OOOO OOO OOO OO | | : : : | 1.5e+07 +-+ : : : | | : : : | 1e+07 +-+ : : : | | : : | | : : | 5e+06 +-+: : | | : : | | : : | 0 +-+O--------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-bdw-de1: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory *************************************************************************************************** lkp-csl-2ap1: 192 threads Intel(R) Xeon(R) CPU @ 2.20GHz with 192G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/runtime/size/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/512G/lkp-csl-2ap1/anon-w-rand/vm-scalability commit: 6768d1b110 ("sched/fair: use load instead of runnable load") df703e3e7a ("sched/fair: evenly spread tasks when not overloaded") 6768d1b11013709b df703e3e7adf42061ec0f44d36a ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:2 -50% :2 dmesg.WARNING:at#for_ip_interrupt_entry/0x :2 100% 2:2 kmsg.ACPI_Error Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 year

1
0
0 / 0

[drop_monitor] 98ffbd6cd2: will-it-scale.per_thread_ops -17.5% regression

by kernel test robot

Greeting, FYI, we noticed a -17.5% regression of will-it-scale.per_thread_ops due to commit: commit: 98ffbd6cd2b25fc6cbb0695e03b4fd43b5e116e6 ("[RFC PATCH net-next 10/12] drop_monitor: Add packet alert mode") url: https://github.com/0day-ci/linux/commits/Ido-Schimmel/drop_monitor-Captur... in testcase: will-it-scale on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory with following parameters: nr_task: 100% mode: thread test: lock1 cpufreq_governor: performance test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase: gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-knm01/lock1/will-it-scale commit: 6bb91c4db1 ("drop_monitor: Require CAP_NET_ADMIN for drop monitor configuration") 98ffbd6cd2 ("drop_monitor: Add packet alert mode") 6bb91c4db105160c 98ffbd6cd2b25fc6cbb0695e03b ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x :4 25% 1:4 dmesg.WARNING:at_ip__fsnotify_parent/0x %stddev %change %stddev \ | \ 5160 ± 2% -17.5% 4259 will-it-scale.per_thread_ops 672.94 ± 5% -14.1% 577.83 will-it-scale.time.user_time 1486179 ± 2% -17.4% 1226919 will-it-scale.workload 29442 -0.7% 29239 proc-vmstat.nr_slab_reclaimable 13523 ± 45% -32.3% 9153 ± 2% softirqs.CPU0.SCHED 2.46 -23.7% 1.87 ± 2% turbostat.RAMWatt 0.00 ± 6% -0.0 0.00 ± 13% mpstat.cpu.all.soft% 0.88 ± 5% -0.1 0.77 mpstat.cpu.all.usr% 1283 ± 2% +26.6% 1624 ± 4% sched_debug.cfs_rq:/.exec_clock.stddev 748596 ± 8% +24.5% 931656 ± 2% sched_debug.cfs_rq:/.min_vruntime.stddev 91.95 ± 9% +17.2% 107.75 ± 7% sched_debug.cfs_rq:/.nr_spread_over.max 7.14 ± 4% +16.3% 8.30 ± 6% sched_debug.cfs_rq:/.nr_spread_over.stddev 647587 ± 11% +16.5% 754309 ± 5% sched_debug.cfs_rq:/.spread0.stddev 6355144 ± 16% +50.5% 9563156 ± 8% sched_debug.cpu.avg_idle.max 1010 ± 3% +36.7% 1381 ± 4% sched_debug.cpu.clock.stddev 1010 ± 3% +36.7% 1381 ± 4% sched_debug.cpu.clock_task.stddev 1464112 ± 14% +31.1% 1918789 ± 5% sched_debug.cpu.max_idle_balance_cost.max 0.00 ± 3% +37.0% 0.00 ± 4% sched_debug.cpu.next_balance.stddev 34552 ± 13% +104.7% 70741 ± 60% sched_debug.cpu.sched_count.max 3147 ± 7% +57.7% 4963 ± 44% sched_debug.cpu.sched_count.stddev 1.00 -0.1 0.94 ± 2% perf-stat.i.branch-miss-rate% 88925913 -7.8% 82025606 perf-stat.i.branch-misses 2.001e+08 ± 2% -6.1% 1.879e+08 perf-stat.i.cache-references 0.21 -0.0 0.19 ± 2% perf-stat.i.iTLB-load-miss-rate% 77284434 -12.1% 67950981 perf-stat.i.iTLB-load-misses 473.52 +10.6% 523.50 ± 2% perf-stat.i.instructions-per-iTLB-miss 1.00 -0.1 0.95 ± 2% perf-stat.overall.branch-miss-rate% 0.21 -0.0 0.19 ± 2% perf-stat.overall.iTLB-load-miss-rate% 473.44 +10.5% 523.39 ± 2% perf-stat.overall.instructions-per-iTLB-miss 7458497 +17.8% 8789581 ± 3% perf-stat.overall.path-length 88598325 -7.8% 81725273 perf-stat.ps.branch-misses 1.994e+08 ± 2% -6.1% 1.872e+08 perf-stat.ps.cache-references 77002361 -12.1% 67703754 perf-stat.ps.iTLB-load-misses 1.10 -0.0 1.05 perf-profile.calltrace.cycles-pp.do_lock_file_wait.fcntl_setlk.do_fcntl.__x64_sys_fcntl.do_syscall_64 0.24 ± 4% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.30 ± 4% -0.0 0.25 ± 5% perf-profile.children.cycles-pp.kmem_cache_free 0.24 ± 10% -0.0 0.20 ± 5% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.20 ± 4% -0.0 0.16 ± 2% perf-profile.children.cycles-pp.locks_release_private 1.10 ± 2% -0.0 1.06 perf-profile.children.cycles-pp.do_lock_file_wait 0.54 ± 2% -0.0 0.49 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 0.19 ± 4% -0.0 0.16 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.09 ± 26% -0.0 0.06 perf-profile.children.cycles-pp.rb_next 0.64 ± 2% -0.0 0.61 ± 2% perf-profile.children.cycles-pp.locks_alloc_lock 0.15 ± 4% -0.0 0.13 ± 6% perf-profile.children.cycles-pp.memset_erms 0.16 ± 2% -0.0 0.15 ± 3% perf-profile.children.cycles-pp._copy_from_user 0.14 ± 6% -0.0 0.12 perf-profile.children.cycles-pp.security_file_lock 0.06 -0.0 0.05 perf-profile.children.cycles-pp.locks_delete_lock_ctx 0.08 +0.0 0.10 ± 9% perf-profile.children.cycles-pp.__might_sleep 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.sched_clock_cpu 0.12 ± 24% +0.1 0.19 ± 2% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.04 ± 57% +0.1 0.11 ± 14% perf-profile.children.cycles-pp.account_process_tick 0.14 ± 27% +0.1 0.21 ± 20% perf-profile.children.cycles-pp.generic_perform_write 0.35 +0.1 0.45 ± 2% perf-profile.children.cycles-pp.file_has_perm 0.32 ± 2% +0.1 0.42 perf-profile.children.cycles-pp.security_file_fcntl 0.00 +0.1 0.10 ± 10% perf-profile.children.cycles-pp.fpregs_assert_state_consistent 0.12 ± 4% +0.1 0.23 perf-profile.children.cycles-pp.avc_has_perm 0.27 ± 12% +0.1 0.38 ± 6% perf-profile.children.cycles-pp.irq_enter 0.15 ± 13% +0.1 0.26 ± 9% perf-profile.children.cycles-pp.rcu_irq_enter 0.37 ± 2% -0.1 0.26 ± 2% perf-profile.self.cycles-pp.fcntl_setlk 0.17 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.do_syscall_64 0.24 ± 5% -0.1 0.18 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.28 ± 4% -0.0 0.23 ± 5% perf-profile.self.cycles-pp.kmem_cache_free 0.17 ± 4% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.09 ± 23% -0.0 0.06 perf-profile.self.cycles-pp.rb_next 0.21 ± 3% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.kmem_cache_alloc 0.10 ± 10% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.11 ± 7% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.locks_release_private 0.13 ± 3% -0.0 0.11 ± 3% perf-profile.self.cycles-pp.memset_erms 0.11 -0.0 0.10 perf-profile.self.cycles-pp.___might_sleep 0.06 ± 6% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.07 +0.0 0.09 ± 10% perf-profile.self.cycles-pp.__might_sleep 0.15 ± 7% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.do_fcntl 0.18 ± 10% +0.0 0.22 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.17 ± 4% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.posix_lock_inode 0.08 ± 34% +0.1 0.14 perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.04 ± 57% +0.1 0.11 ± 14% perf-profile.self.cycles-pp.account_process_tick 0.12 ± 3% +0.1 0.21 ± 2% perf-profile.self.cycles-pp.avc_has_perm 0.00 +0.1 0.10 ± 8% perf-profile.self.cycles-pp.fpregs_assert_state_consistent 0.15 ± 13% +0.1 0.26 ± 9% perf-profile.self.cycles-pp.rcu_irq_enter 3856 +76.0% 6787 ± 23% interrupts.CPU104.NMI:Non-maskable_interrupts 3856 +76.0% 6787 ± 23% interrupts.CPU104.PMI:Performance_monitoring_interrupts 5779 ± 31% -32.1% 3924 interrupts.CPU117.NMI:Non-maskable_interrupts 5779 ± 31% -32.1% 3924 interrupts.CPU117.PMI:Performance_monitoring_interrupts 5750 ± 31% -32.0% 3912 interrupts.CPU119.NMI:Non-maskable_interrupts 5750 ± 31% -32.0% 3912 interrupts.CPU119.PMI:Performance_monitoring_interrupts 6704 ± 23% -42.1% 3880 interrupts.CPU120.NMI:Non-maskable_interrupts 6704 ± 23% -42.1% 3880 interrupts.CPU120.PMI:Performance_monitoring_interrupts 6708 ± 23% -41.9% 3899 interrupts.CPU121.NMI:Non-maskable_interrupts 6708 ± 23% -41.9% 3899 interrupts.CPU121.PMI:Performance_monitoring_interrupts 6705 ± 23% -41.3% 3933 interrupts.CPU122.NMI:Non-maskable_interrupts 6705 ± 23% -41.3% 3933 interrupts.CPU122.PMI:Performance_monitoring_interrupts 5735 ± 31% -32.0% 3899 interrupts.CPU123.NMI:Non-maskable_interrupts 5735 ± 31% -32.0% 3899 interrupts.CPU123.PMI:Performance_monitoring_interrupts 3980 +49.3% 5943 ± 32% interrupts.CPU13.NMI:Non-maskable_interrupts 3980 +49.3% 5943 ± 32% interrupts.CPU13.PMI:Performance_monitoring_interrupts 4838 ± 32% +20.2% 5818 ± 31% interrupts.CPU130.NMI:Non-maskable_interrupts 4838 ± 32% +20.2% 5818 ± 31% interrupts.CPU130.PMI:Performance_monitoring_interrupts 9080 ± 6% -6.6% 8485 ± 5% interrupts.CPU139.CAL:Function_call_interrupts 150.00 ±153% -95.5% 6.75 ± 48% interrupts.CPU141.RES:Rescheduling_interrupts 6758 ± 23% -27.5% 4899 ± 33% interrupts.CPU142.NMI:Non-maskable_interrupts 6758 ± 23% -27.5% 4899 ± 33% interrupts.CPU142.PMI:Performance_monitoring_interrupts 5777 ± 31% -32.2% 3916 interrupts.CPU147.NMI:Non-maskable_interrupts 5777 ± 31% -32.2% 3916 interrupts.CPU147.PMI:Performance_monitoring_interrupts 288.75 ±114% -86.3% 39.50 ± 88% interrupts.CPU151.RES:Rescheduling_interrupts 62.25 ± 85% +223.7% 201.50 ± 35% interrupts.CPU152.RES:Rescheduling_interrupts 6753 ± 23% -42.2% 3906 ± 2% interrupts.CPU156.NMI:Non-maskable_interrupts 6753 ± 23% -42.2% 3906 ± 2% interrupts.CPU156.PMI:Performance_monitoring_interrupts 6704 ± 23% -27.3% 4875 ± 33% interrupts.CPU161.NMI:Non-maskable_interrupts 6704 ± 23% -27.3% 4875 ± 33% interrupts.CPU161.PMI:Performance_monitoring_interrupts 5789 ± 31% -33.1% 3871 interrupts.CPU172.NMI:Non-maskable_interrupts 5789 ± 31% -33.1% 3871 interrupts.CPU172.PMI:Performance_monitoring_interrupts 6685 ± 23% -41.5% 3909 interrupts.CPU181.NMI:Non-maskable_interrupts 6685 ± 23% -41.5% 3909 interrupts.CPU181.PMI:Performance_monitoring_interrupts 3880 +50.5% 5840 ± 32% interrupts.CPU197.NMI:Non-maskable_interrupts 3880 +50.5% 5840 ± 32% interrupts.CPU197.PMI:Performance_monitoring_interrupts 5741 ± 30% -31.8% 3914 interrupts.CPU203.NMI:Non-maskable_interrupts 5741 ± 30% -31.8% 3914 interrupts.CPU203.PMI:Performance_monitoring_interrupts 6703 ± 23% -41.9% 3897 interrupts.CPU205.NMI:Non-maskable_interrupts 6703 ± 23% -41.9% 3897 interrupts.CPU205.PMI:Performance_monitoring_interrupts 9130 ± 6% -7.3% 8463 ± 6% interrupts.CPU211.CAL:Function_call_interrupts 9128 ± 5% -7.8% 8413 ± 6% interrupts.CPU22.CAL:Function_call_interrupts 6729 ± 23% -41.6% 3928 interrupts.CPU225.NMI:Non-maskable_interrupts 6729 ± 23% -41.6% 3928 interrupts.CPU225.PMI:Performance_monitoring_interrupts 54.00 ±147% +253.2% 190.75 ± 77% interrupts.CPU225.RES:Rescheduling_interrupts 5766 ± 31% -32.3% 3905 interrupts.CPU228.NMI:Non-maskable_interrupts 5766 ± 31% -32.3% 3905 interrupts.CPU228.PMI:Performance_monitoring_interrupts 148.00 ± 43% -48.6% 76.00 ± 90% interrupts.CPU232.RES:Rescheduling_interrupts 4751 ± 33% +42.8% 6786 ± 24% interrupts.CPU240.NMI:Non-maskable_interrupts 4751 ± 33% +42.8% 6786 ± 24% interrupts.CPU240.PMI:Performance_monitoring_interrupts 7614 ± 2% -36.2% 4860 ± 33% interrupts.CPU255.NMI:Non-maskable_interrupts 7614 ± 2% -36.2% 4860 ± 33% interrupts.CPU255.PMI:Performance_monitoring_interrupts 4782 ± 33% +21.7% 5820 ± 32% interrupts.CPU264.NMI:Non-maskable_interrupts 4782 ± 33% +21.7% 5820 ± 32% interrupts.CPU264.PMI:Performance_monitoring_interrupts 134.75 ±148% -94.2% 7.75 ± 32% interrupts.CPU268.RES:Rescheduling_interrupts 4870 ± 32% +41.7% 6900 ± 23% interrupts.CPU4.NMI:Non-maskable_interrupts 4870 ± 32% +41.7% 6900 ± 23% interrupts.CPU4.PMI:Performance_monitoring_interrupts 3906 +48.3% 5794 ± 31% interrupts.CPU40.NMI:Non-maskable_interrupts 3906 +48.3% 5794 ± 31% interrupts.CPU40.PMI:Performance_monitoring_interrupts 4816 ± 33% +41.1% 6794 ± 23% interrupts.CPU57.NMI:Non-maskable_interrupts 4816 ± 33% +41.1% 6794 ± 23% interrupts.CPU57.PMI:Performance_monitoring_interrupts 3820 ± 2% +53.3% 5857 ± 33% interrupts.CPU58.NMI:Non-maskable_interrupts 3820 ± 2% +53.3% 5857 ± 33% interrupts.CPU58.PMI:Performance_monitoring_interrupts 180.50 ±110% -85.6% 26.00 ± 73% interrupts.CPU68.RES:Rescheduling_interrupts 180.75 ±100% -90.3% 17.50 ±133% interrupts.CPU69.RES:Rescheduling_interrupts 42.50 ± 63% +330.0% 182.75 ± 18% interrupts.CPU80.RES:Rescheduling_interrupts 5781 ± 31% -32.0% 3932 interrupts.CPU92.NMI:Non-maskable_interrupts 5781 ± 31% -32.0% 3932 interrupts.CPU92.PMI:Performance_monitoring_interrupts 6676 ± 23% -41.2% 3924 interrupts.CPU93.NMI:Non-maskable_interrupts 6676 ± 23% -41.2% 3924 interrupts.CPU93.PMI:Performance_monitoring_interrupts 9099 ± 5% -12.0% 8006 ± 3% interrupts.CPU95.CAL:Function_call_interrupts will-it-scale.per_thread_ops 5400 +-+------------------------------------------------------------------+ |.+.. .+.+..+. .+. .+. + + .+.. .+.+. .+. .+..+. .+. | 5200 +-+ + + +. +. + + + +. + +.+..+ + +.| | + : + | | : + | 5000 +-+ + | | | 4800 +-+ | | | 4600 +-+ | | O | O O O O O O O | 4400 +-+ O O O O O O O O O | | O O O O O O O O | 4200 +-+----------------------------------------------------O----O--------+ will-it-scale.workload 1.55e+06 +-+--------------------------------------------------------------+ |.+. .+.+.+.+.+.+..+. + + .+.. .+.+. .+. .+.+. .+..+ | 1.5e+06 +-+ +. +.+ + + + +. +.+.+ : +.| | : + | 1.45e+06 +-+ + | | | 1.4e+06 +-+ | | | 1.35e+06 +-+ | | | 1.3e+06 +-+ O | O O O O O O O O O O | 1.25e+06 +-O O O O O O O O O | | O O O O O O O | 1.2e+06 +-+--------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 year

1
0
0 / 0

2020

2019

2018

2017

2016

2015

2014

2013

LKP July 2019