December 2019 - LKP - Ml01.01.Org

Re: [PATCH] x86/pat: Fix off-by-one bugs in interval tree search

by Ingo Molnar

* Mariusz Ceier <mceier(a)gmail.com> wrote: > Your patch fixes performance issue on my system and afterwards > /sys/kernel/debug/x86/pat_memtype_list contents are: Great, thanks for testing it! > PAT memtype list: > uncached-minus @ 0xfed90000-0xfed91000 > write-combining @ 0x2000000000-0x2100000000 > write-combining @ 0x2000000000-0x2100000000 > uncached-minus @ 0x2100000000-0x2100001000 Note how the UC- region starts right after the WC region, which triggered the bug on your system. > It's very similar to pat_memtype_list contents after reverting 4 > x86/mm/pat patches affecting performance: > > @@ -1,8 +1,8 @@ > PAT memtype list: > write-back @ 0x55ba4000-0x55ba5000 > write-back @ 0x5e88c000-0x5e8b5000 > -write-back @ 0x5e8b4000-0x5e8b8000 > write-back @ 0x5e8b4000-0x5e8b5000 > +write-back @ 0x5e8b4000-0x5e8b8000 > write-back @ 0x5e8b7000-0x5e8bb000 > write-back @ 0x5e8ba000-0x5e8bc000 > write-back @ 0x5e8bb000-0x5e8be000 > @@ -21,8 +21,8 @@ > uncached-minus @ 0xec260000-0xec264000 > uncached-minus @ 0xec300000-0xec320000 > uncached-minus @ 0xec326000-0xec327000 > -uncached-minus @ 0xf0000000-0xf0001000 > uncached-minus @ 0xf0000000-0xf8000000 > +uncached-minus @ 0xf0000000-0xf0001000 Yes, the ordering of same-start regions is different. I believe the difference is due to how the old rbtree logic inserted subtrees: - while (*node) { - struct memtype *data = rb_entry(*node, struct memtype, rb); - - parent = *node; - if (data->subtree_max_end < newdata->end) - data->subtree_max_end = newdata->end; - if (newdata->start <= data->start) - node = &((*node)->rb_left); - else if (newdata->start > data->start) - node = &((*node)->rb_right); - } - - newdata->subtree_max_end = newdata->end; - rb_link_node(&newdata->rb, parent, node); - rb_insert_augmented(&newdata->rb, root, &memtype_rb_augment_cb); In the new interval-tree logic this is: while (*link) { \ rb_parent = *link; \ parent = rb_entry(rb_parent, ITSTRUCT, ITRB); \ if (parent->ITSUBTREE < last) \ parent->ITSUBTREE = last; \ if (start < ITSTART(parent)) \ link = &parent->ITRB.rb_left; \ else { \ link = &parent->ITRB.rb_right; \ leftmost = false; \ } \ } \ \ node->ITSUBTREE = last; \ rb_link_node(&node->ITRB, rb_parent, link); \ rb_insert_augmented_cached(&node->ITRB, root, \ leftmost, &ITPREFIX ## _augment); \ The old logic was a bit convoluted, but it can be written as: if (newdata->start <= data->start) node = &parent->rb_left; else node = &parent->rb_right; The new logic is, in effect: if (start < data->start) link = &parent->rb_left; else link = &parent->rb_right; Note the "<=" vs. '<' difference - this I believe changes the ordering within the tree. It's still fine as long as this is used consistently, but this changes the internal ordering of the nodes of the tree. Thanks, Ingo

1 year, 2 months

1
0
0 / 0

942437c97f ("y2038: allow disabling time32 system calls"): BUG: kernel hang in test stage

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 942437c97fd9ff23a17c13118f50bd0490f6868c Author: Arnd Bergmann <arnd(a)arndb.de> AuthorDate: Mon Jul 15 11:46:10 2019 +0200 Commit: Arnd Bergmann <arnd(a)arndb.de> CommitDate: Fri Nov 15 14:38:30 2019 +0100 y2038: allow disabling time32 system calls At the moment, the compilation of the old time32 system calls depends purely on the architecture. As systems with new libc based on 64-bit time_t are getting deployed, even architectures that previously supported these (notably x86-32 and arm32 but also many others) no longer depend on them, and removing them from a kernel image results in a smaller kernel binary, the same way we can leave out many other optional system calls. More importantly, on an embedded system that needs to keep working beyond year 2038, any user space program calling these system calls is likely a bug, so removing them from the kernel image does provide an extra debugging help for finding broken applications. I've gone back and forth on hiding this option unless CONFIG_EXPERT is set. This version leaves it visible based on the logic that eventually it will be turned off indefinitely. Acked-by: Christian Brauner <christian.brauner(a)ubuntu.com> Signed-off-by: Arnd Bergmann <arnd(a)arndb.de> bd40a17576 y2038: itimer: change implementation to timespec64 942437c97f y2038: allow disabling time32 system calls +-----------------------------------------------------------+------------+------------+ | | bd40a17576 | 942437c97f | +-----------------------------------------------------------+------------+------------+ | boot_successes | 38 | 0 | | boot_failures | 0 | 19 | | BUG:kernel_hang_in_test_stage | 0 | 15 | | Assertion_failed | 0 | 4 | | Kernel_panic-not_syncing:Attempted_to_kill_init!exitcode= | 0 | 4 | +-----------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level [ 3.739930] rm (184) used greatest stack depth: 6556 bytes left [ 5.754828] mount_root: mounting /dev/root [ 5.762906] urandom-seed: Seed file not found (/etc/urandom.seed) [ 5.767017] sh (172) used greatest stack depth: 6308 bytes left BUG: kernel hang in test stage # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 1875ff320f14afe21731a6e4c7b46dd33e45dfaa v5.4 -- git bisect good 9137b1791b9c92238389cdd1a23ca6d16da3145a # 18:36 G 11 0 0 0 Merge remote-tracking branch 'xtensa/xtensa-for-next' git bisect good 4f718d6625e1302a61524688b87e9bdfaab0c7c0 # 19:40 G 11 0 0 0 Merge remote-tracking branch 'selinux/next' git bisect good a7b0c9fecaeca5272254f581541864e898508d21 # 20:33 G 11 0 0 0 Merge remote-tracking branch 'scsi/for-next' git bisect bad 2c98eb870fbbd8a18f7725a53decea87a303466a # 21:32 B 0 7 23 0 Merge remote-tracking branch 'kspp/for-next/kspp' git bisect good 1f5b50bd3d843e2da5a388096de05b74c8acb4f3 # 22:47 G 11 0 2 2 Merge remote-tracking branch 'userns/for-next' git bisect bad 128a34ad76c10bf861b6410b46e2226f83dda09e # 23:37 B 0 1 18 0 Merge remote-tracking branch 'livepatching/for-next' git bisect good 6481881419b03d1af1229f32bcc5ecd1731ec2be # 00:21 G 11 0 0 0 Merge remote-tracking branch 'ktest/for-next' git bisect bad d7eb0d1dbabe7415854fa69173b05c897d5bbed1 # 01:23 B 0 6 22 0 Merge remote-tracking branch 'y2038/y2038' git bisect good 3e859adf3643c2da9765cd5088170738d7918567 # 05:05 G 10 0 0 0 compat_ioctl: unify copy-in of ppp filters git bisect good 5e0fb1b57bea8d11fe77da2bc80f4c9a67e28318 # 05:50 G 11 0 0 0 y2038: time: avoid timespec usage in settimeofday() git bisect bad b111df8447acdeb4b9220f99d5d4b28f83eb56ad # 06:54 B 1 2 1 3 y2038: alarm: fix half-second cut-off git bisect good bd40a175769d411b2a37e1c087082ac7ee2c15bb # 08:26 G 11 0 1 1 y2038: itimer: change implementation to timespec64 git bisect bad 1c11ca7a0584ddede5b8c93057b40d31e8a96d3d # 09:32 B 0 2 20 2 y2038: fix typo in powerpc vdso "LOPART" git bisect bad 942437c97fd9ff23a17c13118f50bd0490f6868c # 11:17 B 0 3 21 2 y2038: allow disabling time32 system calls # first bad commit: [942437c97fd9ff23a17c13118f50bd0490f6868c] y2038: allow disabling time32 system calls git bisect good bd40a175769d411b2a37e1c087082ac7ee2c15bb # 12:54 G 32 0 0 1 y2038: itimer: change implementation to timespec64 # extra tests with debug options git bisect good 942437c97fd9ff23a17c13118f50bd0490f6868c # 13:51 G 11 0 0 0 y2038: allow disabling time32 system calls # extra tests on revert first bad commit git bisect good 66c1e1f1d2ea1a246dedf3908dae5806bde49691 # 15:19 G 11 0 0 0 Revert "y2038: allow disabling time32 system calls" # good: [66c1e1f1d2ea1a246dedf3908dae5806bde49691] Revert "y2038: allow disabling time32 system calls" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

1 year, 2 months

2
1
0 / 0

[locking/mutex] a0855d24fc: WARNING:at_kernel/locking/mutex.c:#mutex_trylock

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: a0855d24fc22d49cdc25664fb224caee16998683 ("locking/mutex: Complain upon mutex API misuse in IRQ contexts") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: boot on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------------------+------------+------------+ | | 751459043c | a0855d24fc | +------------------------------------------------------+------------+------------+ | boot_successes | 1 | 1 | | boot_failures | 35 | 35 | | BUG:workqueue_lockup-pool | 10 | 9 | | BUG:soft_lockup-CPU##stuck_for#s![dma-fence:#:#] | 24 | 25 | | EIP:to_kthread | 3 | 2 | | Kernel_panic-not_syncing:softlockup:hung_tasks | 25 | 26 | | EIP:thread_signal_callback | 6 | 6 | | calltrace:do_softirq_own_stack | 7 | 12 | | EIP:lock_is_held_type | 10 | 10 | | EIP:debug_lockdep_rcu_enabled | 2 | 5 | | EIP:arch_local_save_flags | 3 | | | EIP:kthread_should_stop | 1 | 1 | | BUG:soft_lockup-CPU##stuck_for#s![rcu_torture_fak:#] | 1 | | | WARNING:at_kernel/locking/mutex.c:#mutex_trylock | 0 | 26 | | EIP:mutex_trylock | 0 | 26 | | WARNING:at_kernel/locking/mutex.c:#mutex_unlock | 0 | 26 | | EIP:mutex_unlock | 0 | 26 | | BUG:soft_lockup-CPU##stuck_for#s![swapper:#] | 0 | 1 | | EIP:rs_modnn | 0 | 1 | | EIP:rcu_read_lock_held | 0 | 1 | +------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 576.348903] watchdog: BUG: soft lockup - CPU#0 stuck for 134s! [dma-fence:0:230] [ 576.348903] Modules linked in: [ 576.348903] irq event stamp: 280712 [ 576.348903] hardirqs last enabled at (280711): [<dda010fb>] trace_hardirqs_on_thunk+0xc/0x10 [ 576.348903] hardirqs last disabled at (280712): [<dda0110b>] trace_hardirqs_off_thunk+0xc/0x10 [ 576.348903] softirqs last enabled at (526): [<de4568db>] __do_softirq+0x283/0x2b1 [ 576.348903] softirqs last disabled at (519): [<dda05bb1>] do_softirq_own_stack+0x21/0x27 [ 576.348903] CPU: 0 PID: 230 Comm: dma-fence:0 Not tainted 5.4.0-rc2-00042-ga0855d24fc22d #1 [ 576.348903] EIP: to_kthread+0x0/0x1a [ 576.348903] Code: 43 08 ff d2 83 c3 14 eb e9 5b 5e 5d c3 3e 8d 74 26 00 55 89 e5 ff 70 2c 68 b8 e7 a1 de 68 00 10 00 00 51 e8 d7 65 a0 00 c9 c3 <f6> 40 16 20 75 0d 55 89 e5 0f 0b 8b 80 28 03 00 00 5d c3 8b 80 28 [ 576.348903] EAX: e9c3b380 EBX: ee6abea8 ECX: dec3053c EDX: e9c3b85c [ 576.348903] ESI: 00000000 EDI: 00000000 EBP: eb3bff4c ESP: eb3bff48 [ 576.348903] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 EFLAGS: 00000246 [ 576.348903] CR0: 80050033 CR2: 00000000 CR3: 1eea9000 CR4: 000406b0 [ 576.348903] Call Trace: [ 576.348903] ? kthread_should_stop+0x12/0x1b [ 576.348903] thread_signal_callback+0x122/0x243 [ 576.348903] kthread+0xdc/0xde [ 576.348903] ? test_signaling+0x7b/0x7b [ 576.348903] ? kthread_create_worker_on_cpu+0x1c/0x1c [ 576.348903] ret_from_fork+0x19/0x24 [ 576.348903] Kernel panic - not syncing: softlockup: hung tasks [ 576.348903] CPU: 0 PID: 230 Comm: dma-fence:0 Tainted: G L 5.4.0-rc2-00042-ga0855d24fc22d #1 [ 576.348903] Call Trace: [ 576.348903] <IRQ> [ 576.348903] dump_stack+0x16/0x18 [ 576.348903] panic+0xaa/0x259 [ 576.348903] watchdog_timer_fn+0x1ec/0x211 [ 576.348903] ? __touch_watchdog+0x1a/0x1a [ 576.348903] __hrtimer_run_queues+0x165/0x256 [ 576.348903] ? __touch_watchdog+0x1a/0x1a [ 576.348903] hrtimer_run_queues+0xc4/0xd9 [ 576.348903] run_local_timers+0xd/0x26 [ 576.348903] update_process_times+0x1c/0x3c [ 576.348903] tick_periodic+0x99/0x9b [ 576.348903] tick_handle_periodic+0x18/0x54 [ 576.348903] timer_interrupt+0x12/0x19 [ 576.348903] __handle_irq_event_percpu+0xa7/0x1ac [ 576.348903] handle_irq_event_percpu+0x1c/0x42 [ 576.348903] handle_irq_event+0x2e/0x47 [ 576.348903] ? irq_set_chained_handler_and_data+0x4f/0x4f [ 576.348903] handle_level_irq+0x57/0x83 [ 576.348903] handle_irq+0x3a/0x45 [ 576.348903] </IRQ> [ 576.348903] do_IRQ+0x6a/0x8f [ 576.348903] common_interrupt+0xfd/0x104 [ 576.348903] EIP: to_kthread+0x0/0x1a [ 576.348903] Code: 43 08 ff d2 83 c3 14 eb e9 5b 5e 5d c3 3e 8d 74 26 00 55 89 e5 ff 70 2c 68 b8 e7 a1 de 68 00 10 00 00 51 e8 d7 65 a0 00 c9 c3 <f6> 40 16 20 75 0d 55 89 e5 0f 0b 8b 80 28 03 00 00 5d c3 8b 80 28 [ 576.348903] EAX: e9c3b380 EBX: ee6abea8 ECX: dec3053c EDX: e9c3b85c [ 576.348903] ESI: 00000000 EDI: 00000000 EBP: eb3bff4c ESP: eb3bff48 [ 576.348903] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 EFLAGS: 00000246 [ 576.348903] ? __modver_version_show+0x1d/0x1d [ 576.348903] ? kthread_should_stop+0x12/0x1b [ 576.348903] thread_signal_callback+0x122/0x243 [ 576.348903] kthread+0xdc/0xde [ 576.348903] ? test_signaling+0x7b/0x7b [ 576.348903] ? kthread_create_worker_on_cpu+0x1c/0x1c [ 576.348903] ret_from_fork+0x19/0x24 [ 576.348903] ------------[ cut here ]------------ [ 576.348903] WARNING: CPU: 0 PID: 230 at kernel/locking/mutex.c:1419 mutex_trylock+0x4b/0x81 [ 576.348903] Modules linked in: [ 576.348903] CPU: 0 PID: 230 Comm: dma-fence:0 Tainted: G L 5.4.0-rc2-00042-ga0855d24fc22d #1 [ 576.348903] EIP: mutex_trylock+0x4b/0x81 [ 576.348903] Code: 74 1c 83 3d 08 5f d7 de 00 75 13 68 68 53 8f de 68 97 76 8e de e8 fe af 5d ff 0f 0b 58 5a a1 d4 af ad de a9 00 ff 1f 00 74 02 <0f> 0b 89 d8 e8 d0 28 60 ff 84 c0 89 c6 74 1b ff 75 04 8d 43 38 6a [ 576.348903] EAX: 00010002 EBX: dec36e60 ECX: 00000000 EDX: e9c3b380 [ 576.348903] ESI: 00000000 EDI: ee659ec8 EBP: ee659e7c ESP: ee659e74 [ 576.348903] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 EFLAGS: 00010006 [ 576.348903] CR0: 80050033 CR2: 00000000 CR3: 1eea9000 CR4: 000406b0 [ 576.348903] Call Trace: [ 576.348903] <IRQ> [ 576.348903] __crash_kexec+0x26/0xa8 [ 576.348903] panic+0xba/0x259 [ 576.348903] watchdog_timer_fn+0x1ec/0x211 [ 576.348903] ? __touch_watchdog+0x1a/0x1a [ 576.348903] __hrtimer_run_queues+0x165/0x256 [ 576.348903] ? __touch_watchdog+0x1a/0x1a [ 576.348903] hrtimer_run_queues+0xc4/0xd9 [ 576.348903] run_local_timers+0xd/0x26 [ 576.348903] update_process_times+0x1c/0x3c [ 576.348903] tick_periodic+0x99/0x9b [ 576.348903] tick_handle_periodic+0x18/0x54 [ 576.348903] timer_interrupt+0x12/0x19 [ 576.348903] __handle_irq_event_percpu+0xa7/0x1ac [ 576.348903] handle_irq_event_percpu+0x1c/0x42 [ 576.348903] handle_irq_event+0x2e/0x47 [ 576.348903] ? irq_set_chained_handler_and_data+0x4f/0x4f [ 576.348903] handle_level_irq+0x57/0x83 [ 576.348903] handle_irq+0x3a/0x45 [ 576.348903] </IRQ> [ 576.348903] do_IRQ+0x6a/0x8f [ 576.348903] common_interrupt+0xfd/0x104 [ 576.348903] EIP: to_kthread+0x0/0x1a [ 576.348903] Code: 43 08 ff d2 83 c3 14 eb e9 5b 5e 5d c3 3e 8d 74 26 00 55 89 e5 ff 70 2c 68 b8 e7 a1 de 68 00 10 00 00 51 e8 d7 65 a0 00 c9 c3 <f6> 40 16 20 75 0d 55 89 e5 0f 0b 8b 80 28 03 00 00 5d c3 8b 80 28 [ 576.348903] EAX: e9c3b380 EBX: ee6abea8 ECX: dec3053c EDX: e9c3b85c [ 576.348903] ESI: 00000000 EDI: 00000000 EBP: eb3bff4c ESP: eb3bff48 [ 576.348903] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068 EFLAGS: 00000246 [ 576.348903] ? __modver_version_show+0x1d/0x1d [ 576.348903] ? kthread_should_stop+0x12/0x1b [ 576.348903] thread_signal_callback+0x122/0x243 [ 576.348903] kthread+0xdc/0xde [ 576.348903] ? test_signaling+0x7b/0x7b [ 576.348903] ? kthread_create_worker_on_cpu+0x1c/0x1c [ 576.348903] ret_from_fork+0x19/0x24 [ 576.348903] irq event stamp: 280712 [ 576.348903] hardirqs last enabled at (280711): [<dda010fb>] trace_hardirqs_on_thunk+0xc/0x10 [ 576.348903] hardirqs last disabled at (280712): [<dda0110b>] trace_hardirqs_off_thunk+0xc/0x10 [ 576.348903] softirqs last enabled at (526): [<de4568db>] __do_softirq+0x283/0x2b1 [ 576.348903] softirqs last disabled at (519): [<dda05bb1>] do_softirq_own_stack+0x21/0x27 [ 576.348903] ---[ end trace a5329ee86dc626c2 ]--- To reproduce: # build kernel cd linux cp config-5.4.0-rc2-00042-ga0855d24fc22d .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

1 year, 2 months

1
0
0 / 0

[tracing] bf8e602186: WARNING:at_kernel/trace/trace.c:#create_trace_option_files

by kenerl test robot

FYI, we noticed the following commit (built with gcc-7): commit: bf8e602186ec402ed937b2cbd6c39a34c0029757 ("tracing: Do not create tracefs files if tracefs lockdown is in effect") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: boot on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +------------------------------------------------------------+------------+------------+ | | 17911ff38a | bf8e602186 | +------------------------------------------------------------+------------+------------+ | boot_successes | 0 | 0 | | boot_failures | 8 | 8 | | WARNING:at_fs/open.c:#do_dentry_open | 8 | 8 | | EIP:do_dentry_open | 8 | 8 | | WARNING:at_kernel/trace/trace.c:#create_trace_option_files | 0 | 8 | | EIP:create_trace_option_files | 0 | 8 | +------------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> [ 1.049546] WARNING: CPU: 0 PID: 1 at kernel/trace/trace.c:8107 create_trace_option_files+0x134/0x148 [ 1.051505] Modules linked in: [ 1.051928] CPU: 0 PID: 1 Comm: swapper Not tainted 5.4.0-rc2-00007-gbf8e602186ec4 #2 [ 1.052316] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 1.052316] EIP: create_trace_option_files+0x134/0x148 [ 1.052316] Code: fe ff ff 89 46 0c 59 58 83 7e 0c 00 75 20 80 3d e3 d3 c7 c1 00 75 17 ff 37 68 7b 54 a6 c1 c6 05 e3 d3 c7 c1 01 e8 46 77 f8 ff <0f> 0b 58 5a 83 c7 08 83 c6 10 eb 94 8d 65 f4 5b 5e 5f 5d c3 55 89 [ 1.052316] EAX: 0000002e EBX: c1bc1900 ECX: 00000000 EDX: 0000020c [ 1.052316] ESI: f0e54f80 EDI: c1bc1d2c EBP: f0cb5f10 ESP: f0cb5ef4 [ 1.052316] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010282 [ 1.052316] CR0: 80050033 CR2: 00000000 CR3: 01d98000 CR4: 000406d0 [ 1.052316] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1.052316] DR6: fffe0ff0 DR7: 00000400 [ 1.052316] Call Trace: [ 1.052316] __update_tracer_options+0x23/0x2c [ 1.052316] tracer_init_tracefs+0x140/0x166 [ 1.052316] ? register_tracer+0x21d/0x21d [ 1.052316] do_one_initcall+0x7c/0x184 [ 1.052316] ? set_debug_rodata+0x2d/0x2d [ 1.052316] ? trace_initcall_level+0x4c/0x50 [ 1.052316] kernel_init_freeable+0x138/0x1fe [ 1.052316] ? rest_init+0xdc/0xdc [ 1.052316] kernel_init+0x8/0xd0 [ 1.052316] ret_from_fork+0x1e/0x28 [ 1.052316] ---[ end trace 42d703653fa60537 ]--- To reproduce: # build kernel cd linux cp config-5.4.0-rc2-00007-gbf8e602186ec4 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, Rong Chen

1 year, 2 months

1
0
0 / 0

3e05ad861b: vm-scalability.median -86.3% regression

by kernel test robot

Greeting, FYI, we noticed a -86.3% regression of vm-scalability.median due to commit: commit: 3e05ad861b9b2b61a1cbfd0d98951579eb3c85e0 ("ZEN: Implement zen-tune v5.4") https://github.com/zen-kernel/zen-kernel 5.4/zen-sauce in testcase: vm-scalability on test machine: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory with following parameters: thp_enabled: never thp_defrag: never nr_task: 8 nr_pmem: 4 priority: 1 test: swap-w-seq cpufreq_governor: performance ucode: 0x16 test-description: The motivation behind this suite is to exercise functions and regions of the mm/ of the Linux kernel which are of interest to us. test-url: https://git.kernel.org/cgit/linux/kernel/git/wfg/vm-scalability.git/ In addition to that, the commit also has significant impact on the following tests: +------------------+------------------------------------------------------------------------+ | testcase: change | fsmark: fsmark.files_per_sec 1609.1% improvement | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | disk=1SSD | | | filesize=16MB | | | fs=xfs | | | iterations=1x | | | nr_directories=16d | | | nr_files_per_directory=256fpd | | | nr_threads=32t | | | sync_method=NoSync | | | test_size=60G | | | ucode=0x500002b | +------------------+------------------------------------------------------------------------+ | testcase: change | fsmark: fsmark.files_per_sec 832.4% improvement | | test machine | 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory | | test parameters | cpufreq_governor=performance | | | disk=1HDD | | | filesize=16MB | | | fs=f2fs | | | iterations=1x | | | nr_directories=16d | | | nr_files_per_directory=256fpd | | | nr_threads=32t | | | sync_method=NoSync | | | test_size=60G | | | ucode=0x500002b | +------------------+------------------------------------------------------------------------+ | testcase: change | fio-basic: fio.write_bw_MBps -15.4% regression | | test machine | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=ext4 | | | ioengine=libaio | | | nr_task=100% | | | runtime=300s | | | rw=write | | | test_size=128G | | | ucode=0x42e | +------------------+------------------------------------------------------------------------+ | testcase: change | vm-scalability: vm-scalability.median -93.8% regression | | test machine | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory | | test parameters | cpufreq_governor=performance | | | nr_pmem=4 | | | nr_task=64 | | | priority=1 | | | test=swap-w-seq | | | thp_defrag=never | | | thp_enabled=never | | | ucode=0x16 | +------------------+------------------------------------------------------------------------+ | testcase: change | fio-basic: fio.write_bw_MBps 228.8% improvement | | test machine | 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=1SSD | | | fs=ext4 | | | ioengine=sync | | | nr_task=100% | | | runtime=300s | | | rw=write | | | test_size=128G | | | ucode=0xb000038 | +------------------+------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <rong.a.chen(a)intel.com> Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/nr_pmem/nr_task/priority/rootfs/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode: gcc-7/performance/x86_64-rhel-7.6/4/8/1/debian-x86_64-2019-11-14.cgz/lkp-hsw-4ex1/swap-w-seq/vm-scalability/never/never/0x16 commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 100% 4:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x 4:4 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 1:4 -25% :4 dmesg.WARNING:at_ip___perf_sw_event/0x :4 25% 1:4 dmesg.page_allocation_failure:order:#,mode:#(GFP_KERNEL),nodemask=(null),cpuset=/,mems_allowed= %stddev %change %stddev \ | \ 46.22 ± 8% -87.4% 5.83 ± 5% vm-scalability.free_time 231850 -86.3% 31877 vm-scalability.median 1827686 -86.1% 254237 vm-scalability.throughput 294.40 +4.8% 308.51 vm-scalability.time.elapsed_time 294.40 +4.8% 308.51 vm-scalability.time.elapsed_time.max 397527 ± 5% -82.7% 68834 ± 10% vm-scalability.time.involuntary_context_switches 4139804 ± 10% -31.7% 2825636 vm-scalability.time.maximum_resident_set_size 98485423 -93.1% 6825583 vm-scalability.time.minor_page_faults 760.00 -58.0% 319.33 ± 5% vm-scalability.time.percent_of_cpu_this_job_got 1665 ± 2% -42.8% 952.65 ± 5% vm-scalability.time.system_time 573.54 -94.0% 34.29 ± 2% vm-scalability.time.user_time 19251 ± 9% -45.4% 10505 ± 16% vm-scalability.time.voluntary_context_switches 4.432e+08 -82.7% 76514024 vm-scalability.workload 2756753 ± 37% +221.9% 8873554 ± 9% cpuidle.C1.time 8563 ± 35% +260.4% 30860 ± 13% cpuidle.POLL.usage 91.69 -3.2% 88.74 iostat.cpu.idle 6.81 +59.6% 10.88 ± 6% iostat.cpu.system 1.50 -74.3% 0.38 ± 7% iostat.cpu.user 0.11 ± 24% -0.1 0.02 ±104% mpstat.cpu.all.soft% 6.74 +4.6 11.37 ± 7% mpstat.cpu.all.sys% 1.50 -1.1 0.41 ± 11% mpstat.cpu.all.usr% 0.01 ± 57% +0.0 0.02 turbostat.C1% 2.452e+08 ± 6% -61.6% 94135899 turbostat.IRQ 0.49 ± 45% -71.4% 0.14 ± 5% turbostat.Pkg%pc2 69.17 -15.0% 58.83 turbostat.RAMWatt 3.485e+09 ± 2% -50.7% 1.717e+09 ± 7% perf-node.node-load-misses 1.439e+09 ± 3% -92.3% 1.111e+08 ± 10% perf-node.node-loads 28.50 -81.3% 5.33 ± 8% perf-node.node-local-load-ratio 30.75 ± 5% -38.2% 19.00 ± 4% perf-node.node-local-store-ratio 1.386e+09 ± 4% -48.5% 7.143e+08 perf-node.node-store-misses 6.308e+08 ± 9% -72.4% 1.743e+08 ± 4% perf-node.node-stores 91.00 -2.2% 89.00 vmstat.cpu.id 123.50 ± 19% +37.7% 170.00 ± 13% vmstat.memory.buff 1.779e+08 -96.4% 6328966 ± 7% vmstat.memory.swpd 11.00 +27.3% 14.00 ± 11% vmstat.procs.r 199.75 -59.1% 81.67 ± 46% vmstat.swap.si 1255814 -97.0% 37410 ± 5% vmstat.swap.so 6094 ± 4% +60.1% 9759 ± 8% vmstat.system.cs 5445 -38.4% 3352 ± 71% syscalls.sys_mmap.min 1.93e+08 ± 10% -9.1e+07 1.022e+08 ± 71% syscalls.sys_mmap.noise.100% 3.126e+08 ± 5% -1.3e+08 1.862e+08 ± 71% syscalls.sys_mmap.noise.2% 2.91e+08 ± 5% -1.2e+08 1.716e+08 ± 71% syscalls.sys_mmap.noise.25% 3.116e+08 ± 5% -1.3e+08 1.855e+08 ± 71% syscalls.sys_mmap.noise.5% 2.617e+08 ± 6% -1.1e+08 1.526e+08 ± 71% syscalls.sys_mmap.noise.50% 2.292e+08 ± 8% -1e+08 1.289e+08 ± 70% syscalls.sys_mmap.noise.75% 2.559e+09 +365.3% 1.191e+10 ± 54% syscalls.sys_read.max 10679 ± 7% -42.8% 6111 ± 51% syscalls.sys_read.med 1960 -20.8% 1553 ± 8% syscalls.sys_read.min 18303468 -36.8% 11568383 ± 3% meminfo.Active 18303386 -36.8% 11568239 ± 3% meminfo.Active(anon) 694.75 ± 22% +311.7% 2860 ± 56% meminfo.AnonHugePages 20960583 -39.3% 12713259 ± 4% meminfo.AnonPages 2658038 -27.8% 1919449 ± 10% meminfo.Inactive 2657850 -27.8% 1918713 ± 10% meminfo.Inactive(anon) 136970 -17.8% 112657 ± 2% meminfo.KReclaimable 20792 +20.8% 25125 ± 2% meminfo.Mapped 1011847 ± 5% +289.4% 3939966 ± 27% meminfo.MemAvailable 1186192 ± 5% +247.8% 4125935 ± 26% meminfo.MemFree 31594374 -9.3% 28654632 ± 3% meminfo.Memused 450849 -89.6% 46670 ± 5% meminfo.PageTables 136970 -17.8% 112657 ± 2% meminfo.SReclaimable 226454 +66.5% 376940 ± 4% meminfo.SUnreclaim 2315 ± 2% +3322.9% 79248 ± 51% meminfo.Shmem 363425 +34.7% 489598 ± 3% meminfo.Slab 102.00 +6.8e+05% 697034 ± 13% meminfo.SwapCached 3.248e+08 +52.4% 4.952e+08 meminfo.SwapFree 111099 +540.3% 711342 ± 29% meminfo.max_used_kB 20266653 ± 9% -89.4% 2143184 ± 12% numa-numastat.node0.local_node 17092726 ± 78% -93.6% 1092549 ± 71% numa-numastat.node0.numa_foreign 20289644 ± 9% -89.3% 2166495 ± 12% numa-numastat.node0.numa_hit 4370210 ± 39% -87.7% 537112 ± 52% numa-numastat.node0.numa_miss 4393242 ± 39% -87.2% 560464 ± 51% numa-numastat.node0.other_node 19284050 ± 30% -91.3% 1673025 ± 6% numa-numastat.node1.local_node 15320346 ± 37% -97.9% 323994 ± 55% numa-numastat.node1.numa_foreign 19308182 ± 30% -91.2% 1691098 ± 7% numa-numastat.node1.numa_hit 7140678 ± 77% -83.7% 1164553 ± 12% numa-numastat.node1.numa_miss 7164810 ± 77% -83.5% 1182625 ± 11% numa-numastat.node1.other_node 11814274 ± 16% -81.2% 2219724 ± 9% numa-numastat.node2.local_node 4407066 ± 28% -83.0% 750305 ±100% numa-numastat.node2.numa_foreign 11838498 ± 16% -81.0% 2247034 ± 8% numa-numastat.node2.numa_hit 14628324 ± 16% -96.2% 561499 ± 56% numa-numastat.node2.numa_miss 14652548 ± 16% -96.0% 588810 ± 55% numa-numastat.node2.other_node 13847898 ± 7% -85.0% 2071206 ± 18% numa-numastat.node3.local_node 2000384 ± 23% -60.3% 793185 ± 74% numa-numastat.node3.numa_foreign 13877156 ± 7% -84.9% 2091059 ± 18% numa-numastat.node3.numa_hit 12681311 ± 8% -94.5% 696869 ± 61% numa-numastat.node3.numa_miss 12710569 ± 8% -94.4% 716723 ± 58% numa-numastat.node3.other_node 77617 ± 3% +1659.8% 1365927 ± 7% slabinfo.Acpi-Parse.active_objs 1071 ± 3% +1721.7% 19510 ± 6% slabinfo.Acpi-Parse.active_slabs 78208 ± 3% +1721.2% 1424296 ± 6% slabinfo.Acpi-Parse.num_objs 1071 ± 3% +1721.7% 19510 ± 6% slabinfo.Acpi-Parse.num_slabs 308.50 ± 7% -32.5% 208.33 ± 11% slabinfo.biovec-128.active_objs 308.50 ± 7% -32.5% 208.33 ± 11% slabinfo.biovec-128.num_objs 1019 ± 6% +11.2% 1133 ± 4% slabinfo.mnt_cache.active_objs 1019 ± 6% +11.2% 1133 ± 4% slabinfo.mnt_cache.num_objs 227.75 ± 29% -66.2% 77.00 ± 15% slabinfo.nfs_read_data.active_objs 227.75 ± 29% -66.2% 77.00 ± 15% slabinfo.nfs_read_data.num_objs 122256 ± 2% -34.3% 80273 ± 6% slabinfo.radix_tree_node.active_objs 2631 ± 5% -41.4% 1541 ± 4% slabinfo.radix_tree_node.active_slabs 123650 -30.2% 86332 ± 4% slabinfo.radix_tree_node.num_objs 2631 ± 5% -41.4% 1541 ± 4% slabinfo.radix_tree_node.num_slabs 3920 +37.7% 5397 slabinfo.scsi_sense_cache.active_objs 3920 +37.7% 5397 slabinfo.scsi_sense_cache.num_objs 1278 ± 2% -30.0% 895.00 ± 6% slabinfo.skbuff_fclone_cache.active_objs 1278 ± 2% -30.0% 895.00 ± 6% slabinfo.skbuff_fclone_cache.num_objs 14232 ± 2% +8988.9% 1293549 ± 8% slabinfo.vmap_area.active_objs 233.75 ± 2% +8918.9% 21081 ± 6% slabinfo.vmap_area.active_slabs 15005 ± 2% +8892.0% 1349243 ± 6% slabinfo.vmap_area.num_objs 233.75 ± 2% +8918.9% 21081 ± 6% slabinfo.vmap_area.num_slabs 273.25 ± 11% -46.2% 147.00 ± 23% slabinfo.xfrm_state.active_objs 273.25 ± 11% -46.2% 147.00 ± 23% slabinfo.xfrm_state.num_objs 11339 ± 13% +28.9% 14616 ± 7% sched_debug.cfs_rq:/.exec_clock.avg 38562 ± 21% +218.2% 122703 ± 3% sched_debug.cfs_rq:/.exec_clock.max 8226 ± 28% +125.3% 18533 ± 3% sched_debug.cfs_rq:/.exec_clock.stddev 11965 ± 30% +123.9% 26793 ± 45% sched_debug.cfs_rq:/.min_vruntime.min 69332 ± 26% -43.1% 39465 ± 14% sched_debug.cfs_rq:/.min_vruntime.stddev 0.28 ± 17% +968.1% 2.94 ± 23% sched_debug.cfs_rq:/.nr_spread_over.avg 3.18 ± 13% +403.9% 16.00 ± 30% sched_debug.cfs_rq:/.nr_spread_over.max 0.66 ± 13% +466.5% 3.75 ± 31% sched_debug.cfs_rq:/.nr_spread_over.stddev 55.34 ± 10% -25.5% 41.23 ± 18% sched_debug.cfs_rq:/.runnable_load_avg.avg 864.71 ± 11% -15.7% 729.11 ± 7% sched_debug.cfs_rq:/.runnable_load_avg.max 184.52 ± 7% -19.5% 148.54 ± 11% sched_debug.cfs_rq:/.runnable_load_avg.stddev 69340 ± 26% -43.1% 39488 ± 14% sched_debug.cfs_rq:/.spread0.stddev 128.04 ± 10% -17.9% 105.18 sched_debug.cfs_rq:/.util_est_enqueued.avg 291.97 ± 4% -15.9% 245.48 ± 2% sched_debug.cfs_rq:/.util_est_enqueued.stddev 940165 -46.9% 498861 sched_debug.cpu.avg_idle.avg 1079015 ± 7% -25.9% 799571 ± 12% sched_debug.cpu.avg_idle.max 162655 ± 9% -46.1% 87662 ± 6% sched_debug.cpu.avg_idle.stddev 183659 ± 6% +19.6% 219628 ± 6% sched_debug.cpu.clock.avg 183677 ± 6% +19.8% 219995 ± 5% sched_debug.cpu.clock.max 183639 ± 6% +19.4% 219208 ± 6% sched_debug.cpu.clock.min 10.78 ± 8% +2091.6% 236.33 ± 59% sched_debug.cpu.clock.stddev 183659 ± 6% +19.6% 219628 ± 6% sched_debug.cpu.clock_task.avg 183677 ± 6% +19.8% 219995 ± 5% sched_debug.cpu.clock_task.max 183639 ± 6% +19.4% 219208 ± 6% sched_debug.cpu.clock_task.min 10.78 ± 8% +2091.5% 236.34 ± 59% sched_debug.cpu.clock_task.stddev 6670 ± 6% -51.1% 3265 ± 5% sched_debug.cpu.curr->pid.max 930.74 ± 4% -14.2% 798.42 ± 2% sched_debug.cpu.curr->pid.stddev 500425 -46.3% 268486 sched_debug.cpu.max_idle_balance_cost.avg 543755 ± 8% -24.0% 413244 ± 14% sched_debug.cpu.max_idle_balance_cost.max 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 4081 ±104% +474.1% 23435 ± 24% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 8% +871.7% 0.00 ± 57% sched_debug.cpu.next_balance.stddev 0.10 ± 7% +19.6% 0.12 ± 7% sched_debug.cpu.nr_running.avg 8136 ± 8% -28.3% 5830 ± 11% sched_debug.cpu.nr_switches.avg 4371 ± 17% -21.0% 3451 ± 9% sched_debug.cpu.nr_switches.stddev 0.01 ± 19% +436.2% 0.03 ± 56% sched_debug.cpu.nr_uninterruptible.avg 6180 ± 10% -37.0% 3891 ± 15% sched_debug.cpu.sched_count.avg 3922 ± 19% -23.7% 2992 ± 10% sched_debug.cpu.sched_count.stddev 982.40 ± 11% +55.2% 1524 ± 20% sched_debug.cpu.sched_goidle.avg 7520 ± 32% +66.4% 12515 ± 22% sched_debug.cpu.sched_goidle.max 1043 ± 19% +41.2% 1473 ± 10% sched_debug.cpu.sched_goidle.stddev 3068 ± 10% -36.5% 1946 ± 16% sched_debug.cpu.ttwu_count.avg 2168 ± 17% -32.1% 1472 ± 6% sched_debug.cpu.ttwu_count.stddev 1780 ± 15% -71.3% 510.64 ± 5% sched_debug.cpu.ttwu_local.avg 6397 ± 15% +68.1% 10753 ± 25% sched_debug.cpu.ttwu_local.max 204.80 ± 29% -62.2% 77.33 ± 16% sched_debug.cpu.ttwu_local.min 183640 ± 6% +19.4% 219189 ± 6% sched_debug.cpu_clk 179329 ± 7% +19.8% 214877 ± 6% sched_debug.ktime 188095 ± 6% +18.2% 222368 ± 5% sched_debug.sched_clk 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 4339533 ± 3% -39.4% 2629862 ± 3% numa-meminfo.node0.Active 4339511 ± 3% -39.4% 2629827 ± 3% numa-meminfo.node0.Active(anon) 238.50 ± 60% +1039.5% 2717 ± 76% numa-meminfo.node0.AnonHugePages 5006302 ± 2% -42.8% 2864276 ± 4% numa-meminfo.node0.AnonPages 255381 +76.2% 449876 ± 7% numa-meminfo.node0.FilePages 667538 -35.8% 428640 ± 4% numa-meminfo.node0.Inactive 667487 -35.8% 428507 ± 4% numa-meminfo.node0.Inactive(anon) 40002 ± 2% -18.1% 32761 ± 6% numa-meminfo.node0.KReclaimable 5528 ± 2% +95.1% 10784 ± 32% numa-meminfo.node0.Mapped 277493 ± 14% +264.0% 1010043 ± 26% numa-meminfo.node0.MemFree 7785521 -9.4% 7052972 ± 3% numa-meminfo.node0.MemUsed 107288 ± 2% -89.2% 11624 ± 6% numa-meminfo.node0.PageTables 40002 ± 2% -18.1% 32761 ± 6% numa-meminfo.node0.SReclaimable 76331 ± 16% +42.5% 108808 ± 11% numa-meminfo.node0.SUnreclaim 1224 ± 23% +1589.3% 20676 ± 45% numa-meminfo.node0.Shmem 4504812 ± 3% -35.3% 2914148 ± 5% numa-meminfo.node1.Active 4504759 ± 3% -35.3% 2914135 ± 5% numa-meminfo.node1.Active(anon) 5174175 ± 3% -37.3% 3245139 ± 5% numa-meminfo.node1.AnonPages 266325 ± 3% +59.8% 425548 ± 5% numa-meminfo.node1.FilePages 669572 -25.1% 501236 ± 6% numa-meminfo.node1.Inactive 669503 -25.2% 501121 ± 6% numa-meminfo.node1.Inactive(anon) 34365 ± 13% -27.9% 24794 ± 9% numa-meminfo.node1.KReclaimable 5348 ± 4% -13.1% 4649 ± 6% numa-meminfo.node1.KernelStack 5099 +68.7% 8604 ± 20% numa-meminfo.node1.Mapped 262692 ± 9% +360.1% 1208713 ± 22% numa-meminfo.node1.MemFree 7986874 -11.8% 7040853 ± 3% numa-meminfo.node1.MemUsed 115349 ± 2% -90.3% 11242 ± 8% numa-meminfo.node1.PageTables 34365 ± 13% -27.9% 24794 ± 9% numa-meminfo.node1.SReclaimable 58815 ± 18% +38.9% 81716 ± 10% numa-meminfo.node1.SUnreclaim 388.75 ± 45% +3941.8% 15712 ± 20% numa-meminfo.node1.Shmem 4747251 -37.5% 2967712 ± 5% numa-meminfo.node2.Active 4747245 -37.5% 2967679 ± 5% numa-meminfo.node2.Active(anon) 5411320 -39.6% 3270127 ± 6% numa-meminfo.node2.AnonPages 263555 ± 4% +68.9% 445205 numa-meminfo.node2.FilePages 663741 -27.9% 478645 ± 11% numa-meminfo.node2.Inactive 663692 -27.9% 478592 ± 11% numa-meminfo.node2.Inactive(anon) 5099 +37.1% 6992 ± 19% numa-meminfo.node2.Mapped 311551 ± 5% +243.1% 1068839 ± 19% numa-meminfo.node2.MemFree 7938015 -9.5% 7180727 ± 2% numa-meminfo.node2.MemUsed 113940 ± 2% -89.9% 11545 ± 5% numa-meminfo.node2.PageTables 46207 ± 11% +96.1% 90608 ± 14% numa-meminfo.node2.SUnreclaim 150.75 ± 58% +13355.6% 20284 ± 69% numa-meminfo.node2.Shmem 79484 ± 9% +49.3% 118674 ± 13% numa-meminfo.node2.Slab 4713363 -38.3% 2909141 ± 3% numa-meminfo.node3.Active 4713360 -38.3% 2909071 ± 3% numa-meminfo.node3.Active(anon) 5372503 -40.2% 3213815 ± 5% numa-meminfo.node3.AnonPages 263686 ± 4% +66.5% 439151 ± 4% numa-meminfo.node3.FilePages 659157 -28.0% 474492 ± 13% numa-meminfo.node3.Inactive 659137 -28.1% 474018 ± 13% numa-meminfo.node3.Inactive(anon) 20.00 ±173% +2268.3% 473.67 ± 92% numa-meminfo.node3.Inactive(file) 330423 ± 8% +223.9% 1070132 ± 15% numa-meminfo.node3.MemFree 7887992 -9.4% 7148283 ± 2% numa-meminfo.node3.MemUsed 114192 ± 3% -90.0% 11432 ± 3% numa-meminfo.node3.PageTables 45094 ± 16% +107.7% 93667 ± 6% numa-meminfo.node3.SUnreclaim 74729 ± 9% +61.0% 120342 ± 6% numa-meminfo.node3.Slab 14.81 ± 14% -18.0% 12.15 ± 16% perf-stat.i.MPKI 3.083e+09 -36.0% 1.974e+09 ± 6% perf-stat.i.branch-instructions 14.08 ± 13% -3.5 10.59 ± 11% perf-stat.i.cache-miss-rate% 23708784 ± 2% -58.9% 9743671 perf-stat.i.cache-misses 1.688e+08 ± 12% -44.9% 93081918 ± 12% perf-stat.i.cache-references 6087 ± 4% +86.7% 11363 perf-stat.i.context-switches 2.10 ± 8% +66.6% 3.50 ± 5% perf-stat.i.cpi 2.237e+10 ± 7% +22.5% 2.74e+10 ± 4% perf-stat.i.cpu-cycles 32.70 ± 36% +148.0% 81.10 ± 3% perf-stat.i.cpu-migrations 1059 ± 7% +173.1% 2892 ± 3% perf-stat.i.cycles-between-cache-misses 0.52 ± 10% -0.2 0.31 ± 25% perf-stat.i.dTLB-load-miss-rate% 18429072 ± 8% -67.2% 6035849 ± 20% perf-stat.i.dTLB-load-misses 3.46e+09 -38.9% 2.114e+09 ± 6% perf-stat.i.dTLB-loads 4537715 ± 10% -62.5% 1702789 ± 25% perf-stat.i.dTLB-store-misses 2.243e+09 ± 7% -61.9% 8.533e+08 ± 9% perf-stat.i.dTLB-stores 74.00 ± 10% -25.8 48.18 ± 30% perf-stat.i.iTLB-load-miss-rate% 3791998 ± 9% -67.7% 1224176 ± 32% perf-stat.i.iTLB-load-misses 1.277e+10 -34.9% 8.309e+09 ± 5% perf-stat.i.instructions 3458 ± 12% +129.5% 7935 ± 35% perf-stat.i.instructions-per-iTLB-miss 0.56 ± 7% -44.5% 0.31 ± 4% perf-stat.i.ipc 339728 -82.7% 58892 ± 5% perf-stat.i.minor-faults 72.00 +19.4 91.39 perf-stat.i.node-load-miss-rate% 11919754 ± 2% -48.9% 6091855 perf-stat.i.node-load-misses 5092184 ± 3% -90.8% 468188 perf-stat.i.node-loads 70.13 +5.6 75.71 perf-stat.i.node-store-miss-rate% 4781459 ± 5% -50.1% 2384157 perf-stat.i.node-store-misses 2215719 ± 7% -66.6% 740244 perf-stat.i.node-stores 339791 -82.6% 59017 ± 5% perf-stat.i.page-faults 14.27 ± 13% -4.3 9.96 ± 10% perf-stat.overall.cache-miss-rate% 1.75 ± 7% +86.6% 3.27 ± 5% perf-stat.overall.cpi 943.87 ± 7% +171.8% 2565 ± 6% perf-stat.overall.cycles-between-cache-misses 0.53 ± 9% -0.2 0.31 ± 30% perf-stat.overall.dTLB-load-miss-rate% 75.42 ± 8% -26.4 48.97 ± 31% perf-stat.overall.iTLB-load-miss-rate% 3396 ± 9% +106.8% 7023 ± 32% perf-stat.overall.instructions-per-iTLB-miss 0.57 ± 7% -46.5% 0.31 ± 5% perf-stat.overall.ipc 70.08 +23.3 93.42 perf-stat.overall.node-load-miss-rate% 68.36 +11.3 79.70 perf-stat.overall.node-store-miss-rate% 8424 ± 2% +237.1% 28402 ± 7% perf-stat.overall.path-length 3.071e+09 -48.0% 1.598e+09 ± 7% perf-stat.ps.branch-instructions 23613472 ± 2% -62.0% 8984355 ± 2% perf-stat.ps.cache-misses 1.681e+08 ± 12% -45.6% 91402164 ± 12% perf-stat.ps.cache-references 6061 ± 4% -9.3% 5498 ± 5% perf-stat.ps.context-switches 18350392 ± 8% -69.5% 5597280 ± 25% perf-stat.ps.dTLB-load-misses 3.446e+09 -47.9% 1.796e+09 ± 7% perf-stat.ps.dTLB-loads 4519736 ± 10% -68.1% 1439823 ± 29% perf-stat.ps.dTLB-store-misses 2.234e+09 ± 7% -61.1% 8.691e+08 ± 8% perf-stat.ps.dTLB-stores 3776596 ± 9% -70.0% 1133072 ± 35% perf-stat.ps.iTLB-load-misses 1.272e+10 -44.5% 7.063e+09 ± 7% perf-stat.ps.instructions 338377 -93.4% 22226 ± 3% perf-stat.ps.minor-faults 11871025 ± 2% -50.7% 5847074 perf-stat.ps.node-load-misses 5069993 ± 3% -91.9% 412375 ± 7% perf-stat.ps.node-loads 4764923 ± 5% -51.4% 2314661 ± 3% perf-stat.ps.node-store-misses 2206822 ± 7% -73.3% 589452 ± 3% perf-stat.ps.node-stores 338442 -93.4% 22286 ± 3% perf-stat.ps.page-faults 3.733e+12 ± 2% -41.8% 2.173e+12 ± 7% perf-stat.total.instructions 380861 ± 4% -96.9% 11776 ± 15% proc-vmstat.allocstall_movable 5280 ± 4% +45.0% 7655 ± 19% proc-vmstat.allocstall_normal 2834038 ± 55% -97.9% 60180 ± 33% proc-vmstat.compact_daemon_free_scanned 1346165 ± 47% -86.6% 180664 ±131% proc-vmstat.compact_daemon_migrate_scanned 3661 ± 95% -99.8% 8.33 ± 14% proc-vmstat.compact_daemon_wake 2929601 ± 52% -97.9% 60180 ± 33% proc-vmstat.compact_free_scanned 266752 ± 41% -95.2% 12762 ± 43% proc-vmstat.compact_isolated 1584266 ± 44% -88.6% 180664 ±131% proc-vmstat.compact_migrate_scanned 452.75 ± 82% -92.3% 34.67 ± 21% proc-vmstat.kswapd_high_wmark_hit_quickly 4277 ± 90% -99.1% 37.67 ± 6% proc-vmstat.kswapd_low_wmark_hit_quickly 4573083 -37.5% 2855944 ± 4% proc-vmstat.nr_active_anon 5236995 -40.0% 3142739 ± 5% proc-vmstat.nr_anon_pages 23889 ± 6% +743.7% 201559 ± 34% proc-vmstat.nr_dirty_background_threshold 47838 ± 6% +953.6% 504024 ± 34% proc-vmstat.nr_dirty_threshold 262238 +72.6% 452544 ± 3% proc-vmstat.nr_file_pages 299754 ± 5% +256.4% 1068440 ± 32% proc-vmstat.nr_free_pages 664056 -28.2% 476653 ± 11% proc-vmstat.nr_inactive_anon 46.75 ± 11% +305.7% 189.67 ± 34% proc-vmstat.nr_inactive_file 163.25 ± 3% +104.6% 334.00 ± 5% proc-vmstat.nr_isolated_anon 5315 +22.8% 6527 proc-vmstat.nr_mapped 112605 -89.7% 11591 ± 5% proc-vmstat.nr_page_table_pages 576.75 ± 2% +3306.5% 19647 ± 53% proc-vmstat.nr_shmem 34157 ± 2% -17.7% 28107 ± 2% proc-vmstat.nr_slab_reclaimable 56613 +66.4% 94220 ± 4% proc-vmstat.nr_slab_unreclaimable 53764745 -95.7% 2292926 ± 5% proc-vmstat.nr_vmscan_write 93294656 -95.2% 4496008 ± 4% proc-vmstat.nr_written 4571365 -37.6% 2854690 ± 4% proc-vmstat.nr_zone_active_anon 665319 -28.4% 476264 ± 12% proc-vmstat.nr_zone_inactive_anon 46.25 ± 10% +310.1% 189.67 ± 34% proc-vmstat.nr_zone_inactive_file 38820524 ± 18% -92.4% 2960034 ± 7% proc-vmstat.numa_foreign 1113 ± 47% +3310.4% 37975 ± 8% proc-vmstat.numa_hint_faults 202.75 ± 76% +6161.9% 12696 ± 36% proc-vmstat.numa_hint_faults_local 65344291 ± 11% -87.4% 8219622 ± 3% proc-vmstat.numa_hit 65243287 ± 11% -87.5% 8130191 ± 3% proc-vmstat.numa_local 38820524 ± 18% -92.4% 2960034 ± 7% proc-vmstat.numa_miss 38921529 ± 18% -92.2% 3049465 ± 7% proc-vmstat.numa_other 45302506 ± 12% -89.7% 4677268 ± 2% proc-vmstat.numa_pte_updates 4750 ± 89% -98.4% 77.33 ± 13% proc-vmstat.pageoutrun 1965706 ± 43% -99.7% 6400 ± 23% proc-vmstat.pgalloc_dma 10382445 ± 11% -95.1% 509165 ± 6% proc-vmstat.pgalloc_dma32 1.183e+08 -90.8% 10852761 ± 2% proc-vmstat.pgalloc_normal 94019717 -95.9% 3818207 ± 3% proc-vmstat.pgdeactivate 99516300 -92.9% 7100462 ± 2% proc-vmstat.pgfault 1.307e+08 -91.8% 10779080 ± 3% proc-vmstat.pgfree 12520 ± 10% -36.1% 7995 ± 54% proc-vmstat.pgmajfault 787.50 ±104% -98.4% 12.33 ± 78% proc-vmstat.pgmigrate_fail 129306 ± 41% -94.9% 6595 ± 38% proc-vmstat.pgmigrate_success 94620510 -96.0% 3821306 ± 3% proc-vmstat.pgrefill 42338990 ± 7% +425.9% 2.227e+08 ± 3% proc-vmstat.pgscan_direct 60752611 -97.0% 1796934 ± 12% proc-vmstat.pgscan_kswapd 32578411 -92.3% 2517344 ± 10% proc-vmstat.pgsteal_direct 60721385 -97.1% 1739303 ± 13% proc-vmstat.pgsteal_kswapd 12803 ± 10% -77.2% 2916 ± 49% proc-vmstat.pswpin 93298288 -98.2% 1690205 ± 7% proc-vmstat.pswpout 143.25 ± 8% +667.2% 1099 ± 96% proc-vmstat.swap_ra 52.00 +1790.4% 983.00 ±104% proc-vmstat.swap_ra_hit 1084114 ± 3% -42.0% 628577 numa-vmstat.node0.nr_active_anon 1250553 ± 2% -45.5% 681208 numa-vmstat.node0.nr_anon_pages 63849 +72.1% 109898 ± 8% numa-vmstat.node0.nr_file_pages 70516 ± 14% +342.3% 311920 ± 31% numa-vmstat.node0.nr_free_pages 166639 -40.7% 98872 ± 7% numa-vmstat.node0.nr_inactive_anon 1377 +93.3% 2662 ± 32% numa-vmstat.node0.nr_mapped 26837 ± 2% -89.5% 2817 ± 9% numa-vmstat.node0.nr_page_table_pages 310.00 ± 23% +1635.5% 5380 ± 42% numa-vmstat.node0.nr_shmem 9826 ± 2% -16.9% 8165 ± 6% numa-vmstat.node0.nr_slab_reclaimable 19082 ± 16% +39.8% 26668 ± 12% numa-vmstat.node0.nr_slab_unreclaimable 12469818 ± 2% -95.7% 530861 ± 8% numa-vmstat.node0.nr_vmscan_write 12469821 ± 2% -94.4% 701283 ± 11% numa-vmstat.node0.nr_written 1083727 ± 3% -42.0% 628283 numa-vmstat.node0.nr_zone_active_anon 166963 -40.8% 98803 ± 7% numa-vmstat.node0.nr_zone_inactive_anon 9322571 ± 84% -91.5% 788811 ± 78% numa-vmstat.node0.numa_foreign 12430629 ± 8% -83.7% 2025212 ± 16% numa-vmstat.node0.numa_hit 12406467 ± 8% -83.9% 2002001 ± 17% numa-vmstat.node0.numa_local 2617336 ± 39% -85.4% 382759 ± 54% numa-vmstat.node0.numa_miss 2641506 ± 39% -84.6% 405973 ± 51% numa-vmstat.node0.numa_other 1125698 ± 3% -38.7% 690330 ± 9% numa-vmstat.node1.nr_active_anon 1292897 ± 3% -40.8% 764966 ± 9% numa-vmstat.node1.nr_anon_pages 66581 ± 3% +56.0% 103898 ± 6% numa-vmstat.node1.nr_file_pages 66383 ± 10% +460.8% 372282 ± 29% numa-vmstat.node1.nr_free_pages 167219 -31.4% 114672 ± 11% numa-vmstat.node1.nr_inactive_anon 5345 ± 4% -13.0% 4650 ± 6% numa-vmstat.node1.nr_kernel_stack 1303 ± 3% +54.3% 2011 ± 3% numa-vmstat.node1.nr_mapped 28875 ± 2% -90.7% 2689 ± 10% numa-vmstat.node1.nr_page_table_pages 97.75 ± 46% +3982.9% 3991 ± 26% numa-vmstat.node1.nr_shmem 8425 ± 13% -28.1% 6057 ± 12% numa-vmstat.node1.nr_slab_reclaimable 13793388 -96.0% 558058 ± 8% numa-vmstat.node1.nr_vmscan_write 13793433 -94.9% 706238 ± 8% numa-vmstat.node1.nr_written 1125261 ± 3% -38.7% 690068 ± 9% numa-vmstat.node1.nr_zone_active_anon 167470 -31.6% 114561 ± 11% numa-vmstat.node1.nr_zone_inactive_anon 8969163 ± 41% -97.5% 220481 ± 63% numa-vmstat.node1.numa_foreign 12036607 ± 29% -88.3% 1410089 ± 4% numa-vmstat.node1.numa_hit 11938330 ± 29% -89.0% 1318629 ± 5% numa-vmstat.node1.numa_local 4043928 ± 82% -79.1% 845837 ± 22% numa-vmstat.node1.numa_miss 4142212 ± 80% -77.4% 937300 ± 18% numa-vmstat.node1.numa_other 1186166 -39.2% 720915 ± 7% numa-vmstat.node2.nr_active_anon 1352038 -41.8% 787352 ± 8% numa-vmstat.node2.nr_anon_pages 65889 ± 4% +64.5% 108363 ± 2% numa-vmstat.node2.nr_file_pages 78835 ± 5% +310.8% 323882 ± 20% numa-vmstat.node2.nr_free_pages 165769 -35.0% 107757 ± 21% numa-vmstat.node2.nr_inactive_anon 28517 ± 2% -90.2% 2802 ± 6% numa-vmstat.node2.nr_page_table_pages 37.50 ± 58% +13228.0% 4998 ± 69% numa-vmstat.node2.nr_shmem 8143 ± 7% -16.1% 6835 ± 11% numa-vmstat.node2.nr_slab_reclaimable 11551 ± 11% +90.3% 21980 ± 12% numa-vmstat.node2.nr_slab_unreclaimable 13669648 ± 2% -95.9% 557388 ± 10% numa-vmstat.node2.nr_vmscan_write 13669665 ± 2% -94.8% 708930 ± 8% numa-vmstat.node2.nr_written 1185741 -39.2% 720658 ± 7% numa-vmstat.node2.nr_zone_active_anon 166089 -35.2% 107681 ± 21% numa-vmstat.node2.nr_zone_inactive_anon 2623218 ± 27% -80.5% 511882 ±100% numa-vmstat.node2.numa_foreign 7670185 ± 14% -73.9% 2005004 ± 11% numa-vmstat.node2.numa_hit 7572087 ± 15% -74.8% 1904742 ± 12% numa-vmstat.node2.numa_local 8417300 ± 17% -95.5% 376219 ± 59% numa-vmstat.node2.numa_miss 8515407 ± 17% -94.4% 476484 ± 49% numa-vmstat.node2.numa_other 14.25 ± 80% -90.6% 1.33 ±141% numa-vmstat.node2.workingset_nodes 1177529 -40.0% 706088 ± 4% numa-vmstat.node3.nr_active_anon 1342172 -42.3% 774015 ± 6% numa-vmstat.node3.nr_anon_pages 65923 ± 4% +62.3% 107007 ± 4% numa-vmstat.node3.nr_file_pages 83660 ± 7% +295.0% 330457 ± 25% numa-vmstat.node3.nr_free_pages 164645 -34.7% 107507 ± 17% numa-vmstat.node3.nr_inactive_anon 28571 ± 3% -90.4% 2752 ± 6% numa-vmstat.node3.nr_page_table_pages 11273 ± 16% +101.0% 22656 ± 8% numa-vmstat.node3.nr_slab_unreclaimable 13895597 ± 2% -96.1% 543450 ± 10% numa-vmstat.node3.nr_vmscan_write 13895599 ± 2% -95.0% 691547 ± 10% numa-vmstat.node3.nr_written 1177056 -40.0% 705805 ± 4% numa-vmstat.node3.nr_zone_active_anon 165015 -34.9% 107434 ± 17% numa-vmstat.node3.nr_zone_inactive_anon 9066546 ± 6% -79.8% 1829027 ± 20% numa-vmstat.node3.numa_hit 8963338 ± 6% -80.6% 1736058 ± 21% numa-vmstat.node3.numa_local 7069916 ± 9% -93.1% 485153 ± 63% numa-vmstat.node3.numa_miss 7173131 ± 9% -91.9% 578125 ± 52% numa-vmstat.node3.numa_other 14.91 ± 8% -12.5 2.44 ± 32% perf-profile.calltrace.cycles-pp.do_access 12.01 ± 8% -9.6 2.44 ± 32% perf-profile.calltrace.cycles-pp.page_fault.do_access 11.57 ± 8% -9.1 2.44 ± 32% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access 11.52 ± 8% -9.1 2.44 ± 32% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.do_access 10.96 ± 8% -8.5 2.44 ± 32% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_access 9.60 ± 8% -6.9 2.68 ± 21% perf-profile.calltrace.cycles-pp.alloc_pages_vma.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 9.52 ± 8% -6.8 2.68 ± 21% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault.handle_mm_fault.__do_page_fault 9.12 ± 8% -6.4 2.68 ± 21% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault.handle_mm_fault 10.84 ± 8% -4.7 6.10 ± 26% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 7.23 ± 9% -4.5 2.68 ± 21% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault 7.14 ± 9% -1.8 5.35 ± 26% perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma 1.02 ± 43% +0.6 1.59 ± 34% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 3.19 ± 9% +0.9 4.09 ± 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.12 ±173% +2.4 2.48 ± 60% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 3.90 ± 28% +2.4 6.26 ± 16% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +2.5 2.47 ± 61% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 7.84 ± 15% +2.5 10.33 ± 6% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 0.00 +2.5 2.51 ± 92% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.new_slab.___slab_alloc.__slab_alloc.kmem_cache_alloc_node 0.00 +2.7 2.67 ± 66% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_swap_page.__handle_mm_fault.handle_mm_fault 0.00 +2.7 2.67 ± 67% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.do_swap_page.__handle_mm_fault 0.00 +2.7 2.67 ± 67% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.do_swap_page 0.00 +2.7 2.68 ± 67% perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_swap_page.__handle_mm_fault.handle_mm_fault.__do_page_fault 0.00 +2.8 2.76 ± 68% perf-profile.calltrace.cycles-pp.do_swap_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 0.00 +2.8 2.78 ± 79% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.new_slab.___slab_alloc.__slab_alloc 0.00 +2.8 2.78 ± 79% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.new_slab.___slab_alloc 0.00 +2.8 2.78 ± 79% perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.new_slab 8.70 ± 16% +3.1 11.79 ± 6% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 66.46 ± 3% +5.9 72.39 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 66.99 ± 3% +5.9 72.93 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 71.77 ± 3% +7.0 78.80 ± 3% perf-profile.calltrace.cycles-pp.secondary_startup_64 71.24 ± 3% +7.1 78.37 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 71.29 ± 3% +7.2 78.44 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 71.29 ± 3% +7.2 78.44 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 3.36 ± 8% +13.6 17.00 ± 25% perf-profile.calltrace.cycles-pp.pageout.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node 0.00 +15.7 15.65 ± 25% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.z3fold_alloc.zswap_frontswap_store.__frontswap_store 0.00 +16.4 16.44 ± 26% perf-profile.calltrace.cycles-pp._raw_spin_lock.z3fold_alloc.zswap_frontswap_store.__frontswap_store.swap_writepage 0.00 +16.5 16.54 ± 26% perf-profile.calltrace.cycles-pp.z3fold_alloc.zswap_frontswap_store.__frontswap_store.swap_writepage.pageout 0.00 +17.0 16.97 ± 25% perf-profile.calltrace.cycles-pp.zswap_frontswap_store.__frontswap_store.swap_writepage.pageout.shrink_page_list 0.00 +17.0 16.98 ± 25% perf-profile.calltrace.cycles-pp.__frontswap_store.swap_writepage.pageout.shrink_page_list.shrink_inactive_list 0.00 +17.0 16.99 ± 25% perf-profile.calltrace.cycles-pp.swap_writepage.pageout.shrink_page_list.shrink_inactive_list.shrink_node_memcg 15.46 ± 8% -13.0 2.44 ± 32% perf-profile.children.cycles-pp.do_access 12.15 ± 8% -5.3 6.89 ± 24% perf-profile.children.cycles-pp.page_fault 11.68 ± 8% -4.8 6.89 ± 24% perf-profile.children.cycles-pp.do_page_fault 11.64 ± 8% -4.8 6.89 ± 24% perf-profile.children.cycles-pp.__do_page_fault 11.07 ± 8% -4.2 6.90 ± 24% perf-profile.children.cycles-pp.handle_mm_fault 10.95 ± 8% -4.0 6.90 ± 24% perf-profile.children.cycles-pp.__handle_mm_fault 9.67 ± 8% -2.8 6.87 ± 26% perf-profile.children.cycles-pp.alloc_pages_vma 2.28 ± 9% -2.0 0.30 ± 48% perf-profile.children.cycles-pp.rmap_walk_anon 3.70 ± 19% -1.5 2.22 ± 37% perf-profile.children.cycles-pp.__softirqentry_text_start 1.54 ± 14% -1.5 0.08 ± 40% perf-profile.children.cycles-pp.rcu_core 1.27 ± 7% -0.9 0.33 ± 50% perf-profile.children.cycles-pp.page_referenced 0.67 ± 8% -0.5 0.12 ± 55% perf-profile.children.cycles-pp.page_vma_mapped_walk 0.67 ± 7% -0.5 0.18 ± 53% perf-profile.children.cycles-pp.page_referenced_one 0.83 ± 12% -0.3 0.50 ± 8% perf-profile.children.cycles-pp.native_irq_return_iret 0.55 ± 12% -0.3 0.23 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.36 ± 6% -0.2 0.12 ± 56% perf-profile.children.cycles-pp.isolate_lru_pages 0.29 ± 3% -0.2 0.09 ± 59% perf-profile.children.cycles-pp.inactive_list_is_low 0.41 ± 8% -0.1 0.26 ± 10% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.64 ± 5% -0.1 0.53 ± 7% perf-profile.children.cycles-pp.find_next_bit 0.21 ± 9% -0.1 0.14 ± 26% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.08 ± 19% +0.0 0.11 ± 11% perf-profile.children.cycles-pp.rcu_irq_exit 0.12 ± 7% +0.0 0.16 ± 10% perf-profile.children.cycles-pp.rcu_idle_exit 0.08 ± 20% +0.1 0.13 ± 23% perf-profile.children.cycles-pp.menu_reflect 0.21 ± 17% +0.1 0.29 ± 8% perf-profile.children.cycles-pp.native_sched_clock 0.22 ± 15% +0.1 0.31 ± 8% perf-profile.children.cycles-pp.sched_clock 0.28 ± 16% +0.1 0.39 ± 10% perf-profile.children.cycles-pp.sched_clock_cpu 0.29 ± 11% +0.1 0.40 ± 14% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.00 +0.1 0.15 ± 74% perf-profile.children.cycles-pp.dup_mm 0.63 ± 9% +0.2 0.78 ± 8% perf-profile.children.cycles-pp.__next_timer_interrupt 0.31 ± 21% +0.2 0.50 ± 20% perf-profile.children.cycles-pp.read_tsc 0.00 +0.2 0.21 ± 47% perf-profile.children.cycles-pp.mm_init 0.00 +0.2 0.21 ± 47% perf-profile.children.cycles-pp.pgd_alloc 0.35 ± 23% +0.2 0.57 ± 18% perf-profile.children.cycles-pp.native_write_msr 0.32 ± 29% +0.2 0.55 ± 18% perf-profile.children.cycles-pp.lapic_next_deadline 0.00 +0.3 0.28 ±100% perf-profile.children.cycles-pp.do_wp_page 0.00 +0.3 0.28 ±100% perf-profile.children.cycles-pp.wp_page_copy 0.00 +0.3 0.30 ± 50% perf-profile.children.cycles-pp.pipe_write 0.00 +0.3 0.34 ± 91% perf-profile.children.cycles-pp.__vmalloc_node 0.00 +0.3 0.34 ± 20% perf-profile.children.cycles-pp.lz4_compress_crypto 0.00 +0.3 0.34 ± 20% perf-profile.children.cycles-pp.LZ4_compress_default 0.00 +0.4 0.36 ± 87% perf-profile.children.cycles-pp.proc_reg_open 0.00 +0.4 0.36 ± 87% perf-profile.children.cycles-pp.single_open_size 0.00 +0.4 0.37 ± 84% perf-profile.children.cycles-pp.open64 0.00 +0.4 0.39 ± 80% perf-profile.children.cycles-pp.__vmalloc_node_range 0.00 +0.4 0.40 ± 74% perf-profile.children.cycles-pp.do_dentry_open 0.00 +0.4 0.41 ± 73% perf-profile.children.cycles-pp.do_sys_open 0.00 +0.4 0.41 ± 73% perf-profile.children.cycles-pp.do_filp_open 0.00 +0.4 0.41 ± 73% perf-profile.children.cycles-pp.path_openat 0.07 ± 17% +0.4 0.50 ± 70% perf-profile.children.cycles-pp.__libc_fork 0.03 ±105% +0.4 0.48 ± 71% perf-profile.children.cycles-pp.__x64_sys_clone 0.00 +0.5 0.46 ± 76% perf-profile.children.cycles-pp.__get_free_pages 0.00 +0.5 0.46 ± 48% perf-profile.children.cycles-pp.ksys_write 0.00 +0.5 0.46 ± 48% perf-profile.children.cycles-pp.vfs_write 0.00 +0.5 0.46 ± 48% perf-profile.children.cycles-pp.new_sync_write 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.disk_check_events 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.sr_block_check_events 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.cdrom_check_events 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.sr_check_events 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.__scsi_execute 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.blk_rq_map_kern 0.00 +0.8 0.80 ± 56% perf-profile.children.cycles-pp.bio_copy_kern 3.24 ± 9% +0.9 4.16 ± 10% perf-profile.children.cycles-pp.menu_select 0.00 +1.2 1.23 ± 79% perf-profile.children.cycles-pp.poll 0.00 +1.5 1.47 ± 67% perf-profile.children.cycles-pp.__x64_sys_poll 0.00 +1.5 1.47 ± 67% perf-profile.children.cycles-pp.do_sys_poll 0.03 ±105% +1.6 1.60 ± 59% perf-profile.children.cycles-pp._do_fork 0.03 ±102% +1.6 1.60 ± 59% perf-profile.children.cycles-pp.copy_process 0.42 ± 15% +2.1 2.48 ± 60% perf-profile.children.cycles-pp.worker_thread 0.39 ± 18% +2.1 2.47 ± 61% perf-profile.children.cycles-pp.process_one_work 8.43 ± 14% +2.1 10.57 ± 6% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 4.18 ± 26% +2.3 6.45 ± 16% perf-profile.children.cycles-pp.hrtimer_interrupt 9.03 ± 14% +2.3 11.37 ± 6% perf-profile.children.cycles-pp.apic_timer_interrupt 0.51 ± 10% +2.5 3.03 ± 67% perf-profile.children.cycles-pp.__slab_alloc 0.51 ± 10% +2.5 3.03 ± 67% perf-profile.children.cycles-pp.___slab_alloc 0.46 ± 11% +2.6 3.03 ± 67% perf-profile.children.cycles-pp.new_slab 0.00 +2.7 2.74 ± 79% perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.00 +3.5 3.47 ± 49% perf-profile.children.cycles-pp.do_swap_page 0.23 ± 11% +4.1 4.31 ± 53% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 0.23 ± 11% +4.1 4.31 ± 53% perf-profile.children.cycles-pp.do_syscall_64 7.37 ± 8% +4.8 12.21 ± 17% perf-profile.children.cycles-pp.try_to_free_pages 7.28 ± 8% +4.9 12.21 ± 17% perf-profile.children.cycles-pp.do_try_to_free_pages 15.73 ± 8% +5.1 20.87 ± 14% perf-profile.children.cycles-pp.shrink_node 15.14 ± 8% +5.7 20.87 ± 14% perf-profile.children.cycles-pp.shrink_node_memcg 67.41 ± 3% +5.8 73.25 ± 5% perf-profile.children.cycles-pp.cpuidle_enter_state 67.43 ± 3% +5.8 73.27 ± 5% perf-profile.children.cycles-pp.cpuidle_enter 13.75 ± 8% +7.0 20.76 ± 14% perf-profile.children.cycles-pp.shrink_inactive_list 13.48 ± 8% +7.0 20.50 ± 14% perf-profile.children.cycles-pp.shrink_page_list 71.77 ± 3% +7.0 78.80 ± 3% perf-profile.children.cycles-pp.secondary_startup_64 71.77 ± 3% +7.0 78.80 ± 3% perf-profile.children.cycles-pp.cpu_startup_entry 71.78 ± 3% +7.0 78.83 ± 3% perf-profile.children.cycles-pp.do_idle 71.29 ± 3% +7.2 78.44 ± 3% perf-profile.children.cycles-pp.start_secondary 3.38 ± 8% +16.5 19.90 ± 13% perf-profile.children.cycles-pp.pageout 0.42 ± 25% +18.0 18.45 ± 13% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.06 ± 9% +18.5 19.57 ± 13% perf-profile.children.cycles-pp._raw_spin_lock 0.00 +19.4 19.39 ± 13% perf-profile.children.cycles-pp.z3fold_alloc 0.39 ± 8% +19.5 19.89 ± 13% perf-profile.children.cycles-pp.swap_writepage 0.27 ± 10% +19.6 19.88 ± 13% perf-profile.children.cycles-pp.__frontswap_store 0.10 ± 11% +19.8 19.87 ± 13% perf-profile.children.cycles-pp.zswap_frontswap_store 0.83 ± 12% -0.3 0.50 ± 8% perf-profile.self.cycles-pp.native_irq_return_iret 0.40 ± 10% -0.3 0.08 ± 53% perf-profile.self.cycles-pp.page_vma_mapped_walk 0.41 ± 10% -0.2 0.21 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.20 ± 13% -0.1 0.10 ± 53% perf-profile.self.cycles-pp.shrink_page_list 0.54 ± 6% -0.1 0.45 ± 10% perf-profile.self.cycles-pp.find_next_bit 0.08 ± 19% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.rcu_irq_exit 0.11 ± 8% +0.0 0.14 ± 17% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.04 ± 58% +0.1 0.09 ± 25% perf-profile.self.cycles-pp.menu_reflect 0.19 ± 17% +0.1 0.27 ± 6% perf-profile.self.cycles-pp.native_sched_clock 0.29 ± 21% +0.2 0.47 ± 21% perf-profile.self.cycles-pp.read_tsc 0.34 ± 24% +0.2 0.57 ± 18% perf-profile.self.cycles-pp.native_write_msr 0.00 +0.3 0.32 ± 19% perf-profile.self.cycles-pp.LZ4_compress_default 0.79 ± 9% +0.3 1.12 ± 23% perf-profile.self.cycles-pp._raw_spin_lock 1.05 ± 15% +0.5 1.59 ± 9% perf-profile.self.cycles-pp.menu_select 0.41 ± 25% +17.8 18.24 ± 13% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 122606 ± 39% -84.8% 18591 ± 7% softirqs.CPU0.RCU 203038 ± 29% -88.5% 23369 ± 14% softirqs.CPU1.RCU 150480 ± 40% -87.3% 19153 ± 12% softirqs.CPU10.RCU 98126 ± 26% -81.1% 18538 ± 13% softirqs.CPU100.RCU 93426 ± 14% -80.7% 18029 ± 4% softirqs.CPU101.RCU 77015 ± 9% -76.6% 18022 ± 9% softirqs.CPU102.RCU 99909 ± 11% -81.1% 18923 ± 15% softirqs.CPU103.RCU 93122 ± 13% -81.3% 17437 ± 13% softirqs.CPU104.RCU 78975 ± 7% -76.6% 18480 ± 8% softirqs.CPU105.RCU 87946 ± 13% -78.9% 18550 ± 6% softirqs.CPU106.RCU 33409 ± 11% +15.7% 38663 ± 3% softirqs.CPU106.SCHED 115027 ± 20% -82.9% 19648 ± 6% softirqs.CPU107.RCU 81078 ± 18% -66.9% 26850 ± 24% softirqs.CPU108.RCU 36747 ± 4% -11.9% 32386 ± 9% softirqs.CPU108.SCHED 69297 ± 15% -75.6% 16921 ± 4% softirqs.CPU109.RCU 165572 ± 34% -88.3% 19328 ± 12% softirqs.CPU11.RCU 69818 ± 21% -73.9% 18202 ± 9% softirqs.CPU110.RCU 80920 ± 28% -79.8% 16354 ± 8% softirqs.CPU111.RCU 79628 ± 10% -78.1% 17463 ± 7% softirqs.CPU112.RCU 72148 ± 6% -75.2% 17882 ± 6% softirqs.CPU113.RCU 75531 ± 22% -75.9% 18175 ± 9% softirqs.CPU114.RCU 72033 ± 17% -74.9% 18113 ± 6% softirqs.CPU115.RCU 96564 ± 20% -82.7% 16672 ± 4% softirqs.CPU116.RCU 71933 ± 6% -74.0% 18715 ± 11% softirqs.CPU117.RCU 75370 ± 18% -75.3% 18640 ± 15% softirqs.CPU118.RCU 87992 ± 20% -78.5% 18945 ± 10% softirqs.CPU119.RCU 128382 ± 23% -84.5% 19897 ± 14% softirqs.CPU12.RCU 69381 ± 25% -74.0% 18022 ± 6% softirqs.CPU120.RCU 80626 ± 22% -76.6% 18842 ± 8% softirqs.CPU121.RCU 70946 ± 19% -71.1% 20518 ± 12% softirqs.CPU122.RCU 62063 ± 12% -66.8% 20631 ± 4% softirqs.CPU123.RCU 75868 ± 20% -76.2% 18022 ± 4% softirqs.CPU124.RCU 81495 ± 13% -76.8% 18935 ± 4% softirqs.CPU125.RCU 70058 ± 6% -75.0% 17531 ± 15% softirqs.CPU126.RCU 81325 ± 26% -79.8% 16436 ± 15% softirqs.CPU127.RCU 86220 ± 33% -80.8% 16592 ± 14% softirqs.CPU128.RCU 68686 ± 20% -77.5% 15475 ± 7% softirqs.CPU129.RCU 132884 ± 33% -85.3% 19578 ± 11% softirqs.CPU13.RCU 74043 ± 25% -75.5% 18144 ± 13% softirqs.CPU130.RCU 59391 ± 14% -48.1% 30845 ± 59% softirqs.CPU131.RCU 81171 ± 22% -79.2% 16866 ± 12% softirqs.CPU132.RCU 93115 ± 24% -81.5% 17209 ± 5% softirqs.CPU133.RCU 87047 ± 12% -81.2% 16340 ± 2% softirqs.CPU134.RCU 69785 ± 19% -76.8% 16158 ± 2% softirqs.CPU135.RCU 95661 ± 40% -82.5% 16781 ± 4% softirqs.CPU136.RCU 70503 ± 9% -76.3% 16692 ± 7% softirqs.CPU137.RCU 75369 ± 14% -77.7% 16824 ± 5% softirqs.CPU138.RCU 79699 ± 21% -78.2% 17410 ± 5% softirqs.CPU139.RCU 136157 ± 39% -83.1% 23004 ± 22% softirqs.CPU14.RCU 76216 ± 26% -78.5% 16395 softirqs.CPU140.RCU 90114 ± 27% -78.3% 19521 ± 19% softirqs.CPU141.RCU 65557 ± 14% -70.4% 19378 ± 17% softirqs.CPU142.RCU 90425 ± 12% -81.1% 17051 ± 2% softirqs.CPU143.RCU 146804 ± 31% -85.6% 21205 ± 17% softirqs.CPU15.RCU 176665 ± 42% -89.5% 18487 ± 6% softirqs.CPU16.RCU 166802 ± 40% -89.8% 17010 ± 5% softirqs.CPU17.RCU 185563 ± 17% -87.3% 23609 ± 18% softirqs.CPU18.RCU 141768 ± 26% -83.3% 23614 ± 22% softirqs.CPU19.RCU 156993 ± 33% -87.8% 19088 ± 10% softirqs.CPU2.RCU 141610 ± 16% -85.6% 20417 ± 14% softirqs.CPU20.RCU 148436 ± 24% -87.4% 18663 ± 2% softirqs.CPU21.RCU 139031 ± 31% -85.9% 19632 ± 4% softirqs.CPU22.RCU 146175 ± 26% -85.5% 21181 ± 12% softirqs.CPU23.RCU 31417 ± 8% +15.9% 36417 ± 5% softirqs.CPU23.SCHED 127829 ± 20% -84.2% 20230 ± 12% softirqs.CPU24.RCU 116097 ± 28% -81.8% 21132 ± 9% softirqs.CPU25.RCU 139619 ± 3% -86.3% 19168 softirqs.CPU26.RCU 135700 ± 20% -85.4% 19847 ± 8% softirqs.CPU27.RCU 32348 ± 3% +12.2% 36305 ± 3% softirqs.CPU27.SCHED 121814 ± 20% -85.1% 18147 ± 3% softirqs.CPU28.RCU 126002 ± 17% -85.0% 18951 ± 5% softirqs.CPU29.RCU 160372 ± 27% -87.1% 20621 ± 9% softirqs.CPU3.RCU 118673 ± 18% -85.3% 17488 ± 5% softirqs.CPU30.RCU 144613 ± 17% -86.3% 19837 ± 11% softirqs.CPU31.RCU 32959 ± 2% +13.1% 37286 ± 3% softirqs.CPU31.SCHED 123185 ± 27% -84.9% 18635 ± 5% softirqs.CPU32.RCU 127766 ± 29% -85.6% 18430 ± 8% softirqs.CPU33.RCU 123713 ± 28% -85.5% 17984 ± 4% softirqs.CPU34.RCU 157712 ± 22% -87.8% 19239 softirqs.CPU35.RCU 33338 ± 5% +12.6% 37524 ± 3% softirqs.CPU35.SCHED 120549 ± 22% -82.0% 21718 ± 4% softirqs.CPU36.RCU 31740 ± 6% +12.3% 35630 ± 5% softirqs.CPU36.SCHED 116845 ± 15% -85.2% 17268 ± 5% softirqs.CPU37.RCU 95905 ± 25% -81.0% 18236 ± 14% softirqs.CPU38.RCU 101435 ± 17% -82.3% 17940 ± 8% softirqs.CPU39.RCU 144406 ± 35% -86.1% 20108 ± 13% softirqs.CPU4.RCU 136655 ± 16% -87.2% 17550 ± 8% softirqs.CPU40.RCU 33988 ± 5% +12.2% 38130 ± 2% softirqs.CPU40.SCHED 83889 ± 21% -77.5% 18878 softirqs.CPU41.RCU 101687 ± 26% -80.7% 19622 ± 6% softirqs.CPU42.RCU 111109 ± 28% -83.3% 18601 ± 3% softirqs.CPU43.RCU 103519 ± 25% -82.4% 18187 ± 7% softirqs.CPU44.RCU 103654 ± 26% -78.7% 22042 ± 10% softirqs.CPU45.RCU 98151 ± 41% -79.5% 20146 ± 13% softirqs.CPU46.RCU 107163 ± 28% -82.8% 18463 ± 2% softirqs.CPU47.RCU 110162 ± 14% -82.5% 19289 ± 6% softirqs.CPU48.RCU 96132 ± 22% -81.3% 18018 ± 6% softirqs.CPU49.RCU 144339 ± 42% -85.8% 20501 ± 16% softirqs.CPU5.RCU 106013 ± 28% -79.9% 21333 ± 15% softirqs.CPU50.RCU 89289 ± 19% -77.3% 20308 ± 9% softirqs.CPU51.RCU 95463 ± 24% -77.5% 21482 ± 5% softirqs.CPU52.RCU 105183 ± 21% -83.0% 17871 ± 2% softirqs.CPU53.RCU 108932 ± 24% -79.9% 21897 ± 14% softirqs.CPU54.RCU 32507 ± 5% +9.0% 35428 ± 3% softirqs.CPU54.SCHED 125463 ± 49% -84.5% 19488 ± 19% softirqs.CPU55.RCU 109702 ± 14% -83.0% 18691 ± 20% softirqs.CPU56.RCU 98004 ± 36% -82.8% 16903 ± 3% softirqs.CPU57.RCU 98802 ± 32% -81.6% 18168 ± 6% softirqs.CPU58.RCU 86125 ± 25% -76.6% 20188 ± 22% softirqs.CPU59.RCU 142191 ± 30% -84.7% 21779 ± 13% softirqs.CPU6.RCU 91832 ± 18% -80.8% 17587 ± 8% softirqs.CPU60.RCU 95250 ± 24% -80.9% 18193 ± 8% softirqs.CPU61.RCU 110906 ± 28% -83.0% 18890 ± 10% softirqs.CPU62.RCU 96808 ± 14% -82.4% 17038 ± 2% softirqs.CPU63.RCU 103911 ± 18% -85.0% 15608 ± 3% softirqs.CPU64.RCU 111434 ± 13% -76.0% 26761 ± 49% softirqs.CPU65.RCU 120328 ± 30% -86.4% 16376 ± 7% softirqs.CPU66.RCU 114254 ± 17% -85.4% 16657 ± 10% softirqs.CPU67.RCU 94384 ± 22% -82.0% 16982 ± 4% softirqs.CPU68.RCU 103606 ± 24% -70.9% 30127 ± 46% softirqs.CPU69.RCU 152746 ± 18% -87.5% 19092 ± 8% softirqs.CPU7.RCU 88543 ± 9% -77.8% 19671 ± 23% softirqs.CPU70.RCU 91807 ± 14% -76.0% 22015 ± 31% softirqs.CPU71.RCU 91668 ± 17% -75.1% 22825 ± 19% softirqs.CPU72.RCU 122358 ± 29% -84.3% 19219 ± 6% softirqs.CPU73.RCU 119056 ± 6% -84.1% 18892 ± 5% softirqs.CPU74.RCU 76091 ± 12% -73.3% 20309 ± 24% softirqs.CPU75.RCU 91371 ± 17% -78.1% 20011 ± 8% softirqs.CPU76.RCU 87961 ± 18% -79.7% 17850 ± 8% softirqs.CPU77.RCU 101265 ± 25% -77.7% 22620 ± 29% softirqs.CPU78.RCU 111370 ± 13% -83.0% 18961 ± 9% softirqs.CPU79.RCU 138698 ± 41% -87.1% 17837 ± 7% softirqs.CPU8.RCU 95723 ± 36% -82.6% 16658 ± 8% softirqs.CPU80.RCU 96671 ± 21% -82.1% 17290 ± 10% softirqs.CPU81.RCU 94733 ± 30% -81.2% 17825 ± 9% softirqs.CPU82.RCU 100387 ± 30% -80.6% 19431 ± 4% softirqs.CPU83.RCU 80467 ± 23% -77.2% 18371 ± 7% softirqs.CPU84.RCU 103293 ± 28% -82.2% 18342 ± 13% softirqs.CPU85.RCU 85568 ± 16% -79.2% 17784 ± 4% softirqs.CPU86.RCU 104826 ± 29% -81.5% 19385 ± 2% softirqs.CPU87.RCU 117826 ± 29% -80.6% 22900 ± 9% softirqs.CPU88.RCU 100578 ± 23% -81.5% 18600 ± 9% softirqs.CPU89.RCU 175695 ± 36% -89.3% 18713 ± 13% softirqs.CPU9.RCU 84614 ± 29% -77.4% 19127 ± 11% softirqs.CPU90.RCU 94919 ± 18% -79.9% 19054 ± 21% softirqs.CPU91.RCU 71979 ± 3% -71.7% 20342 ± 10% softirqs.CPU92.RCU 91785 ± 33% -82.3% 16216 ± 5% softirqs.CPU93.RCU 85976 ± 21% -79.1% 17984 ± 3% softirqs.CPU94.RCU 91839 ± 16% -81.7% 16781 ± 7% softirqs.CPU95.RCU 86493 ± 29% -78.4% 18653 ± 11% softirqs.CPU96.RCU 95683 ± 7% -79.4% 19730 ± 16% softirqs.CPU97.RCU 83164 ± 22% -79.4% 17127 ± 6% softirqs.CPU98.RCU 80882 ± 12% -77.3% 18394 ± 8% softirqs.CPU99.RCU 15122287 ± 11% -81.8% 2749702 ± 4% softirqs.RCU 603.75 ±105% -72.1% 168.33 ± 8% interrupts.54:PCI-MSI.1572873-edge.eth0-TxRx-9 523.00 ± 79% -68.1% 166.67 ± 9% interrupts.59:PCI-MSI.1572878-edge.eth0-TxRx-14 81871075 -98.0% 1625185 ± 2% interrupts.CAL:Function_call_interrupts 849480 ± 68% -98.5% 12359 ± 26% interrupts.CPU0.CAL:Function_call_interrupts 916640 ± 68% -98.8% 11136 ± 28% interrupts.CPU0.TLB:TLB_shootdowns 1673791 ± 63% -99.2% 14087 ± 34% interrupts.CPU1.CAL:Function_call_interrupts 4570 ± 33% -49.8% 2295 ± 32% interrupts.CPU1.RES:Rescheduling_interrupts 1783806 ± 63% -99.3% 12954 ± 38% interrupts.CPU1.TLB:TLB_shootdowns 1056163 ± 65% -98.2% 19126 ± 63% interrupts.CPU10.CAL:Function_call_interrupts 1122817 ± 66% -98.4% 18041 ± 68% interrupts.CPU10.TLB:TLB_shootdowns 396.00 ± 44% +2610.9% 10735 ± 63% interrupts.CPU100.NMI:Non-maskable_interrupts 396.00 ± 44% +2610.9% 10735 ± 63% interrupts.CPU100.PMI:Performance_monitoring_interrupts 451819 ± 72% -98.1% 8554 ± 13% interrupts.CPU101.CAL:Function_call_interrupts 480717 ± 73% -98.5% 7315 ± 15% interrupts.CPU101.TLB:TLB_shootdowns 265399 ± 58% -97.9% 5696 ± 29% interrupts.CPU102.CAL:Function_call_interrupts 289.75 ± 28% +2381.2% 7189 ± 70% interrupts.CPU102.NMI:Non-maskable_interrupts 289.75 ± 28% +2381.2% 7189 ± 70% interrupts.CPU102.PMI:Performance_monitoring_interrupts 280959 ± 58% -98.4% 4447 ± 37% interrupts.CPU102.TLB:TLB_shootdowns 395.00 ± 14% +1306.2% 5554 ± 66% interrupts.CPU103.NMI:Non-maskable_interrupts 395.00 ± 14% +1306.2% 5554 ± 66% interrupts.CPU103.PMI:Performance_monitoring_interrupts 261286 ± 48% -98.7% 3280 ± 19% interrupts.CPU105.CAL:Function_call_interrupts 278056 ± 49% -99.3% 1967 ± 25% interrupts.CPU105.TLB:TLB_shootdowns 470198 ± 74% -98.5% 7167 ± 39% interrupts.CPU106.CAL:Function_call_interrupts 339.50 ± 27% +2637.9% 9295 ± 67% interrupts.CPU106.NMI:Non-maskable_interrupts 339.50 ± 27% +2637.9% 9295 ± 67% interrupts.CPU106.PMI:Performance_monitoring_interrupts 506596 ± 74% -98.8% 5949 ± 49% interrupts.CPU106.TLB:TLB_shootdowns 915159 ± 61% -99.0% 9525 ± 30% interrupts.CPU107.CAL:Function_call_interrupts 973233 ± 61% -99.1% 8357 ± 34% interrupts.CPU107.TLB:TLB_shootdowns 337201 ± 97% -94.8% 17569 ± 20% interrupts.CPU108.CAL:Function_call_interrupts 488.00 ± 51% +4125.8% 20621 ± 76% interrupts.CPU108.NMI:Non-maskable_interrupts 488.00 ± 51% +4125.8% 20621 ± 76% interrupts.CPU108.PMI:Performance_monitoring_interrupts 357853 ± 98% -95.4% 16619 ± 21% interrupts.CPU108.TLB:TLB_shootdowns 769.00 ± 13% +102.0% 1553 ± 38% interrupts.CPU109.RES:Rescheduling_interrupts 1370570 ± 62% -99.0% 14377 ± 50% interrupts.CPU11.CAL:Function_call_interrupts 1023 ± 66% +261.4% 3696 ± 53% interrupts.CPU11.NMI:Non-maskable_interrupts 1023 ± 66% +261.4% 3696 ± 53% interrupts.CPU11.PMI:Performance_monitoring_interrupts 1452507 ± 62% -99.1% 13360 ± 55% interrupts.CPU11.TLB:TLB_shootdowns 202682 ± 64% -92.8% 14671 ± 47% interrupts.CPU110.CAL:Function_call_interrupts 214672 ± 66% -93.7% 13608 ± 51% interrupts.CPU110.TLB:TLB_shootdowns 220346 ± 67% -94.5% 12066 ± 67% interrupts.CPU111.CAL:Function_call_interrupts 231609 ± 67% -95.3% 10836 ± 75% interrupts.CPU111.TLB:TLB_shootdowns 360536 ± 7% -96.9% 11207 ± 50% interrupts.CPU113.CAL:Function_call_interrupts 388026 ± 8% -97.4% 10044 ± 59% interrupts.CPU113.TLB:TLB_shootdowns 226426 ± 94% -95.6% 10070 ± 20% interrupts.CPU114.CAL:Function_call_interrupts 241420 ± 96% -96.3% 8853 ± 23% interrupts.CPU114.TLB:TLB_shootdowns 191959 ± 74% -94.2% 11192 ± 45% interrupts.CPU115.CAL:Function_call_interrupts 719.00 ± 41% +48.5% 1068 ± 12% interrupts.CPU115.RES:Rescheduling_interrupts 201336 ± 76% -95.0% 10008 ± 52% interrupts.CPU115.TLB:TLB_shootdowns 667.50 ± 13% +62.6% 1085 ± 26% interrupts.CPU117.RES:Rescheduling_interrupts 133001 ± 76% -88.1% 15791 ± 53% interrupts.CPU118.CAL:Function_call_interrupts 139621 ± 78% -89.4% 14864 ± 60% interrupts.CPU118.TLB:TLB_shootdowns 225506 ± 83% -94.5% 12418 ± 44% interrupts.CPU119.CAL:Function_call_interrupts 237295 ± 84% -95.3% 11205 ± 50% interrupts.CPU119.TLB:TLB_shootdowns 795307 ± 33% -98.3% 13685 ± 61% interrupts.CPU12.CAL:Function_call_interrupts 839259 ± 34% -98.5% 12571 ± 66% interrupts.CPU12.TLB:TLB_shootdowns 121059 ± 51% -91.6% 10168 ± 75% interrupts.CPU120.CAL:Function_call_interrupts 126869 ± 52% -92.9% 8976 ± 87% interrupts.CPU120.TLB:TLB_shootdowns 63281 ± 91% -86.7% 8391 ± 58% interrupts.CPU121.CAL:Function_call_interrupts 65227 ± 94% -88.7% 7352 ± 72% interrupts.CPU121.TLB:TLB_shootdowns 139835 ± 49% -90.7% 12954 ± 40% interrupts.CPU122.CAL:Function_call_interrupts 772.75 ± 33% +1116.2% 9398 ± 64% interrupts.CPU122.NMI:Non-maskable_interrupts 772.75 ± 33% +1116.2% 9398 ± 64% interrupts.CPU122.PMI:Performance_monitoring_interrupts 718.50 ± 15% +73.9% 1249 ± 16% interrupts.CPU122.RES:Rescheduling_interrupts 147558 ± 50% -92.0% 11827 ± 44% interrupts.CPU122.TLB:TLB_shootdowns 497.75 ± 22% +122.3% 1106 ± 10% interrupts.CPU123.RES:Rescheduling_interrupts 149678 ± 82% -94.6% 8147 ± 46% interrupts.CPU124.CAL:Function_call_interrupts 410.50 ± 46% +1774.7% 7695 ± 66% interrupts.CPU124.NMI:Non-maskable_interrupts 410.50 ± 46% +1774.7% 7695 ± 66% interrupts.CPU124.PMI:Performance_monitoring_interrupts 158193 ± 83% -95.6% 6929 ± 57% interrupts.CPU124.TLB:TLB_shootdowns 379553 ± 39% -95.9% 15621 ± 38% interrupts.CPU125.CAL:Function_call_interrupts 402724 ± 39% -96.4% 14560 ± 42% interrupts.CPU125.TLB:TLB_shootdowns 286100 ± 53% -95.8% 12114 ± 10% interrupts.CPU126.CAL:Function_call_interrupts 303685 ± 54% -96.4% 10959 ± 13% interrupts.CPU126.TLB:TLB_shootdowns 199563 ± 51% -94.0% 11896 ± 27% interrupts.CPU127.CAL:Function_call_interrupts 532.75 ± 61% +1691.0% 9541 ± 89% interrupts.CPU127.NMI:Non-maskable_interrupts 532.75 ± 61% +1691.0% 9541 ± 89% interrupts.CPU127.PMI:Performance_monitoring_interrupts 209264 ± 52% -94.9% 10740 ± 31% interrupts.CPU127.TLB:TLB_shootdowns 317414 ± 95% -97.3% 8414 ± 54% interrupts.CPU128.CAL:Function_call_interrupts 351005 ± 98% -97.9% 7203 ± 64% interrupts.CPU128.TLB:TLB_shootdowns 99448 ± 59% -89.8% 10118 ± 49% interrupts.CPU129.CAL:Function_call_interrupts 102781 ± 60% -91.3% 8937 ± 58% interrupts.CPU129.TLB:TLB_shootdowns 772971 ± 64% -98.4% 12140 ± 58% interrupts.CPU13.CAL:Function_call_interrupts 819963 ± 65% -98.7% 11029 ± 64% interrupts.CPU13.TLB:TLB_shootdowns 64152 ± 71% -86.6% 8594 ± 43% interrupts.CPU130.CAL:Function_call_interrupts 508.00 ± 48% +97.4% 1003 ± 33% interrupts.CPU130.RES:Rescheduling_interrupts 65483 ± 74% -88.7% 7372 ± 50% interrupts.CPU130.TLB:TLB_shootdowns 72850 ± 54% -89.1% 7964 ± 43% interrupts.CPU131.CAL:Function_call_interrupts 550.25 ± 39% +55.0% 853.00 ± 11% interrupts.CPU131.RES:Rescheduling_interrupts 74903 ± 56% -90.9% 6779 ± 52% interrupts.CPU131.TLB:TLB_shootdowns 261747 ± 52% -96.2% 9923 ± 45% interrupts.CPU132.CAL:Function_call_interrupts 295.50 ± 25% +3663.9% 11122 ± 85% interrupts.CPU132.NMI:Non-maskable_interrupts 295.50 ± 25% +3663.9% 11122 ± 85% interrupts.CPU132.PMI:Performance_monitoring_interrupts 277227 ± 52% -96.9% 8699 ± 52% interrupts.CPU132.TLB:TLB_shootdowns 286476 ± 69% -96.7% 9388 ± 55% interrupts.CPU134.CAL:Function_call_interrupts 303867 ± 70% -97.3% 8160 ± 66% interrupts.CPU134.TLB:TLB_shootdowns 145156 ±108% -94.3% 8338 ± 53% interrupts.CPU135.CAL:Function_call_interrupts 354.25 ± 48% +1738.2% 6511 ± 74% interrupts.CPU135.NMI:Non-maskable_interrupts 354.25 ± 48% +1738.2% 6511 ± 74% interrupts.CPU135.PMI:Performance_monitoring_interrupts 151225 ±109% -95.3% 7148 ± 64% interrupts.CPU135.TLB:TLB_shootdowns 392.25 ± 44% +3897.0% 15678 ± 70% interrupts.CPU137.NMI:Non-maskable_interrupts 392.25 ± 44% +3897.0% 15678 ± 70% interrupts.CPU137.PMI:Performance_monitoring_interrupts 199456 ± 45% -97.1% 5883 ± 65% interrupts.CPU138.CAL:Function_call_interrupts 217728 ± 47% -97.9% 4647 ± 86% interrupts.CPU138.TLB:TLB_shootdowns 294075 ±110% -96.5% 10197 ± 81% interrupts.CPU139.CAL:Function_call_interrupts 310749 ±111% -97.1% 9084 ± 95% interrupts.CPU139.TLB:TLB_shootdowns 523.00 ± 79% -68.1% 166.67 ± 9% interrupts.CPU14.59:PCI-MSI.1572878-edge.eth0-TxRx-14 745929 ± 67% -98.4% 12023 ± 58% interrupts.CPU14.CAL:Function_call_interrupts 2028 ± 45% -43.5% 1146 ± 22% interrupts.CPU14.RES:Rescheduling_interrupts 790069 ± 68% -98.6% 10856 ± 65% interrupts.CPU14.TLB:TLB_shootdowns 248766 ±108% -96.7% 8211 ± 51% interrupts.CPU141.CAL:Function_call_interrupts 260228 ±108% -97.3% 6995 ± 62% interrupts.CPU141.TLB:TLB_shootdowns 476.00 ± 40% +104.3% 972.33 ± 13% interrupts.CPU142.RES:Rescheduling_interrupts 586208 ± 42% -97.8% 12830 ± 72% interrupts.CPU143.CAL:Function_call_interrupts 623556 ± 42% -98.1% 11740 ± 81% interrupts.CPU143.TLB:TLB_shootdowns 992404 ± 57% -98.7% 12737 ± 38% interrupts.CPU15.CAL:Function_call_interrupts 1052603 ± 58% -98.9% 11616 ± 42% interrupts.CPU15.TLB:TLB_shootdowns 1677250 ± 59% -99.1% 14619 ± 54% interrupts.CPU16.CAL:Function_call_interrupts 1778293 ± 59% -99.2% 13492 ± 59% interrupts.CPU16.TLB:TLB_shootdowns 1223311 ± 61% -98.8% 15177 ± 51% interrupts.CPU17.CAL:Function_call_interrupts 2879 ± 36% -47.7% 1505 ± 14% interrupts.CPU17.RES:Rescheduling_interrupts 1295129 ± 62% -98.9% 14135 ± 55% interrupts.CPU17.TLB:TLB_shootdowns 1564988 ± 28% -99.3% 10674 ± 18% interrupts.CPU18.CAL:Function_call_interrupts 562.50 ± 72% +1407.6% 8480 ± 54% interrupts.CPU18.NMI:Non-maskable_interrupts 562.50 ± 72% +1407.6% 8480 ± 54% interrupts.CPU18.PMI:Performance_monitoring_interrupts 3001 ± 21% -60.0% 1199 ± 23% interrupts.CPU18.RES:Rescheduling_interrupts 1674735 ± 28% -99.4% 9500 ± 19% interrupts.CPU18.TLB:TLB_shootdowns 1230199 ± 53% -99.0% 12270 ± 20% interrupts.CPU19.CAL:Function_call_interrupts 2384 ± 37% -58.6% 986.67 ± 16% interrupts.CPU19.RES:Rescheduling_interrupts 1315132 ± 53% -99.2% 11145 ± 21% interrupts.CPU19.TLB:TLB_shootdowns 1239868 ± 52% -98.7% 15566 ± 3% interrupts.CPU2.CAL:Function_call_interrupts 1307900 ± 52% -98.9% 14479 ± 3% interrupts.CPU2.TLB:TLB_shootdowns 1174374 ± 27% -98.9% 13198 ± 22% interrupts.CPU20.CAL:Function_call_interrupts 2291 ± 23% -55.6% 1017 ± 14% interrupts.CPU20.RES:Rescheduling_interrupts 1261995 ± 28% -99.0% 12040 ± 24% interrupts.CPU20.TLB:TLB_shootdowns 1163562 ± 72% -99.0% 12151 ± 17% interrupts.CPU21.CAL:Function_call_interrupts 1242814 ± 72% -99.1% 11040 ± 20% interrupts.CPU21.TLB:TLB_shootdowns 935698 ± 52% -98.7% 11798 ± 20% interrupts.CPU22.CAL:Function_call_interrupts 2077 ± 16% -39.0% 1268 ± 5% interrupts.CPU22.RES:Rescheduling_interrupts 997742 ± 53% -98.9% 10643 ± 23% interrupts.CPU22.TLB:TLB_shootdowns 962574 ± 41% -99.1% 9130 ± 28% interrupts.CPU23.CAL:Function_call_interrupts 2487 ± 25% -55.7% 1103 ± 26% interrupts.CPU23.RES:Rescheduling_interrupts 1031301 ± 41% -99.2% 7958 ± 35% interrupts.CPU23.TLB:TLB_shootdowns 983055 ± 34% -99.1% 9145 ± 17% interrupts.CPU24.CAL:Function_call_interrupts 2020 ± 8% -43.4% 1143 ± 14% interrupts.CPU24.RES:Rescheduling_interrupts 1055794 ± 34% -99.2% 7933 ± 20% interrupts.CPU24.TLB:TLB_shootdowns 712727 ± 59% -98.6% 9881 ± 30% interrupts.CPU25.CAL:Function_call_interrupts 762308 ± 59% -98.9% 8687 ± 35% interrupts.CPU25.TLB:TLB_shootdowns 1088593 ± 25% -99.3% 7636 ± 10% interrupts.CPU26.CAL:Function_call_interrupts 2177 ± 28% -54.9% 982.33 ± 15% interrupts.CPU26.RES:Rescheduling_interrupts 1160016 ± 25% -99.4% 6583 ± 16% interrupts.CPU26.TLB:TLB_shootdowns 1035163 ± 32% -99.0% 9871 ± 36% interrupts.CPU27.CAL:Function_call_interrupts 2313 ± 26% -57.4% 986.33 ± 28% interrupts.CPU27.RES:Rescheduling_interrupts 1107088 ± 31% -99.2% 8640 ± 43% interrupts.CPU27.TLB:TLB_shootdowns 897104 ± 51% -99.2% 6742 ± 21% interrupts.CPU28.CAL:Function_call_interrupts 907.00 ± 61% +309.7% 3716 ± 48% interrupts.CPU28.NMI:Non-maskable_interrupts 907.00 ± 61% +309.7% 3716 ± 48% interrupts.CPU28.PMI:Performance_monitoring_interrupts 959438 ± 51% -99.4% 5514 ± 27% interrupts.CPU28.TLB:TLB_shootdowns 900084 ± 27% -99.4% 5596 ± 24% interrupts.CPU29.CAL:Function_call_interrupts 1976 ± 24% -56.8% 853.33 ± 32% interrupts.CPU29.RES:Rescheduling_interrupts 959656 ± 26% -99.5% 4341 ± 35% interrupts.CPU29.TLB:TLB_shootdowns 1194729 ± 56% -98.5% 17430 ± 39% interrupts.CPU3.CAL:Function_call_interrupts 525.75 ± 85% +556.6% 3452 ± 47% interrupts.CPU3.NMI:Non-maskable_interrupts 525.75 ± 85% +556.6% 3452 ± 47% interrupts.CPU3.PMI:Performance_monitoring_interrupts 1261693 ± 57% -98.7% 16395 ± 42% interrupts.CPU3.TLB:TLB_shootdowns 787739 ± 33% -99.1% 6919 ± 20% interrupts.CPU30.CAL:Function_call_interrupts 849132 ± 33% -99.3% 5737 ± 24% interrupts.CPU30.TLB:TLB_shootdowns 1135046 ± 55% -99.4% 6855 ± 30% interrupts.CPU31.CAL:Function_call_interrupts 1211676 ± 55% -99.5% 5602 ± 39% interrupts.CPU31.TLB:TLB_shootdowns 972935 ± 43% -99.0% 9546 ± 27% interrupts.CPU32.CAL:Function_call_interrupts 1818 ± 23% -44.3% 1012 ± 15% interrupts.CPU32.RES:Rescheduling_interrupts 1051259 ± 44% -99.2% 8453 ± 33% interrupts.CPU32.TLB:TLB_shootdowns 923884 ± 71% -99.0% 9111 ± 54% interrupts.CPU33.CAL:Function_call_interrupts 986655 ± 71% -99.2% 7952 ± 63% interrupts.CPU33.TLB:TLB_shootdowns 829706 ± 59% -99.0% 8204 ± 30% interrupts.CPU34.CAL:Function_call_interrupts 883394 ± 59% -99.2% 7222 ± 34% interrupts.CPU34.TLB:TLB_shootdowns 1545345 ± 48% -99.3% 11243 ± 28% interrupts.CPU35.CAL:Function_call_interrupts 2611 ± 32% -41.4% 1530 ± 31% interrupts.CPU35.RES:Rescheduling_interrupts 1648616 ± 48% -99.4% 10099 ± 31% interrupts.CPU35.TLB:TLB_shootdowns 743486 ± 25% -96.9% 23407 ± 22% interrupts.CPU36.CAL:Function_call_interrupts 709.75 ± 79% +1988.3% 14822 ± 84% interrupts.CPU36.NMI:Non-maskable_interrupts 709.75 ± 79% +1988.3% 14822 ± 84% interrupts.CPU36.PMI:Performance_monitoring_interrupts 1670 ± 21% -24.5% 1262 ± 23% interrupts.CPU36.RES:Rescheduling_interrupts 793538 ± 25% -97.2% 22365 ± 23% interrupts.CPU36.TLB:TLB_shootdowns 512017 ± 43% -98.3% 8886 ± 40% interrupts.CPU37.CAL:Function_call_interrupts 694.00 ± 31% +618.8% 4988 ± 77% interrupts.CPU37.NMI:Non-maskable_interrupts 694.00 ± 31% +618.8% 4988 ± 77% interrupts.CPU37.PMI:Performance_monitoring_interrupts 1553 ± 19% -29.2% 1099 ± 28% interrupts.CPU37.RES:Rescheduling_interrupts 542917 ± 44% -98.6% 7663 ± 48% interrupts.CPU37.TLB:TLB_shootdowns 410157 ± 52% -96.4% 14951 ± 53% interrupts.CPU38.CAL:Function_call_interrupts 433249 ± 52% -96.8% 13784 ± 57% interrupts.CPU38.TLB:TLB_shootdowns 326913 ± 19% -96.2% 12392 ± 30% interrupts.CPU39.CAL:Function_call_interrupts 1361 ± 13% -31.8% 928.67 ± 20% interrupts.CPU39.RES:Rescheduling_interrupts 344502 ± 19% -96.7% 11332 ± 34% interrupts.CPU39.TLB:TLB_shootdowns 961062 ± 66% -98.2% 17399 ± 31% interrupts.CPU4.CAL:Function_call_interrupts 1018054 ± 66% -98.4% 16319 ± 34% interrupts.CPU4.TLB:TLB_shootdowns 811854 ± 11% -98.9% 9229 ± 24% interrupts.CPU40.CAL:Function_call_interrupts 2403 ± 25% -53.2% 1125 ± 27% interrupts.CPU40.RES:Rescheduling_interrupts 867263 ± 10% -99.1% 8044 ± 29% interrupts.CPU40.TLB:TLB_shootdowns 287216 ± 45% -96.1% 11286 ± 60% interrupts.CPU41.CAL:Function_call_interrupts 309210 ± 46% -96.7% 10138 ± 69% interrupts.CPU41.TLB:TLB_shootdowns 470893 ± 55% -96.3% 17210 ± 58% interrupts.CPU42.CAL:Function_call_interrupts 503992 ± 55% -96.8% 16080 ± 61% interrupts.CPU42.TLB:TLB_shootdowns 536532 ± 33% -97.5% 13371 ± 56% interrupts.CPU43.CAL:Function_call_interrupts 570162 ± 33% -97.8% 12291 ± 62% interrupts.CPU43.TLB:TLB_shootdowns 383745 ± 84% -97.3% 10222 ± 73% interrupts.CPU44.CAL:Function_call_interrupts 408337 ± 84% -97.7% 9202 ± 83% interrupts.CPU44.TLB:TLB_shootdowns 397985 ± 42% -96.8% 12896 ± 42% interrupts.CPU45.CAL:Function_call_interrupts 353.00 ± 31% +2152.7% 7952 ± 84% interrupts.CPU45.NMI:Non-maskable_interrupts 353.00 ± 31% +2152.7% 7952 ± 84% interrupts.CPU45.PMI:Performance_monitoring_interrupts 420434 ± 41% -97.2% 11900 ± 48% interrupts.CPU45.TLB:TLB_shootdowns 379477 ± 95% -95.4% 17413 ± 71% interrupts.CPU46.CAL:Function_call_interrupts 401864 ± 95% -95.9% 16430 ± 77% interrupts.CPU46.TLB:TLB_shootdowns 464073 ± 58% -98.0% 9210 ± 34% interrupts.CPU47.CAL:Function_call_interrupts 1484 ± 27% -36.9% 936.00 ± 16% interrupts.CPU47.RES:Rescheduling_interrupts 491939 ± 59% -98.4% 8033 ± 41% interrupts.CPU47.TLB:TLB_shootdowns 531744 ± 47% -97.0% 16180 ± 75% interrupts.CPU48.CAL:Function_call_interrupts 1563 ± 19% -43.5% 884.00 ± 18% interrupts.CPU48.RES:Rescheduling_interrupts 564429 ± 48% -97.3% 15078 ± 82% interrupts.CPU48.TLB:TLB_shootdowns 322097 ± 71% -95.7% 13797 ± 22% interrupts.CPU49.CAL:Function_call_interrupts 340460 ± 71% -96.3% 12681 ± 25% interrupts.CPU49.TLB:TLB_shootdowns 943457 ± 71% -98.2% 16513 ± 58% interrupts.CPU5.CAL:Function_call_interrupts 933.25 ± 69% +591.6% 6454 ± 51% interrupts.CPU5.NMI:Non-maskable_interrupts 933.25 ± 69% +591.6% 6454 ± 51% interrupts.CPU5.PMI:Performance_monitoring_interrupts 3127 ± 38% -53.3% 1461 ± 28% interrupts.CPU5.RES:Rescheduling_interrupts 1001711 ± 72% -98.5% 15424 ± 63% interrupts.CPU5.TLB:TLB_shootdowns 362917 ± 75% -96.1% 14110 ± 36% interrupts.CPU50.CAL:Function_call_interrupts 815.00 ± 71% +1328.5% 11642 ± 59% interrupts.CPU50.NMI:Non-maskable_interrupts 815.00 ± 71% +1328.5% 11642 ± 59% interrupts.CPU50.PMI:Performance_monitoring_interrupts 384929 ± 75% -96.5% 13325 ± 43% interrupts.CPU50.TLB:TLB_shootdowns 445928 ± 42% -97.0% 13267 ± 37% interrupts.CPU51.CAL:Function_call_interrupts 309.00 ± 39% +2831.8% 9059 ± 66% interrupts.CPU51.NMI:Non-maskable_interrupts 309.00 ± 39% +2831.8% 9059 ± 66% interrupts.CPU51.PMI:Performance_monitoring_interrupts 1836 ± 60% -43.7% 1033 ± 28% interrupts.CPU51.RES:Rescheduling_interrupts 474465 ± 42% -97.3% 12603 ± 42% interrupts.CPU51.TLB:TLB_shootdowns 399958 ± 46% -96.5% 14160 ± 49% interrupts.CPU52.CAL:Function_call_interrupts 422881 ± 46% -96.9% 13077 ± 55% interrupts.CPU52.TLB:TLB_shootdowns 689403 ± 46% -98.5% 10119 ± 47% interrupts.CPU53.CAL:Function_call_interrupts 735469 ± 47% -98.8% 8963 ± 55% interrupts.CPU53.TLB:TLB_shootdowns 617414 ± 38% -97.9% 12768 ± 28% interrupts.CPU54.CAL:Function_call_interrupts 667.25 ± 46% +1629.6% 11541 ± 62% interrupts.CPU54.NMI:Non-maskable_interrupts 667.25 ± 46% +1629.6% 11541 ± 62% interrupts.CPU54.PMI:Performance_monitoring_interrupts 658336 ± 39% -98.2% 11672 ± 32% interrupts.CPU54.TLB:TLB_shootdowns 546653 ± 89% -97.5% 13738 ± 13% interrupts.CPU55.CAL:Function_call_interrupts 774.75 ± 41% +721.2% 6362 ± 94% interrupts.CPU55.NMI:Non-maskable_interrupts 774.75 ± 41% +721.2% 6362 ± 94% interrupts.CPU55.PMI:Performance_monitoring_interrupts 580070 ± 90% -97.8% 12580 ± 15% interrupts.CPU55.TLB:TLB_shootdowns 409611 ± 49% -97.3% 11007 ± 49% interrupts.CPU56.CAL:Function_call_interrupts 1345 ± 20% -20.8% 1066 ± 26% interrupts.CPU56.RES:Rescheduling_interrupts 432727 ± 49% -97.7% 9815 ± 57% interrupts.CPU56.TLB:TLB_shootdowns 351492 ± 68% -97.6% 8567 ± 63% interrupts.CPU57.CAL:Function_call_interrupts 372086 ± 68% -98.0% 7404 ± 75% interrupts.CPU57.TLB:TLB_shootdowns 331546 ± 52% -97.3% 8874 ± 43% interrupts.CPU58.CAL:Function_call_interrupts 394.25 ± 47% +2800.7% 11436 ± 86% interrupts.CPU58.NMI:Non-maskable_interrupts 394.25 ± 47% +2800.7% 11436 ± 86% interrupts.CPU58.PMI:Performance_monitoring_interrupts 352515 ± 52% -97.8% 7710 ± 50% interrupts.CPU58.TLB:TLB_shootdowns 1017645 ± 49% -98.6% 14448 ± 45% interrupts.CPU6.CAL:Function_call_interrupts 1077506 ± 49% -98.8% 13383 ± 49% interrupts.CPU6.TLB:TLB_shootdowns 447523 ± 42% -97.6% 10874 ± 32% interrupts.CPU60.CAL:Function_call_interrupts 474946 ± 42% -98.0% 9696 ± 37% interrupts.CPU60.TLB:TLB_shootdowns 195026 ± 43% -95.2% 9316 ± 84% interrupts.CPU61.CAL:Function_call_interrupts 204326 ± 44% -96.0% 8100 ±100% interrupts.CPU61.TLB:TLB_shootdowns 506257 ± 37% -97.6% 12160 ± 52% interrupts.CPU62.CAL:Function_call_interrupts 536017 ± 38% -98.0% 10978 ± 60% interrupts.CPU62.TLB:TLB_shootdowns 284965 ± 31% -96.6% 9596 ± 58% interrupts.CPU63.CAL:Function_call_interrupts 302145 ± 32% -97.2% 8379 ± 68% interrupts.CPU63.TLB:TLB_shootdowns 479734 ± 54% -98.1% 9072 ± 56% interrupts.CPU64.CAL:Function_call_interrupts 508462 ± 54% -98.4% 7910 ± 67% interrupts.CPU64.TLB:TLB_shootdowns 579804 ± 47% -98.5% 8486 ± 16% interrupts.CPU65.CAL:Function_call_interrupts 386.25 ± 42% +3442.4% 13682 ±105% interrupts.CPU65.NMI:Non-maskable_interrupts 386.25 ± 42% +3442.4% 13682 ±105% interrupts.CPU65.PMI:Performance_monitoring_interrupts 1275 ± 21% -30.7% 884.00 ± 7% interrupts.CPU65.RES:Rescheduling_interrupts 618622 ± 48% -98.8% 7303 ± 22% interrupts.CPU65.TLB:TLB_shootdowns 591246 ± 51% -98.6% 8565 ± 51% interrupts.CPU66.CAL:Function_call_interrupts 629018 ± 52% -98.8% 7334 ± 62% interrupts.CPU66.TLB:TLB_shootdowns 662137 ± 52% -98.5% 10092 ± 45% interrupts.CPU67.CAL:Function_call_interrupts 1641 ± 7% -37.6% 1024 ± 27% interrupts.CPU67.RES:Rescheduling_interrupts 703434 ± 52% -98.7% 8912 ± 54% interrupts.CPU67.TLB:TLB_shootdowns 297911 ± 33% -96.5% 10478 ± 61% interrupts.CPU68.CAL:Function_call_interrupts 377.00 ± 62% +5955.2% 22828 ± 94% interrupts.CPU68.NMI:Non-maskable_interrupts 377.00 ± 62% +5955.2% 22828 ± 94% interrupts.CPU68.PMI:Performance_monitoring_interrupts 314898 ± 33% -97.0% 9293 ± 71% interrupts.CPU68.TLB:TLB_shootdowns 426139 ± 42% -97.7% 9808 ± 33% interrupts.CPU69.CAL:Function_call_interrupts 450916 ± 43% -98.1% 8712 ± 38% interrupts.CPU69.TLB:TLB_shootdowns 1227894 ± 28% -98.6% 16727 ± 33% interrupts.CPU7.CAL:Function_call_interrupts 2755 ± 22% -41.5% 1611 ± 21% interrupts.CPU7.RES:Rescheduling_interrupts 1301072 ± 29% -98.8% 15683 ± 35% interrupts.CPU7.TLB:TLB_shootdowns 363690 ± 39% -97.4% 9440 ± 48% interrupts.CPU70.CAL:Function_call_interrupts 386135 ± 40% -97.9% 8253 ± 57% interrupts.CPU70.TLB:TLB_shootdowns 406370 ± 30% -97.6% 9598 ± 52% interrupts.CPU71.CAL:Function_call_interrupts 1555 ± 16% +56.5% 2433 ± 22% interrupts.CPU71.RES:Rescheduling_interrupts 431220 ± 31% -98.0% 8419 ± 62% interrupts.CPU71.TLB:TLB_shootdowns 379521 ± 43% -96.1% 14799 ± 56% interrupts.CPU72.CAL:Function_call_interrupts 401269 ± 43% -96.6% 13719 ± 61% interrupts.CPU72.TLB:TLB_shootdowns 853406 ± 53% -98.5% 13118 ± 56% interrupts.CPU73.CAL:Function_call_interrupts 904450 ± 53% -98.7% 12014 ± 62% interrupts.CPU73.TLB:TLB_shootdowns 929431 ± 17% -98.4% 15108 ± 49% interrupts.CPU74.CAL:Function_call_interrupts 1953 ± 24% -38.2% 1208 ± 15% interrupts.CPU74.RES:Rescheduling_interrupts 980216 ± 17% -98.6% 14050 ± 54% interrupts.CPU74.TLB:TLB_shootdowns 378639 ± 55% -96.6% 12982 ± 41% interrupts.CPU75.CAL:Function_call_interrupts 761.50 ± 67% +555.7% 4993 ± 68% interrupts.CPU75.NMI:Non-maskable_interrupts 761.50 ± 67% +555.7% 4993 ± 68% interrupts.CPU75.PMI:Performance_monitoring_interrupts 396850 ± 55% -97.0% 11861 ± 45% interrupts.CPU75.TLB:TLB_shootdowns 458962 ± 58% -97.5% 11531 ± 32% interrupts.CPU76.CAL:Function_call_interrupts 485627 ± 59% -97.9% 10373 ± 35% interrupts.CPU76.TLB:TLB_shootdowns 281173 ± 82% -93.1% 19292 ± 37% interrupts.CPU77.CAL:Function_call_interrupts 861.25 ± 70% +625.0% 6244 ± 54% interrupts.CPU77.NMI:Non-maskable_interrupts 861.25 ± 70% +625.0% 6244 ± 54% interrupts.CPU77.PMI:Performance_monitoring_interrupts 297326 ± 83% -93.8% 18476 ± 39% interrupts.CPU77.TLB:TLB_shootdowns 640811 ± 43% -98.5% 9471 ± 30% interrupts.CPU78.CAL:Function_call_interrupts 678615 ± 43% -98.8% 8256 ± 34% interrupts.CPU78.TLB:TLB_shootdowns 688480 ± 26% -98.3% 11407 ± 36% interrupts.CPU79.CAL:Function_call_interrupts 1512 ± 15% -25.0% 1133 ± 19% interrupts.CPU79.RES:Rescheduling_interrupts 727783 ± 26% -98.6% 10309 ± 39% interrupts.CPU79.TLB:TLB_shootdowns 1027219 ± 72% -98.6% 14687 ± 56% interrupts.CPU8.CAL:Function_call_interrupts 545.50 ± 47% +763.6% 4711 ± 74% interrupts.CPU8.NMI:Non-maskable_interrupts 545.50 ± 47% +763.6% 4711 ± 74% interrupts.CPU8.PMI:Performance_monitoring_interrupts 1088136 ± 73% -98.8% 13533 ± 61% interrupts.CPU8.TLB:TLB_shootdowns 567067 ± 89% -97.9% 12111 ± 26% interrupts.CPU80.CAL:Function_call_interrupts 463.75 ± 76% +929.3% 4773 ± 71% interrupts.CPU80.NMI:Non-maskable_interrupts 463.75 ± 76% +929.3% 4773 ± 71% interrupts.CPU80.PMI:Performance_monitoring_interrupts 600387 ± 90% -98.2% 10968 ± 28% interrupts.CPU80.TLB:TLB_shootdowns 499954 ± 43% -97.2% 13756 ± 12% interrupts.CPU81.CAL:Function_call_interrupts 656.50 ± 34% +724.5% 5412 ± 76% interrupts.CPU81.NMI:Non-maskable_interrupts 656.50 ± 34% +724.5% 5412 ± 76% interrupts.CPU81.PMI:Performance_monitoring_interrupts 529479 ± 43% -97.6% 12630 ± 14% interrupts.CPU81.TLB:TLB_shootdowns 443999 ± 80% -97.8% 9792 ± 44% interrupts.CPU82.CAL:Function_call_interrupts 468595 ± 81% -98.2% 8582 ± 50% interrupts.CPU82.TLB:TLB_shootdowns 615661 ± 65% -98.0% 12037 ± 32% interrupts.CPU83.CAL:Function_call_interrupts 650630 ± 65% -98.3% 10861 ± 35% interrupts.CPU83.TLB:TLB_shootdowns 317778 ± 63% -96.7% 10627 ± 9% interrupts.CPU84.CAL:Function_call_interrupts 334372 ± 64% -97.2% 9524 ± 10% interrupts.CPU84.TLB:TLB_shootdowns 546581 ± 63% -98.3% 9310 ± 40% interrupts.CPU85.CAL:Function_call_interrupts 1311 ± 28% -39.0% 799.67 ± 27% interrupts.CPU85.RES:Rescheduling_interrupts 574583 ± 63% -98.6% 8083 ± 46% interrupts.CPU85.TLB:TLB_shootdowns 419213 ± 51% -98.0% 8407 ± 31% interrupts.CPU86.CAL:Function_call_interrupts 443869 ± 51% -98.4% 7177 ± 36% interrupts.CPU86.TLB:TLB_shootdowns 531049 ± 63% -98.2% 9730 ± 47% interrupts.CPU87.CAL:Function_call_interrupts 563628 ± 63% -98.5% 8564 ± 54% interrupts.CPU87.TLB:TLB_shootdowns 907469 ± 55% -98.9% 9870 ± 48% interrupts.CPU88.CAL:Function_call_interrupts 954396 ± 55% -99.1% 8694 ± 55% interrupts.CPU88.TLB:TLB_shootdowns 627701 ± 48% -98.3% 10603 ± 47% interrupts.CPU89.CAL:Function_call_interrupts 665632 ± 48% -98.6% 9438 ± 52% interrupts.CPU89.TLB:TLB_shootdowns 603.75 ±105% -72.1% 168.33 ± 8% interrupts.CPU9.54:PCI-MSI.1572873-edge.eth0-TxRx-9 1395838 ± 39% -98.7% 18681 ± 60% interrupts.CPU9.CAL:Function_call_interrupts 763.75 ± 33% +661.8% 5818 ± 93% interrupts.CPU9.NMI:Non-maskable_interrupts 763.75 ± 33% +661.8% 5818 ± 93% interrupts.CPU9.PMI:Performance_monitoring_interrupts 3455 ± 55% -53.1% 1619 ± 39% interrupts.CPU9.RES:Rescheduling_interrupts 1480077 ± 39% -98.8% 17602 ± 64% interrupts.CPU9.TLB:TLB_shootdowns 854.25 ± 25% +925.6% 8761 ± 88% interrupts.CPU90.NMI:Non-maskable_interrupts 854.25 ± 25% +925.6% 8761 ± 88% interrupts.CPU90.PMI:Performance_monitoring_interrupts 545942 ± 51% -97.9% 11661 ± 31% interrupts.CPU91.CAL:Function_call_interrupts 580139 ± 51% -98.2% 10555 ± 33% interrupts.CPU91.TLB:TLB_shootdowns 291246 ± 40% -96.7% 9728 ± 44% interrupts.CPU92.CAL:Function_call_interrupts 308723 ± 41% -97.2% 8636 ± 53% interrupts.CPU92.TLB:TLB_shootdowns 573160 ±115% -98.0% 11689 ± 21% interrupts.CPU93.CAL:Function_call_interrupts 608315 ±116% -98.3% 10620 ± 23% interrupts.CPU93.TLB:TLB_shootdowns 464446 ± 69% -97.2% 12863 ± 53% interrupts.CPU94.CAL:Function_call_interrupts 493711 ± 70% -97.6% 11665 ± 58% interrupts.CPU94.TLB:TLB_shootdowns 401393 ± 68% -98.5% 5850 ± 21% interrupts.CPU95.CAL:Function_call_interrupts 427084 ± 69% -98.9% 4632 ± 29% interrupts.CPU95.TLB:TLB_shootdowns 486077 ± 57% -98.4% 7564 ± 19% interrupts.CPU97.CAL:Function_call_interrupts 605.25 ± 68% +2436.9% 15354 ± 72% interrupts.CPU97.NMI:Non-maskable_interrupts 605.25 ± 68% +2436.9% 15354 ± 72% interrupts.CPU97.PMI:Performance_monitoring_interrupts 515849 ± 58% -98.8% 6337 ± 24% interrupts.CPU97.TLB:TLB_shootdowns 490328 ± 68% -98.1% 9077 ± 39% interrupts.CPU98.CAL:Function_call_interrupts 523353 ± 68% -98.5% 7952 ± 45% interrupts.CPU98.TLB:TLB_shootdowns 425492 ± 43% -98.0% 8654 ± 33% interrupts.CPU99.CAL:Function_call_interrupts 451818 ± 43% -98.4% 7411 ± 39% interrupts.CPU99.TLB:TLB_shootdowns 87005273 -98.3% 1460829 ± 3% interrupts.TLB:TLB_shootdowns vm-scalability.time.user_time 700 +-+-------------------------------------------------------------------+ | | 600 +-+ .++.++ .+ | | +.++.+.++.++.+ + .+.++. .+.++.++. +.++. +.++.++.+.++ +.| 500 +-+ ++ ++ +.+ +.+ | | | 400 +-+ | | | 300 +-+ | | | 200 +-+ | | | 100 +-+ | O OO O O OO O OO O OO O OO OO O OO OO O O O | 0 +-+---O------O--O-----O----------------------O------------------------+ vm-scalability.time.system_time 1800 +-+------------------------------------------------------------------+ |.++.+ .++.+.++.++. +.+.++.++. +.+.++. +. +.++.+. +.++.+ .+.++.++.++.| 1600 +-+ + + + + + + + | 1400 +-+ | | | 1200 +-+ O | 1000 +-O O O O O O O O | | O OO O O O O O O | 800 O-+ O O O OO O O | 600 +-+ | | | 400 +-+ | 200 +-+ | | | 0 +-+---O------O--O-----O---------------------O------------------------+ vm-scalability.time.percent_of_cpu_this_job_got 800 +-+-------------------------------------------------------------------+ |.++.++.+.++.++.+.++.++.++.+.++.++.+.++.++.+.++.++.+.++.++.++.+.++.++.| 700 +-+ | 600 +-+ | | | 500 +-+ | | | 400 +-+ | | O O O OO O O OO O | 300 +-+O OO O O OO O OO O OO O O | 200 O-+ | | | 100 +-+ | | | 0 +-+---O------O--O-----O----------------------O------------------------+ vm-scalability.time.maximum_resident_set_size 5e+06 +-+---------------------------------------------------------------+ 4.5e+06 +-+ + | | :: | 4e+06 +-+ +. .+ +.+ .+ +. +. : :.+| 3.5e+06 +-++.+ + .++.++.++. .++.++ +.++.+ +.++.++ +.++.+ + ++ + | | + ++ | 3e+06 O-OO O OO O O OO O OO OO OO OO OO OO O OO | 2.5e+06 +-+ | 2e+06 +-+ | | | 1.5e+06 +-+ | 1e+06 +-+ | | | 500000 +-+ | 0 +-+---O-----O--O-----O--------------------O-----------------------+ vm-scalability.time.minor_page_faults 1e+08 +-+-----------------------------------------------------------------+ 9e+07 +-+ | | | 8e+07 +-+ | 7e+07 +-+ | | | 6e+07 +-+ | 5e+07 +-+ | 4e+07 +-+ | | | 3e+07 +-+ | 2e+07 +-+ | | | 1e+07 O-OO O OO O O O OO O OO OO OO O OO OO O OO | 0 +-+---O-----O--O------O--------------------O------------------------+ vm-scalability.time.involuntary_context_switches 450000 +-+----------------------------------------------------------------+ | +. +.++. +.++. +. .+ .+ .+ .+ +. +. .+ .++ +. ++.| 400000 +-+ + + ++.+ ++ + +.+ + +.++.+ + ++ + + : + + | 350000 +-+ + + | | | 300000 +-+ | 250000 +-+ | | | 200000 +-+ | 150000 +-+ | | | 100000 +-O O OO | 50000 O-+O O OO O O OO OO OO O OO OO OO O O | | | 0 +-+---O-----O--O-----O---------------------O-----------------------+ perf-node.node-stores 9e+08 +-+-----------------------------------------------------------------+ | + | 8e+08 +-+ + : | 7e+08 +-+ :: :: .+ | | + .+ :: : : + .+ +. .+ +. + +. + + + : | 6e+08 +-++ : : + .++.+.+ : + + : : + : + : + : + + : : +.+ : | | :.+ + :+ :.+ :+ +.+ : + : : +.| 5e+08 +-+ + + + + + ++ | | | 4e+08 +-+ | 3e+08 +-+ | | | 2e+08 +-+ O O O O O O OO O O | O OO O O OO OO OO O O O O O O OO | 1e+08 +-+------------O---------------------------O------------------------+ perf-node.node-store-misses 1.5e+09 +-+---------------------------------------------------------------+ 1.4e+09 +-++ .+ +.+ ++ +.+ ++ ++ | |+ + .+ + +. +. : : + + +. : : + + .++.+ + + | 1.3e+09 +-+ ++ : .+ + + +.++ ++.+ + +.++ ++ + +| 1.2e+09 +-+ +.++ | | | 1.1e+09 +-+ | 1e+09 +-+ | 9e+08 +-+ | | | 8e+08 +-+ O O O O O O | 7e+08 O-OO O OO OO O O O O OO O OO O O OO | | | 6e+08 +-+ O O O | 5e+08 +-+---------------------------------------O-----------------------+ vm-scalability.throughput 2e+06 +-+---------------------------------------------------------------+ |.++.++.+ .++.++.++. +.++.++.++.++.++.++.++.++.++.++.++.++.++.+ .+| 1.8e+06 +-+ + + + | 1.6e+06 +-+ | | | 1.4e+06 +-+ | 1.2e+06 +-+ | | | 1e+06 +-+ | 800000 +-+ | | | 600000 +-+ | 400000 +-+ | | | 200000 O-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO-OO--------------------+ vm-scalability.free_time 55 +-+--------------------------------------------------------------------+ 50 +-+ .+ .+ | | .+.++.+. + : +.+ + .+. + : | 45 +-+ .++ + :+ : .+.+ +.+ + +.+. +.+.++ + :.| 40 +-++ + +.++ +.+.+ +.++.+ ++.+ + | 35 +-+ | 30 +-+ | | | 25 +-+ | 20 +-+ | 15 +-+ | 10 +-+ | | O O OO O O | 5 O-+O OO O OO OO O OO OO O OO O OO OO O O O | 0 +-+--------------------------------------------------------------------+ vm-scalability.median 250000 +-+----------------------------------------------------------------+ |.++.++.++.++.++.++. +.++.++.++.+.++.++.++.++.++.++.++.++.++.++.++.| | + | 200000 +-+ | | | | | 150000 +-+ | | | 100000 +-+ | | | | | 50000 +-+ | O OO OO OO OO OO OO OO OO OO OO O OO OO OO OO O | | | 0 +-+----------------------------------------------------------------+ vm-scalability.workload 4.5e+08 +-+---------------------------------------------------------------+ | | 4e+08 +-+ | 3.5e+08 +-+ | | | 3e+08 +-+ | | | 2.5e+08 +-+ | | | 2e+08 +-+ | 1.5e+08 +-+ | | | 1e+08 +-+ | O OO OO OO OO OO OO OO OO OO OO OO OO OO OO OO | 5e+07 +-+---------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-7/performance/1SSD/16MB/xfs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-2019-09-23.cgz/NoSync/lkp-csl-2sp7/60G/fsmark/0x500002b commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- %stddev %change %stddev \ | \ 213530 ± 6% -17.1% 176946 ± 3% fsmark.app_overhead 141.28 ± 25% +1609.1% 2414 fsmark.files_per_sec 2828 -2.0% 2772 fsmark.time.maximum_resident_set_size 48.75 ± 2% -17.3% 40.33 fsmark.time.percent_of_cpu_this_job_got 66.37 ± 2% -17.6% 54.70 fsmark.time.system_time 36378 -95.4% 1678 ± 4% fsmark.time.voluntary_context_switches 19.21 -5.1% 18.24 ± 6% boot-time.dhcp 8.17 -7.2 0.93 mpstat.cpu.all.iowait% 4911670 ± 5% -100.0% 0.00 numa-numastat.node0.numa_foreign 4911670 ± 5% -100.0% 0.00 numa-numastat.node1.numa_miss 4911702 ± 5% -99.9% 6293 ±140% numa-numastat.node1.other_node 4810521 ± 5% -100.0% 0.00 numa-vmstat.node0.numa_foreign 4810600 ± 5% -100.0% 0.00 numa-vmstat.node1.numa_miss 4980554 ± 5% -96.5% 175777 ± 4% numa-vmstat.node1.numa_other 91.00 +7.7% 98.00 vmstat.cpu.id 8.00 -100.0% 0.00 vmstat.procs.b 2552 -22.4% 1980 vmstat.system.cs 7659590 ± 67% -87.6% 951609 ± 20% cpuidle.C1.time 70426 ± 38% -51.2% 34365 ± 7% cpuidle.C1.usage 2823007 ± 73% -57.4% 1203288 ± 10% cpuidle.POLL.time 22890 ± 13% -23.2% 17580 ± 3% cpuidle.POLL.usage 1396 ± 6% -6.8% 1302 ± 6% slabinfo.kmalloc-rcl-96.active_objs 1396 ± 6% -6.8% 1302 ± 6% slabinfo.kmalloc-rcl-96.num_objs 344.75 ± 14% -35.4% 222.67 ± 13% slabinfo.xfrm_state.active_objs 344.75 ± 14% -35.4% 222.67 ± 13% slabinfo.xfrm_state.num_objs 68.25 ± 15% -17.9% 56.00 turbostat.Avg_MHz 4.71 ± 19% -1.0 3.72 turbostat.Busy% 66815 ± 38% -58.5% 27727 ± 7% turbostat.C1 93.61 ± 3% -4.9% 89.06 turbostat.PkgWatt 91.08 +7.8% 98.23 iostat.cpu.idle 8.05 -88.6% 0.92 iostat.cpu.iowait 15.79 ±173% -100.0% 0.00 iostat.sda.avgqu-sz.max 41.72 ±173% -100.0% 0.00 iostat.sda.await.max 17.62 ±173% -100.0% 0.00 iostat.sda.r_await.max 0.27 ±173% -100.0% 0.00 iostat.sda.svctm.max 41.72 ±173% -100.0% 0.00 iostat.sda.w_await.max 1.21 ± 57% +68.9% 2.05 iostat.sdb.wrqm/s 6902065 +8.6% 7498469 proc-vmstat.nr_dirty 6494293 +100.2% 13003066 proc-vmstat.nr_dirty_background_threshold 13004465 +150.0% 32515607 proc-vmstat.nr_dirty_threshold 14902299 +3.8% 15475261 proc-vmstat.nr_file_pages 50695632 -1.1% 50115309 proc-vmstat.nr_free_pages 14627724 +3.9% 15201051 proc-vmstat.nr_inactive_file 1196 -5.3% 1133 proc-vmstat.nr_page_table_pages 53073 +2.5% 54382 proc-vmstat.nr_slab_reclaimable 14627724 +3.9% 15201051 proc-vmstat.nr_zone_inactive_file 6924527 +8.6% 7520765 proc-vmstat.nr_zone_write_pending 4911670 ± 5% -100.0% 0.00 proc-vmstat.numa_foreign 11272999 ± 2% +43.5% 16181547 proc-vmstat.numa_hit 11241838 ± 2% +43.7% 16150474 proc-vmstat.numa_local 4911670 ± 5% -100.0% 0.00 proc-vmstat.numa_miss 4942831 ± 5% -99.4% 31073 proc-vmstat.numa_other 605288 ± 46% +62.4% 982801 ± 21% sched_debug.cfs_rq:/.load.max 110012 ± 20% +35.8% 149419 ± 15% sched_debug.cfs_rq:/.load.stddev 1.85 ± 73% +131.2% 4.28 ± 6% sched_debug.cfs_rq:/.removed.util_avg.avg 14.22 ± 59% +71.5% 24.40 ± 6% sched_debug.cfs_rq:/.removed.util_avg.stddev 604628 ± 46% +62.5% 982801 ± 21% sched_debug.cfs_rq:/.runnable_weight.max 109963 ± 20% +35.9% 149420 ± 15% sched_debug.cfs_rq:/.runnable_weight.stddev 182.26 ± 3% -5.8% 171.67 ± 5% sched_debug.cfs_rq:/.util_avg.stddev 859467 -47.5% 451140 sched_debug.cpu.avg_idle.avg 1050621 ± 6% -46.8% 559208 ± 8% sched_debug.cpu.avg_idle.max 179785 ± 8% -53.1% 84392 ± 9% sched_debug.cpu.avg_idle.stddev 502355 -50.1% 250635 sched_debug.cpu.max_idle_balance_cost.avg 595239 ± 16% -50.1% 297142 ± 12% sched_debug.cpu.max_idle_balance_cost.max 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 3625 -14.7% 3093 sched_debug.cpu.nr_switches.avg 8.56 ± 13% -14.8% 7.29 ± 11% sched_debug.cpu.nr_uninterruptible.stddev 1812 -30.1% 1266 sched_debug.cpu.sched_count.avg 876.95 -31.5% 600.57 sched_debug.cpu.sched_goidle.avg 911.99 -29.9% 639.51 sched_debug.cpu.ttwu_count.avg 650.33 -39.6% 392.92 sched_debug.cpu.ttwu_local.avg 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 4.59e+08 ± 2% -4.3% 4.394e+08 ± 3% perf-stat.i.branch-instructions 1984351 ± 36% -49.3% 1005882 ± 11% perf-stat.i.cache-misses 30076098 ± 73% -74.8% 7591943 ± 7% perf-stat.i.cache-references 2477 -23.3% 1899 perf-stat.i.context-switches 4.704e+09 ± 22% -25.0% 3.529e+09 ± 3% perf-stat.i.cpu-cycles 2929 ± 23% +43.9% 4217 perf-stat.i.cycles-between-cache-misses 0.20 ± 96% -0.2 0.01 ± 15% perf-stat.i.dTLB-load-miss-rate% 677095 ± 96% -95.8% 28535 ± 31% perf-stat.i.dTLB-load-misses 5.269e+08 -5.1% 5e+08 ± 3% perf-stat.i.dTLB-loads 107155 ± 91% -91.0% 9662 ± 14% perf-stat.i.dTLB-store-misses 2.543e+08 ± 2% -5.8% 2.397e+08 ± 2% perf-stat.i.dTLB-stores 52.36 -1.5 50.83 perf-stat.i.iTLB-load-miss-rate% 2.234e+09 ± 2% -4.9% 2.125e+09 ± 3% perf-stat.i.instructions 78.56 ± 2% +4.4 82.97 ± 2% perf-stat.i.node-load-miss-rate% 52109 ± 8% -28.4% 37317 ± 11% perf-stat.i.node-loads 56.57 ± 22% +30.6 87.13 ± 3% perf-stat.i.node-store-miss-rate% 127934 ± 5% -60.3% 50785 ±112% perf-stat.i.node-stores 13.46 ± 73% -73.5% 3.57 ± 4% perf-stat.overall.MPKI 2510 ± 15% +41.0% 3540 ± 8% perf-stat.overall.cycles-between-cache-misses 53.32 -1.7 51.60 perf-stat.overall.iTLB-load-miss-rate% 4.554e+08 ± 2% -4.3% 4.359e+08 ± 3% perf-stat.ps.branch-instructions 1969672 ± 36% -49.3% 998526 ± 11% perf-stat.ps.cache-misses 29852128 ± 73% -74.8% 7534678 ± 7% perf-stat.ps.cache-references 2458 -23.3% 1885 perf-stat.ps.context-switches 4.668e+09 ± 22% -25.0% 3.502e+09 ± 3% perf-stat.ps.cpu-cycles 672093 ± 96% -95.8% 28338 ± 31% perf-stat.ps.dTLB-load-misses 5.228e+08 -5.1% 4.961e+08 ± 3% perf-stat.ps.dTLB-loads 106362 ± 91% -91.0% 9591 ± 14% perf-stat.ps.dTLB-store-misses 2.524e+08 ± 2% -5.8% 2.378e+08 ± 2% perf-stat.ps.dTLB-stores 2.217e+09 ± 2% -4.9% 2.108e+09 ± 3% perf-stat.ps.instructions 51702 ± 8% -28.4% 37035 ± 11% perf-stat.ps.node-loads 2.964e+11 ± 2% -4.4% 2.833e+11 ± 3% perf-stat.total.instructions 80.76 ± 3% -3.3 77.45 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 81.61 ± 2% -3.1 78.54 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 96.58 -1.3 95.29 perf-profile.calltrace.cycles-pp.secondary_startup_64 2.48 ± 9% -0.8 1.63 ± 12% perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 2.16 ± 8% -0.8 1.36 ± 14% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 1.28 ± 16% -0.3 0.99 ± 18% perf-profile.calltrace.cycles-pp.ktime_get.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt 1.59 ± 7% +0.4 1.98 ± 7% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.33 ± 19% +0.4 1.74 perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.35 ±102% +0.5 0.84 ± 15% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.xfs_destroy_ioend.xfs_end_ioend.xfs_end_io 0.00 +0.6 0.58 ± 5% perf-profile.calltrace.cycles-pp.native_write_msr.__intel_pmu_enable_all.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt 0.00 +0.6 0.64 ± 4% perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 3.72 ± 15% +0.8 4.49 ± 3% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 82.91 ± 2% -3.5 79.44 perf-profile.children.cycles-pp.cpuidle_enter_state 82.99 ± 2% -3.5 79.52 perf-profile.children.cycles-pp.cpuidle_enter 96.58 -1.3 95.29 perf-profile.children.cycles-pp.secondary_startup_64 96.58 -1.3 95.29 perf-profile.children.cycles-pp.cpu_startup_entry 96.69 -1.3 95.41 perf-profile.children.cycles-pp.do_idle 2.53 ± 9% -0.9 1.66 ± 10% perf-profile.children.cycles-pp.irq_enter 2.20 ± 8% -0.8 1.39 ± 13% perf-profile.children.cycles-pp.tick_irq_enter 0.53 ± 42% -0.4 0.10 ± 17% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.60 ± 39% -0.3 0.33 ± 6% perf-profile.children.cycles-pp.enqueue_hrtimer 0.26 ± 53% -0.2 0.09 ± 18% perf-profile.children.cycles-pp.hrtimer_forward 0.24 ± 33% -0.1 0.12 ± 29% perf-profile.children.cycles-pp.rb_insert_color 0.11 ± 41% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.intel_pmu_disable_all 0.21 ± 6% +0.0 0.25 perf-profile.children.cycles-pp.rcu_eqs_exit 0.09 ± 26% +0.0 0.13 ± 14% perf-profile.children.cycles-pp.smp_call_function_single 0.10 ± 28% +0.0 0.13 ± 17% perf-profile.children.cycles-pp.perf_event_read 0.01 ±173% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.fbcon_redraw 0.01 ±173% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.con_scroll 0.01 ±173% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.lf 0.01 ±173% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.fbcon_scroll 0.01 ±173% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.vt_console_print 0.06 ± 26% +0.0 0.11 ± 12% perf-profile.children.cycles-pp.schedule 0.01 ±173% +0.0 0.06 ± 13% perf-profile.children.cycles-pp.__wake_up_common 0.00 +0.1 0.05 perf-profile.children.cycles-pp.vm_mmap_pgoff 0.24 ± 7% +0.1 0.30 ± 12% perf-profile.children.cycles-pp.call_cpuidle 0.08 ± 24% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.__perf_event_read_value 0.10 ± 20% +0.1 0.17 ± 12% perf-profile.children.cycles-pp.update_dl_rq_load_avg 0.07 ± 64% +0.1 0.15 ± 24% perf-profile.children.cycles-pp.__mod_node_page_state 0.31 ± 8% +0.1 0.40 ± 19% perf-profile.children.cycles-pp.rcu_eqs_enter 0.60 ± 7% +0.1 0.72 ± 4% perf-profile.children.cycles-pp._raw_spin_lock 0.22 ± 42% +0.2 0.38 ± 5% perf-profile.children.cycles-pp.trigger_load_balance 0.41 ± 20% +0.2 0.65 ± 4% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.59 ± 29% +0.3 0.88 ± 14% perf-profile.children.cycles-pp.test_clear_page_writeback 1.64 ± 7% +0.4 2.01 ± 7% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 1.38 ± 18% +0.4 1.76 perf-profile.children.cycles-pp.scheduler_tick 0.92 ± 18% +0.4 1.37 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.53 ± 42% -0.4 0.10 ± 17% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.32 ± 49% -0.2 0.15 ± 10% perf-profile.self.cycles-pp.timerqueue_add 0.26 ± 53% -0.2 0.08 ± 20% perf-profile.self.cycles-pp.hrtimer_forward 0.20 ± 36% -0.1 0.11 ± 26% perf-profile.self.cycles-pp.rb_insert_color 0.22 ± 13% -0.1 0.14 ± 20% perf-profile.self.cycles-pp.irq_exit 0.12 ± 13% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.cpuidle_governor_latency_req 0.07 ± 11% -0.0 0.04 ± 73% perf-profile.self.cycles-pp.enqueue_hrtimer 0.10 ± 11% -0.0 0.07 perf-profile.self.cycles-pp.find_busiest_group 0.08 ± 23% +0.0 0.12 ± 10% perf-profile.self.cycles-pp.smp_call_function_single 0.20 ± 12% +0.1 0.25 ± 9% perf-profile.self.cycles-pp.load_balance 0.10 ± 21% +0.1 0.15 ± 11% perf-profile.self.cycles-pp.update_dl_rq_load_avg 0.07 ± 15% +0.1 0.13 ± 31% perf-profile.self.cycles-pp.page_mapping 0.24 ± 8% +0.1 0.30 ± 13% perf-profile.self.cycles-pp.call_cpuidle 0.07 ± 64% +0.1 0.15 ± 24% perf-profile.self.cycles-pp.__mod_node_page_state 0.30 ± 10% +0.1 0.39 ± 19% perf-profile.self.cycles-pp.rcu_eqs_enter 0.59 ± 7% +0.1 0.71 ± 3% perf-profile.self.cycles-pp._raw_spin_lock 3.36 ± 14% +0.6 3.98 perf-profile.self.cycles-pp.cpuidle_enter_state 0.25 ±173% +65100.0% 163.00 ± 79% interrupts.86:PCI-MSI.31981619-edge.i40e-eth0-TxRx-50 290.50 ± 36% -44.5% 161.33 ± 23% interrupts.CPU0.NMI:Non-maskable_interrupts 290.50 ± 36% -44.5% 161.33 ± 23% interrupts.CPU0.PMI:Performance_monitoring_interrupts 5404 ±111% -91.4% 463.67 ± 94% interrupts.CPU1.RES:Rescheduling_interrupts 204.50 ± 24% -40.8% 121.00 ± 32% interrupts.CPU12.NMI:Non-maskable_interrupts 204.50 ± 24% -40.8% 121.00 ± 32% interrupts.CPU12.PMI:Performance_monitoring_interrupts 200.00 ± 16% -35.0% 130.00 ± 30% interrupts.CPU13.NMI:Non-maskable_interrupts 200.00 ± 16% -35.0% 130.00 ± 30% interrupts.CPU13.PMI:Performance_monitoring_interrupts 208.50 ± 22% -34.1% 137.33 ± 27% interrupts.CPU14.NMI:Non-maskable_interrupts 208.50 ± 22% -34.1% 137.33 ± 27% interrupts.CPU14.PMI:Performance_monitoring_interrupts 215.75 ± 23% -44.8% 119.00 ± 28% interrupts.CPU15.NMI:Non-maskable_interrupts 215.75 ± 23% -44.8% 119.00 ± 28% interrupts.CPU15.PMI:Performance_monitoring_interrupts 93.75 ±159% -97.2% 2.67 ± 17% interrupts.CPU18.RES:Rescheduling_interrupts 212.25 ± 24% -42.8% 121.33 ± 24% interrupts.CPU2.NMI:Non-maskable_interrupts 212.25 ± 24% -42.8% 121.33 ± 24% interrupts.CPU2.PMI:Performance_monitoring_interrupts 259.75 ± 37% -42.9% 148.33 ± 4% interrupts.CPU29.NMI:Non-maskable_interrupts 259.75 ± 37% -42.9% 148.33 ± 4% interrupts.CPU29.PMI:Performance_monitoring_interrupts 39.75 ±153% -97.5% 1.00 ± 81% interrupts.CPU36.RES:Rescheduling_interrupts 244.00 ± 39% -23.5% 186.67 ± 39% interrupts.CPU38.NMI:Non-maskable_interrupts 244.00 ± 39% -23.5% 186.67 ± 39% interrupts.CPU38.PMI:Performance_monitoring_interrupts 263.50 ± 42% -47.4% 138.67 ± 33% interrupts.CPU4.NMI:Non-maskable_interrupts 263.50 ± 42% -47.4% 138.67 ± 33% interrupts.CPU4.PMI:Performance_monitoring_interrupts 256.00 ± 42% -42.3% 147.67 ± 8% interrupts.CPU41.NMI:Non-maskable_interrupts 256.00 ± 42% -42.3% 147.67 ± 8% interrupts.CPU41.PMI:Performance_monitoring_interrupts 213.75 ± 30% -42.6% 122.67 ± 23% interrupts.CPU5.NMI:Non-maskable_interrupts 213.75 ± 30% -42.6% 122.67 ± 23% interrupts.CPU5.PMI:Performance_monitoring_interrupts 2077 ± 17% -20.0% 1663 interrupts.CPU51.CAL:Function_call_interrupts 2068 ± 18% -18.7% 1682 interrupts.CPU62.CAL:Function_call_interrupts 203.75 ± 18% -28.2% 146.33 ± 2% interrupts.CPU62.NMI:Non-maskable_interrupts 203.75 ± 18% -28.2% 146.33 ± 2% interrupts.CPU62.PMI:Performance_monitoring_interrupts 2068 ± 18% -18.7% 1682 interrupts.CPU63.CAL:Function_call_interrupts 2064 ± 18% -22.7% 1595 ± 5% interrupts.CPU65.CAL:Function_call_interrupts 208.25 ± 19% -26.2% 153.67 ± 5% interrupts.CPU66.NMI:Non-maskable_interrupts 208.25 ± 19% -26.2% 153.67 ± 5% interrupts.CPU66.PMI:Performance_monitoring_interrupts 2064 ± 18% -19.6% 1658 interrupts.CPU67.CAL:Function_call_interrupts 2065 ± 18% -20.8% 1636 interrupts.CPU68.CAL:Function_call_interrupts 200.25 ± 20% -35.9% 128.33 ± 28% interrupts.CPU7.NMI:Non-maskable_interrupts 200.25 ± 20% -35.9% 128.33 ± 28% interrupts.CPU7.PMI:Performance_monitoring_interrupts 297.50 ± 43% -47.0% 157.67 ± 8% interrupts.CPU70.NMI:Non-maskable_interrupts 297.50 ± 43% -47.0% 157.67 ± 8% interrupts.CPU70.PMI:Performance_monitoring_interrupts 267.50 ± 37% -44.2% 149.33 ± 3% interrupts.CPU72.NMI:Non-maskable_interrupts 267.50 ± 37% -44.2% 149.33 ± 3% interrupts.CPU72.PMI:Performance_monitoring_interrupts 255.75 ± 38% -47.3% 134.67 ± 4% interrupts.CPU76.NMI:Non-maskable_interrupts 255.75 ± 38% -47.3% 134.67 ± 4% interrupts.CPU76.PMI:Performance_monitoring_interrupts 261.50 ± 39% -40.9% 154.67 ± 5% interrupts.CPU77.NMI:Non-maskable_interrupts 261.50 ± 39% -40.9% 154.67 ± 5% interrupts.CPU77.PMI:Performance_monitoring_interrupts 263.00 ± 44% -52.1% 126.00 ± 38% interrupts.CPU79.NMI:Non-maskable_interrupts 263.00 ± 44% -52.1% 126.00 ± 38% interrupts.CPU79.PMI:Performance_monitoring_interrupts 259.50 ± 40% -50.8% 127.67 ± 34% interrupts.CPU84.NMI:Non-maskable_interrupts 259.50 ± 40% -50.8% 127.67 ± 34% interrupts.CPU84.PMI:Performance_monitoring_interrupts 253.75 ± 40% -41.1% 149.33 ± 48% interrupts.CPU86.NMI:Non-maskable_interrupts 253.75 ± 40% -41.1% 149.33 ± 48% interrupts.CPU86.PMI:Performance_monitoring_interrupts 250.75 ± 44% -49.6% 126.33 ± 31% interrupts.CPU88.NMI:Non-maskable_interrupts 250.75 ± 44% -49.6% 126.33 ± 31% interrupts.CPU88.PMI:Performance_monitoring_interrupts 252.25 ± 42% -51.9% 121.33 ± 33% interrupts.CPU89.NMI:Non-maskable_interrupts 252.25 ± 42% -51.9% 121.33 ± 33% interrupts.CPU89.PMI:Performance_monitoring_interrupts 245.75 ± 40% -50.8% 121.00 ± 34% interrupts.CPU90.NMI:Non-maskable_interrupts 245.75 ± 40% -50.8% 121.00 ± 34% interrupts.CPU90.PMI:Performance_monitoring_interrupts 261.00 ± 42% -46.4% 140.00 interrupts.CPU91.NMI:Non-maskable_interrupts 261.00 ± 42% -46.4% 140.00 interrupts.CPU91.PMI:Performance_monitoring_interrupts 256.75 ± 39% -42.7% 147.00 ± 6% interrupts.CPU92.NMI:Non-maskable_interrupts 256.75 ± 39% -42.7% 147.00 ± 6% interrupts.CPU92.PMI:Performance_monitoring_interrupts 255.50 ± 39% -40.4% 152.33 ± 7% interrupts.CPU93.NMI:Non-maskable_interrupts 255.50 ± 39% -40.4% 152.33 ± 7% interrupts.CPU93.PMI:Performance_monitoring_interrupts 270.75 ± 41% -45.0% 149.00 ± 4% interrupts.CPU94.NMI:Non-maskable_interrupts 270.75 ± 41% -45.0% 149.00 ± 4% interrupts.CPU94.PMI:Performance_monitoring_interrupts 14178 ± 5% -24.3% 10726 ± 13% softirqs.CPU10.RCU 14327 ± 5% -25.3% 10709 ± 13% softirqs.CPU11.RCU 13657 ± 6% -22.7% 10555 ± 13% softirqs.CPU12.RCU 13837 ± 5% -23.1% 10640 ± 15% softirqs.CPU13.RCU 14189 ± 5% -24.5% 10711 ± 15% softirqs.CPU14.RCU 14009 ± 3% -22.0% 10921 ± 15% softirqs.CPU15.RCU 13125 ± 4% -14.6% 11210 ± 14% softirqs.CPU17.RCU 12992 ± 5% -17.9% 10663 ± 12% softirqs.CPU19.RCU 13489 ± 2% -20.0% 10787 ± 14% softirqs.CPU2.RCU 13290 ± 4% -18.1% 10883 ± 11% softirqs.CPU20.RCU 12513 ± 4% -13.9% 10776 ± 12% softirqs.CPU21.RCU 13147 ± 6% -19.9% 10526 ± 19% softirqs.CPU23.RCU 13959 ± 7% -20.4% 11106 ± 15% softirqs.CPU24.RCU 19261 ± 3% -8.8% 17559 softirqs.CPU24.SCHED 12910 ± 9% -17.4% 10662 ± 5% softirqs.CPU27.RCU 63260 ± 13% -17.9% 51933 ± 7% softirqs.CPU27.TIMER 58283 ± 11% -11.8% 51424 ± 6% softirqs.CPU28.TIMER 58243 ± 12% -11.7% 51441 ± 6% softirqs.CPU29.TIMER 16519 ± 18% -36.8% 10444 ± 30% softirqs.CPU3.RCU 14396 ± 9% -29.2% 10196 ± 6% softirqs.CPU32.RCU 13800 ± 11% -23.8% 10513 ± 8% softirqs.CPU33.RCU 14106 ± 10% -25.4% 10522 ± 7% softirqs.CPU34.RCU 14153 ± 10% -22.6% 10956 ± 12% softirqs.CPU35.RCU 13465 ± 12% -15.1% 11433 ± 12% softirqs.CPU36.RCU 13993 ± 11% -25.1% 10477 ± 7% softirqs.CPU37.RCU 13908 ± 9% -25.1% 10413 ± 8% softirqs.CPU38.RCU 13697 ± 11% -21.5% 10749 ± 4% softirqs.CPU39.RCU 15844 ± 16% -31.9% 10786 ± 14% softirqs.CPU4.RCU 21525 ± 13% -14.8% 18350 ± 4% softirqs.CPU4.SCHED 13962 ± 11% -23.6% 10666 ± 5% softirqs.CPU40.RCU 14388 ± 11% -26.7% 10541 ± 8% softirqs.CPU41.RCU 13635 ± 13% -27.6% 9868 ± 8% softirqs.CPU42.RCU 14048 ± 11% -21.3% 11053 ± 11% softirqs.CPU43.RCU 14344 ± 11% -26.8% 10497 ± 8% softirqs.CPU44.RCU 58230 ± 11% -13.4% 50409 ± 6% softirqs.CPU44.TIMER 14276 ± 10% -25.1% 10690 ± 5% softirqs.CPU45.RCU 14280 ± 11% -25.9% 10581 ± 5% softirqs.CPU46.RCU 13720 ± 8% -23.1% 10556 ± 6% softirqs.CPU47.RCU 13096 ± 4% -16.5% 10936 ± 16% softirqs.CPU48.RCU 13617 ± 4% -22.6% 10535 ± 11% softirqs.CPU49.RCU 14186 ± 6% -20.2% 11317 ± 8% softirqs.CPU5.RCU 14099 ± 3% -24.2% 10691 ± 11% softirqs.CPU50.RCU 13393 ± 8% -20.4% 10667 ± 13% softirqs.CPU52.RCU 14321 ± 4% -22.5% 11099 ± 17% softirqs.CPU53.RCU 14290 ± 5% -20.4% 11377 ± 11% softirqs.CPU54.RCU 13816 ± 5% -22.3% 10738 ± 14% softirqs.CPU55.RCU 14216 ± 5% -25.7% 10562 ± 15% softirqs.CPU56.RCU 13828 ± 5% -21.0% 10921 ± 16% softirqs.CPU57.RCU 14420 ± 7% -25.4% 10759 ± 14% softirqs.CPU58.RCU 14181 ± 5% -24.5% 10704 ± 14% softirqs.CPU59.RCU 14292 ± 5% -25.0% 10718 ± 13% softirqs.CPU6.RCU 13769 ± 4% -19.8% 11045 ± 10% softirqs.CPU60.RCU 13739 ± 3% -20.9% 10866 ± 14% softirqs.CPU61.RCU 14338 ± 7% -24.9% 10767 ± 14% softirqs.CPU62.RCU 13912 ± 3% -24.3% 10537 ± 14% softirqs.CPU63.RCU 12922 ± 5% -18.2% 10573 ± 13% softirqs.CPU64.RCU 13176 ± 8% -18.6% 10722 ± 9% softirqs.CPU65.RCU 13092 ± 8% -20.6% 10390 ± 14% softirqs.CPU67.RCU 12796 ± 3% -32.8% 8594 ± 43% softirqs.CPU68.RCU 12979 ± 8% -20.8% 10274 ± 12% softirqs.CPU69.RCU 13999 ± 4% -22.7% 10823 ± 13% softirqs.CPU7.RCU 12760 ± 6% -18.1% 10446 ± 12% softirqs.CPU70.RCU 12198 ± 14% -26.6% 8948 ± 22% softirqs.CPU75.RCU 56460 ± 13% -13.0% 49127 ± 7% softirqs.CPU76.TIMER 56294 ± 13% -13.0% 48997 ± 6% softirqs.CPU77.TIMER 14222 ± 5% -26.4% 10472 ± 12% softirqs.CPU8.RCU 13604 ± 10% -14.6% 11615 ± 9% softirqs.CPU81.RCU 14079 ± 11% -27.0% 10277 ± 8% softirqs.CPU82.RCU 13936 ± 11% -26.5% 10244 ± 8% softirqs.CPU83.RCU 14220 ± 13% -28.1% 10230 ± 7% softirqs.CPU85.RCU 13573 ± 9% -25.4% 10128 ± 8% softirqs.CPU86.RCU 13399 ± 12% -22.7% 10356 ± 3% softirqs.CPU87.RCU 13987 ± 11% -26.8% 10245 ± 7% softirqs.CPU88.RCU 14106 ± 12% -23.7% 10761 softirqs.CPU89.RCU 13853 ± 4% -23.6% 10577 ± 13% softirqs.CPU9.RCU 13260 ± 11% -22.1% 10325 ± 8% softirqs.CPU90.RCU 15140 ± 16% -33.6% 10054 ± 8% softirqs.CPU91.RCU 17734 ± 5% -7.0% 16486 ± 4% softirqs.CPU91.SCHED 14199 ± 12% -28.5% 10153 ± 7% softirqs.CPU92.RCU 14249 ± 12% -25.3% 10641 ± 11% softirqs.CPU93.RCU 13965 ± 11% -22.7% 10789 ± 8% softirqs.CPU94.RCU 1301155 ± 6% -21.3% 1023388 ± 10% softirqs.RCU *************************************************************************************************** lkp-csl-2sp7: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-7/performance/1HDD/16MB/f2fs/1x/x86_64-rhel-7.6/16d/256fpd/32t/debian-x86_64-2019-09-23.cgz/NoSync/lkp-csl-2sp7/60G/fsmark/0x500002b commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- %stddev %change %stddev \ | \ 207487 ± 2% -24.5% 156557 ± 3% fsmark.app_overhead 37.10 +832.4% 345.93 ± 3% fsmark.files_per_sec 83.25 ± 3% +22.5% 102.00 ± 3% fsmark.time.percent_of_cpu_this_job_got 289.97 ± 2% +23.3% 357.53 ± 3% fsmark.time.system_time 137357 -73.7% 36165 fsmark.time.voluntary_context_switches 28459340 +9.4% 31135240 meminfo.Dirty 1013333 ±149% -91.1% 90145 ± 7% turbostat.C1 9.46 ± 2% -8.5 0.97 ± 7% mpstat.cpu.all.iowait% 0.93 ± 3% +0.2 1.18 ± 3% mpstat.cpu.all.sys% 62658734 ±154% -95.9% 2589839 ± 15% cpuidle.C1.time 1017527 ±148% -90.7% 94277 ± 9% cpuidle.C1.usage 133577 ± 89% -62.8% 49722 ± 4% cpuidle.POLL.usage 89.25 +8.7% 97.00 vmstat.cpu.id 9.00 -97.2% 0.25 ±173% vmstat.procs.b 2202 -29.0% 1564 vmstat.system.cs 3.00 ±173% +2.2e+05% 6602 ±165% softirqs.CPU19.BLOCK 33584 ± 4% -15.5% 28392 ± 11% softirqs.CPU3.RCU 43095 ± 3% +5.9% 45643 ± 5% softirqs.CPU43.SCHED 30539 ± 5% -14.1% 26236 ± 8% softirqs.CPU53.RCU 3490367 ± 14% -100.0% 0.00 numa-numastat.node0.numa_foreign 94454 ±123% -86.9% 12407 ±106% numa-numastat.node0.other_node 3380548 ± 23% +211.7% 10536792 ± 29% numa-numastat.node1.local_node 3383810 ± 22% +211.9% 10555638 ± 29% numa-numastat.node1.numa_hit 3490367 ± 14% -100.0% 0.00 numa-numastat.node1.numa_miss 3493632 ± 14% -99.5% 18845 ± 69% numa-numastat.node1.other_node 89.52 +9.1% 97.71 iostat.cpu.idle 9.41 ± 2% -89.7% 0.97 ± 7% iostat.cpu.iowait 0.96 ± 2% +25.7% 1.21 ± 3% iostat.cpu.system 140.86 ±173% +634.2% 1034 ±101% iostat.sda.await.max 74.25 ±173% +448.5% 407.24 ±168% iostat.sda.r_await.max 140.86 ±173% +634.2% 1034 ±101% iostat.sda.w_await.max 1225 ± 57% -99.9% 0.85 ±173% iostat.sdb.r_await.max 85549 ± 8% -51.4% 41552 ± 59% numa-meminfo.node0.Active(file) 155085 ± 5% -39.2% 94346 ± 30% numa-meminfo.node0.KReclaimable 2939 ± 13% -22.9% 2267 ± 17% numa-meminfo.node0.PageTables 155085 ± 5% -39.2% 94346 ± 30% numa-meminfo.node0.SReclaimable 257098 ± 2% -26.5% 188849 ± 16% numa-meminfo.node0.Slab 20863 ± 34% +227.4% 68312 ± 35% numa-meminfo.node1.Active(file) 137672 ± 19% -29.8% 96587 ± 27% numa-meminfo.node1.AnonHugePages 59732 ± 13% +111.6% 126382 ± 22% numa-meminfo.node1.KReclaimable 59732 ± 13% +111.6% 126382 ± 22% numa-meminfo.node1.SReclaimable 132316 ± 4% +55.3% 205519 ± 15% numa-meminfo.node1.Slab 16.03 ±118% -72.2% 4.46 perf-stat.i.MPKI 2179 -30.0% 1526 perf-stat.i.context-switches 3588 ± 32% +60.5% 5760 ± 3% perf-stat.i.cycles-between-cache-misses 55.93 ± 8% -5.5 50.45 perf-stat.i.iTLB-load-miss-rate% 84.24 +3.7 87.98 perf-stat.i.node-load-miss-rate% 65.18 ± 12% +23.8 88.94 perf-stat.i.node-store-miss-rate% 144399 ± 4% +36.4% 196999 ± 5% perf-stat.i.node-stores 84.95 +1.8 86.80 perf-stat.overall.node-load-miss-rate% 60.52 -8.0 52.53 ± 3% perf-stat.overall.node-store-miss-rate% 2172 -29.9% 1521 perf-stat.ps.context-switches 144175 ± 4% +36.1% 196209 ± 5% perf-stat.ps.node-stores 21387 ± 8% -51.5% 10377 ± 58% numa-vmstat.node0.nr_active_file 734.25 ± 13% -23.0% 565.25 ± 17% numa-vmstat.node0.nr_page_table_pages 38748 ± 5% -39.2% 23576 ± 30% numa-vmstat.node0.nr_slab_reclaimable 21387 ± 8% -51.5% 10377 ± 58% numa-vmstat.node0.nr_zone_active_file 3408738 ± 14% -100.0% 0.00 numa-vmstat.node0.numa_foreign 5214 ± 34% +227.7% 17087 ± 36% numa-vmstat.node1.nr_active_file 14932 ± 13% +111.6% 31599 ± 22% numa-vmstat.node1.nr_slab_reclaimable 5214 ± 34% +227.7% 17087 ± 36% numa-vmstat.node1.nr_zone_active_file 3413078 ± 20% +214.0% 10715494 ± 29% numa-vmstat.node1.numa_hit 3239704 ± 21% +224.9% 10526475 ± 29% numa-vmstat.node1.numa_local 3408755 ± 14% -100.0% 0.00 numa-vmstat.node1.numa_miss 3582130 ± 13% -94.7% 189019 ± 6% numa-vmstat.node1.numa_other 26585 +3.4% 27484 proc-vmstat.nr_active_file 7108139 +9.6% 7787129 proc-vmstat.nr_dirty 6494961 +100.2% 13005676 proc-vmstat.nr_dirty_background_threshold 13005805 +150.1% 32522132 proc-vmstat.nr_dirty_threshold 14934015 +4.3% 15578971 proc-vmstat.nr_file_pages 50671481 -1.3% 50025580 proc-vmstat.nr_free_pages 14632035 +4.4% 15276391 proc-vmstat.nr_inactive_file 1197 -6.2% 1123 proc-vmstat.nr_page_table_pages 5959 ± 2% -2.8% 5795 proc-vmstat.nr_shmem 53692 +2.8% 55196 proc-vmstat.nr_slab_reclaimable 26585 +3.4% 27484 proc-vmstat.nr_zone_active_file 14632035 +4.4% 15276391 proc-vmstat.nr_zone_inactive_file 7130078 +9.5% 7808827 proc-vmstat.nr_zone_write_pending 3556800 ± 11% -100.0% 0.00 proc-vmstat.numa_foreign 13113098 ± 2% +27.1% 16668341 proc-vmstat.numa_hit 13081660 ± 3% +27.2% 16637084 proc-vmstat.numa_local 3556800 ± 11% -100.0% 0.00 proc-vmstat.numa_miss 3588237 ± 11% -99.1% 31257 proc-vmstat.numa_other 8353 ± 2% -4.4% 7982 proc-vmstat.pgactivate 807.96 ± 12% -25.0% 606.17 ± 6% sched_debug.cfs_rq:/.load_avg.max 161.13 ± 7% -15.2% 136.56 ± 8% sched_debug.cfs_rq:/.load_avg.stddev 262344 ± 4% +26.2% 331200 ± 3% sched_debug.cfs_rq:/.min_vruntime.max 87252 ± 4% +24.9% 109020 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev 590.83 ± 15% -28.5% 422.50 ± 10% sched_debug.cfs_rq:/.runnable_load_avg.max 104.27 ± 8% -23.3% 79.93 ± 18% sched_debug.cfs_rq:/.runnable_load_avg.stddev 87252 ± 4% +24.9% 109021 ± 7% sched_debug.cfs_rq:/.spread0.stddev 914115 -48.5% 470421 sched_debug.cpu.avg_idle.avg 1018444 ± 3% -44.0% 569875 ± 11% sched_debug.cpu.avg_idle.max 136783 ± 20% -50.0% 68366 ± 18% sched_debug.cpu.avg_idle.stddev 501309 -49.7% 251975 sched_debug.cpu.max_idle_balance_cost.avg 521214 ± 4% -42.5% 299871 ± 19% sched_debug.cpu.max_idle_balance_cost.max 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 6323 -28.4% 4524 sched_debug.cpu.nr_switches.avg 6684 ± 5% -19.0% 5412 ± 10% sched_debug.cpu.nr_switches.stddev 0.08 ± 2% -71.2% 0.02 ± 17% sched_debug.cpu.nr_uninterruptible.avg 4672 -39.0% 2851 sched_debug.cpu.sched_count.avg 6408 ± 4% -20.0% 5125 ± 11% sched_debug.cpu.sched_count.stddev 2297 -39.7% 1384 sched_debug.cpu.sched_goidle.avg 3150 ± 4% -20.6% 2503 ± 11% sched_debug.cpu.sched_goidle.stddev 2327 -39.2% 1416 sched_debug.cpu.ttwu_count.avg 1122 -52.2% 536.60 sched_debug.cpu.ttwu_local.avg 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 74.50 ±168% -99.3% 0.50 ±173% interrupts.48:PCI-MSI.31981581-edge.i40e-eth0-TxRx-12 106.00 ± 84% -100.0% 0.00 interrupts.50:PCI-MSI.31981583-edge.i40e-eth0-TxRx-14 137.25 ± 99% -95.3% 6.50 ± 53% interrupts.CPU12.RES:Rescheduling_interrupts 227.75 ± 36% -39.8% 137.00 ± 16% interrupts.CPU29.NMI:Non-maskable_interrupts 227.75 ± 36% -39.8% 137.00 ± 16% interrupts.CPU29.PMI:Performance_monitoring_interrupts 81.75 ±147% -95.4% 3.75 ± 78% interrupts.CPU32.RES:Rescheduling_interrupts 231.00 ± 41% -40.4% 137.75 ± 17% interrupts.CPU34.NMI:Non-maskable_interrupts 231.00 ± 41% -40.4% 137.75 ± 17% interrupts.CPU34.PMI:Performance_monitoring_interrupts 237.50 ± 32% -38.9% 145.00 ± 25% interrupts.CPU39.NMI:Non-maskable_interrupts 237.50 ± 32% -38.9% 145.00 ± 25% interrupts.CPU39.PMI:Performance_monitoring_interrupts 226.75 ± 34% -47.5% 119.00 ± 35% interrupts.CPU4.NMI:Non-maskable_interrupts 226.75 ± 34% -47.5% 119.00 ± 35% interrupts.CPU4.PMI:Performance_monitoring_interrupts 237.00 ± 43% -51.6% 114.75 ± 36% interrupts.CPU48.NMI:Non-maskable_interrupts 237.00 ± 43% -51.6% 114.75 ± 36% interrupts.CPU48.PMI:Performance_monitoring_interrupts 229.25 ± 41% -51.4% 111.50 ± 31% interrupts.CPU49.NMI:Non-maskable_interrupts 229.25 ± 41% -51.4% 111.50 ± 31% interrupts.CPU49.PMI:Performance_monitoring_interrupts 224.25 ± 35% -58.1% 94.00 ± 31% interrupts.CPU50.NMI:Non-maskable_interrupts 224.25 ± 35% -58.1% 94.00 ± 31% interrupts.CPU50.PMI:Performance_monitoring_interrupts 233.00 ± 38% -57.8% 98.25 ± 22% interrupts.CPU51.NMI:Non-maskable_interrupts 233.00 ± 38% -57.8% 98.25 ± 22% interrupts.CPU51.PMI:Performance_monitoring_interrupts 221.25 ± 38% -54.9% 99.75 ± 31% interrupts.CPU52.NMI:Non-maskable_interrupts 221.25 ± 38% -54.9% 99.75 ± 31% interrupts.CPU52.PMI:Performance_monitoring_interrupts 205.75 ± 33% -48.4% 106.25 ± 31% interrupts.CPU54.NMI:Non-maskable_interrupts 205.75 ± 33% -48.4% 106.25 ± 31% interrupts.CPU54.PMI:Performance_monitoring_interrupts 218.25 ± 29% -49.3% 110.75 ± 33% interrupts.CPU59.NMI:Non-maskable_interrupts 218.25 ± 29% -49.3% 110.75 ± 33% interrupts.CPU59.PMI:Performance_monitoring_interrupts 217.25 ± 23% -52.7% 102.75 ± 51% interrupts.CPU62.NMI:Non-maskable_interrupts 217.25 ± 23% -52.7% 102.75 ± 51% interrupts.CPU62.PMI:Performance_monitoring_interrupts 242.00 ± 34% -49.8% 121.50 ± 34% interrupts.CPU68.NMI:Non-maskable_interrupts 242.00 ± 34% -49.8% 121.50 ± 34% interrupts.CPU68.PMI:Performance_monitoring_interrupts 208.25 ± 38% -42.0% 120.75 ± 31% interrupts.CPU7.NMI:Non-maskable_interrupts 208.25 ± 38% -42.0% 120.75 ± 31% interrupts.CPU7.PMI:Performance_monitoring_interrupts 275.50 ± 25% -57.3% 117.75 ± 35% interrupts.CPU71.NMI:Non-maskable_interrupts 275.50 ± 25% -57.3% 117.75 ± 35% interrupts.CPU71.PMI:Performance_monitoring_interrupts 248.50 ± 30% -43.3% 141.00 ± 23% interrupts.CPU72.NMI:Non-maskable_interrupts 248.50 ± 30% -43.3% 141.00 ± 23% interrupts.CPU72.PMI:Performance_monitoring_interrupts 1.75 ±173% +5600.0% 99.75 ±150% interrupts.CPU75.RES:Rescheduling_interrupts 1.00 ±100% +20250.0% 203.50 ±165% interrupts.CPU76.RES:Rescheduling_interrupts 0.00 +2.2e+104% 221.75 ±169% interrupts.CPU77.RES:Rescheduling_interrupts 21680 ± 36% -37.0% 13655 ± 9% interrupts.NMI:Non-maskable_interrupts 21680 ± 36% -37.0% 13655 ± 9% interrupts.PMI:Performance_monitoring_interrupts 0.39 ± 57% +0.3 0.67 ± 10% perf-profile.calltrace.cycles-pp.arch_cpu_idle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.52 ± 14% +0.4 1.94 ± 4% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.59 ± 62% +0.5 1.05 ± 12% perf-profile.calltrace.cycles-pp.run_local_timers.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.13 ±173% +0.5 0.65 ± 10% perf-profile.calltrace.cycles-pp.tsc_verify_tsc_adjust.arch_cpu_idle_enter.do_idle.cpu_startup_entry.start_secondary 0.13 ±173% +0.6 0.72 ± 23% perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages 1.45 ± 59% +1.4 2.81 ± 2% perf-profile.calltrace.cycles-pp.timekeeping_max_deferment.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 5.50 ± 21% +1.7 7.18 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 10.82 ± 13% +2.8 13.59 ± 5% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 0.34 ± 53% -0.3 0.03 ±100% perf-profile.children.cycles-pp.new_sync_write 0.34 ± 53% -0.3 0.03 ±100% perf-profile.children.cycles-pp.ksys_write 0.34 ± 53% -0.3 0.03 ±100% perf-profile.children.cycles-pp.vfs_write 0.35 ± 51% -0.3 0.03 ±100% perf-profile.children.cycles-pp.write 0.32 ± 38% -0.2 0.08 ± 10% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.15 ± 38% -0.1 0.03 ±100% perf-profile.children.cycles-pp.tick_check_broadcast_expired 0.84 ± 7% -0.1 0.71 ± 11% perf-profile.children.cycles-pp.console_unlock 0.75 ± 5% -0.1 0.64 ± 13% perf-profile.children.cycles-pp.wait_for_xmitr 0.75 ± 4% -0.1 0.65 ± 11% perf-profile.children.cycles-pp.uart_console_write 0.77 ± 5% -0.1 0.66 ± 12% perf-profile.children.cycles-pp.serial8250_console_write 0.73 ± 4% -0.1 0.62 ± 13% perf-profile.children.cycles-pp.serial8250_console_putchar 0.11 ± 65% -0.1 0.03 ±100% perf-profile.children.cycles-pp.irq_work_needs_cpu 0.21 ± 7% -0.1 0.14 ± 18% perf-profile.children.cycles-pp.memcpy_erms 0.21 ± 8% -0.1 0.14 ± 18% perf-profile.children.cycles-pp.drm_fb_helper_dirty_work 0.07 ± 7% -0.0 0.03 ±100% perf-profile.children.cycles-pp.x86_pmu_disable 0.50 ± 7% +0.2 0.67 ± 10% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust 0.51 ± 7% +0.2 0.69 ± 10% perf-profile.children.cycles-pp.arch_cpu_idle_enter 0.41 ± 22% +0.2 0.61 ± 13% perf-profile.children.cycles-pp.__intel_pmu_enable_all 1.56 ± 12% +0.4 1.97 ± 3% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 1.53 ± 47% +1.3 2.84 perf-profile.children.cycles-pp.timekeeping_max_deferment 6.81 ± 19% +1.6 8.40 ± 4% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 5.61 ± 20% +1.7 7.28 ± 3% perf-profile.children.cycles-pp.tick_nohz_next_event 11.00 ± 13% +2.8 13.76 ± 5% perf-profile.children.cycles-pp.menu_select 7.66 ± 40% +2.9 10.61 ± 2% perf-profile.children.cycles-pp.ktime_get 0.32 ± 38% -0.2 0.08 ± 10% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.21 ± 7% -0.1 0.14 ± 18% perf-profile.self.cycles-pp.memcpy_erms 0.06 ± 60% +0.0 0.10 ± 10% perf-profile.self.cycles-pp.cpuidle_enter 0.62 ± 24% +0.2 0.78 ± 4% perf-profile.self.cycles-pp.__softirqentry_text_start 0.45 ± 15% +0.2 0.65 ± 11% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust 0.41 ± 25% +0.3 0.68 ± 11% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.77 ± 17% +0.4 1.14 ± 6% perf-profile.self.cycles-pp.tick_nohz_next_event 3.18 ± 20% +0.7 3.92 ± 2% perf-profile.self.cycles-pp.cpuidle_enter_state 1.52 ± 48% +1.3 2.84 perf-profile.self.cycles-pp.timekeeping_max_deferment 6.49 ± 43% +2.8 9.32 ± 3% perf-profile.self.cycles-pp.ktime_get *************************************************************************************************** lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1HDD/ext4/libaio/x86_64-rhel-7.6/100%/debian-x86_64-2019-11-14.cgz/300s/write/lkp-ivb-2ep1/128G/fio-basic/0x42e commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x %stddev %change %stddev \ | \ 28.56 ± 14% -22.6 5.98 ± 12% fio.latency_100ms% 0.25 ± 8% -0.2 0.02 ± 47% fio.latency_20ms% 0.73 +0.4 1.15 ± 13% fio.latency_250ms% 0.01 ± 9% +9.6 9.57 ± 12% fio.latency_2ms% 0.01 +0.2 0.24 ± 20% fio.latency_4ms% 17.73 ± 8% +15.0 32.70 ± 12% fio.latency_750us% 0.27 ± 4% -0.1 0.14 ± 6% fio.latency_>=2000ms% 385.62 +8.7% 419.34 fio.time.elapsed_time 385.62 +8.7% 419.34 fio.time.elapsed_time.max 99365986 +6.9% 1.063e+08 fio.time.file_system_outputs 1945 ± 3% +69.2% 3291 ± 4% fio.time.involuntary_context_switches 34.75 +53.2% 53.25 ± 3% fio.time.percent_of_cpu_this_job_got 114.87 +74.3% 200.26 ± 4% fio.time.system_time 21.53 ± 2% +18.9% 25.60 ± 2% fio.time.user_time 263007 ± 5% -25.9% 194806 ± 8% fio.time.voluntary_context_switches 12420747 +6.9% 13282337 fio.workload 148.65 -15.4% 125.72 fio.write_bw_MBps 57147392 -17.4% 47185920 ± 2% fio.write_clat_90%_us 61341696 ± 2% -11.3% 54394880 ± 5% fio.write_clat_95%_us 1.277e+08 +27.7% 1.631e+08 ± 9% fio.write_clat_99%_us 35406116 -43.2% 20095509 fio.write_clat_mean_us 1.267e+08 +22.5% 1.552e+08 ± 2% fio.write_clat_stddev 38053 -15.4% 32184 fio.write_iops 1140007 -43.3% 646192 fio.write_slat_mean_us 22923702 +20.1% 27528884 ± 2% fio.write_slat_stddev 41.91 -29.4% 29.59 ± 2% iostat.cpu.idle 57.60 +20.4% 69.36 iostat.cpu.iowait 15846790 ± 15% +62.9% 25808727 ± 13% cpuidle.C1E.usage 13758474 ± 27% -59.3% 5598154 ± 65% cpuidle.C3.usage 5.555e+09 ± 11% -19.6% 4.468e+09 ± 7% cpuidle.C6.time 152936 ± 63% -78.2% 33344 ± 61% numa-numastat.node0.numa_miss 159740 ± 62% -73.2% 42830 ± 47% numa-numastat.node0.other_node 152936 ± 63% -78.2% 33344 ± 61% numa-numastat.node1.numa_foreign 41.62 -12.4 29.26 ± 2% mpstat.cpu.all.idle% 57.90 +11.8 69.69 mpstat.cpu.all.iowait% 0.38 +0.5 0.90 ± 4% mpstat.cpu.all.sys% 0.08 ± 2% +0.0 0.13 mpstat.cpu.all.usr% 41.50 -30.1% 29.00 ± 3% vmstat.cpu.id 56.75 +21.6% 69.00 vmstat.cpu.wa 128261 -1.6% 126247 vmstat.io.bo 32638075 +32.1% 43113546 ± 2% vmstat.memory.cache 32546493 -32.2% 22066885 ± 4% vmstat.memory.free 31.50 +22.2% 38.50 vmstat.procs.b 3411 ± 4% -17.2% 2825 ± 6% vmstat.system.cs 15846466 ± 15% +62.9% 25808277 ± 13% turbostat.C1E 13758155 ± 27% -59.3% 5597714 ± 65% turbostat.C3 29.88 ± 11% -7.8 22.10 ± 6% turbostat.C6% 13.68 ± 20% -60.8% 5.37 ± 50% turbostat.CPU%c6 74.95 ± 2% +4.1% 78.05 ± 2% turbostat.CorWatt 11.18 ± 25% -71.8% 3.16 ±102% turbostat.Pkg%pc2 0.10 ± 47% -85.4% 0.01 ±173% turbostat.Pkg%pc3 0.07 ± 48% -82.1% 0.01 ±173% turbostat.Pkg%pc6 103.68 ± 2% +3.0% 106.79 turbostat.PkgWatt 7.50 +1.1% 7.57 turbostat.RAMWatt 31797979 +32.0% 41987949 ± 2% meminfo.Cached 2246330 +12.6% 2530086 meminfo.Committed_AS 7112704 ± 6% +4.0% 7398400 ± 7% meminfo.DirectMap2M 9273585 +99.7% 18516564 meminfo.Dirty 30704128 +33.2% 40895517 ± 2% meminfo.Inactive 362019 +15.6% 418524 meminfo.Inactive(anon) 30342109 +33.4% 40476992 ± 2% meminfo.Inactive(file) 919250 +30.3% 1197987 meminfo.KReclaimable 372616 +15.3% 429563 meminfo.Mapped 32467447 -32.3% 21995088 ± 4% meminfo.MemFree 33378851 +31.4% 43851210 ± 2% meminfo.Memused 919250 +30.3% 1197987 meminfo.SReclaimable 367488 +15.5% 424291 meminfo.Shmem 1044343 +26.8% 1324225 meminfo.Slab 263713 +174.6% 724109 meminfo.Writeback 206.00 ± 2% +18.7% 244.50 ± 12% interrupts.42:PCI-MSI.2621448-edge.eth0-TxRx-7 481.50 ± 12% -29.5% 339.50 ± 29% interrupts.CPU11.NMI:Non-maskable_interrupts 481.50 ± 12% -29.5% 339.50 ± 29% interrupts.CPU11.PMI:Performance_monitoring_interrupts 431.75 ± 33% -42.4% 248.50 ± 24% interrupts.CPU12.NMI:Non-maskable_interrupts 431.75 ± 33% -42.4% 248.50 ± 24% interrupts.CPU12.PMI:Performance_monitoring_interrupts 4925 ± 11% +15.8% 5704 ± 7% interrupts.CPU13.CAL:Function_call_interrupts 546.50 ± 29% -62.4% 205.25 ± 59% interrupts.CPU13.RES:Rescheduling_interrupts 4560 ± 6% +31.1% 5978 ± 19% interrupts.CPU19.CAL:Function_call_interrupts 407.50 ± 60% -81.2% 76.50 ± 40% interrupts.CPU19.RES:Rescheduling_interrupts 288.50 ± 45% -77.2% 65.75 ± 22% interrupts.CPU21.RES:Rescheduling_interrupts 157.25 ± 43% -61.4% 60.75 ± 70% interrupts.CPU29.RES:Rescheduling_interrupts 206.00 ± 2% +18.7% 244.50 ± 12% interrupts.CPU31.42:PCI-MSI.2621448-edge.eth0-TxRx-7 4469 ± 7% +39.0% 6210 ± 22% interrupts.CPU36.CAL:Function_call_interrupts 494.75 ± 65% -72.0% 138.50 ± 40% interrupts.CPU36.RES:Rescheduling_interrupts 4316 ± 7% +22.2% 5274 ± 10% interrupts.CPU40.CAL:Function_call_interrupts 270.00 ± 44% -63.8% 97.75 ± 87% interrupts.CPU40.RES:Rescheduling_interrupts 495.50 ± 39% -79.3% 102.75 ± 61% interrupts.CPU45.RES:Rescheduling_interrupts 16584 ± 11% -30.8% 11482 ± 16% interrupts.RES:Rescheduling_interrupts 4618717 ± 3% +101.4% 9302547 numa-meminfo.node0.Dirty 16354663 ± 3% +28.5% 21008472 ± 2% numa-meminfo.node0.FilePages 15811819 ± 3% +29.5% 20470740 ± 2% numa-meminfo.node0.Inactive 15710516 ± 4% +29.6% 20360981 numa-meminfo.node0.Inactive(file) 469767 ± 3% +27.1% 596863 ± 2% numa-meminfo.node0.KReclaimable 15710137 ± 4% -30.5% 10924214 ± 5% numa-meminfo.node0.MemFree 17155509 ± 3% +27.9% 21941432 ± 2% numa-meminfo.node0.MemUsed 469767 ± 3% +27.1% 596863 ± 2% numa-meminfo.node0.SReclaimable 536435 ± 3% +23.4% 661977 ± 2% numa-meminfo.node0.Slab 127732 ± 2% +173.0% 348765 ± 4% numa-meminfo.node0.Writeback 4655423 ± 3% +98.3% 9231636 ± 3% numa-meminfo.node1.Dirty 15447018 ± 3% +36.1% 21022313 ± 2% numa-meminfo.node1.FilePages 14892867 ± 3% +37.4% 20464806 ± 3% numa-meminfo.node1.Inactive 14632190 ± 4% +37.7% 20155562 ± 3% numa-meminfo.node1.Inactive(file) 449292 ± 3% +34.0% 602274 ± 2% numa-meminfo.node1.KReclaimable 16756430 ± 3% -34.2% 11029554 ± 5% numa-meminfo.node1.MemFree 16224220 ± 3% +35.3% 21951097 ± 2% numa-meminfo.node1.MemUsed 449292 ± 3% +34.0% 602274 ± 2% numa-meminfo.node1.SReclaimable 507722 ± 3% +30.7% 663404 numa-meminfo.node1.Slab 135362 ± 5% +178.0% 376303 ± 3% numa-meminfo.node1.Writeback 4444421 ± 3% +25.2% 5562521 ± 3% numa-vmstat.node0.nr_dirtied 1155371 ± 3% +101.2% 2324533 numa-vmstat.node0.nr_dirty 4090377 ± 3% +28.4% 5250862 ± 2% numa-vmstat.node0.nr_file_pages 3925740 ± 4% -30.4% 2732344 ± 5% numa-vmstat.node0.nr_free_pages 3929385 ± 4% +29.5% 5088978 numa-vmstat.node0.nr_inactive_file 117498 ± 3% +26.9% 149157 ± 2% numa-vmstat.node0.nr_slab_reclaimable 31827 ± 2% +174.6% 87400 ± 4% numa-vmstat.node0.nr_writeback 3929385 ± 4% +29.5% 5088978 numa-vmstat.node0.nr_zone_inactive_file 1187493 ± 3% +103.1% 2412048 numa-vmstat.node0.nr_zone_write_pending 5087678 ± 5% +23.9% 6305813 ± 6% numa-vmstat.node0.numa_hit 5080612 ± 5% +23.9% 6295655 ± 6% numa-vmstat.node0.numa_local 146231 ± 62% -77.6% 32778 ± 61% numa-vmstat.node0.numa_miss 4434100 ± 3% +39.2% 6170063 ± 4% numa-vmstat.node1.nr_dirtied 1164610 ± 3% +98.1% 2307168 ± 3% numa-vmstat.node1.nr_dirty 3863740 ± 3% +36.0% 5254625 ± 2% numa-vmstat.node1.nr_file_pages 4187032 ± 3% -34.1% 2758350 ± 5% numa-vmstat.node1.nr_free_pages 3660023 ± 4% +37.6% 5037997 ± 3% numa-vmstat.node1.nr_inactive_file 112388 ± 3% +33.9% 150514 ± 2% numa-vmstat.node1.nr_slab_reclaimable 33693 ± 4% +179.1% 94036 ± 3% numa-vmstat.node1.nr_writeback 3660023 ± 4% +37.6% 5037997 ± 3% numa-vmstat.node1.nr_zone_inactive_file 1198596 ± 3% +100.3% 2401291 ± 3% numa-vmstat.node1.nr_zone_write_pending 146232 ± 62% -77.6% 32778 ± 61% numa-vmstat.node1.numa_foreign 5176677 ± 5% +37.1% 7096463 ± 6% numa-vmstat.node1.numa_hit 5011522 ± 6% +38.4% 6934030 ± 6% numa-vmstat.node1.numa_local 444.00 ± 3% +25.4% 556.75 ± 5% slabinfo.biovec-128.active_objs 444.00 ± 3% +25.4% 556.75 ± 5% slabinfo.biovec-128.num_objs 734.25 ± 4% +45.8% 1070 ± 3% slabinfo.biovec-64.active_objs 734.25 ± 4% +45.8% 1070 ± 3% slabinfo.biovec-64.num_objs 295.50 ± 18% +68.0% 496.50 ± 12% slabinfo.biovec-max.active_objs 319.25 ± 16% +63.0% 520.25 ± 13% slabinfo.biovec-max.num_objs 700.00 ± 8% +56.5% 1095 ± 2% slabinfo.blkdev_ioc.active_objs 700.00 ± 8% +56.5% 1095 ± 2% slabinfo.blkdev_ioc.num_objs 7592132 +33.4% 10126644 ± 2% slabinfo.buffer_head.active_objs 196264 +32.6% 260282 ± 2% slabinfo.buffer_head.active_slabs 7654320 +32.6% 10151024 ± 2% slabinfo.buffer_head.num_objs 196264 +32.6% 260282 ± 2% slabinfo.buffer_head.num_slabs 2158 ± 12% -14.3% 1849 ± 14% slabinfo.eventpoll_pwq.active_objs 2158 ± 12% -14.3% 1849 ± 14% slabinfo.eventpoll_pwq.num_objs 1211 ± 9% +20.1% 1454 ± 5% slabinfo.ext4_extent_status.active_objs 1211 ± 9% +20.1% 1454 ± 5% slabinfo.ext4_extent_status.num_objs 884.75 ± 8% +199.0% 2645 ± 17% slabinfo.ext4_io_end.active_objs 884.75 ± 8% +199.0% 2645 ± 17% slabinfo.ext4_io_end.num_objs 2314 ± 2% +14.8% 2655 ± 3% slabinfo.pool_workqueue.active_objs 2314 ± 2% +14.8% 2655 ± 2% slabinfo.pool_workqueue.num_objs 136721 +29.6% 177228 slabinfo.radix_tree_node.active_objs 2502 +27.6% 3193 slabinfo.radix_tree_node.active_slabs 140165 +27.6% 178868 slabinfo.radix_tree_node.num_objs 2502 +27.6% 3193 slabinfo.radix_tree_node.num_slabs 1024 +51.6% 1552 slabinfo.scsi_sense_cache.active_objs 1024 +51.6% 1552 slabinfo.scsi_sense_cache.num_objs 1761 ± 3% +29.5% 2280 ± 5% slabinfo.skbuff_fclone_cache.active_objs 1761 ± 3% +29.5% 2280 ± 5% slabinfo.skbuff_fclone_cache.num_objs 12421560 +6.9% 13283012 proc-vmstat.nr_dirtied 2318632 +99.7% 4631102 proc-vmstat.nr_dirty 1559352 +99.2% 3105629 proc-vmstat.nr_dirty_background_threshold 3122517 +148.7% 7765972 proc-vmstat.nr_dirty_threshold 7951217 +32.1% 10502150 ± 2% proc-vmstat.nr_file_pages 8115739 -32.3% 5494098 ± 4% proc-vmstat.nr_free_pages 90484 +15.6% 104626 proc-vmstat.nr_inactive_anon 7586540 +33.4% 10123741 ± 2% proc-vmstat.nr_inactive_file 11900 +2.3% 12177 proc-vmstat.nr_kernel_stack 93248 +15.3% 107512 proc-vmstat.nr_mapped 2085 +6.9% 2229 proc-vmstat.nr_page_table_pages 91851 +15.5% 106068 proc-vmstat.nr_shmem 229797 +30.4% 299628 ± 2% proc-vmstat.nr_slab_reclaimable 65830 +175.3% 181251 proc-vmstat.nr_writeback 12421560 +6.9% 13282996 proc-vmstat.nr_written 90484 +15.6% 104626 proc-vmstat.nr_zone_inactive_anon 7586540 +33.4% 10123741 ± 2% proc-vmstat.nr_zone_inactive_file 2385078 +101.8% 4812521 proc-vmstat.nr_zone_write_pending 226534 ± 44% -85.3% 33344 ± 61% proc-vmstat.numa_foreign 2658 ± 3% -32.5% 1794 ± 12% proc-vmstat.numa_hint_faults 1355 ± 4% -60.2% 539.25 ± 26% proc-vmstat.numa_hint_faults_local 13464638 +8.4% 14600391 proc-vmstat.numa_hit 13448439 +8.4% 14584391 proc-vmstat.numa_local 226534 ± 44% -85.3% 33344 ± 61% proc-vmstat.numa_miss 242733 ± 41% -79.7% 49344 ± 41% proc-vmstat.numa_other 1685 ± 20% -25.6% 1253 ± 7% proc-vmstat.numa_pages_migrated 4928 ± 57% -47.5% 2587 ± 7% proc-vmstat.numa_pte_updates 13778418 +6.9% 14722437 proc-vmstat.pgalloc_normal 1091426 +7.7% 1175859 proc-vmstat.pgfault 13552030 ± 2% +7.4% 14558129 proc-vmstat.pgfree 1685 ± 20% -25.6% 1253 ± 7% proc-vmstat.pgmigrate_success 49829011 +6.9% 53275846 proc-vmstat.pgpgout 204046 ± 14% +167.4% 545721 ± 7% proc-vmstat.pgrotated 34.16 ± 3% -30.4% 23.79 ± 16% perf-stat.i.MPKI 2.032e+08 +7.9% 2.193e+08 perf-stat.i.branch-instructions 4.66 ± 2% -1.0 3.67 ± 3% perf-stat.i.branch-miss-rate% 7209528 -22.0% 5624742 ± 2% perf-stat.i.branch-misses 5217888 -14.9% 4442724 ± 3% perf-stat.i.cache-misses 28886731 ± 2% -29.2% 20439061 ± 14% perf-stat.i.cache-references 3402 ± 4% -18.9% 2761 ± 5% perf-stat.i.context-switches 4.76 ± 2% -19.1% 3.85 ± 9% perf-stat.i.cpi 8.66 ± 7% -24.5% 6.54 ± 3% perf-stat.i.cpu-migrations 918.81 ± 2% +146.1% 2261 ± 7% perf-stat.i.cycles-between-cache-misses 0.90 ± 3% -0.3 0.63 ± 6% perf-stat.i.dTLB-load-miss-rate% 2588584 ± 3% -12.5% 2266173 ± 6% perf-stat.i.dTLB-load-misses 0.15 ± 10% -0.0 0.12 ± 5% perf-stat.i.dTLB-store-miss-rate% 454795 ± 6% -18.0% 373105 ± 2% perf-stat.i.dTLB-store-misses 2.861e+08 ± 3% -17.4% 2.364e+08 ± 7% perf-stat.i.dTLB-stores 9.973e+08 +6.3% 1.06e+09 perf-stat.i.instructions 1073 ± 3% -9.0% 977.08 ± 2% perf-stat.i.instructions-per-iTLB-miss 0.24 +31.4% 0.32 ± 13% perf-stat.i.ipc 310076 +8.4% 335970 ± 5% perf-stat.i.node-loads 13.69 ± 7% +5.3 18.96 ± 10% perf-stat.i.node-store-miss-rate% 287029 ± 5% +23.8% 355408 ± 5% perf-stat.i.node-store-misses 28.96 ± 2% -33.4% 19.28 ± 14% perf-stat.overall.MPKI 3.55 -1.0 2.56 ± 3% perf-stat.overall.branch-miss-rate% 4.22 ± 2% -11.5% 3.74 ± 7% perf-stat.overall.cpi 0.90 ± 3% -0.1 0.77 ± 6% perf-stat.overall.dTLB-load-miss-rate% 11.82 ± 4% +2.7 14.51 ± 4% perf-stat.overall.node-store-miss-rate% 30943 +8.0% 33430 perf-stat.overall.path-length 2.027e+08 +8.0% 2.189e+08 perf-stat.ps.branch-instructions 7191170 -22.0% 5612177 ± 2% perf-stat.ps.branch-misses 5204770 -14.8% 4433929 ± 3% perf-stat.ps.cache-misses 28812682 ± 2% -29.2% 20395260 ± 14% perf-stat.ps.cache-references 3393 ± 4% -18.8% 2754 ± 5% perf-stat.ps.context-switches 8.64 ± 7% -24.5% 6.52 ± 3% perf-stat.ps.cpu-migrations 2582032 ± 3% -12.4% 2261755 ± 6% perf-stat.ps.dTLB-load-misses 453645 ± 6% -17.9% 372354 ± 2% perf-stat.ps.dTLB-store-misses 2.854e+08 ± 3% -17.3% 2.359e+08 ± 7% perf-stat.ps.dTLB-stores 9.948e+08 +6.3% 1.058e+09 perf-stat.ps.instructions 309286 +8.4% 335263 ± 5% perf-stat.ps.node-loads 286345 ± 5% +23.9% 354808 ± 5% perf-stat.ps.node-store-misses 3.843e+11 +15.5% 4.44e+11 perf-stat.total.instructions 92834 +44.6% 134241 ± 10% softirqs.CPU0.RCU 96210 +41.3% 135902 ± 9% softirqs.CPU1.RCU 102417 ± 5% +33.3% 136522 ± 2% softirqs.CPU10.RCU 102855 ± 2% +31.0% 134702 ± 8% softirqs.CPU11.RCU 90216 +38.4% 124897 ± 8% softirqs.CPU12.RCU 48053 ± 3% +8.4% 52087 softirqs.CPU12.SCHED 93157 +35.3% 126033 ± 6% softirqs.CPU13.RCU 91743 +33.9% 122809 ± 4% softirqs.CPU14.RCU 90784 +35.5% 122992 ± 6% softirqs.CPU15.RCU 93452 ± 2% +38.8% 129716 ± 5% softirqs.CPU16.RCU 95886 ± 2% +39.3% 133576 ± 6% softirqs.CPU17.RCU 99936 ± 2% +30.6% 130521 ± 5% softirqs.CPU18.RCU 102974 +30.2% 134121 ± 3% softirqs.CPU19.RCU 95928 +40.8% 135098 ± 7% softirqs.CPU2.RCU 100739 ± 2% +25.3% 126223 ± 4% softirqs.CPU20.RCU 107688 +22.8% 132213 ± 4% softirqs.CPU21.RCU 105205 +24.1% 130520 ± 3% softirqs.CPU22.RCU 102666 ± 2% +29.7% 133203 ± 4% softirqs.CPU23.RCU 91275 +37.9% 125857 ± 8% softirqs.CPU24.RCU 96242 +28.1% 123267 ± 9% softirqs.CPU25.RCU 96994 +38.5% 134298 ± 8% softirqs.CPU26.RCU 96537 ± 4% +40.6% 135744 ± 7% softirqs.CPU27.RCU 97235 ± 2% +31.9% 128261 ± 5% softirqs.CPU28.RCU 99328 +32.5% 131564 ± 5% softirqs.CPU29.RCU 101675 ± 3% +34.7% 136996 ± 6% softirqs.CPU3.RCU 97250 +35.0% 131258 ± 5% softirqs.CPU30.RCU 95333 +39.4% 132933 ± 6% softirqs.CPU31.RCU 92225 ± 2% +36.6% 125995 ± 6% softirqs.CPU32.RCU 94777 +32.4% 125465 ± 7% softirqs.CPU33.RCU 98071 +31.1% 128524 ± 4% softirqs.CPU34.RCU 96363 ± 2% +34.5% 129585 ± 4% softirqs.CPU35.RCU 91442 +48.4% 135696 ± 9% softirqs.CPU36.RCU 47007 ± 3% +7.7% 50640 ± 2% softirqs.CPU36.SCHED 95833 +43.4% 137424 ± 8% softirqs.CPU37.RCU 96444 +40.0% 135045 ± 7% softirqs.CPU38.RCU 97483 +38.5% 134993 ± 7% softirqs.CPU39.RCU 96291 +31.0% 126101 ± 7% softirqs.CPU4.RCU 99450 +38.4% 137663 ± 6% softirqs.CPU40.RCU 98131 ± 2% +41.9% 139295 ± 8% softirqs.CPU41.RCU 102150 +35.6% 138513 ± 4% softirqs.CPU42.RCU 104087 +36.5% 142062 ± 4% softirqs.CPU43.RCU 100013 +31.4% 131430 ± 4% softirqs.CPU44.RCU 104323 ± 3% +30.3% 135896 ± 5% softirqs.CPU45.RCU 105295 +26.6% 133313 ± 4% softirqs.CPU46.RCU 100966 ± 2% +28.3% 129533 ± 10% softirqs.CPU47.RCU 99327 +30.3% 129414 ± 5% softirqs.CPU5.RCU 99025 +35.7% 134399 ± 4% softirqs.CPU6.RCU 99818 +35.1% 134895 ± 6% softirqs.CPU7.RCU 102828 +33.0% 136710 ± 5% softirqs.CPU8.RCU 103045 +36.6% 140801 ± 4% softirqs.CPU9.RCU 4713974 +34.6% 6346250 ± 5% softirqs.RCU 1898 ± 4% +110.6% 3999 ± 3% sched_debug.cfs_rq:/.exec_clock.avg 2456 ± 5% +96.3% 4821 ± 3% sched_debug.cfs_rq:/.exec_clock.max 1618 ± 5% +124.6% 3633 ± 4% sched_debug.cfs_rq:/.exec_clock.min 179.43 ± 8% +48.0% 265.47 ± 17% sched_debug.cfs_rq:/.exec_clock.stddev 11645 ± 6% +18.8% 13834 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg 0.19 ± 43% +93.5% 0.37 ± 18% sched_debug.cfs_rq:/.nr_spread_over.avg 1.29 ± 42% +75.0% 2.25 ± 8% sched_debug.cfs_rq:/.nr_spread_over.max 0.30 ± 43% +62.2% 0.48 ± 10% sched_debug.cfs_rq:/.nr_spread_over.stddev 15.03 ± 29% -50.5% 7.44 ± 43% sched_debug.cfs_rq:/.removed.load_avg.avg 279.77 ± 40% -52.6% 132.57 ± 5% sched_debug.cfs_rq:/.removed.load_avg.max 58.84 ± 19% -49.7% 29.58 ± 17% sched_debug.cfs_rq:/.removed.load_avg.stddev 694.48 ± 29% -50.7% 342.42 ± 43% sched_debug.cfs_rq:/.removed.runnable_sum.avg 12952 ± 40% -52.8% 6108 ± 5% sched_debug.cfs_rq:/.removed.runnable_sum.max 2719 ± 19% -49.9% 1362 ± 17% sched_debug.cfs_rq:/.removed.runnable_sum.stddev 5.45 ± 21% -55.3% 2.43 ± 57% sched_debug.cfs_rq:/.removed.util_avg.avg 124.68 ± 34% -59.9% 50.03 ± 18% sched_debug.cfs_rq:/.removed.util_avg.max 22.75 ± 22% -57.8% 9.60 ± 30% sched_debug.cfs_rq:/.removed.util_avg.stddev 111.20 ± 7% -17.7% 91.50 ± 4% sched_debug.cfs_rq:/.util_avg.avg 910639 -49.4% 460497 sched_debug.cpu.avg_idle.avg 1000000 -50.0% 500000 sched_debug.cpu.avg_idle.max 503719 ± 13% -68.0% 160964 ± 19% sched_debug.cpu.avg_idle.min 102546 ± 11% -35.8% 65864 ± 7% sched_debug.cpu.avg_idle.stddev 198884 ± 7% +22.4% 243507 ± 5% sched_debug.cpu.clock.avg 198886 ± 7% +22.4% 243509 ± 5% sched_debug.cpu.clock.max 198881 ± 7% +22.4% 243505 ± 5% sched_debug.cpu.clock.min 1.36 ± 4% -12.0% 1.20 ± 3% sched_debug.cpu.clock.stddev 198884 ± 7% +22.4% 243507 ± 5% sched_debug.cpu.clock_task.avg 198886 ± 7% +22.4% 243509 ± 5% sched_debug.cpu.clock_task.max 198881 ± 7% +22.4% 243505 ± 5% sched_debug.cpu.clock_task.min 1.36 ± 4% -12.0% 1.20 ± 3% sched_debug.cpu.clock_task.stddev 5811 ± 5% +19.5% 6946 ± 4% sched_debug.cpu.curr->pid.max 944.59 ± 3% +14.6% 1082 ± 4% sched_debug.cpu.curr->pid.stddev 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.avg 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.max 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 0.00 ± 6% -14.1% 0.00 ± 10% sched_debug.cpu.next_balance.stddev 0.09 ± 4% -13.7% 0.08 ± 3% sched_debug.cpu.nr_running.avg 0.26 ± 3% -8.6% 0.24 sched_debug.cpu.nr_running.stddev 12984 ± 6% +8.4% 14071 ± 5% sched_debug.cpu.nr_switches.avg 11083 ± 7% +9.6% 12148 ± 5% sched_debug.cpu.sched_count.avg 5450 ± 9% -9.3% 4943 ± 2% sched_debug.cpu.sched_count.min 5478 ± 17% +25.6% 6880 ± 17% sched_debug.cpu.sched_count.stddev 2681 ± 9% -11.1% 2384 sched_debug.cpu.sched_goidle.min 2647 ± 18% +28.0% 3388 ± 16% sched_debug.cpu.sched_goidle.stddev 5499 ± 7% +10.1% 6054 ± 5% sched_debug.cpu.ttwu_count.avg 2298 ± 10% -18.6% 1869 ± 9% sched_debug.cpu.ttwu_local.min 198882 ± 7% +22.4% 243505 ± 5% sched_debug.cpu_clk 196302 ± 7% +22.7% 240926 ± 5% sched_debug.ktime 199298 ± 7% +22.4% 243922 ± 5% sched_debug.sched_clk 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 7.26 ± 6% -1.8 5.42 ± 19% perf-profile.calltrace.cycles-pp.io_submit 6.61 ± 5% -1.7 4.92 ± 20% perf-profile.calltrace.cycles-pp.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 6.48 ± 5% -1.7 4.81 ± 21% perf-profile.calltrace.cycles-pp.io_submit_one.__x64_sys_io_submit.do_syscall_64.entry_SYSCALL_64_after_hwframe.io_submit 1.21 ± 12% -0.4 0.80 ± 7% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.aio_write 1.19 ± 13% -0.4 0.79 ± 6% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter 1.17 ± 13% -0.4 0.78 ± 6% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter 0.64 ± 5% -0.4 0.27 ±100% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write 1.18 ± 7% -0.3 0.90 ± 15% perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter 1.16 ± 6% -0.3 0.89 ± 15% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter 0.97 ± 9% -0.1 0.83 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.62 ± 12% +0.2 0.83 ± 8% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt 0.90 ± 4% +0.3 1.17 ± 9% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 0.66 ± 6% +0.4 1.05 ± 14% perf-profile.calltrace.cycles-pp.rcu_core.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 2.60 ± 3% +0.5 3.07 ± 11% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 1.63 ± 6% +0.5 2.10 ± 11% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 1.16 ± 2% +0.5 1.64 ± 6% perf-profile.calltrace.cycles-pp.memcpy_erms.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread 1.23 +0.5 1.75 ± 4% perf-profile.calltrace.cycles-pp.drm_fb_helper_dirty_work.process_one_work.worker_thread.kthread.ret_from_fork 1.66 ± 6% +0.6 2.25 ± 8% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 1.73 ± 6% +0.6 2.33 ± 7% perf-profile.calltrace.cycles-pp.ret_from_fork 1.73 ± 6% +0.6 2.33 ± 7% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 1.64 ± 6% +0.6 2.23 ± 7% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 3.33 ± 3% +0.6 3.94 ± 11% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.26 ±100% +0.7 0.93 ± 25% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.14 ±173% +0.7 0.89 ± 26% perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 6.29 ± 3% +0.8 7.06 ± 7% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 3.51 ± 2% +1.1 4.62 ± 8% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 18.08 ± 3% +1.6 19.64 ± 2% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 15.80 ± 2% +1.6 17.41 ± 2% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 4.50 +1.6 6.11 ± 11% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 7.30 ± 6% -1.9 5.44 ± 19% perf-profile.children.cycles-pp.io_submit 8.53 ± 6% -1.8 6.72 ± 16% perf-profile.children.cycles-pp.do_syscall_64 8.55 ± 6% -1.8 6.76 ± 16% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.61 ± 5% -1.7 4.93 ± 20% perf-profile.children.cycles-pp.__x64_sys_io_submit 6.49 ± 5% -1.7 4.81 ± 21% perf-profile.children.cycles-pp.io_submit_one 1.21 ± 12% -0.4 0.81 ± 7% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 1.19 ± 13% -0.4 0.79 ± 6% perf-profile.children.cycles-pp.copyin 1.20 ± 13% -0.4 0.80 ± 7% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 1.18 ± 7% -0.3 0.90 ± 15% perf-profile.children.cycles-pp.grab_cache_page_write_begin 1.17 ± 7% -0.3 0.89 ± 14% perf-profile.children.cycles-pp.pagecache_get_page 0.64 ± 5% -0.2 0.47 ± 16% perf-profile.children.cycles-pp.add_to_page_cache_lru 0.47 ± 11% -0.2 0.31 ± 10% perf-profile.children.cycles-pp.ext4_block_write_begin 0.52 ± 10% -0.2 0.36 ± 12% perf-profile.children.cycles-pp.generic_write_end 0.46 ± 10% -0.1 0.33 ± 11% perf-profile.children.cycles-pp.block_write_end 0.45 ± 12% -0.1 0.32 ± 10% perf-profile.children.cycles-pp.__block_commit_write 0.31 ± 15% -0.1 0.19 ± 20% perf-profile.children.cycles-pp.mark_buffer_dirty 0.29 ± 4% -0.1 0.17 ± 6% perf-profile.children.cycles-pp.kmem_cache_alloc 0.37 ± 9% -0.1 0.26 ± 12% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.39 ± 8% -0.1 0.29 ± 25% perf-profile.children.cycles-pp.call_timer_fn 0.40 ± 8% -0.1 0.30 ± 8% perf-profile.children.cycles-pp.__add_to_page_cache_locked 0.26 ± 17% -0.1 0.17 ± 15% perf-profile.children.cycles-pp.create_empty_buffers 0.21 ± 13% -0.1 0.13 ± 19% perf-profile.children.cycles-pp.xas_load 0.35 ± 3% -0.1 0.27 ± 13% perf-profile.children.cycles-pp.entry_SYSCALL_64 0.21 ± 21% -0.1 0.12 ± 14% perf-profile.children.cycles-pp.alloc_page_buffers 0.23 ± 20% -0.1 0.15 ± 21% perf-profile.children.cycles-pp.__set_page_dirty 0.42 ± 12% -0.1 0.35 ± 12% perf-profile.children.cycles-pp.do_io_getevents 0.18 ± 10% -0.1 0.11 ± 18% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited 0.26 ± 10% -0.1 0.20 ± 13% perf-profile.children.cycles-pp.__schedule 0.16 ± 19% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.alloc_buffer_head 0.14 ± 19% -0.1 0.08 ± 27% perf-profile.children.cycles-pp.schedule_timeout 0.19 ± 11% -0.1 0.14 ± 15% perf-profile.children.cycles-pp.rb_insert_color 0.15 ± 7% -0.1 0.10 ± 21% perf-profile.children.cycles-pp.fio_gettime 0.17 ± 6% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.ext4_da_get_block_prep 0.13 ± 8% -0.1 0.08 ± 23% perf-profile.children.cycles-pp.find_get_entry 0.19 ± 15% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.stack_trace_save_tsk 0.18 ± 13% -0.0 0.14 ± 6% perf-profile.children.cycles-pp.arch_stack_walk 0.15 ± 10% -0.0 0.10 ± 17% perf-profile.children.cycles-pp.td_io_queue 0.08 ± 13% -0.0 0.05 ± 60% perf-profile.children.cycles-pp.ext4_es_lookup_extent 0.10 ± 14% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.mem_cgroup_commit_charge 0.09 ± 16% +0.0 0.12 ± 14% perf-profile.children.cycles-pp.cpumask_next_and 0.11 ± 10% +0.0 0.15 ± 10% perf-profile.children.cycles-pp.update_rt_rq_load_avg 0.06 ± 60% +0.0 0.10 ± 15% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.12 ± 16% +0.0 0.17 ± 16% perf-profile.children.cycles-pp.perf_event_task_tick 0.03 ±100% +0.1 0.08 ± 10% perf-profile.children.cycles-pp.cpuidle_select 0.16 ± 6% +0.1 0.21 ± 9% perf-profile.children.cycles-pp.tick_nohz_tick_stopped 0.16 ± 7% +0.1 0.24 ± 12% perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit 0.79 ± 2% +0.1 0.88 ± 5% perf-profile.children.cycles-pp.find_next_bit 0.29 ± 10% +0.1 0.39 ± 12% perf-profile.children.cycles-pp.rcu_idle_exit 0.12 ± 8% +0.1 0.22 ± 5% perf-profile.children.cycles-pp.rcu_eqs_exit 0.55 ± 5% +0.1 0.65 ± 13% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.78 ± 9% +0.1 0.89 ± 6% perf-profile.children.cycles-pp.io_serial_in 0.36 ± 18% +0.1 0.48 ± 9% perf-profile.children.cycles-pp.update_sd_lb_stats 0.41 ± 18% +0.2 0.56 ± 10% perf-profile.children.cycles-pp.find_busiest_group 0.49 ± 2% +0.2 0.65 ± 24% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 1.95 ± 2% +0.2 2.14 ± 10% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.65 ± 11% +0.2 0.87 ± 8% perf-profile.children.cycles-pp.load_balance 0.33 ± 9% +0.2 0.57 ± 12% perf-profile.children.cycles-pp.note_gp_changes 0.93 ± 4% +0.3 1.22 ± 10% perf-profile.children.cycles-pp.rebalance_domains 0.69 ± 6% +0.4 1.09 ± 14% perf-profile.children.cycles-pp.rcu_core 0.49 ± 18% +0.4 0.93 ± 26% perf-profile.children.cycles-pp.tick_nohz_irq_exit 1.67 ± 7% +0.5 2.16 ± 12% perf-profile.children.cycles-pp.clockevents_program_event 1.22 +0.5 1.74 ± 5% perf-profile.children.cycles-pp.memcpy_erms 1.23 +0.5 1.75 ± 4% perf-profile.children.cycles-pp.drm_fb_helper_dirty_work 2.69 ± 2% +0.5 3.21 ± 10% perf-profile.children.cycles-pp.tick_nohz_next_event 1.74 ± 6% +0.6 2.33 ± 7% perf-profile.children.cycles-pp.ret_from_fork 1.66 ± 6% +0.6 2.25 ± 8% perf-profile.children.cycles-pp.worker_thread 1.73 ± 6% +0.6 2.33 ± 7% perf-profile.children.cycles-pp.kthread 1.64 ± 6% +0.6 2.23 ± 7% perf-profile.children.cycles-pp.process_one_work 1.91 ± 5% +0.7 2.59 ± 12% perf-profile.children.cycles-pp.ktime_get 3.40 ± 2% +0.7 4.08 ± 11% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 6.46 ± 2% +0.8 7.30 ± 6% perf-profile.children.cycles-pp.menu_select 3.82 ± 4% +1.3 5.12 ± 8% perf-profile.children.cycles-pp.__softirqentry_text_start 16.23 ± 2% +1.7 17.91 ± 2% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 17.52 ± 2% +1.7 19.26 ± 2% perf-profile.children.cycles-pp.apic_timer_interrupt 4.85 ± 2% +1.8 6.67 ± 10% perf-profile.children.cycles-pp.irq_exit 1.19 ± 14% -0.4 0.80 ± 7% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.37 ± 9% -0.1 0.26 ± 13% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.14 ± 13% -0.1 0.07 ± 61% perf-profile.self.cycles-pp.kmem_cache_alloc 0.33 ± 4% -0.1 0.26 ± 15% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.25 ± 5% -0.1 0.17 ± 16% perf-profile.self.cycles-pp.smp_apic_timer_interrupt 0.28 ± 8% -0.1 0.21 ± 11% perf-profile.self.cycles-pp.tick_sched_timer 0.16 ± 17% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.xas_load 0.14 ± 8% -0.0 0.10 ± 17% perf-profile.self.cycles-pp.fio_gettime 0.13 ± 14% -0.0 0.09 ± 13% perf-profile.self.cycles-pp.io_submit_one 0.17 ± 10% -0.0 0.12 ± 16% perf-profile.self.cycles-pp.rb_insert_color 0.09 ± 8% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.update_rt_rq_load_avg 0.34 +0.0 0.38 perf-profile.self.cycles-pp.__hrtimer_run_queues 0.03 ±100% +0.0 0.07 ± 12% perf-profile.self.cycles-pp.cpuidle_select 0.12 ± 17% +0.0 0.17 ± 16% perf-profile.self.cycles-pp.perf_event_task_tick 0.10 ± 17% +0.1 0.16 ± 13% perf-profile.self.cycles-pp.tick_nohz_tick_stopped 0.12 ± 5% +0.1 0.18 ± 11% perf-profile.self.cycles-pp.rebalance_domains 0.01 ±173% +0.1 0.08 ± 31% perf-profile.self.cycles-pp.rcu_eqs_exit 0.16 ± 6% +0.1 0.24 ± 12% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.70 ± 5% +0.1 0.78 ± 6% perf-profile.self.cycles-pp.find_next_bit 0.22 ± 13% +0.1 0.32 ± 13% perf-profile.self.cycles-pp.update_sd_lb_stats 0.78 ± 9% +0.1 0.89 ± 6% perf-profile.self.cycles-pp.io_serial_in 0.43 ± 7% +0.1 0.56 ± 13% perf-profile.self.cycles-pp.tick_nohz_next_event 0.47 ± 11% +0.1 0.62 ± 2% perf-profile.self.cycles-pp.do_idle 0.18 ± 24% +0.2 0.33 ± 9% perf-profile.self.cycles-pp.note_gp_changes 1.13 ± 7% +0.5 1.63 ± 5% perf-profile.self.cycles-pp.memcpy_erms *************************************************************************************************** lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_pmem/nr_task/priority/rootfs/tbox_group/test/testcase/thp_defrag/thp_enabled/ucode: gcc-7/performance/x86_64-rhel-7.6/4/64/1/debian-x86_64-2019-11-14.cgz/lkp-hsw-4ex1/swap-w-seq/vm-scalability/never/never/0x16 commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_interrupt_entry/0x 4:4 -100% :4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x 3:4 -75% :4 dmesg.WARNING:at_ip__slab_free/0x %stddev %change %stddev \ | \ 17.61 ± 16% -76.0% 4.22 ± 2% vm-scalability.free_time 62939 -93.8% 3918 ± 2% vm-scalability.median 12.50 ± 10% -69.4% 3.82 ± 13% vm-scalability.stddev 4156059 ± 2% -93.9% 255064 vm-scalability.throughput 150.71 ± 2% +103.9% 307.32 vm-scalability.time.elapsed_time 150.71 ± 2% +103.9% 307.32 vm-scalability.time.elapsed_time.max 1794938 ± 9% -82.1% 320896 ± 11% vm-scalability.time.involuntary_context_switches 1450485 ± 17% -66.4% 487715 ± 8% vm-scalability.time.maximum_resident_set_size 98488869 -93.0% 6896039 ± 2% vm-scalability.time.minor_page_faults 4923 ± 2% -44.5% 2732 ± 9% vm-scalability.time.percent_of_cpu_this_job_got 6767 ± 4% +23.5% 8355 ± 9% vm-scalability.time.system_time 655.28 -93.1% 44.97 vm-scalability.time.user_time 66324 ± 11% -70.9% 19284 ± 10% vm-scalability.time.voluntary_context_switches 4.432e+08 -82.7% 76779117 vm-scalability.workload 3.19 ± 3% -81.2% 0.60 ± 16% iostat.cpu.user 7.042e+09 ± 51% +203.6% 2.138e+10 ± 36% cpuidle.C3.time 16070353 ± 54% +204.0% 48850208 ± 25% cpuidle.C3.usage 0.00 ±142% +0.0 0.01 ± 71% mpstat.cpu.all.iowait% 0.51 ± 11% -0.5 0.00 ± 26% mpstat.cpu.all.soft% 36.56 +8.7 45.28 ± 10% mpstat.cpu.all.sys% 3.24 ± 3% -2.7 0.50 ± 26% mpstat.cpu.all.usr% 4.909e+09 ± 2% -63.3% 1.803e+09 ± 2% perf-node.node-load-misses 1.304e+09 ± 3% -91.2% 1.145e+08 ± 12% perf-node.node-loads 20.50 ± 4% -73.2% 5.50 ± 15% perf-node.node-local-load-ratio 49.75 -60.3% 19.75 ± 2% perf-node.node-local-store-ratio 1.767e+09 ± 2% -59.6% 7.135e+08 perf-node.node-store-misses 1.775e+09 ± 2% -89.8% 1.813e+08 ± 2% perf-node.node-stores 104.25 ± 14% -40.3% 62.25 ± 20% vmstat.memory.buff 1312560 -11.3% 1164717 vmstat.memory.cache 1.548e+08 ± 2% -95.1% 7649606 ± 21% vmstat.memory.swpd 912.25 ± 2% -91.4% 78.75 ± 40% vmstat.swap.si 2399897 -98.2% 42592 ± 31% vmstat.swap.so 28854 ± 7% -59.3% 11746 ± 33% vmstat.system.cs 511.50 -26.1% 378.25 ± 7% turbostat.Avg_MHz 42.73 -11.1 31.61 ± 7% turbostat.Busy% 16066383 ± 54% +204.2% 48875909 ± 25% turbostat.C3 1.991e+08 ± 2% -53.9% 91725606 turbostat.IRQ 0.36 ± 24% -51.7% 0.17 ± 16% turbostat.Pkg%pc2 189.06 -8.1% 173.81 turbostat.PkgWatt 83.86 -30.9% 57.93 turbostat.RAMWatt 17963146 -38.3% 11083518 ± 3% meminfo.Active 17963065 -38.3% 11083226 ± 3% meminfo.Active(anon) 588.00 ± 28% +392.4% 2895 ± 31% meminfo.AnonHugePages 20605635 -40.3% 12298605 ± 3% meminfo.AnonPages 2650666 -24.4% 2003821 ± 5% meminfo.Inactive 2650493 -24.4% 2003364 ± 5% meminfo.Inactive(anon) 260010 ± 4% -56.8% 112427 ± 4% meminfo.KReclaimable 22096 +16.9% 25838 meminfo.Mapped 1283021 ± 6% +203.6% 3895652 ± 14% meminfo.MemAvailable 1396082 ± 5% +192.4% 4081951 ± 14% meminfo.MemFree 31384484 -8.6% 28698616 ± 2% meminfo.Memused 386048 ± 2% -87.5% 48385 ± 2% meminfo.PageTables 260010 ± 4% -56.8% 112427 ± 4% meminfo.SReclaimable 226846 +71.7% 389488 ± 2% meminfo.SUnreclaim 3399 ± 11% +212.9% 10639 ± 42% meminfo.Shmem 240.25 ± 9% +3e+05% 721703 ± 8% meminfo.SwapCached 3.485e+08 +42.0% 4.948e+08 meminfo.SwapFree 222522 ± 2% +382.2% 1072922 ± 20% meminfo.max_used_kB 88227 +1491.9% 1404480 ± 4% slabinfo.Acpi-Parse.active_objs 1210 +1601.3% 20598 ± 3% slabinfo.Acpi-Parse.active_slabs 88412 +1600.8% 1503700 ± 3% slabinfo.Acpi-Parse.num_objs 1210 +1601.3% 20598 ± 3% slabinfo.Acpi-Parse.num_slabs 18609 ± 3% +9.8% 20440 ± 4% slabinfo.kmalloc-512.num_objs 19763 ± 6% -37.3% 12398 ± 3% slabinfo.proc_inode_cache.active_objs 574.25 ± 13% -50.9% 282.00 ± 3% slabinfo.proc_inode_cache.active_slabs 20306 ± 5% -37.9% 12617 ± 2% slabinfo.proc_inode_cache.num_objs 574.25 ± 13% -50.9% 282.00 ± 3% slabinfo.proc_inode_cache.num_slabs 314685 ± 6% -73.2% 84365 ± 5% slabinfo.radix_tree_node.active_objs 13466 ± 14% -88.5% 1550 ± 6% slabinfo.radix_tree_node.active_slabs 315881 ± 6% -72.5% 86829 ± 6% slabinfo.radix_tree_node.num_objs 13466 ± 14% -88.5% 1550 ± 6% slabinfo.radix_tree_node.num_slabs 3888 +39.9% 5440 slabinfo.scsi_sense_cache.active_objs 3888 +39.9% 5440 slabinfo.scsi_sense_cache.num_objs 13264 +10013.3% 1341508 ± 5% slabinfo.vmap_area.active_objs 209.50 +10620.0% 22458 ± 3% slabinfo.vmap_area.active_slabs 13453 +10583.9% 1437385 ± 3% slabinfo.vmap_area.num_objs 209.50 +10620.0% 22458 ± 3% slabinfo.vmap_area.num_slabs 17577672 ± 14% -85.5% 2555856 ± 2% numa-numastat.node0.local_node 18174681 ± 25% -92.3% 1393014 ± 43% numa-numastat.node0.numa_foreign 17606911 ± 14% -85.4% 2573908 ± 2% numa-numastat.node0.numa_hit 4897187 ± 17% -97.1% 139796 ± 37% numa-numastat.node0.numa_miss 4926429 ± 17% -96.8% 157870 ± 40% numa-numastat.node0.other_node 16881406 ± 20% -87.7% 2076300 ± 17% numa-numastat.node1.local_node 8681777 ± 66% -89.9% 877168 ± 55% numa-numastat.node1.numa_foreign 16903128 ± 20% -87.6% 2092922 ± 17% numa-numastat.node1.numa_hit 12011542 ± 16% -93.9% 727801 ± 53% numa-numastat.node1.numa_miss 12033265 ± 16% -93.8% 744454 ± 50% numa-numastat.node1.other_node 16971463 ± 19% -88.4% 1969502 ± 25% numa-numastat.node2.local_node 5917967 ± 46% -89.2% 636924 ± 67% numa-numastat.node2.numa_foreign 16995319 ± 19% -88.3% 1996383 ± 25% numa-numastat.node2.numa_hit 10651608 ± 34% -91.6% 892057 ± 55% numa-numastat.node2.numa_miss 10675466 ± 34% -91.4% 918981 ± 54% numa-numastat.node2.other_node 18415256 ± 24% -91.7% 1532521 ± 22% numa-numastat.node3.local_node 4966169 ± 66% -96.5% 173950 ± 42% numa-numastat.node3.numa_foreign 18435690 ± 24% -91.5% 1559777 ± 21% numa-numastat.node3.numa_hit 10180257 ± 41% -87.0% 1321403 ± 22% numa-numastat.node3.numa_miss 10200691 ± 41% -86.8% 1348671 ± 22% numa-numastat.node3.other_node 90104 ± 6% -93.7% 5696 ± 48% syscalls.sys_close.max 1398 ± 2% -24.6% 1054 ± 2% syscalls.sys_close.med 50733844 ± 6% -4.4e+07 6692932 ± 34% syscalls.sys_close.noise.100% 70188309 ± 4% -5.4e+07 16449674 ± 10% syscalls.sys_close.noise.2% 68032137 ± 4% -5.5e+07 12974683 ± 11% syscalls.sys_close.noise.25% 70087310 ± 4% -5.4e+07 16286916 ± 10% syscalls.sys_close.noise.5% 63199296 ± 5% -5.3e+07 9760150 ± 20% syscalls.sys_close.noise.50% 56784706 ± 5% -4.9e+07 7965791 ± 28% syscalls.sys_close.noise.75% 192606 ± 43% -100.0% 0.00 syscalls.sys_mmap.max 9408 ± 2% -100.0% 0.00 syscalls.sys_mmap.med 5526 ± 2% -100.0% 0.00 syscalls.sys_mmap.min 2.557e+08 ± 8% -2.6e+08 0.00 syscalls.sys_mmap.noise.100% 3.674e+08 ± 6% -3.7e+08 0.00 syscalls.sys_mmap.noise.2% 3.481e+08 ± 6% -3.5e+08 0.00 syscalls.sys_mmap.noise.25% 3.665e+08 ± 6% -3.7e+08 0.00 syscalls.sys_mmap.noise.5% 3.179e+08 ± 7% -3.2e+08 0.00 syscalls.sys_mmap.noise.50% 2.879e+08 ± 7% -2.9e+08 0.00 syscalls.sys_mmap.noise.75% 2.494e+10 -100.0% 107756 ± 13% syscalls.sys_open.max 12591 +88.5% 23735 ± 17% syscalls.sys_open.med 7793 +10.5% 8608 syscalls.sys_open.min 3.789e+12 ± 8% -3.8e+12 3.455e+08 ± 28% syscalls.sys_open.noise.100% 3.789e+12 ± 8% -3.8e+12 5.935e+08 ± 20% syscalls.sys_open.noise.2% 3.789e+12 ± 8% -3.8e+12 5.224e+08 ± 23% syscalls.sys_open.noise.25% 3.789e+12 ± 8% -3.8e+12 5.907e+08 ± 20% syscalls.sys_open.noise.5% 3.789e+12 ± 8% -3.8e+12 4.383e+08 ± 27% syscalls.sys_open.noise.50% 3.789e+12 ± 8% -3.8e+12 3.752e+08 ± 28% syscalls.sys_open.noise.75% 5.619e+09 ± 25% -100.0% 618791 ± 46% syscalls.sys_read.max 11896 ± 7% -82.9% 2039 ± 5% syscalls.sys_read.med 1.374e+12 ± 6% -1.4e+12 2.054e+09 ± 6% syscalls.sys_read.noise.100% 1.374e+12 ± 6% -1.4e+12 2.064e+09 ± 6% syscalls.sys_read.noise.2% 1.374e+12 ± 6% -1.4e+12 2.063e+09 ± 6% syscalls.sys_read.noise.25% 1.374e+12 ± 6% -1.4e+12 2.064e+09 ± 6% syscalls.sys_read.noise.5% 1.374e+12 ± 6% -1.4e+12 2.062e+09 ± 6% syscalls.sys_read.noise.50% 1.374e+12 ± 6% -1.4e+12 2.06e+09 ± 6% syscalls.sys_read.noise.75% 1.289e+08 ± 51% -100.0% 0.00 syscalls.sys_write.max 5989 ± 41% -100.0% 0.00 syscalls.sys_write.med 2097 -100.0% 0.00 syscalls.sys_write.min 1.333e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.100% 1.338e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.2% 1.338e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.25% 1.338e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.5% 1.334e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.50% 1.333e+10 ± 32% -1.3e+10 0.00 syscalls.sys_write.noise.75% 51.75 ± 2% -38.3 13.48 ±173% perf-profile.calltrace.cycles-pp.do_access 48.42 ± 2% -35.0 13.46 ±173% perf-profile.calltrace.cycles-pp.page_fault.do_access 47.80 ± 3% -34.3 13.46 ±173% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.do_access 47.71 ± 3% -34.3 13.46 ±173% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.do_access 47.30 ± 3% -33.8 13.46 ±173% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.do_access 44.88 ± 3% -31.0 13.87 ±173% perf-profile.calltrace.cycles-pp.alloc_pages_vma.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault 41.74 ± 5% -27.9 13.84 ±173% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault.handle_mm_fault.__do_page_fault 40.70 ± 5% -26.9 13.84 ±173% perf-profile.calltrace.cycles-pp.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault.handle_mm_fault 37.04 ± 2% -23.1 13.90 ±173% perf-profile.calltrace.cycles-pp.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma.__handle_mm_fault 36.71 ± 3% -22.6 14.11 ±173% perf-profile.calltrace.cycles-pp.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask.alloc_pages_vma 36.52 ± 3% -21.8 14.75 ±173% perf-profile.calltrace.cycles-pp.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath.__alloc_pages_nodemask 34.80 ± 3% -20.1 14.75 ±173% perf-profile.calltrace.cycles-pp.shrink_node_memcg.shrink_node.do_try_to_free_pages.try_to_free_pages.__alloc_pages_slowpath 31.06 ± 4% -16.3 14.74 ±173% perf-profile.calltrace.cycles-pp.shrink_inactive_list.shrink_node_memcg.shrink_node.do_try_to_free_pages.try_to_free_pages 28.98 ± 4% -14.2 14.74 ±173% perf-profile.calltrace.cycles-pp.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node.do_try_to_free_pages 9.80 ± 13% -9.8 0.00 perf-profile.calltrace.cycles-pp.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node 8.68 ± 7% -8.7 0.00 perf-profile.calltrace.cycles-pp.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node 8.66 ± 7% -8.7 0.00 perf-profile.calltrace.cycles-pp.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list.shrink_node_memcg 8.56 ± 7% -8.6 0.00 perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list.shrink_inactive_list 8.14 ± 8% -8.1 0.00 perf-profile.calltrace.cycles-pp.on_each_cpu_mask.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty.shrink_page_list 7.99 ± 8% -8.0 0.00 perf-profile.calltrace.cycles-pp.smp_call_function_single.on_each_cpu_mask.on_each_cpu_cond_mask.arch_tlbbatch_flush.try_to_unmap_flush_dirty 8.87 ± 3% -7.8 1.02 ±173% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 8.87 ± 3% -7.8 1.03 ±173% perf-profile.calltrace.cycles-pp.ret_from_fork 7.32 ± 19% -7.3 0.00 perf-profile.calltrace.cycles-pp.add_to_swap_cache.add_to_swap.shrink_page_list.shrink_inactive_list.shrink_node_memcg 5.74 ± 10% -5.7 0.00 perf-profile.calltrace.cycles-pp.__remove_mapping.shrink_page_list.shrink_inactive_list.shrink_node_memcg.shrink_node 52.40 ± 2% -38.9 13.48 ±173% perf-profile.children.cycles-pp.do_access 45.11 ± 3% -30.8 14.36 ±173% perf-profile.children.cycles-pp.alloc_pages_vma 45.97 ± 2% -30.1 15.84 ±173% perf-profile.children.cycles-pp.__alloc_pages_nodemask 44.33 ± 2% -28.5 15.84 ±173% perf-profile.children.cycles-pp.__alloc_pages_slowpath 40.91 ± 3% -24.2 16.68 ±173% perf-profile.children.cycles-pp.shrink_node 39.17 ± 4% -22.5 16.67 ±173% perf-profile.children.cycles-pp.shrink_node_memcg 37.69 ± 3% -21.9 15.82 ±173% perf-profile.children.cycles-pp.try_to_free_pages 37.37 ± 3% -21.5 15.82 ±173% perf-profile.children.cycles-pp.do_try_to_free_pages 35.23 ± 4% -18.6 16.66 ±173% perf-profile.children.cycles-pp.shrink_inactive_list 32.95 ± 4% -16.3 16.66 ±173% perf-profile.children.cycles-pp.shrink_page_list 9.95 ± 13% -9.9 0.00 perf-profile.children.cycles-pp.add_to_swap 8.78 ± 7% -8.8 0.03 ±173% perf-profile.children.cycles-pp.try_to_unmap_flush_dirty 8.77 ± 7% -8.7 0.03 ±173% perf-profile.children.cycles-pp.arch_tlbbatch_flush 8.70 ± 7% -8.7 0.03 ±173% perf-profile.children.cycles-pp.on_each_cpu_cond_mask 8.27 ± 8% -8.2 0.02 ±173% perf-profile.children.cycles-pp.on_each_cpu_mask 8.21 ± 8% -8.2 0.03 ±173% perf-profile.children.cycles-pp.smp_call_function_single 8.87 ± 3% -7.8 1.02 ±173% perf-profile.children.cycles-pp.kthread 8.87 ± 4% -7.8 1.04 ±173% perf-profile.children.cycles-pp.ret_from_fork 7.67 ± 19% -7.7 0.00 perf-profile.children.cycles-pp.add_to_swap_cache 7.61 ± 12% -7.5 0.10 ±173% perf-profile.children.cycles-pp.get_page_from_freelist 6.04 ± 10% -6.0 0.01 ±173% perf-profile.children.cycles-pp.__remove_mapping 10.11 ± 12% +6.3 16.39 ±173% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 5.84 ± 11% +10.7 16.50 ±173% perf-profile.children.cycles-pp._raw_spin_lock 7.46 ± 9% -7.4 0.02 ±173% perf-profile.self.cycles-pp.smp_call_function_single 10.10 ± 12% +6.1 16.18 ±173% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 4329827 -46.0% 2337051 ± 10% numa-meminfo.node0.Active 4329808 -46.0% 2336843 ± 10% numa-meminfo.node0.Active(anon) 4992078 -48.7% 2562281 ± 12% numa-meminfo.node0.AnonPages 260335 ± 3% +68.8% 439568 ± 5% numa-meminfo.node0.FilePages 666055 -37.5% 416131 ± 11% numa-meminfo.node0.Inactive 666013 -37.5% 415958 ± 11% numa-meminfo.node0.Inactive(anon) 74249 ± 14% -57.2% 31763 ± 10% numa-meminfo.node0.KReclaimable 6334 ± 10% +15.6% 7323 ± 13% numa-meminfo.node0.KernelStack 5819 ± 3% +75.9% 10238 numa-meminfo.node0.Mapped 356516 ± 8% +235.6% 1196383 ± 20% numa-meminfo.node0.MemFree 7706498 -10.9% 6866632 ± 3% numa-meminfo.node0.MemUsed 80605 ± 13% -85.6% 11598 ± 9% numa-meminfo.node0.PageTables 74249 ± 14% -57.2% 31763 ± 10% numa-meminfo.node0.SReclaimable 68688 ± 6% +69.5% 116397 ± 11% numa-meminfo.node0.SUnreclaim 1417 ± 57% +403.3% 7131 ± 14% numa-meminfo.node0.Shmem 4589192 ± 3% -43.6% 2589650 ± 7% numa-meminfo.node1.Active 4589182 ± 3% -43.6% 2589589 ± 7% numa-meminfo.node1.Active(anon) 5250722 ± 2% -45.5% 2861734 ± 8% numa-meminfo.node1.AnonPages 260329 ± 2% +60.6% 418001 ± 2% numa-meminfo.node1.FilePages 663023 ± 3% -32.3% 449130 ± 14% numa-meminfo.node1.Inactive 662957 ± 3% -32.3% 448937 ± 14% numa-meminfo.node1.Inactive(anon) 61247 ± 17% -57.3% 26140 ± 6% numa-meminfo.node1.KReclaimable 344774 ± 15% +292.5% 1353279 ± 21% numa-meminfo.node1.MemFree 7904793 -12.8% 6896287 ± 4% numa-meminfo.node1.MemUsed 103272 ± 9% -88.7% 11650 ± 6% numa-meminfo.node1.PageTables 61247 ± 17% -57.3% 26140 ± 6% numa-meminfo.node1.SReclaimable 52333 ± 20% +84.2% 96422 ± 11% numa-meminfo.node1.SUnreclaim 4515529 ± 4% -39.4% 2735238 ± 7% numa-meminfo.node2.Active 4515494 ± 4% -39.4% 2735218 ± 7% numa-meminfo.node2.Active(anon) 5183552 ± 3% -41.5% 3030983 ± 7% numa-meminfo.node2.AnonPages 262131 ± 3% +61.1% 422260 ± 8% numa-meminfo.node2.FilePages 669419 -28.4% 479415 ± 10% numa-meminfo.node2.Inactive 669381 -28.4% 479276 ± 10% numa-meminfo.node2.Inactive(anon) 61005 ± 12% -51.6% 29551 ± 12% numa-meminfo.node2.KReclaimable 323022 ± 9% +331.1% 1392530 ± 23% numa-meminfo.node2.MemFree 7926544 -13.5% 6857036 ± 4% numa-meminfo.node2.MemUsed 98827 -88.6% 11310 ± 8% numa-meminfo.node2.PageTables 61005 ± 12% -51.6% 29551 ± 12% numa-meminfo.node2.SReclaimable 52987 ± 27% +59.3% 84423 ± 7% numa-meminfo.node2.SUnreclaim 4560396 ± 4% -39.2% 2774139 ± 3% numa-meminfo.node3.Active 4560379 ± 4% -39.2% 2774119 ± 3% numa-meminfo.node3.Active(anon) 5220719 ± 3% -40.3% 3115767 ± 4% numa-meminfo.node3.AnonPages 267193 ± 4% +51.2% 403915 ± 5% numa-meminfo.node3.FilePages 661103 ± 2% -26.1% 488255 ± 11% numa-meminfo.node3.Inactive 661080 ± 2% -26.1% 488229 ± 11% numa-meminfo.node3.Inactive(anon) 59354 ± 11% -59.2% 24210 ± 13% numa-meminfo.node3.KReclaimable 336489 ± 5% +300.8% 1348753 ± 20% numa-meminfo.node3.MemFree 7881925 -12.8% 6869661 ± 3% numa-meminfo.node3.MemUsed 103777 ± 12% -89.7% 10718 ± 3% numa-meminfo.node3.PageTables 59354 ± 11% -59.2% 24210 ± 13% numa-meminfo.node3.SReclaimable 52814 ± 23% +55.5% 82151 ± 4% numa-meminfo.node3.SUnreclaim 53595 ± 4% +108.5% 111746 ± 17% sched_debug.cfs_rq:/.exec_clock.max 44554 ± 14% +49.7% 66682 ± 19% sched_debug.cfs_rq:/.load.avg 1563170 ± 3% -33.1% 1045091 ± 38% sched_debug.cfs_rq:/.min_vruntime.avg 3177061 -39.6% 1918710 ± 40% sched_debug.cfs_rq:/.min_vruntime.max 957373 ± 6% -66.3% 323068 ± 46% sched_debug.cfs_rq:/.min_vruntime.stddev 4.83 ± 34% +10503.0% 512.62 ± 41% sched_debug.cfs_rq:/.nr_spread_over.avg 28.08 ± 31% +3211.6% 930.00 ± 43% sched_debug.cfs_rq:/.nr_spread_over.max 5.45 ± 37% +2701.1% 152.72 ± 51% sched_debug.cfs_rq:/.nr_spread_over.stddev 44503 ± 14% +49.8% 66656 ± 19% sched_debug.cfs_rq:/.runnable_weight.avg -1150416 -96.9% -35376 sched_debug.cfs_rq:/.spread0.avg -2466210 -69.4% -754726 sched_debug.cfs_rq:/.spread0.min 957970 ± 6% -64.3% 342227 ± 46% sched_debug.cfs_rq:/.spread0.stddev 878053 -42.5% 504620 sched_debug.cpu.avg_idle.avg 1381357 ± 38% -50.5% 684086 ± 2% sched_debug.cpu.avg_idle.max 215030 ± 16% -58.4% 89380 ± 6% sched_debug.cpu.avg_idle.stddev 115976 +62.2% 188124 ± 13% sched_debug.cpu.clock.avg 116043 +65.1% 191587 ± 13% sched_debug.cpu.clock.max 115899 +58.6% 183767 ± 13% sched_debug.cpu.clock.min 43.31 ± 17% +5110.4% 2256 ± 50% sched_debug.cpu.clock.stddev 115976 +62.2% 188124 ± 13% sched_debug.cpu.clock_task.avg 116043 +65.1% 191587 ± 13% sched_debug.cpu.clock_task.max 115899 +58.6% 183767 ± 13% sched_debug.cpu.clock_task.min 43.31 ± 17% +5110.4% 2256 ± 50% sched_debug.cpu.clock_task.stddev 4115 ± 6% -28.4% 2945 ± 3% sched_debug.cpu.curr->pid.max 1214 -14.8% 1034 ± 4% sched_debug.cpu.curr->pid.stddev 503244 -45.6% 273574 sched_debug.cpu.max_idle_balance_cost.avg 800322 ± 53% -53.7% 370523 sched_debug.cpu.max_idle_balance_cost.max 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 0.00 ± 13% +4112.8% 0.00 ± 50% sched_debug.cpu.next_balance.stddev 16443 ± 8% -67.6% 5324 ± 20% sched_debug.cpu.nr_switches.avg 4039 ± 4% -40.2% 2416 ± 28% sched_debug.cpu.nr_switches.min 9116 ± 7% -65.3% 3160 ± 16% sched_debug.cpu.nr_switches.stddev 0.01 ± 19% +1266.7% 0.07 ± 63% sched_debug.cpu.nr_uninterruptible.avg 16.33 ± 6% -13.8% 14.07 ± 3% sched_debug.cpu.nr_uninterruptible.stddev 14501 ± 10% -76.4% 3415 ± 33% sched_debug.cpu.sched_count.avg 2846 ± 9% -50.1% 1419 ± 45% sched_debug.cpu.sched_count.min 8516 ± 7% -71.2% 2452 ± 25% sched_debug.cpu.sched_count.stddev 680.34 ± 3% +50.7% 1025 ± 28% sched_debug.cpu.sched_goidle.avg 4040 ± 11% +204.9% 12321 ± 37% sched_debug.cpu.sched_goidle.max 537.76 ± 11% +127.4% 1222 ± 27% sched_debug.cpu.sched_goidle.stddev 7325 ± 10% -77.1% 1677 ± 33% sched_debug.cpu.ttwu_count.avg 1661 ± 16% -64.8% 585.62 ± 50% sched_debug.cpu.ttwu_count.min 3741 ± 11% -67.4% 1220 ± 23% sched_debug.cpu.ttwu_count.stddev 2912 ± 9% -83.9% 468.94 ± 28% sched_debug.cpu.ttwu_local.avg 500.08 ± 9% -80.4% 98.04 ± 44% sched_debug.cpu.ttwu_local.min 1750 ± 8% -46.9% 929.33 ± 31% sched_debug.cpu.ttwu_local.stddev 115899 +58.6% 183767 ± 13% sched_debug.cpu_clk 111590 +60.8% 179456 ± 13% sched_debug.ktime 119393 +57.6% 188219 ± 12% sched_debug.sched_clk 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 9.49 ± 3% -61.5% 3.65 ± 54% perf-stat.i.MPKI 45240326 ± 4% -56.2% 19807119 ± 37% perf-stat.i.branch-misses 22.42 -5.3 17.15 ± 20% perf-stat.i.cache-miss-rate% 66478385 -78.2% 14465455 ± 47% perf-stat.i.cache-misses 2.954e+08 -63.8% 1.071e+08 ± 64% perf-stat.i.cache-references 29848 ± 8% -51.2% 14551 ± 16% perf-stat.i.context-switches 2.53 +46.1% 3.70 perf-stat.i.cpi 297.34 ± 6% +116.7% 644.32 ± 27% perf-stat.i.cpu-migrations 1337 ± 6% +508.0% 8129 ± 13% perf-stat.i.cycles-between-cache-misses 0.43 ± 11% -0.3 0.10 ± 39% perf-stat.i.dTLB-load-miss-rate% 38712590 ± 9% -80.9% 7390481 ± 62% perf-stat.i.dTLB-load-misses 0.18 -0.0 0.14 ± 14% perf-stat.i.dTLB-store-miss-rate% 7054225 -75.7% 1713274 ± 65% perf-stat.i.dTLB-store-misses 3.646e+09 -70.8% 1.063e+09 ± 69% perf-stat.i.dTLB-stores 74.52 ± 2% -45.2 29.29 ± 38% perf-stat.i.iTLB-load-miss-rate% 5296601 ± 2% -78.1% 1158896 ± 78% perf-stat.i.iTLB-load-misses 7242 ± 6% +837.7% 67911 ± 48% perf-stat.i.instructions-per-iTLB-miss 0.41 -33.3% 0.28 perf-stat.i.ipc 140.45 ± 14% -78.8% 29.77 ± 68% perf-stat.i.major-faults 666959 ± 2% -91.1% 59313 ± 5% perf-stat.i.minor-faults 79.72 +14.2 93.89 perf-stat.i.node-load-miss-rate% 32758157 ± 2% -72.0% 9162090 ± 47% perf-stat.i.node-load-misses 8807005 -93.5% 573675 ± 40% perf-stat.i.node-loads 54.33 +20.7 75.00 perf-stat.i.node-store-miss-rate% 11853895 ± 3% -71.2% 3414343 ± 49% perf-stat.i.node-store-misses 12055174 -91.1% 1070993 ± 37% perf-stat.i.node-stores 667419 ± 2% -91.0% 59826 ± 5% perf-stat.i.page-faults 9.40 -47.1% 4.97 ± 10% perf-stat.overall.MPKI 0.60 ± 5% -0.2 0.41 ± 30% perf-stat.overall.branch-miss-rate% 22.52 -9.8 12.69 ± 13% perf-stat.overall.cache-miss-rate% 2.35 +47.1% 3.46 perf-stat.overall.cpi 1112 ± 2% +403.4% 5599 ± 8% perf-stat.overall.cycles-between-cache-misses 0.47 ± 10% -0.3 0.14 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.19 -0.0 0.15 ± 12% perf-stat.overall.dTLB-store-miss-rate% 78.80 ± 2% -35.7 43.10 ± 18% perf-stat.overall.iTLB-load-miss-rate% 5952 +216.2% 18818 ± 14% perf-stat.overall.instructions-per-iTLB-miss 0.42 -32.1% 0.29 perf-stat.overall.ipc 78.82 +15.3 94.11 perf-stat.overall.node-load-miss-rate% 49.72 ± 2% +28.9 78.60 perf-stat.overall.node-store-miss-rate% 10467 ± 2% +491.5% 61917 ± 7% perf-stat.overall.path-length 7.485e+09 -50.5% 3.704e+09 ± 9% perf-stat.ps.branch-instructions 44621026 ± 4% -66.6% 14920712 ± 31% perf-stat.ps.branch-misses 65271966 -85.3% 9594206 perf-stat.ps.cache-misses 2.898e+08 -73.5% 76820933 ± 12% perf-stat.ps.cache-references 29229 ± 7% -79.0% 6150 ± 7% perf-stat.ps.context-switches 7.259e+10 -26.0% 5.372e+10 ± 8% perf-stat.ps.cpu-cycles 291.59 ± 5% -21.1% 230.19 ± 21% perf-stat.ps.cpu-migrations 37840178 ± 9% -86.1% 5274649 ± 11% perf-stat.ps.dTLB-load-misses 8.076e+09 -53.8% 3.728e+09 ± 8% perf-stat.ps.dTLB-loads 6908013 -83.3% 1152725 ± 15% perf-stat.ps.dTLB-store-misses 3.574e+09 -78.3% 7.744e+08 ± 8% perf-stat.ps.dTLB-stores 5181335 -83.7% 845503 ± 18% perf-stat.ps.iTLB-load-misses 1395935 ± 9% -20.1% 1115727 ± 13% perf-stat.ps.iTLB-loads 3.084e+10 -49.7% 1.551e+10 ± 8% perf-stat.ps.instructions 137.26 ± 14% -90.7% 12.74 ± 49% perf-stat.ps.major-faults 652076 ± 2% -96.8% 20654 ± 5% perf-stat.ps.minor-faults 32148538 ± 2% -81.3% 6004642 perf-stat.ps.node-load-misses 8631275 ± 2% -95.6% 376483 ± 10% perf-stat.ps.node-loads 11688310 ± 3% -80.1% 2329722 ± 4% perf-stat.ps.node-store-misses 11813097 -94.6% 634518 ± 5% perf-stat.ps.node-stores 652546 ± 2% -96.8% 20835 ± 5% perf-stat.ps.page-faults 853471 -97.6% 20576 ± 15% proc-vmstat.allocstall_movable 6909 ± 15% -38.6% 4244 ± 18% proc-vmstat.allocstall_normal 3133959 ± 35% -97.1% 89457 ± 56% proc-vmstat.compact_daemon_free_scanned 818614 ± 26% -84.0% 130740 ±168% proc-vmstat.compact_daemon_migrate_scanned 582.25 ±130% -97.9% 12.25 ± 27% proc-vmstat.compact_daemon_wake 2999 ± 38% -76.8% 694.50 ± 40% proc-vmstat.compact_fail 3305553 ± 34% -97.3% 89457 ± 56% proc-vmstat.compact_free_scanned 238885 ± 23% -96.5% 8428 ± 43% proc-vmstat.compact_isolated 1630990 ± 20% -92.0% 130740 ±168% proc-vmstat.compact_migrate_scanned 3016 ± 37% -77.0% 695.25 ± 40% proc-vmstat.compact_stall 167.25 ±106% -84.6% 25.75 ± 36% proc-vmstat.kswapd_high_wmark_hit_quickly 877.75 ±104% -96.1% 34.50 ± 47% proc-vmstat.kswapd_low_wmark_hit_quickly 4498214 -40.6% 2670391 proc-vmstat.nr_active_anon 19.75 ± 6% +283.5% 75.75 ±102% proc-vmstat.nr_active_file 5161207 -42.6% 2960450 proc-vmstat.nr_anon_pages 27874 ± 4% +716.9% 227716 ± 9% proc-vmstat.nr_dirty_background_threshold 55819 ± 4% +920.1% 569430 ± 9% proc-vmstat.nr_dirty_threshold 262480 +64.1% 430753 ± 3% proc-vmstat.nr_file_pages 339879 ± 3% +252.9% 1199399 ± 8% proc-vmstat.nr_free_pages 664876 -28.6% 475030 ± 5% proc-vmstat.nr_inactive_anon 42.75 ± 6% +193.6% 125.50 ± 24% proc-vmstat.nr_inactive_file 809.75 ± 2% +43.4% 1161 ± 13% proc-vmstat.nr_isolated_anon 5658 +19.8% 6780 ± 5% proc-vmstat.nr_mapped 96944 ± 3% -88.0% 11672 ± 2% proc-vmstat.nr_page_table_pages 845.75 ± 13% +273.3% 3157 ± 59% proc-vmstat.nr_shmem 64214 ± 3% -56.3% 28085 ± 4% proc-vmstat.nr_slab_reclaimable 56710 +69.3% 96026 ± 2% proc-vmstat.nr_slab_unreclaimable 58066797 -95.7% 2482316 ± 5% proc-vmstat.nr_vmscan_write 92058231 -95.0% 4637520 ± 6% proc-vmstat.nr_written 4492236 -40.6% 2667270 proc-vmstat.nr_zone_active_anon 19.25 ± 4% +296.1% 76.25 ±100% proc-vmstat.nr_zone_active_file 666758 -28.9% 474019 ± 5% proc-vmstat.nr_zone_inactive_anon 42.25 ± 5% +191.1% 123.00 ± 24% proc-vmstat.nr_zone_inactive_file 37740595 ± 7% -91.8% 3081058 ± 10% proc-vmstat.numa_foreign 4216 ± 61% +322.7% 17823 ± 32% proc-vmstat.numa_hint_faults 68.25 ± 45% +10235.2% 7053 ± 62% proc-vmstat.numa_hint_faults_local 69969181 ± 2% -88.2% 8255598 ± 3% proc-vmstat.numa_hit 69873924 ± 2% -88.3% 8165013 ± 3% proc-vmstat.numa_local 37740595 ± 7% -91.8% 3081058 ± 10% proc-vmstat.numa_miss 37835852 ± 7% -91.6% 3171643 ± 9% proc-vmstat.numa_other 39446530 ± 11% -87.1% 5073809 ± 6% proc-vmstat.numa_pte_updates 1050 ±104% -93.8% 65.00 ± 19% proc-vmstat.pageoutrun 66731 ± 7% -70.9% 19390 ± 23% proc-vmstat.pgactivate 749996 ± 17% -99.3% 5564 ± 28% proc-vmstat.pgalloc_dma 8660763 ± 7% -94.2% 506250 ± 4% proc-vmstat.pgalloc_dma32 1.195e+08 ± 2% -90.8% 10970916 ± 2% proc-vmstat.pgalloc_normal 93121825 -95.9% 3848016 ± 4% proc-vmstat.pgdeactivate 99039595 -92.9% 7036256 ± 3% proc-vmstat.pgfault 1.288e+08 -91.3% 11209163 ± 4% proc-vmstat.pgfree 22171 ± 10% -73.1% 5956 ± 26% proc-vmstat.pgmajfault 119215 ± 22% -95.5% 5412 ± 42% proc-vmstat.pgmigrate_success 93314793 -95.9% 3848330 ± 4% proc-vmstat.pgrefill 91553858 ± 3% +130.5% 2.11e+08 ± 5% proc-vmstat.pgscan_direct 14329232 ± 3% -91.6% 1199879 ± 25% proc-vmstat.pgscan_kswapd 192954 ±133% -100.0% 0.00 proc-vmstat.pgskip_normal 77752861 -95.8% 3242366 ± 17% proc-vmstat.pgsteal_direct 14314039 ± 3% -91.8% 1168153 ± 26% proc-vmstat.pgsteal_kswapd 30330 ± 6% -90.9% 2755 ± 59% proc-vmstat.pswpin 92062246 -98.1% 1752839 ± 9% proc-vmstat.pswpout 6003 ± 37% -99.2% 49.50 ± 8% proc-vmstat.swap_ra 3719 ± 38% -99.5% 18.50 ± 26% proc-vmstat.swap_ra_hit 1079479 -47.9% 562842 ± 10% numa-vmstat.node0.nr_active_anon 1244149 -50.9% 610979 ± 11% numa-vmstat.node0.nr_anon_pages 65081 ± 3% +65.9% 107940 ± 7% numa-vmstat.node0.nr_file_pages 92743 ± 9% +275.0% 347789 ± 18% numa-vmstat.node0.nr_free_pages 165606 -42.7% 94832 ± 12% numa-vmstat.node0.nr_inactive_anon 163.50 ± 10% +26.5% 206.75 ± 7% numa-vmstat.node0.nr_isolated_anon 6332 ± 10% +15.2% 7295 ± 13% numa-vmstat.node0.nr_kernel_stack 1475 ± 4% +73.2% 2555 numa-vmstat.node0.nr_mapped 19985 ± 13% -85.9% 2812 ± 11% numa-vmstat.node0.nr_page_table_pages 353.00 ± 56% +439.4% 1904 ± 12% numa-vmstat.node0.nr_shmem 18640 ± 15% -57.6% 7898 ± 10% numa-vmstat.node0.nr_slab_reclaimable 17170 ± 6% +66.0% 28498 ± 12% numa-vmstat.node0.nr_slab_unreclaimable 11339775 ± 8% -95.1% 557216 ± 7% numa-vmstat.node0.nr_vmscan_write 11339785 ± 8% -93.2% 772604 ± 10% numa-vmstat.node0.nr_written 1078097 -47.9% 562212 ± 10% numa-vmstat.node0.nr_zone_active_anon 165645 -42.9% 94563 ± 12% numa-vmstat.node0.nr_zone_inactive_anon 10833432 ± 20% -90.6% 1016984 ± 43% numa-vmstat.node0.numa_foreign 11368384 ± 11% -78.8% 2411490 ± 5% numa-vmstat.node0.numa_hit 11338851 ± 11% -78.9% 2396881 ± 5% numa-vmstat.node0.numa_local 3220466 ± 11% -97.3% 86801 ± 28% numa-vmstat.node0.numa_miss 3250022 ± 11% -96.9% 101418 ± 32% numa-vmstat.node0.numa_other 1144015 ± 3% -45.4% 624641 ± 8% numa-vmstat.node1.nr_active_anon 1308950 ± 2% -47.7% 684070 ± 10% numa-vmstat.node1.nr_anon_pages 65078 ± 2% +56.5% 101862 ± 3% numa-vmstat.node1.nr_file_pages 89819 ± 16% +339.7% 394892 ± 18% numa-vmstat.node1.nr_free_pages 165324 ± 3% -38.1% 102280 ± 20% numa-vmstat.node1.nr_inactive_anon 25582 ± 8% -89.0% 2824 ± 6% numa-vmstat.node1.nr_page_table_pages 61.00 ± 30% +254.5% 216.25 ±102% numa-vmstat.node1.nr_shmem 15272 ± 15% -57.9% 6427 ± 5% numa-vmstat.node1.nr_slab_reclaimable 13082 ± 20% +79.2% 23438 ± 11% numa-vmstat.node1.nr_slab_unreclaimable 15505760 ± 2% -96.1% 597228 ± 11% numa-vmstat.node1.nr_vmscan_write 15505764 ± 2% -95.0% 782744 ± 9% numa-vmstat.node1.nr_written 1142502 ± 3% -45.4% 624049 ± 8% numa-vmstat.node1.nr_zone_active_anon 165905 ± 3% -38.5% 101962 ± 20% numa-vmstat.node1.nr_zone_inactive_anon 5185520 ± 55% -87.2% 663100 ± 57% numa-vmstat.node1.numa_foreign 10989467 ± 16% -82.4% 1938286 ± 14% numa-vmstat.node1.numa_hit 10893913 ± 16% -83.0% 1848232 ± 15% numa-vmstat.node1.numa_local 7480409 ± 13% -93.0% 525453 ± 57% numa-vmstat.node1.numa_miss 7575999 ± 13% -91.9% 615517 ± 48% numa-vmstat.node1.numa_other 1126040 ± 4% -41.5% 658560 ± 8% numa-vmstat.node2.nr_active_anon 1292651 ± 3% -44.3% 719404 ± 8% numa-vmstat.node2.nr_anon_pages 65530 ± 3% +57.4% 103165 ± 9% numa-vmstat.node2.nr_file_pages 84064 ± 11% +383.7% 406632 ± 22% numa-vmstat.node2.nr_free_pages 166954 -34.8% 108855 ± 13% numa-vmstat.node2.nr_inactive_anon 24494 -88.8% 2746 ± 9% numa-vmstat.node2.nr_page_table_pages 15155 ± 11% -52.1% 7256 ± 12% numa-vmstat.node2.nr_slab_reclaimable 13245 ± 27% +55.3% 20576 ± 8% numa-vmstat.node2.nr_slab_unreclaimable 15002550 -95.9% 613047 ± 7% numa-vmstat.node2.nr_vmscan_write 15002556 -94.7% 788175 ± 7% numa-vmstat.node2.nr_written 1124427 ± 4% -41.5% 657862 ± 8% numa-vmstat.node2.nr_zone_active_anon 167651 -35.3% 108435 ± 13% numa-vmstat.node2.nr_zone_inactive_anon 3983276 ± 48% -88.2% 469967 ± 66% numa-vmstat.node2.numa_foreign 11396819 ± 20% -84.8% 1735877 ± 25% numa-vmstat.node2.numa_hit 11298966 ± 20% -85.5% 1642162 ± 26% numa-vmstat.node2.numa_local 6390588 ± 38% -89.6% 661903 ± 58% numa-vmstat.node2.numa_miss 6488475 ± 37% -88.4% 755628 ± 51% numa-vmstat.node2.numa_other 1136490 ± 4% -41.6% 663729 ± 4% numa-vmstat.node3.nr_active_anon 1301163 ± 3% -43.0% 741565 ± 4% numa-vmstat.node3.nr_anon_pages 66788 ± 4% +47.3% 98372 ± 6% numa-vmstat.node3.nr_file_pages 87998 ± 8% +352.3% 398056 ± 17% numa-vmstat.node3.nr_free_pages 164879 ± 2% -31.7% 112582 ± 13% numa-vmstat.node3.nr_inactive_anon 25741 ± 12% -89.9% 2587 ± 5% numa-vmstat.node3.nr_page_table_pages 14885 ± 12% -60.4% 5897 ± 14% numa-vmstat.node3.nr_slab_reclaimable 13200 ± 23% +51.3% 19973 ± 4% numa-vmstat.node3.nr_slab_unreclaimable 15613755 ± 4% -96.0% 616788 ± 7% numa-vmstat.node3.nr_vmscan_write 15613761 ± 4% -95.0% 781041 ± 8% numa-vmstat.node3.nr_written 1134916 ± 4% -41.6% 663173 ± 4% numa-vmstat.node3.nr_zone_active_anon 165543 ± 2% -32.3% 112086 ± 13% numa-vmstat.node3.nr_zone_inactive_anon 3251058 ± 62% -96.3% 120489 ± 48% numa-vmstat.node3.numa_foreign 12300437 ± 23% -88.7% 1395358 ± 20% numa-vmstat.node3.numa_hit 12206583 ± 23% -89.3% 1300144 ± 21% numa-vmstat.node3.numa_local 6166200 ± 40% -83.7% 1005747 ± 28% numa-vmstat.node3.numa_miss 6260087 ± 39% -82.4% 1100966 ± 25% numa-vmstat.node3.numa_other 148500 ± 19% -87.7% 18280 ± 10% softirqs.CPU0.RCU 17858 ± 16% +110.6% 37610 ± 5% softirqs.CPU0.SCHED 66390 ± 6% +74.0% 115536 ± 13% softirqs.CPU0.TIMER 143319 ± 13% -85.8% 20416 ± 21% softirqs.CPU1.RCU 12471 ± 31% +116.1% 26946 ± 13% softirqs.CPU1.SCHED 64688 ± 11% +74.1% 112626 ± 14% softirqs.CPU1.TIMER 158245 ± 15% -89.4% 16848 ± 14% softirqs.CPU10.RCU 7139 ± 34% +272.1% 26563 ± 12% softirqs.CPU10.SCHED 62529 ± 10% +80.2% 112653 ± 11% softirqs.CPU10.TIMER 51975 ± 71% -72.7% 14210 ± 4% softirqs.CPU100.RCU 16021 ± 20% +95.3% 31284 ± 6% softirqs.CPU100.SCHED 61521 ± 17% +82.3% 112124 ± 13% softirqs.CPU100.TIMER 55289 ± 64% -74.1% 14298 ± 9% softirqs.CPU101.RCU 14405 ± 24% +116.4% 31167 ± 5% softirqs.CPU101.SCHED 61371 ± 16% +81.0% 111096 ± 12% softirqs.CPU101.TIMER 64943 ± 66% -77.2% 14807 ± 8% softirqs.CPU102.RCU 14576 ± 25% +110.5% 30690 softirqs.CPU102.SCHED 61443 ± 18% +73.6% 106662 ± 10% softirqs.CPU102.TIMER 56913 ± 71% -74.4% 14548 ± 12% softirqs.CPU103.RCU 14999 ± 23% +107.7% 31152 ± 8% softirqs.CPU103.SCHED 60710 ± 17% +79.6% 109008 ± 10% softirqs.CPU103.TIMER 53357 ± 67% -72.2% 14815 ± 9% softirqs.CPU104.RCU 15438 ± 23% +104.7% 31595 ± 5% softirqs.CPU104.SCHED 60199 ± 17% +87.5% 112876 ± 11% softirqs.CPU104.TIMER 52763 ± 65% -71.3% 15162 ± 7% softirqs.CPU105.RCU 14867 ± 17% +111.7% 31469 ± 4% softirqs.CPU105.SCHED 60043 ± 17% +85.9% 111641 ± 13% softirqs.CPU105.TIMER 51265 ± 70% -71.6% 14549 ± 8% softirqs.CPU106.RCU 15449 ± 24% +103.7% 31472 ± 2% softirqs.CPU106.SCHED 61030 ± 16% +81.5% 110754 ± 11% softirqs.CPU106.TIMER 52653 ± 60% -73.4% 13989 ± 8% softirqs.CPU107.RCU 14764 ± 20% +117.8% 32153 ± 3% softirqs.CPU107.SCHED 61155 ± 16% +80.8% 110561 ± 11% softirqs.CPU107.TIMER 44960 ± 58% -67.3% 14687 ± 34% softirqs.CPU108.RCU 16242 ± 18% +87.1% 30382 ± 15% softirqs.CPU108.SCHED 68723 ± 12% +66.4% 114358 ± 12% softirqs.CPU108.TIMER 48128 ± 42% -74.2% 12434 ± 14% softirqs.CPU109.RCU 58701 ± 17% +84.1% 108099 ± 8% softirqs.CPU109.TIMER 161831 ± 13% -89.4% 17195 ± 15% softirqs.CPU11.RCU 8327 ± 40% +224.8% 27045 ± 20% softirqs.CPU11.SCHED 62461 ± 12% +78.1% 111265 ± 14% softirqs.CPU11.TIMER 44523 ± 52% -72.1% 12432 ± 10% softirqs.CPU110.RCU 15380 ± 18% +100.9% 30900 ± 7% softirqs.CPU110.SCHED 57841 ± 15% +88.5% 109008 ± 9% softirqs.CPU110.TIMER 39980 ± 66% -69.6% 12142 ± 9% softirqs.CPU111.RCU 17226 ± 11% +85.4% 31934 ± 7% softirqs.CPU111.SCHED 59163 ± 16% +87.8% 111088 ± 10% softirqs.CPU111.TIMER 44882 ± 58% -67.5% 14577 ± 13% softirqs.CPU112.RCU 17531 ± 12% +74.5% 30588 ± 9% softirqs.CPU112.SCHED 58977 ± 17% +89.5% 111769 ± 10% softirqs.CPU112.TIMER 42304 ± 35% -67.9% 13599 ± 3% softirqs.CPU113.RCU 16961 ± 14% +85.7% 31489 ± 8% softirqs.CPU113.SCHED 57316 ± 16% +90.6% 109253 ± 11% softirqs.CPU113.TIMER 40603 ± 48% -66.8% 13489 ± 7% softirqs.CPU114.RCU 17596 ± 8% +80.8% 31821 ± 9% softirqs.CPU114.SCHED 57479 ± 17% +90.1% 109273 ± 11% softirqs.CPU114.TIMER 38979 ± 52% -64.4% 13885 ± 13% softirqs.CPU115.RCU 17650 ± 9% +77.1% 31250 ± 8% softirqs.CPU115.SCHED 55459 ± 17% +114.8% 119143 ± 20% softirqs.CPU115.TIMER 38862 ± 45% -67.6% 12596 ± 5% softirqs.CPU116.RCU 17624 ± 9% +88.4% 33198 ± 6% softirqs.CPU116.SCHED 58562 ± 16% +94.3% 113798 ± 11% softirqs.CPU116.TIMER 36619 ± 27% -63.1% 13517 ± 8% softirqs.CPU117.RCU 17288 ± 9% +84.9% 31967 ± 8% softirqs.CPU117.SCHED 59297 ± 16% +89.8% 112547 ± 12% softirqs.CPU117.TIMER 42777 ± 57% -65.5% 14779 ± 8% softirqs.CPU118.RCU 15874 ± 12% +100.9% 31889 ± 4% softirqs.CPU118.SCHED 57772 ± 16% +94.2% 112179 ± 14% softirqs.CPU118.TIMER 37634 ± 43% -62.7% 14036 ± 9% softirqs.CPU119.RCU 16704 ± 18% +88.4% 31464 ± 11% softirqs.CPU119.SCHED 58553 ± 16% +84.7% 108138 ± 9% softirqs.CPU119.TIMER 159594 ± 14% -89.7% 16481 ± 18% softirqs.CPU12.RCU 8647 ± 42% +208.2% 26649 ± 19% softirqs.CPU12.SCHED 62630 ± 13% +75.2% 109705 ± 14% softirqs.CPU12.TIMER 36255 ± 44% -62.1% 13726 ± 9% softirqs.CPU120.RCU 17122 ± 8% +85.1% 31693 ± 9% softirqs.CPU120.SCHED 59082 ± 17% +87.9% 111002 ± 10% softirqs.CPU120.TIMER 39008 ± 42% -62.1% 14787 ± 11% softirqs.CPU121.RCU 18168 ± 10% +74.0% 31609 ± 11% softirqs.CPU121.SCHED 59598 ± 17% +90.9% 113755 ± 13% softirqs.CPU121.TIMER 38742 ± 36% -65.9% 13229 ± 14% softirqs.CPU122.RCU 15884 ± 13% +99.6% 31710 ± 14% softirqs.CPU122.SCHED 59139 ± 14% +90.8% 112826 ± 12% softirqs.CPU122.TIMER 38259 ± 49% -57.7% 16165 ± 17% softirqs.CPU123.RCU 17026 ± 16% +82.2% 31015 ± 12% softirqs.CPU123.SCHED 60160 ± 16% +86.1% 111963 ± 11% softirqs.CPU123.TIMER 38639 ± 33% -64.4% 13772 ± 10% softirqs.CPU124.RCU 17160 ± 15% +83.2% 31429 ± 9% softirqs.CPU124.SCHED 59669 ± 16% +85.4% 110651 ± 10% softirqs.CPU124.TIMER 45107 ± 60% -69.5% 13735 ± 12% softirqs.CPU125.RCU 17242 ± 10% +85.3% 31942 ± 10% softirqs.CPU125.SCHED 59485 ± 17% +87.8% 111686 ± 11% softirqs.CPU125.TIMER 58838 ± 86% -78.7% 12552 ± 5% softirqs.CPU126.RCU 15522 ± 27% +101.0% 31201 ± 8% softirqs.CPU126.SCHED 59876 ± 15% +129.3% 137299 ± 17% softirqs.CPU126.TIMER 57748 ± 88% -78.6% 12348 ± 15% softirqs.CPU127.RCU 16508 ± 20% +83.6% 30303 ± 7% softirqs.CPU127.SCHED 59775 ± 15% +80.4% 107848 ± 12% softirqs.CPU127.TIMER 60225 ± 69% -80.8% 11589 ± 9% softirqs.CPU128.RCU 16116 ± 16% +99.8% 32198 ± 5% softirqs.CPU128.SCHED 59586 ± 15% +80.4% 107499 ± 9% softirqs.CPU128.TIMER 59447 ± 72% -78.1% 12992 ± 9% softirqs.CPU129.RCU 15505 ± 26% +121.5% 34341 ± 4% softirqs.CPU129.SCHED 58958 ± 14% +91.2% 112740 ± 12% softirqs.CPU129.TIMER 154961 ± 18% -89.1% 16885 ± 20% softirqs.CPU13.RCU 9747 ± 52% +174.2% 26731 ± 18% softirqs.CPU13.SCHED 63566 ± 10% +75.1% 111291 ± 11% softirqs.CPU13.TIMER 52198 ± 78% -74.3% 13432 ± 11% softirqs.CPU130.RCU 16744 ± 17% +90.5% 31893 ± 10% softirqs.CPU130.SCHED 57112 ± 11% +87.3% 106987 ± 10% softirqs.CPU130.TIMER 58087 ± 63% -80.4% 11378 ± 9% softirqs.CPU131.RCU 16177 ± 19% +114.1% 34632 softirqs.CPU131.SCHED 60025 ± 13% +82.2% 109374 ± 10% softirqs.CPU131.TIMER 55006 ± 77% -78.6% 11767 ± 8% softirqs.CPU132.RCU 16494 ± 17% +99.6% 32918 ± 4% softirqs.CPU132.SCHED 59868 ± 13% +82.7% 109391 ± 10% softirqs.CPU132.TIMER 52985 ± 76% -75.2% 13118 ± 16% softirqs.CPU133.RCU 15975 ± 17% +96.3% 31354 ± 9% softirqs.CPU133.SCHED 59848 ± 12% +77.7% 106325 ± 8% softirqs.CPU133.TIMER 51423 ± 71% -76.8% 11913 ± 9% softirqs.CPU134.RCU 60239 ± 15% +86.1% 112088 ± 11% softirqs.CPU134.TIMER 58680 ± 69% -79.4% 12065 ± 17% softirqs.CPU135.RCU 15399 ± 27% +117.7% 33530 ± 7% softirqs.CPU135.SCHED 59646 ± 15% +91.8% 114419 ± 12% softirqs.CPU135.TIMER 59344 ± 57% -79.3% 12270 ± 13% softirqs.CPU136.RCU 16577 ± 15% +99.9% 33140 ± 6% softirqs.CPU136.SCHED 60564 ± 14% +81.2% 109717 ± 9% softirqs.CPU136.TIMER 53272 ± 69% -77.5% 12005 ± 9% softirqs.CPU137.RCU 15042 ± 28% +133.3% 35098 ± 2% softirqs.CPU137.SCHED 61160 ± 15% +86.6% 114147 ± 12% softirqs.CPU137.TIMER 50441 ± 71% -74.7% 12785 ± 9% softirqs.CPU138.RCU 15214 ± 24% +120.2% 33499 ± 6% softirqs.CPU138.SCHED 59597 ± 15% +88.8% 112539 ± 14% softirqs.CPU138.TIMER 51925 ± 61% -73.8% 13586 ± 14% softirqs.CPU139.RCU 16718 ± 14% +78.4% 29818 ± 12% softirqs.CPU139.SCHED 59680 ± 15% +87.2% 111693 ± 11% softirqs.CPU139.TIMER 158338 ± 15% -90.4% 15267 ± 14% softirqs.CPU14.RCU 6872 ± 36% +301.7% 27608 ± 15% softirqs.CPU14.SCHED 62444 ± 11% +78.7% 111617 ± 16% softirqs.CPU14.TIMER 51555 ± 58% -75.9% 12413 ± 9% softirqs.CPU140.RCU 16164 ± 11% +107.6% 33556 ± 2% softirqs.CPU140.SCHED 60149 ± 14% +82.5% 109751 ± 10% softirqs.CPU140.TIMER 57032 ± 81% -77.9% 12576 ± 9% softirqs.CPU141.RCU 17207 ± 13% +96.4% 33797 ± 7% softirqs.CPU141.SCHED 61584 ± 15% +84.2% 113415 ± 13% softirqs.CPU141.TIMER 48404 ± 66% -73.3% 12921 ± 15% softirqs.CPU142.RCU 14895 ± 6% +122.0% 33075 ± 6% softirqs.CPU142.SCHED 61970 ± 13% +78.3% 110491 ± 10% softirqs.CPU142.TIMER 53676 ± 59% -71.6% 15256 ± 20% softirqs.CPU143.RCU 15189 ± 20% +66.1% 25226 ± 17% softirqs.CPU143.SCHED 61406 ± 11% +82.3% 111972 ± 10% softirqs.CPU143.TIMER 159128 ± 15% -89.2% 17185 ± 13% softirqs.CPU15.RCU 8270 ± 41% +236.0% 27786 ± 8% softirqs.CPU15.SCHED 62582 ± 12% +79.5% 112330 ± 13% softirqs.CPU15.TIMER 156612 ± 14% -90.8% 14437 ± 14% softirqs.CPU16.RCU 8541 ± 46% +216.9% 27064 ± 11% softirqs.CPU16.SCHED 63236 ± 13% +76.0% 111303 ± 13% softirqs.CPU16.TIMER 158496 ± 16% -91.3% 13798 ± 14% softirqs.CPU17.RCU 7652 ± 51% +278.7% 28978 ± 9% softirqs.CPU17.SCHED 63016 ± 12% +81.7% 114498 ± 15% softirqs.CPU17.TIMER 111902 ± 19% -82.9% 19182 ± 16% softirqs.CPU18.RCU 12335 ± 18% +129.0% 28253 ± 10% softirqs.CPU18.SCHED 63057 ± 15% +74.6% 110102 ± 12% softirqs.CPU18.TIMER 110348 ± 20% -85.4% 16109 ± 2% softirqs.CPU19.RCU 11999 ± 23% +138.4% 28604 ± 4% softirqs.CPU19.SCHED 61933 ± 14% +75.8% 108906 ± 10% softirqs.CPU19.TIMER 155072 ± 17% -89.4% 16429 ± 13% softirqs.CPU2.RCU 7419 ± 28% +262.8% 26915 ± 11% softirqs.CPU2.SCHED 62650 ± 9% +75.2% 109746 ± 14% softirqs.CPU2.TIMER 107320 ± 24% -86.6% 14358 ± 7% softirqs.CPU20.RCU 12967 ± 17% +97.1% 25556 ± 23% softirqs.CPU20.SCHED 62015 ± 15% +76.5% 109477 ± 13% softirqs.CPU20.TIMER 111967 ± 18% -83.7% 18219 ± 18% softirqs.CPU21.RCU 12765 ± 21% +136.3% 30162 ± 14% softirqs.CPU21.SCHED 61643 ± 16% +88.1% 115927 ± 15% softirqs.CPU21.TIMER 104906 ± 30% -85.1% 15663 ± 8% softirqs.CPU22.RCU 13100 ± 21% +127.1% 29751 softirqs.CPU22.SCHED 61691 ± 16% +80.4% 111272 ± 12% softirqs.CPU22.TIMER 111456 ± 21% -81.0% 21201 ± 41% softirqs.CPU23.RCU 12023 ± 19% +112.6% 25565 ± 28% softirqs.CPU23.SCHED 60192 ± 16% +82.7% 109972 ± 12% softirqs.CPU23.TIMER 113756 ± 19% -86.3% 15589 ± 7% softirqs.CPU24.RCU 11746 ± 19% +153.1% 29732 ± 2% softirqs.CPU24.SCHED 60923 ± 16% +84.0% 112093 ± 12% softirqs.CPU24.TIMER 106907 ± 26% -85.4% 15622 ± 7% softirqs.CPU25.RCU 11714 ± 19% +146.9% 28920 softirqs.CPU25.SCHED 62148 ± 15% +76.8% 109902 ± 11% softirqs.CPU25.TIMER 109217 ± 26% -86.0% 15287 ± 13% softirqs.CPU26.RCU 12526 ± 20% +132.5% 29118 ± 7% softirqs.CPU26.SCHED 61152 ± 16% +80.4% 110309 ± 10% softirqs.CPU26.TIMER 106674 ± 22% -86.1% 14869 ± 7% softirqs.CPU27.RCU 12446 ± 25% +153.6% 31566 ± 2% softirqs.CPU27.SCHED 62325 ± 13% +84.3% 114859 ± 13% softirqs.CPU27.TIMER 118516 ± 21% -87.0% 15349 ± 13% softirqs.CPU28.RCU 12017 ± 23% +149.6% 29998 ± 10% softirqs.CPU28.SCHED 62263 ± 16% +80.3% 112233 ± 12% softirqs.CPU28.TIMER 105772 ± 28% -85.1% 15793 ± 9% softirqs.CPU29.RCU 12949 ± 19% +122.0% 28749 ± 7% softirqs.CPU29.SCHED 62928 ± 15% +75.1% 110167 ± 13% softirqs.CPU29.TIMER 152500 ± 18% -88.1% 18110 ± 16% softirqs.CPU3.RCU 8121 ± 31% +192.0% 23716 ± 20% softirqs.CPU3.SCHED 63602 ± 8% +68.7% 107290 ± 13% softirqs.CPU3.TIMER 104250 ± 30% -84.9% 15719 ± 10% softirqs.CPU30.RCU 13072 ± 17% +117.8% 28478 ± 7% softirqs.CPU30.SCHED 62284 ± 16% +71.4% 106727 ± 9% softirqs.CPU30.TIMER 113558 ± 22% -86.1% 15790 ± 13% softirqs.CPU31.RCU 12753 ± 14% +139.4% 30537 ± 3% softirqs.CPU31.SCHED 63368 ± 16% +74.7% 110692 ± 9% softirqs.CPU31.TIMER 111744 ± 24% -88.2% 13187 ± 5% softirqs.CPU32.RCU 12703 ± 21% +142.7% 30833 ± 6% softirqs.CPU32.SCHED 61578 ± 16% +80.0% 110841 ± 10% softirqs.CPU32.TIMER 110779 ± 25% -86.3% 15188 ± 8% softirqs.CPU33.RCU 13048 ± 20% +126.8% 29594 ± 8% softirqs.CPU33.SCHED 62259 ± 17% +78.0% 110803 ± 13% softirqs.CPU33.TIMER 108641 ± 27% -87.6% 13515 ± 4% softirqs.CPU34.RCU 11704 ± 17% +164.7% 30986 ± 2% softirqs.CPU34.SCHED 64380 ± 10% +72.8% 111244 ± 10% softirqs.CPU34.TIMER 108403 ± 23% -87.0% 14126 ± 4% softirqs.CPU35.RCU 12696 ± 22% +146.2% 31262 ± 2% softirqs.CPU35.SCHED 62249 ± 15% +79.0% 111411 ± 11% softirqs.CPU35.TIMER 98380 ± 38% -82.5% 17192 ± 15% softirqs.CPU36.RCU 12910 ± 20% +122.9% 28778 ± 13% softirqs.CPU36.SCHED 62142 ± 11% +83.5% 114020 ± 11% softirqs.CPU36.TIMER 100859 ± 33% -85.1% 15013 ± 13% softirqs.CPU37.RCU 13080 ± 18% +138.9% 31243 ± 11% softirqs.CPU37.SCHED 59497 ± 15% +83.3% 109036 ± 9% softirqs.CPU37.TIMER 97655 ± 40% -84.9% 14790 ± 8% softirqs.CPU38.RCU 13182 ± 25% +121.6% 29215 ± 10% softirqs.CPU38.SCHED 60659 ± 10% +80.6% 109527 ± 10% softirqs.CPU38.TIMER 104662 ± 29% -85.6% 15083 ± 9% softirqs.CPU39.RCU 12493 ± 25% +144.6% 30553 ± 11% softirqs.CPU39.SCHED 58747 ± 15% +88.8% 110929 ± 12% softirqs.CPU39.TIMER 156827 ± 17% -88.5% 18111 ± 8% softirqs.CPU4.RCU 7358 ± 44% +256.2% 26212 ± 10% softirqs.CPU4.SCHED 63026 ± 9% +77.3% 111773 ± 14% softirqs.CPU4.TIMER 108483 ± 29% -86.6% 14545 ± 9% softirqs.CPU40.RCU 12804 ± 26% +147.9% 31740 ± 6% softirqs.CPU40.SCHED 59092 ± 15% +91.0% 112873 ± 10% softirqs.CPU40.TIMER 100472 ± 36% -84.6% 15501 ± 4% softirqs.CPU41.RCU 13986 ± 17% +118.8% 30599 ± 9% softirqs.CPU41.SCHED 58005 ± 15% +90.2% 110346 ± 11% softirqs.CPU41.TIMER 105700 ± 32% -85.7% 15105 ± 7% softirqs.CPU42.RCU 13316 ± 22% +122.1% 29574 ± 9% softirqs.CPU42.SCHED 58757 ± 16% +85.1% 108781 ± 11% softirqs.CPU42.TIMER 109930 ± 28% -85.7% 15756 ± 12% softirqs.CPU43.RCU 13316 ± 12% +104.9% 27287 ± 11% softirqs.CPU43.SCHED 63407 ± 19% +62.6% 103118 ± 8% softirqs.CPU43.TIMER 103971 ± 30% -84.4% 16271 ± 12% softirqs.CPU44.RCU 13004 ± 30% +131.7% 30131 ± 7% softirqs.CPU44.SCHED 58980 ± 15% +93.9% 114384 ± 12% softirqs.CPU44.TIMER 110066 ± 32% -86.3% 15032 ± 5% softirqs.CPU45.RCU 12973 ± 17% +133.4% 30272 ± 7% softirqs.CPU45.SCHED 64564 ± 22% +74.1% 112411 ± 13% softirqs.CPU45.TIMER 108804 ± 22% -86.1% 15130 ± 4% softirqs.CPU46.RCU 13575 ± 17% +120.5% 29929 ± 8% softirqs.CPU46.SCHED 60320 ± 17% +86.7% 112609 ± 14% softirqs.CPU46.TIMER 101742 ± 35% -85.1% 15126 ± 10% softirqs.CPU47.RCU 14210 ± 17% +104.1% 29006 ± 10% softirqs.CPU47.SCHED 59629 ± 15% +79.8% 107203 ± 9% softirqs.CPU47.TIMER 99168 ± 39% -85.0% 14921 ± 17% softirqs.CPU48.RCU 14040 ± 22% +104.7% 28735 ± 14% softirqs.CPU48.SCHED 59847 ± 16% +85.8% 111210 ± 8% softirqs.CPU48.TIMER 97763 ± 35% -83.5% 16172 ± 19% softirqs.CPU49.RCU 13694 ± 15% +120.8% 30236 ± 15% softirqs.CPU49.SCHED 59453 ± 16% +93.8% 115218 ± 12% softirqs.CPU49.TIMER 141559 ± 18% -88.3% 16604 ± 15% softirqs.CPU5.RCU 7789 ± 40% +223.5% 25199 ± 24% softirqs.CPU5.SCHED 63586 ± 10% +73.4% 110254 ± 12% softirqs.CPU5.TIMER 94641 ± 39% -84.7% 14472 ± 25% softirqs.CPU50.RCU 59938 ± 16% +89.3% 113447 ± 10% softirqs.CPU50.TIMER 94452 ± 40% -77.7% 21094 ± 47% softirqs.CPU51.RCU 14050 ± 17% +96.7% 27639 ± 14% softirqs.CPU51.SCHED 59616 ± 16% +88.6% 112418 ± 11% softirqs.CPU51.TIMER 90924 ± 42% -83.7% 14791 ± 2% softirqs.CPU52.RCU 14066 ± 19% +109.8% 29510 ± 8% softirqs.CPU52.SCHED 59965 ± 15% +84.0% 110350 ± 11% softirqs.CPU52.TIMER 91145 ± 30% -84.8% 13823 ± 6% softirqs.CPU53.RCU 12915 ± 18% +136.1% 30490 ± 9% softirqs.CPU53.SCHED 60391 ± 14% +83.2% 110614 ± 10% softirqs.CPU53.TIMER 107181 ± 38% -81.1% 20290 ± 11% softirqs.CPU54.RCU 13172 ± 17% +106.1% 27148 ± 4% softirqs.CPU54.SCHED 61594 ± 13% +81.2% 111615 ± 6% softirqs.CPU54.TIMER 97873 ± 41% -78.8% 20702 ± 49% softirqs.CPU55.RCU 12950 ± 21% +113.8% 27688 ± 11% softirqs.CPU55.SCHED 60134 ± 13% +82.3% 109610 ± 11% softirqs.CPU55.TIMER 112575 ± 36% -87.1% 14513 ± 13% softirqs.CPU56.RCU 12484 ± 24% +132.9% 29080 ± 9% softirqs.CPU56.SCHED 60003 ± 12% +79.1% 107437 ± 9% softirqs.CPU56.TIMER 104995 ± 35% -86.1% 14636 ± 10% softirqs.CPU57.RCU 13488 ± 21% +121.9% 29924 ± 10% softirqs.CPU57.SCHED 60017 ± 13% +83.8% 110336 ± 14% softirqs.CPU57.TIMER 104935 ± 38% -86.3% 14364 ± 12% softirqs.CPU58.RCU 12724 ± 27% +145.4% 31227 ± 10% softirqs.CPU58.SCHED 66812 ± 18% +62.2% 108389 ± 8% softirqs.CPU58.TIMER 108792 ± 37% -86.6% 14543 ± 12% softirqs.CPU59.RCU 13357 ± 26% +126.5% 30251 ± 9% softirqs.CPU59.SCHED 60019 ± 13% +80.3% 108225 ± 10% softirqs.CPU59.TIMER 157899 ± 17% -89.4% 16667 ± 22% softirqs.CPU6.RCU 7072 ± 45% +273.2% 26396 ± 11% softirqs.CPU6.SCHED 61708 ± 12% +78.9% 110372 ± 11% softirqs.CPU6.TIMER 98202 ± 43% -85.9% 13841 ± 8% softirqs.CPU60.RCU 13142 ± 27% +143.1% 31954 ± 4% softirqs.CPU60.SCHED 60763 ± 9% +81.6% 110338 ± 11% softirqs.CPU60.TIMER 106760 ± 39% -86.1% 14830 ± 9% softirqs.CPU61.RCU 13280 ± 26% +124.7% 29834 ± 5% softirqs.CPU61.SCHED 59780 ± 13% +79.2% 107147 ± 9% softirqs.CPU61.TIMER 100012 ± 43% -85.5% 14533 ± 11% softirqs.CPU62.RCU 13730 ± 37% +127.5% 31238 ± 6% softirqs.CPU62.SCHED 60060 ± 14% +87.4% 112559 ± 11% softirqs.CPU62.TIMER 99644 ± 41% -83.1% 16814 ± 24% softirqs.CPU63.RCU 13920 ± 20% +113.0% 29653 ± 8% softirqs.CPU63.SCHED 61483 ± 11% +85.2% 113838 ± 12% softirqs.CPU63.TIMER 98673 ± 41% -86.4% 13371 ± 5% softirqs.CPU64.RCU 12874 ± 30% +146.2% 31698 ± 7% softirqs.CPU64.SCHED 59958 ± 14% +81.8% 109014 ± 10% softirqs.CPU64.TIMER 95038 ± 48% -87.0% 12392 ± 9% softirqs.CPU65.RCU 13766 ± 17% +150.0% 34411 softirqs.CPU65.SCHED 62256 ± 15% +83.8% 114419 ± 12% softirqs.CPU65.TIMER 98059 ± 41% -85.8% 13956 ± 9% softirqs.CPU66.RCU 13840 ± 27% +134.6% 32474 ± 6% softirqs.CPU66.SCHED 60586 ± 15% +87.0% 113313 ± 12% softirqs.CPU66.TIMER 98441 ± 43% -80.6% 19078 ± 36% softirqs.CPU67.RCU 13545 ± 22% +100.8% 27197 ± 19% softirqs.CPU67.SCHED 60113 ± 14% +83.7% 110443 ± 9% softirqs.CPU67.TIMER 99589 ± 42% -85.7% 14240 ± 8% softirqs.CPU68.RCU 13429 ± 29% +123.6% 30022 ± 8% softirqs.CPU68.SCHED 60597 ± 14% +80.8% 109558 ± 10% softirqs.CPU68.TIMER 92903 ± 50% -86.1% 12958 ± 6% softirqs.CPU69.RCU 12046 ± 29% +182.4% 34013 ± 5% softirqs.CPU69.SCHED 60955 ± 15% +86.7% 113828 ± 13% softirqs.CPU69.TIMER 164069 ± 14% -89.8% 16783 ± 14% softirqs.CPU7.RCU 7474 ± 40% +259.7% 26886 ± 10% softirqs.CPU7.SCHED 62306 ± 10% +76.9% 110239 ± 10% softirqs.CPU7.TIMER 93281 ± 44% -86.0% 13025 ± 8% softirqs.CPU70.RCU 13869 ± 32% +135.3% 32634 ± 5% softirqs.CPU70.SCHED 61791 ± 15% +78.1% 110050 ± 10% softirqs.CPU70.TIMER 90837 ± 45% -83.7% 14846 ± 14% softirqs.CPU71.RCU 13831 ± 18% +117.7% 30112 ± 9% softirqs.CPU71.SCHED 61682 ± 12% +79.3% 110607 ± 10% softirqs.CPU71.TIMER 123391 ± 36% -83.3% 20639 ± 13% softirqs.CPU72.RCU 10985 ± 41% +147.9% 27232 ± 21% softirqs.CPU72.SCHED 68480 ± 10% +71.9% 117714 ± 12% softirqs.CPU72.TIMER 113235 ± 37% -86.1% 15735 ± 12% softirqs.CPU73.RCU 10529 ± 54% +169.5% 28374 ± 16% softirqs.CPU73.SCHED 63871 ± 11% +75.0% 111795 ± 16% softirqs.CPU73.TIMER 112668 ± 39% -86.7% 14959 ± 15% softirqs.CPU74.RCU 11747 ± 42% +134.7% 27565 ± 15% softirqs.CPU74.SCHED 63357 ± 10% +72.7% 109401 ± 15% softirqs.CPU74.TIMER 105725 ± 36% -83.6% 17299 ± 28% softirqs.CPU75.RCU 9945 ± 51% +182.2% 28070 ± 7% softirqs.CPU75.SCHED 62141 ± 11% +78.4% 110853 ± 15% softirqs.CPU75.TIMER 105233 ± 37% -85.4% 15384 ± 10% softirqs.CPU76.RCU 11290 ± 49% +158.2% 29148 ± 3% softirqs.CPU76.SCHED 65298 ± 18% +74.6% 114041 ± 14% softirqs.CPU76.TIMER 104140 ± 43% -82.4% 18310 ± 31% softirqs.CPU77.RCU 64348 ± 11% +68.4% 108371 ± 10% softirqs.CPU77.TIMER 102544 ± 44% -82.4% 18029 ± 19% softirqs.CPU78.RCU 12037 ± 47% +107.5% 24978 ± 12% softirqs.CPU78.SCHED 64923 ± 14% +66.7% 108229 ± 12% softirqs.CPU78.TIMER 96336 ± 45% -75.5% 23635 ± 54% softirqs.CPU79.RCU 11778 ± 45% +104.1% 24034 ± 20% softirqs.CPU79.SCHED 62315 ± 10% +74.6% 108789 ± 11% softirqs.CPU79.TIMER 152851 ± 12% -88.6% 17363 ± 13% softirqs.CPU8.RCU 6669 ± 29% +282.5% 25509 ± 15% softirqs.CPU8.SCHED 62228 ± 10% +73.0% 107643 ± 10% softirqs.CPU8.TIMER 100159 ± 45% -83.3% 16770 ± 9% softirqs.CPU80.RCU 11133 ± 55% +147.7% 27581 ± 11% softirqs.CPU80.SCHED 62818 ± 10% +70.0% 106816 ± 12% softirqs.CPU80.TIMER 100214 ± 41% -84.1% 15970 ± 13% softirqs.CPU81.RCU 10962 ± 51% +167.1% 29280 ± 10% softirqs.CPU81.SCHED 62520 ± 10% +81.0% 113151 ± 13% softirqs.CPU81.TIMER 96685 ± 43% -81.3% 18033 ± 28% softirqs.CPU82.RCU 11223 ± 44% +138.2% 26738 ± 21% softirqs.CPU82.SCHED 61842 ± 11% +80.1% 111386 ± 11% softirqs.CPU82.TIMER 92424 ± 46% -82.4% 16229 ± 17% softirqs.CPU83.RCU 10849 ± 46% +158.7% 28069 ± 15% softirqs.CPU83.SCHED 62273 ± 10% +77.4% 110449 ± 14% softirqs.CPU83.TIMER 93512 ± 45% -83.5% 15439 ± 16% softirqs.CPU84.RCU 10190 ± 44% +139.0% 24354 ± 27% softirqs.CPU84.SCHED 62666 ± 9% +74.1% 109074 ± 13% softirqs.CPU84.TIMER 100596 ± 39% -83.8% 16338 ± 16% softirqs.CPU85.RCU 10554 ± 42% +157.4% 27166 ± 16% softirqs.CPU85.SCHED 61517 ± 12% +75.6% 108013 ± 12% softirqs.CPU85.TIMER 89535 ± 43% -82.5% 15707 ± 13% softirqs.CPU86.RCU 11981 ± 45% +141.5% 28934 ± 14% softirqs.CPU86.SCHED 62434 ± 11% +102.2% 126265 ± 19% softirqs.CPU86.TIMER 95204 ± 47% -82.8% 16413 ± 13% softirqs.CPU87.RCU 11032 ± 48% +156.0% 28241 ± 13% softirqs.CPU87.SCHED 61763 ± 11% +77.6% 109690 ± 13% softirqs.CPU87.TIMER 86945 ± 41% -81.0% 16553 ± 12% softirqs.CPU88.RCU 9268 ± 37% +211.4% 28864 ± 9% softirqs.CPU88.SCHED 62534 ± 11% +79.5% 112262 ± 14% softirqs.CPU88.TIMER 92979 ± 44% -80.7% 17961 ± 14% softirqs.CPU89.RCU 11846 ± 37% +131.6% 27433 ± 14% softirqs.CPU89.SCHED 63164 ± 15% +77.1% 111847 ± 16% softirqs.CPU89.TIMER 152947 ± 16% -89.5% 16089 ± 15% softirqs.CPU9.RCU 7789 ± 29% +252.5% 27457 ± 12% softirqs.CPU9.SCHED 62082 ± 10% +79.1% 111176 ± 13% softirqs.CPU9.TIMER 62677 ± 68% -75.2% 15542 ± 22% softirqs.CPU90.RCU 13906 ± 31% +110.2% 29227 ± 9% softirqs.CPU90.SCHED 62952 ± 17% +73.3% 109111 ± 10% softirqs.CPU90.TIMER 69697 ± 65% -79.6% 14199 ± 14% softirqs.CPU91.RCU 14063 ± 21% +109.0% 29398 ± 7% softirqs.CPU91.SCHED 61170 ± 15% +75.4% 107313 ± 10% softirqs.CPU91.TIMER 63874 ± 67% -71.7% 18095 ± 59% softirqs.CPU92.RCU 13638 ± 26% +117.2% 29618 ± 19% softirqs.CPU92.SCHED 61177 ± 15% +85.2% 113289 ± 13% softirqs.CPU92.TIMER 60204 ± 66% -77.2% 13740 ± 15% softirqs.CPU93.RCU 14278 ± 23% +121.5% 31632 ± 9% softirqs.CPU93.SCHED 60784 ± 17% +86.9% 113616 ± 14% softirqs.CPU93.TIMER 56970 ± 73% -76.2% 13564 ± 11% softirqs.CPU94.RCU 14801 ± 27% +108.0% 30781 ± 4% softirqs.CPU94.SCHED 60880 ± 16% +82.5% 111094 ± 12% softirqs.CPU94.TIMER 61290 ± 61% -78.6% 13124 ± 9% softirqs.CPU95.RCU 15518 ± 21% +87.7% 29121 ± 11% softirqs.CPU95.SCHED 60328 ± 16% +86.1% 112243 ± 12% softirqs.CPU95.TIMER 55646 ± 69% -74.8% 14041 ± 8% softirqs.CPU96.RCU 15853 ± 24% +92.7% 30545 ± 6% softirqs.CPU96.SCHED 61611 ± 17% +81.1% 111583 ± 11% softirqs.CPU96.TIMER 56069 ± 71% -74.2% 14442 ± 10% softirqs.CPU97.RCU 15054 ± 23% +99.1% 29967 ± 9% softirqs.CPU97.SCHED 62276 ± 14% +76.1% 109676 ± 11% softirqs.CPU97.TIMER 54992 ± 83% -70.9% 15990 ± 17% softirqs.CPU98.RCU 15081 ± 28% +95.1% 29425 ± 5% softirqs.CPU98.SCHED 60059 ± 16% +83.4% 110133 ± 11% softirqs.CPU98.TIMER 58340 ± 75% -75.3% 14398 ± 5% softirqs.CPU99.RCU 14366 ± 21% +116.4% 31088 ± 2% softirqs.CPU99.SCHED 61874 ± 13% +81.7% 112405 ± 12% softirqs.CPU99.TIMER 8716 ± 2% +44.1% 12556 ± 2% softirqs.NET_RX 12972555 ± 7% -83.1% 2188021 ± 5% softirqs.RCU 1930732 ± 5% +122.3% 4291149 ± 3% softirqs.SCHED 8819686 ± 13% +81.6% 16019120 ± 11% softirqs.TIMER 75.00 +156.0% 192.00 ± 36% interrupts.100:PCI-MSI.1572919-edge.eth0-TxRx-55 74.50 ± 2% +104.0% 152.00 interrupts.101:PCI-MSI.1572920-edge.eth0-TxRx-56 74.50 ± 2% +103.0% 151.25 interrupts.102:PCI-MSI.1572921-edge.eth0-TxRx-57 75.00 ± 3% +101.0% 150.75 interrupts.103:PCI-MSI.1572922-edge.eth0-TxRx-58 75.00 ± 3% +102.0% 151.50 interrupts.105:PCI-MSI.1572924-edge.eth0-TxRx-60 76.50 ± 3% +98.4% 151.75 interrupts.106:PCI-MSI.1572925-edge.eth0-TxRx-61 77.25 ± 5% +95.8% 151.25 interrupts.107:PCI-MSI.1572926-edge.eth0-TxRx-62 298.50 ± 25% +273.2% 1114 ±100% interrupts.45:PCI-MSI.1572864-edge.eth0-TxRx-0 132.50 ± 72% +100.4% 265.50 ± 42% interrupts.47:PCI-MSI.1572866-edge.eth0-TxRx-2 93.00 ± 31% +209.9% 288.25 ± 48% interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3 80.75 ± 5% +203.7% 245.25 ± 53% interrupts.51:PCI-MSI.1572870-edge.eth0-TxRx-6 80.25 ± 3% +406.2% 406.25 ± 75% interrupts.52:PCI-MSI.1572871-edge.eth0-TxRx-7 87.50 ± 9% +175.1% 240.75 ± 52% interrupts.55:PCI-MSI.1572874-edge.eth0-TxRx-10 87.50 ± 17% +79.4% 157.00 ± 3% interrupts.56:PCI-MSI.1572875-edge.eth0-TxRx-11 122.25 ± 63% +949.5% 1283 ± 57% interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15 75.25 +101.7% 151.75 interrupts.61:PCI-MSI.1572880-edge.eth0-TxRx-16 74.75 +105.0% 153.25 ± 2% interrupts.62:PCI-MSI.1572881-edge.eth0-TxRx-17 75.75 +99.3% 151.00 interrupts.63:PCI-MSI.1572882-edge.eth0-TxRx-18 80.25 ± 5% +91.3% 153.50 ± 3% interrupts.64:PCI-MSI.1572883-edge.eth0-TxRx-19 79.25 ± 5% +90.2% 150.75 interrupts.65:PCI-MSI.1572884-edge.eth0-TxRx-20 80.00 ± 7% +89.7% 151.75 interrupts.66:PCI-MSI.1572885-edge.eth0-TxRx-21 74.50 ± 2% +103.7% 151.75 interrupts.67:PCI-MSI.1572886-edge.eth0-TxRx-22 75.75 ± 3% +100.3% 151.75 interrupts.69:PCI-MSI.1572888-edge.eth0-TxRx-24 78.25 ± 8% +92.7% 150.75 interrupts.70:PCI-MSI.1572889-edge.eth0-TxRx-25 76.50 ± 3% +104.9% 156.75 ± 3% interrupts.71:PCI-MSI.1572890-edge.eth0-TxRx-26 77.75 ± 7% +93.9% 150.75 interrupts.72:PCI-MSI.1572891-edge.eth0-TxRx-27 84.00 ± 11% +80.4% 151.50 interrupts.73:PCI-MSI.1572892-edge.eth0-TxRx-28 76.75 +96.7% 151.00 interrupts.74:PCI-MSI.1572893-edge.eth0-TxRx-29 75.75 ± 2% +99.3% 151.00 interrupts.76:PCI-MSI.1572895-edge.eth0-TxRx-31 74.50 ± 2% +102.3% 150.75 interrupts.77:PCI-MSI.1572896-edge.eth0-TxRx-32 76.75 ± 3% +102.6% 155.50 ± 5% interrupts.78:PCI-MSI.1572897-edge.eth0-TxRx-33 75.25 +100.7% 151.00 interrupts.79:PCI-MSI.1572898-edge.eth0-TxRx-34 79.00 ± 8% +91.1% 151.00 interrupts.80:PCI-MSI.1572899-edge.eth0-TxRx-35 105.25 ± 30% +50.8% 158.75 ± 7% interrupts.81:PCI-MSI.1572900-edge.eth0-TxRx-36 75.25 ± 3% +101.0% 151.25 interrupts.82:PCI-MSI.1572901-edge.eth0-TxRx-37 75.00 +118.0% 163.50 ± 12% interrupts.83:PCI-MSI.1572902-edge.eth0-TxRx-38 75.25 ± 3% +100.3% 150.75 interrupts.84:PCI-MSI.1572903-edge.eth0-TxRx-39 75.25 ± 3% +101.0% 151.25 interrupts.85:PCI-MSI.1572904-edge.eth0-TxRx-40 74.50 ± 2% +102.3% 150.75 interrupts.86:PCI-MSI.1572905-edge.eth0-TxRx-41 75.00 ± 2% +101.7% 151.25 interrupts.87:PCI-MSI.1572906-edge.eth0-TxRx-42 75.25 ± 3% +128.9% 172.25 ± 21% interrupts.88:PCI-MSI.1572907-edge.eth0-TxRx-43 77.75 ± 4% +95.2% 151.75 interrupts.89:PCI-MSI.1572908-edge.eth0-TxRx-44 74.50 ± 2% +137.6% 177.00 ± 19% interrupts.90:PCI-MSI.1572909-edge.eth0-TxRx-45 75.25 ± 3% +100.7% 151.00 interrupts.91:PCI-MSI.1572910-edge.eth0-TxRx-46 74.50 ± 2% +103.0% 151.25 interrupts.92:PCI-MSI.1572911-edge.eth0-TxRx-47 74.75 ± 2% +102.7% 151.50 interrupts.93:PCI-MSI.1572912-edge.eth0-TxRx-48 77.50 ± 6% +96.8% 152.50 interrupts.94:PCI-MSI.1572913-edge.eth0-TxRx-49 81.50 ± 9% +131.6% 188.75 ± 34% interrupts.95:PCI-MSI.1572914-edge.eth0-TxRx-50 80.75 ± 5% +86.7% 150.75 interrupts.96:PCI-MSI.1572915-edge.eth0-TxRx-51 74.50 ± 2% +105.0% 152.75 interrupts.97:PCI-MSI.1572916-edge.eth0-TxRx-52 75.25 ± 3% +100.7% 151.00 interrupts.98:PCI-MSI.1572917-edge.eth0-TxRx-53 74.50 ± 2% +105.0% 152.75 interrupts.99:PCI-MSI.1572918-edge.eth0-TxRx-54 74682381 ± 2% -97.7% 1694822 ± 9% interrupts.CAL:Function_call_interrupts 298.50 ± 25% +273.2% 1114 ±100% interrupts.CPU0.45:PCI-MSI.1572864-edge.eth0-TxRx-0 914498 ± 12% -98.2% 16149 ± 20% interrupts.CPU0.CAL:Function_call_interrupts 280346 ± 4% +117.1% 608594 interrupts.CPU0.LOC:Local_timer_interrupts 17327 ± 22% -71.0% 5024 ± 19% interrupts.CPU0.RES:Rescheduling_interrupts 1021391 ± 11% -98.5% 15097 ± 21% interrupts.CPU0.TLB:TLB_shootdowns 909443 ± 7% -98.2% 16490 ± 21% interrupts.CPU1.CAL:Function_call_interrupts 282063 ± 3% +115.6% 608152 interrupts.CPU1.LOC:Local_timer_interrupts 19343 ± 24% -77.0% 4445 ± 22% interrupts.CPU1.RES:Rescheduling_interrupts 1008464 ± 6% -98.5% 15466 ± 23% interrupts.CPU1.TLB:TLB_shootdowns 87.50 ± 9% +175.1% 240.75 ± 52% interrupts.CPU10.55:PCI-MSI.1572874-edge.eth0-TxRx-10 967023 ± 8% -98.3% 16202 ± 23% interrupts.CPU10.CAL:Function_call_interrupts 283153 ± 4% +115.5% 610267 interrupts.CPU10.LOC:Local_timer_interrupts 17721 ± 20% -77.7% 3959 ± 21% interrupts.CPU10.RES:Rescheduling_interrupts 1073390 ± 7% -98.6% 15176 ± 25% interrupts.CPU10.TLB:TLB_shootdowns 269519 ± 91% -96.2% 10134 ± 16% interrupts.CPU100.CAL:Function_call_interrupts 277314 ± 15% +120.7% 612017 interrupts.CPU100.LOC:Local_timer_interrupts 291603 ± 91% -96.9% 9025 ± 19% interrupts.CPU100.TLB:TLB_shootdowns 291672 ± 87% -95.9% 12020 ± 17% interrupts.CPU101.CAL:Function_call_interrupts 263578 ± 15% +131.3% 609776 interrupts.CPU101.LOC:Local_timer_interrupts 322674 ± 87% -96.6% 10924 ± 18% interrupts.CPU101.TLB:TLB_shootdowns 282315 ± 96% -96.0% 11253 ± 21% interrupts.CPU102.CAL:Function_call_interrupts 266211 ± 15% +130.1% 612517 interrupts.CPU102.LOC:Local_timer_interrupts 307668 ± 96% -96.7% 10133 ± 24% interrupts.CPU102.TLB:TLB_shootdowns 286906 ± 78% -96.3% 10720 ± 17% interrupts.CPU103.CAL:Function_call_interrupts 268266 ± 14% +128.3% 612363 interrupts.CPU103.LOC:Local_timer_interrupts 6378 ± 87% -65.8% 2179 ± 18% interrupts.CPU103.RES:Rescheduling_interrupts 313542 ± 78% -96.9% 9609 ± 19% interrupts.CPU103.TLB:TLB_shootdowns 296983 ± 92% -96.6% 10188 ± 16% interrupts.CPU104.CAL:Function_call_interrupts 275192 ± 16% +123.2% 614344 interrupts.CPU104.LOC:Local_timer_interrupts 6515 ± 95% -65.1% 2275 ± 17% interrupts.CPU104.RES:Rescheduling_interrupts 320665 ± 92% -97.2% 9077 ± 19% interrupts.CPU104.TLB:TLB_shootdowns 296917 ± 88% -97.2% 8326 ± 39% interrupts.CPU105.CAL:Function_call_interrupts 267868 ± 15% +128.7% 612494 interrupts.CPU105.LOC:Local_timer_interrupts 1492 ± 65% -71.3% 428.25 ±173% interrupts.CPU105.NMI:Non-maskable_interrupts 1492 ± 65% -71.3% 428.25 ±173% interrupts.CPU105.PMI:Performance_monitoring_interrupts 326205 ± 87% -97.8% 7227 ± 47% interrupts.CPU105.TLB:TLB_shootdowns 302413 ± 88% -96.8% 9573 ± 30% interrupts.CPU106.CAL:Function_call_interrupts 277351 ± 13% +118.6% 606167 interrupts.CPU106.LOC:Local_timer_interrupts 333260 ± 89% -97.5% 8472 ± 35% interrupts.CPU106.TLB:TLB_shootdowns 269281 ± 60% -96.1% 10489 ± 15% interrupts.CPU107.CAL:Function_call_interrupts 274257 ± 15% +122.8% 610956 interrupts.CPU107.LOC:Local_timer_interrupts 5792 ± 79% -60.8% 2273 ± 17% interrupts.CPU107.RES:Rescheduling_interrupts 298181 ± 62% -96.9% 9384 ± 18% interrupts.CPU107.TLB:TLB_shootdowns 274771 ± 73% -96.2% 10543 ± 43% interrupts.CPU108.CAL:Function_call_interrupts 286626 ± 8% +113.1% 610692 interrupts.CPU108.LOC:Local_timer_interrupts 905.25 ± 20% -79.1% 189.00 ±173% interrupts.CPU108.NMI:Non-maskable_interrupts 905.25 ± 20% -79.1% 189.00 ±173% interrupts.CPU108.PMI:Performance_monitoring_interrupts 304852 ± 76% -96.9% 9447 ± 49% interrupts.CPU108.TLB:TLB_shootdowns 267182 ± 57% -96.6% 9172 ± 63% interrupts.CPU109.CAL:Function_call_interrupts 290439 ± 7% +110.9% 612547 interrupts.CPU109.LOC:Local_timer_interrupts 293957 ± 56% -97.3% 8021 ± 73% interrupts.CPU109.TLB:TLB_shootdowns 87.50 ± 17% +79.4% 157.00 ± 3% interrupts.CPU11.56:PCI-MSI.1572875-edge.eth0-TxRx-11 894660 ± 8% -98.0% 17586 ± 39% interrupts.CPU11.CAL:Function_call_interrupts 287488 ± 7% +111.8% 608811 interrupts.CPU11.LOC:Local_timer_interrupts 2155 ± 37% -75.1% 535.75 ±173% interrupts.CPU11.NMI:Non-maskable_interrupts 2155 ± 37% -75.1% 535.75 ±173% interrupts.CPU11.PMI:Performance_monitoring_interrupts 17329 ± 15% -77.3% 3931 ± 23% interrupts.CPU11.RES:Rescheduling_interrupts 984228 ± 7% -98.3% 16551 ± 42% interrupts.CPU11.TLB:TLB_shootdowns 276755 ± 58% -95.6% 12179 ± 36% interrupts.CPU110.CAL:Function_call_interrupts 279776 ± 9% +118.9% 612464 interrupts.CPU110.LOC:Local_timer_interrupts 1321 ± 34% -78.4% 285.75 ±173% interrupts.CPU110.NMI:Non-maskable_interrupts 1321 ± 34% -78.4% 285.75 ±173% interrupts.CPU110.PMI:Performance_monitoring_interrupts 313463 ± 58% -96.5% 11071 ± 40% interrupts.CPU110.TLB:TLB_shootdowns 221726 ± 64% -95.6% 9685 ± 46% interrupts.CPU111.CAL:Function_call_interrupts 292303 ± 5% +109.5% 612424 interrupts.CPU111.LOC:Local_timer_interrupts 241241 ± 64% -96.4% 8590 ± 52% interrupts.CPU111.TLB:TLB_shootdowns 210200 ± 62% -94.5% 11607 ± 31% interrupts.CPU112.CAL:Function_call_interrupts 292675 ± 5% +109.5% 613263 interrupts.CPU112.LOC:Local_timer_interrupts 1273 ± 46% -80.1% 253.50 ±173% interrupts.CPU112.NMI:Non-maskable_interrupts 1273 ± 46% -80.1% 253.50 ±173% interrupts.CPU112.PMI:Performance_monitoring_interrupts 228976 ± 62% -95.4% 10500 ± 35% interrupts.CPU112.TLB:TLB_shootdowns 223663 ± 63% -94.8% 11647 ± 28% interrupts.CPU113.CAL:Function_call_interrupts 285838 ± 10% +113.7% 610827 interrupts.CPU113.LOC:Local_timer_interrupts 248575 ± 66% -95.7% 10569 ± 32% interrupts.CPU113.TLB:TLB_shootdowns 186560 ± 47% -94.2% 10784 ± 40% interrupts.CPU114.CAL:Function_call_interrupts 291542 ± 6% +110.6% 614097 interrupts.CPU114.LOC:Local_timer_interrupts 1354 ± 42% -73.8% 354.75 ±173% interrupts.CPU114.NMI:Non-maskable_interrupts 1354 ± 42% -73.8% 354.75 ±173% interrupts.CPU114.PMI:Performance_monitoring_interrupts 203076 ± 48% -95.2% 9662 ± 45% interrupts.CPU114.TLB:TLB_shootdowns 221747 ± 56% -95.0% 11107 ± 38% interrupts.CPU115.CAL:Function_call_interrupts 286817 ± 9% +113.1% 611160 interrupts.CPU115.LOC:Local_timer_interrupts 1344 ± 42% -86.9% 175.75 ±173% interrupts.CPU115.NMI:Non-maskable_interrupts 1344 ± 42% -86.9% 175.75 ±173% interrupts.CPU115.PMI:Performance_monitoring_interrupts 247437 ± 59% -96.0% 10007 ± 43% interrupts.CPU115.TLB:TLB_shootdowns 207635 ± 57% -95.5% 9307 ± 47% interrupts.CPU116.CAL:Function_call_interrupts 294323 ± 4% +108.1% 612615 interrupts.CPU116.LOC:Local_timer_interrupts 994.75 ± 64% -79.9% 199.50 ±173% interrupts.CPU116.NMI:Non-maskable_interrupts 994.75 ± 64% -79.9% 199.50 ±173% interrupts.CPU116.PMI:Performance_monitoring_interrupts 4119 ± 51% -44.6% 2282 ± 38% interrupts.CPU116.RES:Rescheduling_interrupts 225292 ± 58% -96.4% 8177 ± 55% interrupts.CPU116.TLB:TLB_shootdowns 197047 ± 45% -94.4% 11115 ± 33% interrupts.CPU117.CAL:Function_call_interrupts 286330 ± 9% +114.0% 612849 interrupts.CPU117.LOC:Local_timer_interrupts 214135 ± 45% -95.3% 9996 ± 36% interrupts.CPU117.TLB:TLB_shootdowns 249591 ± 49% -96.6% 8376 ± 40% interrupts.CPU118.CAL:Function_call_interrupts 282487 ± 7% +116.2% 610709 interrupts.CPU118.LOC:Local_timer_interrupts 1467 ± 41% -89.9% 148.75 ±173% interrupts.CPU118.NMI:Non-maskable_interrupts 1467 ± 41% -89.9% 148.75 ±173% interrupts.CPU118.PMI:Performance_monitoring_interrupts 278627 ± 47% -97.4% 7259 ± 47% interrupts.CPU118.TLB:TLB_shootdowns 227464 ± 66% -96.4% 8104 ± 45% interrupts.CPU119.CAL:Function_call_interrupts 286525 ± 9% +114.0% 613265 interrupts.CPU119.LOC:Local_timer_interrupts 1100 ± 49% -89.9% 111.00 ±173% interrupts.CPU119.NMI:Non-maskable_interrupts 1100 ± 49% -89.9% 111.00 ±173% interrupts.CPU119.PMI:Performance_monitoring_interrupts 250689 ± 68% -97.2% 6988 ± 54% interrupts.CPU119.TLB:TLB_shootdowns 910218 ± 12% -97.9% 18993 ± 34% interrupts.CPU12.CAL:Function_call_interrupts 292193 ± 5% +108.4% 608902 interrupts.CPU12.LOC:Local_timer_interrupts 2446 ± 13% -89.0% 269.50 ±173% interrupts.CPU12.NMI:Non-maskable_interrupts 2446 ± 13% -89.0% 269.50 ±173% interrupts.CPU12.PMI:Performance_monitoring_interrupts 17337 ± 21% -77.8% 3856 ± 25% interrupts.CPU12.RES:Rescheduling_interrupts 1005979 ± 12% -98.2% 17975 ± 36% interrupts.CPU12.TLB:TLB_shootdowns 221222 ± 51% -96.1% 8640 ± 60% interrupts.CPU120.CAL:Function_call_interrupts 282436 ± 7% +116.7% 612115 interrupts.CPU120.LOC:Local_timer_interrupts 869.00 ± 49% -81.7% 158.75 ±173% interrupts.CPU120.NMI:Non-maskable_interrupts 869.00 ± 49% -81.7% 158.75 ±173% interrupts.CPU120.PMI:Performance_monitoring_interrupts 249558 ± 50% -97.0% 7539 ± 70% interrupts.CPU120.TLB:TLB_shootdowns 199507 ± 56% -94.7% 10491 ± 38% interrupts.CPU121.CAL:Function_call_interrupts 286909 ± 9% +113.1% 611343 interrupts.CPU121.LOC:Local_timer_interrupts 1434 ± 38% -89.2% 154.50 ±173% interrupts.CPU121.NMI:Non-maskable_interrupts 1434 ± 38% -89.2% 154.50 ±173% interrupts.CPU121.PMI:Performance_monitoring_interrupts 219815 ± 56% -95.7% 9391 ± 43% interrupts.CPU121.TLB:TLB_shootdowns 255129 ± 42% -96.4% 9212 ± 54% interrupts.CPU122.CAL:Function_call_interrupts 276573 ± 8% +121.5% 612542 interrupts.CPU122.LOC:Local_timer_interrupts 3699 ± 41% -38.6% 2270 ± 41% interrupts.CPU122.RES:Rescheduling_interrupts 292981 ± 44% -97.2% 8103 ± 62% interrupts.CPU122.TLB:TLB_shootdowns 217347 ± 66% -95.6% 9586 ± 54% interrupts.CPU123.CAL:Function_call_interrupts 284138 ± 10% +115.4% 611925 interrupts.CPU123.LOC:Local_timer_interrupts 965.25 ± 63% -84.7% 148.00 ±173% interrupts.CPU123.NMI:Non-maskable_interrupts 965.25 ± 63% -84.7% 148.00 ±173% interrupts.CPU123.PMI:Performance_monitoring_interrupts 243187 ± 68% -96.5% 8456 ± 62% interrupts.CPU123.TLB:TLB_shootdowns 223027 ± 58% -95.4% 10365 ± 50% interrupts.CPU124.CAL:Function_call_interrupts 285647 ± 10% +114.2% 611947 interrupts.CPU124.LOC:Local_timer_interrupts 1436 ± 60% -80.2% 285.00 ±173% interrupts.CPU124.NMI:Non-maskable_interrupts 1436 ± 60% -80.2% 285.00 ±173% interrupts.CPU124.PMI:Performance_monitoring_interrupts 249245 ± 61% -96.3% 9262 ± 57% interrupts.CPU124.TLB:TLB_shootdowns 195393 ± 41% -95.1% 9578 ± 69% interrupts.CPU125.CAL:Function_call_interrupts 286776 ± 9% +113.0% 610855 interrupts.CPU125.LOC:Local_timer_interrupts 4495 ± 47% -46.8% 2390 ± 37% interrupts.CPU125.RES:Rescheduling_interrupts 210269 ± 41% -96.0% 8486 ± 79% interrupts.CPU125.TLB:TLB_shootdowns 320707 ± 81% -97.3% 8610 ± 31% interrupts.CPU126.CAL:Function_call_interrupts 270205 ± 14% +126.5% 612144 interrupts.CPU126.LOC:Local_timer_interrupts 5986 ± 80% -67.8% 1929 ± 3% interrupts.CPU126.RES:Rescheduling_interrupts 350384 ± 81% -97.8% 7535 ± 37% interrupts.CPU126.TLB:TLB_shootdowns 293072 ± 79% -97.2% 8262 ± 16% interrupts.CPU127.CAL:Function_call_interrupts 281750 ± 12% +116.5% 610044 interrupts.CPU127.LOC:Local_timer_interrupts 1084 ± 30% -80.2% 214.75 ±173% interrupts.CPU127.NMI:Non-maskable_interrupts 1084 ± 30% -80.2% 214.75 ±173% interrupts.CPU127.PMI:Performance_monitoring_interrupts 5495 ± 88% -68.1% 1750 ± 13% interrupts.CPU127.RES:Rescheduling_interrupts 319049 ± 79% -97.8% 7149 ± 18% interrupts.CPU127.TLB:TLB_shootdowns 311232 ± 69% -97.7% 7248 ± 8% interrupts.CPU128.CAL:Function_call_interrupts 278575 ± 14% +119.0% 610071 interrupts.CPU128.LOC:Local_timer_interrupts 5886 ± 68% -67.6% 1908 ± 7% interrupts.CPU128.RES:Rescheduling_interrupts 338691 ± 69% -98.2% 6091 ± 8% interrupts.CPU128.TLB:TLB_shootdowns 314267 ± 72% -98.1% 5840 ± 29% interrupts.CPU129.CAL:Function_call_interrupts 270295 ± 14% +127.0% 613704 interrupts.CPU129.LOC:Local_timer_interrupts 5654 ± 72% -67.2% 1852 ± 10% interrupts.CPU129.RES:Rescheduling_interrupts 342768 ± 72% -98.6% 4732 ± 36% interrupts.CPU129.TLB:TLB_shootdowns 896870 ± 11% -98.1% 16952 ± 35% interrupts.CPU13.CAL:Function_call_interrupts 279444 ± 5% +117.6% 607983 interrupts.CPU13.LOC:Local_timer_interrupts 2058 ± 15% -86.7% 273.00 ±173% interrupts.CPU13.NMI:Non-maskable_interrupts 2058 ± 15% -86.7% 273.00 ±173% interrupts.CPU13.PMI:Performance_monitoring_interrupts 20397 ± 39% -81.1% 3858 ± 27% interrupts.CPU13.RES:Rescheduling_interrupts 998200 ± 11% -98.4% 15914 ± 38% interrupts.CPU13.TLB:TLB_shootdowns 283507 ± 70% -96.6% 9557 ± 69% interrupts.CPU130.CAL:Function_call_interrupts 276066 ± 15% +122.3% 613584 interrupts.CPU130.LOC:Local_timer_interrupts 1240 ± 17% -87.3% 157.25 ±173% interrupts.CPU130.NMI:Non-maskable_interrupts 1240 ± 17% -87.3% 157.25 ±173% interrupts.CPU130.PMI:Performance_monitoring_interrupts 5019 ± 80% -67.0% 1657 ± 11% interrupts.CPU130.RES:Rescheduling_interrupts 310421 ± 70% -97.3% 8484 ± 80% interrupts.CPU130.TLB:TLB_shootdowns 268465 ± 70% -97.7% 6061 ± 20% interrupts.CPU131.CAL:Function_call_interrupts 273953 ± 15% +122.6% 609689 interrupts.CPU131.LOC:Local_timer_interrupts 1714 ± 45% -88.5% 197.75 ±173% interrupts.CPU131.NMI:Non-maskable_interrupts 1714 ± 45% -88.5% 197.75 ±173% interrupts.CPU131.PMI:Performance_monitoring_interrupts 4887 ± 58% -67.1% 1607 ± 11% interrupts.CPU131.RES:Rescheduling_interrupts 298280 ± 73% -98.3% 4943 ± 23% interrupts.CPU131.TLB:TLB_shootdowns 286408 ± 78% -96.8% 9121 ± 29% interrupts.CPU132.CAL:Function_call_interrupts 274954 ± 16% +123.1% 613425 interrupts.CPU132.LOC:Local_timer_interrupts 988.75 ± 7% -78.7% 210.75 ±173% interrupts.CPU132.NMI:Non-maskable_interrupts 988.75 ± 7% -78.7% 210.75 ±173% interrupts.CPU132.PMI:Performance_monitoring_interrupts 5424 ± 77% -68.6% 1703 ± 19% interrupts.CPU132.RES:Rescheduling_interrupts 314842 ± 78% -97.5% 8013 ± 33% interrupts.CPU132.TLB:TLB_shootdowns 296934 ± 59% -96.9% 9119 ± 51% interrupts.CPU133.CAL:Function_call_interrupts 273477 ± 15% +123.9% 612314 interrupts.CPU133.LOC:Local_timer_interrupts 5032 ± 78% -65.9% 1714 ± 4% interrupts.CPU133.RES:Rescheduling_interrupts 332213 ± 56% -97.6% 8002 ± 57% interrupts.CPU133.TLB:TLB_shootdowns 269174 ± 62% -96.6% 9280 ± 34% interrupts.CPU134.CAL:Function_call_interrupts 275457 ± 16% +122.7% 613514 interrupts.CPU134.LOC:Local_timer_interrupts 23121 ±132% -92.8% 1667 ± 5% interrupts.CPU134.RES:Rescheduling_interrupts 297939 ± 60% -97.3% 8179 ± 39% interrupts.CPU134.TLB:TLB_shootdowns 304830 ± 73% -97.9% 6461 ± 32% interrupts.CPU135.CAL:Function_call_interrupts 266542 ± 15% +130.1% 613417 interrupts.CPU135.LOC:Local_timer_interrupts 1629 ± 34% -87.2% 208.75 ±173% interrupts.CPU135.NMI:Non-maskable_interrupts 1629 ± 34% -87.2% 208.75 ±173% interrupts.CPU135.PMI:Performance_monitoring_interrupts 5423 ± 65% -70.1% 1624 ± 5% interrupts.CPU135.RES:Rescheduling_interrupts 333755 ± 74% -98.4% 5348 ± 38% interrupts.CPU135.TLB:TLB_shootdowns 293065 ± 64% -97.2% 8101 ± 34% interrupts.CPU136.CAL:Function_call_interrupts 277110 ± 15% +120.3% 610447 interrupts.CPU136.LOC:Local_timer_interrupts 5399 ± 67% -67.1% 1776 interrupts.CPU136.RES:Rescheduling_interrupts 319744 ± 64% -97.8% 6994 ± 40% interrupts.CPU136.TLB:TLB_shootdowns 314300 ± 69% -97.6% 7414 ± 35% interrupts.CPU137.CAL:Function_call_interrupts 263007 ± 13% +132.4% 611317 interrupts.CPU137.LOC:Local_timer_interrupts 917.25 ± 23% -78.7% 195.25 ±173% interrupts.CPU137.NMI:Non-maskable_interrupts 917.25 ± 23% -78.7% 195.25 ±173% interrupts.CPU137.PMI:Performance_monitoring_interrupts 4262 ± 72% -60.5% 1684 ± 8% interrupts.CPU137.RES:Rescheduling_interrupts 362554 ± 70% -98.3% 6304 ± 41% interrupts.CPU137.TLB:TLB_shootdowns 296839 ± 64% -96.9% 9319 ± 50% interrupts.CPU138.CAL:Function_call_interrupts 264453 ± 17% +132.2% 614073 interrupts.CPU138.LOC:Local_timer_interrupts 4639 ± 78% -60.0% 1853 ± 8% interrupts.CPU138.RES:Rescheduling_interrupts 329188 ± 63% -97.5% 8184 ± 58% interrupts.CPU138.TLB:TLB_shootdowns 273595 ± 69% -97.6% 6579 ± 19% interrupts.CPU139.CAL:Function_call_interrupts 280505 ± 13% +117.5% 610220 interrupts.CPU139.LOC:Local_timer_interrupts 5066 ± 72% -66.1% 1716 ± 4% interrupts.CPU139.RES:Rescheduling_interrupts 299908 ± 68% -98.2% 5454 ± 24% interrupts.CPU139.TLB:TLB_shootdowns 949929 ± 2% -98.2% 17323 ± 22% interrupts.CPU14.CAL:Function_call_interrupts 276071 ± 2% +119.9% 607061 interrupts.CPU14.LOC:Local_timer_interrupts 1478 ± 27% -81.4% 274.50 ±173% interrupts.CPU14.NMI:Non-maskable_interrupts 1478 ± 27% -81.4% 274.50 ±173% interrupts.CPU14.PMI:Performance_monitoring_interrupts 16967 ± 23% -77.2% 3868 ± 24% interrupts.CPU14.RES:Rescheduling_interrupts 1060256 ± 3% -98.5% 16286 ± 23% interrupts.CPU14.TLB:TLB_shootdowns 268862 ± 64% -97.3% 7361 ± 17% interrupts.CPU140.CAL:Function_call_interrupts 272574 ± 14% +125.2% 613808 interrupts.CPU140.LOC:Local_timer_interrupts 1098 ± 40% -80.5% 214.25 ±173% interrupts.CPU140.NMI:Non-maskable_interrupts 1098 ± 40% -80.5% 214.25 ±173% interrupts.CPU140.PMI:Performance_monitoring_interrupts 4746 ± 75% -61.7% 1819 ± 3% interrupts.CPU140.RES:Rescheduling_interrupts 297616 ± 63% -97.9% 6238 ± 21% interrupts.CPU140.TLB:TLB_shootdowns 243590 ± 54% -95.8% 10219 ± 54% interrupts.CPU141.CAL:Function_call_interrupts 282721 ± 11% +116.5% 612168 interrupts.CPU141.LOC:Local_timer_interrupts 1112 ± 45% -84.4% 173.00 ±173% interrupts.CPU141.NMI:Non-maskable_interrupts 1112 ± 45% -84.4% 173.00 ±173% interrupts.CPU141.PMI:Performance_monitoring_interrupts 4972 ± 77% -65.1% 1735 ± 11% interrupts.CPU141.RES:Rescheduling_interrupts 264794 ± 54% -96.6% 9118 ± 62% interrupts.CPU141.TLB:TLB_shootdowns 316276 ± 44% -96.7% 10543 ± 32% interrupts.CPU142.CAL:Function_call_interrupts 265522 ± 16% +131.1% 613546 interrupts.CPU142.LOC:Local_timer_interrupts 366015 ± 41% -97.4% 9513 ± 36% interrupts.CPU142.TLB:TLB_shootdowns 285901 ± 51% -97.0% 8435 ± 34% interrupts.CPU143.CAL:Function_call_interrupts 265658 ± 14% +129.7% 610109 interrupts.CPU143.LOC:Local_timer_interrupts 1285 ± 64% -83.2% 215.50 ±173% interrupts.CPU143.NMI:Non-maskable_interrupts 1285 ± 64% -83.2% 215.50 ±173% interrupts.CPU143.PMI:Performance_monitoring_interrupts 4872 ± 70% -61.4% 1879 ± 13% interrupts.CPU143.RES:Rescheduling_interrupts 320459 ± 48% -97.7% 7335 ± 38% interrupts.CPU143.TLB:TLB_shootdowns 122.25 ± 63% +949.5% 1283 ± 57% interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15 930715 ± 8% -98.3% 15706 ± 22% interrupts.CPU15.CAL:Function_call_interrupts 288293 ± 5% +111.2% 608914 interrupts.CPU15.LOC:Local_timer_interrupts 17785 ± 19% -78.3% 3862 ± 22% interrupts.CPU15.RES:Rescheduling_interrupts 1032325 ± 7% -98.6% 14665 ± 24% interrupts.CPU15.TLB:TLB_shootdowns 75.25 +101.7% 151.75 interrupts.CPU16.61:PCI-MSI.1572880-edge.eth0-TxRx-16 846100 ± 14% -98.1% 15849 ± 23% interrupts.CPU16.CAL:Function_call_interrupts 290352 ± 5% +109.0% 606827 interrupts.CPU16.LOC:Local_timer_interrupts 16641 ± 26% -77.3% 3776 ± 24% interrupts.CPU16.RES:Rescheduling_interrupts 937358 ± 13% -98.4% 14790 ± 25% interrupts.CPU16.TLB:TLB_shootdowns 74.75 +105.0% 153.25 ± 2% interrupts.CPU17.62:PCI-MSI.1572881-edge.eth0-TxRx-17 949351 ± 13% -98.2% 17097 ± 17% interrupts.CPU17.CAL:Function_call_interrupts 285597 ± 7% +113.2% 609036 interrupts.CPU17.LOC:Local_timer_interrupts 16663 ± 20% -76.8% 3868 ± 21% interrupts.CPU17.RES:Rescheduling_interrupts 1060495 ± 14% -98.5% 16045 ± 19% interrupts.CPU17.TLB:TLB_shootdowns 75.75 +99.3% 151.00 interrupts.CPU18.63:PCI-MSI.1572882-edge.eth0-TxRx-18 645547 ± 17% -98.3% 10717 ± 29% interrupts.CPU18.CAL:Function_call_interrupts 278062 ± 13% +121.1% 614758 interrupts.CPU18.LOC:Local_timer_interrupts 880.25 ± 25% -73.2% 235.75 ±172% interrupts.CPU18.NMI:Non-maskable_interrupts 880.25 ± 25% -73.2% 235.75 ±172% interrupts.CPU18.PMI:Performance_monitoring_interrupts 10884 ± 43% -73.0% 2943 ± 16% interrupts.CPU18.RES:Rescheduling_interrupts 707614 ± 17% -98.6% 9629 ± 33% interrupts.CPU18.TLB:TLB_shootdowns 80.25 ± 5% +91.3% 153.50 ± 3% interrupts.CPU19.64:PCI-MSI.1572883-edge.eth0-TxRx-19 673179 ± 30% -98.1% 12592 ± 26% interrupts.CPU19.CAL:Function_call_interrupts 275442 ± 15% +122.8% 613740 interrupts.CPU19.LOC:Local_timer_interrupts 10948 ± 52% -73.9% 2852 ± 20% interrupts.CPU19.RES:Rescheduling_interrupts 746991 ± 30% -98.5% 11521 ± 29% interrupts.CPU19.TLB:TLB_shootdowns 132.50 ± 72% +100.4% 265.50 ± 42% interrupts.CPU2.47:PCI-MSI.1572866-edge.eth0-TxRx-2 933273 ± 11% -98.2% 17177 ± 26% interrupts.CPU2.CAL:Function_call_interrupts 271018 ± 5% +123.8% 606417 interrupts.CPU2.LOC:Local_timer_interrupts 2413 ± 26% -76.8% 559.50 ±173% interrupts.CPU2.NMI:Non-maskable_interrupts 2413 ± 26% -76.8% 559.50 ±173% interrupts.CPU2.PMI:Performance_monitoring_interrupts 16329 ± 19% -75.7% 3961 ± 24% interrupts.CPU2.RES:Rescheduling_interrupts 1038537 ± 11% -98.4% 16147 ± 27% interrupts.CPU2.TLB:TLB_shootdowns 79.25 ± 5% +90.2% 150.75 interrupts.CPU20.65:PCI-MSI.1572884-edge.eth0-TxRx-20 609982 ± 23% -98.0% 12313 ± 28% interrupts.CPU20.CAL:Function_call_interrupts 275379 ± 15% +121.6% 610307 interrupts.CPU20.LOC:Local_timer_interrupts 10528 ± 50% -64.6% 3727 ± 48% interrupts.CPU20.RES:Rescheduling_interrupts 670139 ± 23% -98.3% 11230 ± 32% interrupts.CPU20.TLB:TLB_shootdowns 80.00 ± 7% +89.7% 151.75 interrupts.CPU21.66:PCI-MSI.1572885-edge.eth0-TxRx-21 607863 ± 26% -98.1% 11742 ± 37% interrupts.CPU21.CAL:Function_call_interrupts 276045 ± 15% +120.9% 609811 interrupts.CPU21.LOC:Local_timer_interrupts 1186 ± 49% -83.1% 201.00 ±173% interrupts.CPU21.NMI:Non-maskable_interrupts 1186 ± 49% -83.1% 201.00 ±173% interrupts.CPU21.PMI:Performance_monitoring_interrupts 10980 ± 48% -73.8% 2874 ± 19% interrupts.CPU21.RES:Rescheduling_interrupts 664177 ± 25% -98.4% 10632 ± 41% interrupts.CPU21.TLB:TLB_shootdowns 74.50 ± 2% +103.7% 151.75 interrupts.CPU22.67:PCI-MSI.1572886-edge.eth0-TxRx-22 600537 ± 24% -98.3% 10181 ± 22% interrupts.CPU22.CAL:Function_call_interrupts 275077 ± 16% +121.2% 608414 interrupts.CPU22.LOC:Local_timer_interrupts 10789 ± 51% -75.6% 2628 ± 24% interrupts.CPU22.RES:Rescheduling_interrupts 655143 ± 24% -98.6% 9095 ± 26% interrupts.CPU22.TLB:TLB_shootdowns 630405 ± 23% -98.6% 8847 ± 32% interrupts.CPU23.CAL:Function_call_interrupts 279751 ± 12% +119.1% 612931 interrupts.CPU23.LOC:Local_timer_interrupts 1528 ± 17% -86.0% 214.25 ±172% interrupts.CPU23.NMI:Non-maskable_interrupts 1528 ± 17% -86.0% 214.25 ±172% interrupts.CPU23.PMI:Performance_monitoring_interrupts 11182 ± 48% -74.7% 2823 ± 20% interrupts.CPU23.RES:Rescheduling_interrupts 688294 ± 23% -98.9% 7748 ± 38% interrupts.CPU23.TLB:TLB_shootdowns 75.75 ± 3% +100.3% 151.75 interrupts.CPU24.69:PCI-MSI.1572888-edge.eth0-TxRx-24 652396 ± 18% -98.5% 9977 ± 40% interrupts.CPU24.CAL:Function_call_interrupts 270545 ± 14% +125.8% 610813 interrupts.CPU24.LOC:Local_timer_interrupts 11363 ± 46% -74.9% 2850 ± 15% interrupts.CPU24.RES:Rescheduling_interrupts 723683 ± 16% -98.8% 8872 ± 46% interrupts.CPU24.TLB:TLB_shootdowns 78.25 ± 8% +92.7% 150.75 interrupts.CPU25.70:PCI-MSI.1572889-edge.eth0-TxRx-25 647055 ± 17% -98.1% 12227 ± 15% interrupts.CPU25.CAL:Function_call_interrupts 270960 ± 16% +125.3% 610355 interrupts.CPU25.LOC:Local_timer_interrupts 1591 ± 23% -87.1% 205.75 ±173% interrupts.CPU25.NMI:Non-maskable_interrupts 1591 ± 23% -87.1% 205.75 ±173% interrupts.CPU25.PMI:Performance_monitoring_interrupts 10595 ± 43% -72.6% 2899 ± 14% interrupts.CPU25.RES:Rescheduling_interrupts 715159 ± 16% -98.4% 11161 ± 16% interrupts.CPU25.TLB:TLB_shootdowns 76.50 ± 3% +104.9% 156.75 ± 3% interrupts.CPU26.71:PCI-MSI.1572890-edge.eth0-TxRx-26 667160 ± 27% -98.2% 11849 ± 23% interrupts.CPU26.CAL:Function_call_interrupts 271990 ± 15% +124.3% 610136 interrupts.CPU26.LOC:Local_timer_interrupts 1457 ± 20% -84.3% 228.75 ±173% interrupts.CPU26.NMI:Non-maskable_interrupts 1457 ± 20% -84.3% 228.75 ±173% interrupts.CPU26.PMI:Performance_monitoring_interrupts 10385 ± 43% -73.9% 2711 ± 14% interrupts.CPU26.RES:Rescheduling_interrupts 747329 ± 29% -98.6% 10721 ± 26% interrupts.CPU26.TLB:TLB_shootdowns 77.75 ± 7% +93.9% 150.75 interrupts.CPU27.72:PCI-MSI.1572891-edge.eth0-TxRx-27 643390 ± 23% -98.3% 10665 ± 14% interrupts.CPU27.CAL:Function_call_interrupts 269670 ± 12% +127.0% 612112 interrupts.CPU27.LOC:Local_timer_interrupts 9744 ± 44% -69.0% 3016 ± 15% interrupts.CPU27.RES:Rescheduling_interrupts 716367 ± 23% -98.7% 9600 ± 16% interrupts.CPU27.TLB:TLB_shootdowns 84.00 ± 11% +80.4% 151.50 interrupts.CPU28.73:PCI-MSI.1572892-edge.eth0-TxRx-28 674461 ± 22% -98.5% 10449 ± 17% interrupts.CPU28.CAL:Function_call_interrupts 267114 ± 15% +129.9% 614200 interrupts.CPU28.LOC:Local_timer_interrupts 1122 ± 41% -80.0% 224.25 ±173% interrupts.CPU28.NMI:Non-maskable_interrupts 1122 ± 41% -80.0% 224.25 ±173% interrupts.CPU28.PMI:Performance_monitoring_interrupts 10858 ± 52% -73.1% 2919 ± 16% interrupts.CPU28.RES:Rescheduling_interrupts 742288 ± 21% -98.7% 9361 ± 19% interrupts.CPU28.TLB:TLB_shootdowns 76.75 +96.7% 151.00 interrupts.CPU29.74:PCI-MSI.1572893-edge.eth0-TxRx-29 618163 ± 31% -98.0% 12304 ± 17% interrupts.CPU29.CAL:Function_call_interrupts 273122 ± 17% +123.7% 610915 interrupts.CPU29.LOC:Local_timer_interrupts 1545 ± 30% -75.7% 375.00 ±173% interrupts.CPU29.NMI:Non-maskable_interrupts 1545 ± 30% -75.7% 375.00 ±173% interrupts.CPU29.PMI:Performance_monitoring_interrupts 10589 ± 54% -74.3% 2718 ± 21% interrupts.CPU29.RES:Rescheduling_interrupts 677771 ± 31% -98.3% 11207 ± 20% interrupts.CPU29.TLB:TLB_shootdowns 93.00 ± 31% +209.9% 288.25 ± 48% interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3 923756 ± 7% -98.0% 18900 ± 24% interrupts.CPU3.CAL:Function_call_interrupts 284849 ± 4% +114.3% 610562 interrupts.CPU3.LOC:Local_timer_interrupts 17053 ± 22% -75.9% 4113 ± 20% interrupts.CPU3.RES:Rescheduling_interrupts 1025520 ± 6% -98.3% 17871 ± 25% interrupts.CPU3.TLB:TLB_shootdowns 606421 ± 26% -97.5% 14896 ± 28% interrupts.CPU30.CAL:Function_call_interrupts 273717 ± 17% +123.6% 612130 interrupts.CPU30.LOC:Local_timer_interrupts 10123 ± 56% -69.9% 3044 ± 8% interrupts.CPU30.RES:Rescheduling_interrupts 678043 ± 25% -98.0% 13799 ± 30% interrupts.CPU30.TLB:TLB_shootdowns 75.75 ± 2% +99.3% 151.00 interrupts.CPU31.76:PCI-MSI.1572895-edge.eth0-TxRx-31 614682 ± 21% -98.1% 11811 ± 27% interrupts.CPU31.CAL:Function_call_interrupts 273565 ± 15% +124.0% 612786 interrupts.CPU31.LOC:Local_timer_interrupts 1863 ± 32% -77.6% 418.25 ±173% interrupts.CPU31.NMI:Non-maskable_interrupts 1863 ± 32% -77.6% 418.25 ±173% interrupts.CPU31.PMI:Performance_monitoring_interrupts 11072 ± 50% -74.9% 2780 ± 18% interrupts.CPU31.RES:Rescheduling_interrupts 673714 ± 20% -98.4% 10709 ± 30% interrupts.CPU31.TLB:TLB_shootdowns 74.50 ± 2% +102.3% 150.75 interrupts.CPU32.77:PCI-MSI.1572896-edge.eth0-TxRx-32 657006 ± 30% -98.2% 12098 ± 21% interrupts.CPU32.CAL:Function_call_interrupts 274908 ± 16% +121.9% 610112 interrupts.CPU32.LOC:Local_timer_interrupts 11320 ± 48% -76.3% 2680 ± 22% interrupts.CPU32.RES:Rescheduling_interrupts 719498 ± 30% -98.5% 10981 ± 24% interrupts.CPU32.TLB:TLB_shootdowns 76.75 ± 3% +102.6% 155.50 ± 5% interrupts.CPU33.78:PCI-MSI.1572897-edge.eth0-TxRx-33 663241 ± 24% -98.4% 10893 ± 22% interrupts.CPU33.CAL:Function_call_interrupts 278400 ± 13% +118.6% 608674 interrupts.CPU33.LOC:Local_timer_interrupts 11796 ± 47% -76.2% 2812 ± 15% interrupts.CPU33.RES:Rescheduling_interrupts 731432 ± 23% -98.7% 9798 ± 26% interrupts.CPU33.TLB:TLB_shootdowns 75.25 +100.7% 151.00 interrupts.CPU34.79:PCI-MSI.1572898-edge.eth0-TxRx-34 642251 ± 24% -98.4% 10101 ± 30% interrupts.CPU34.CAL:Function_call_interrupts 268454 ± 16% +128.7% 613994 interrupts.CPU34.LOC:Local_timer_interrupts 9906 ± 49% -73.7% 2609 ± 23% interrupts.CPU34.RES:Rescheduling_interrupts 718139 ± 24% -98.7% 8999 ± 35% interrupts.CPU34.TLB:TLB_shootdowns 79.00 ± 8% +91.1% 151.00 interrupts.CPU35.80:PCI-MSI.1572899-edge.eth0-TxRx-35 599308 ± 24% -98.1% 11108 ± 13% interrupts.CPU35.CAL:Function_call_interrupts 280035 ± 11% +119.0% 613246 interrupts.CPU35.LOC:Local_timer_interrupts 11089 ± 49% -75.9% 2677 ± 16% interrupts.CPU35.RES:Rescheduling_interrupts 653661 ± 23% -98.5% 10004 ± 15% interrupts.CPU35.TLB:TLB_shootdowns 105.25 ± 30% +50.8% 158.75 ± 7% interrupts.CPU36.81:PCI-MSI.1572900-edge.eth0-TxRx-36 576675 ± 34% -97.8% 12733 ± 38% interrupts.CPU36.CAL:Function_call_interrupts 280316 ± 7% +117.8% 610536 interrupts.CPU36.LOC:Local_timer_interrupts 8217 ± 46% -65.1% 2868 ± 31% interrupts.CPU36.RES:Rescheduling_interrupts 639697 ± 34% -98.2% 11682 ± 42% interrupts.CPU36.TLB:TLB_shootdowns 75.25 ± 3% +101.0% 151.25 interrupts.CPU37.82:PCI-MSI.1572901-edge.eth0-TxRx-37 585255 ± 34% -98.1% 11315 ± 48% interrupts.CPU37.CAL:Function_call_interrupts 282162 ± 7% +117.3% 613248 interrupts.CPU37.LOC:Local_timer_interrupts 1790 ± 41% -82.4% 314.75 ±173% interrupts.CPU37.NMI:Non-maskable_interrupts 1790 ± 41% -82.4% 314.75 ±173% interrupts.CPU37.PMI:Performance_monitoring_interrupts 8908 ± 46% -66.8% 2955 ± 28% interrupts.CPU37.RES:Rescheduling_interrupts 646131 ± 34% -98.4% 10235 ± 54% interrupts.CPU37.TLB:TLB_shootdowns 75.00 +118.0% 163.50 ± 12% interrupts.CPU38.83:PCI-MSI.1572902-edge.eth0-TxRx-38 571165 ± 35% -97.9% 11914 ± 43% interrupts.CPU38.CAL:Function_call_interrupts 284027 ± 7% +115.6% 612253 interrupts.CPU38.LOC:Local_timer_interrupts 869.75 ± 28% -83.4% 144.00 ±173% interrupts.CPU38.NMI:Non-maskable_interrupts 869.75 ± 28% -83.4% 144.00 ±173% interrupts.CPU38.PMI:Performance_monitoring_interrupts 8272 ± 46% -66.5% 2774 ± 29% interrupts.CPU38.RES:Rescheduling_interrupts 629458 ± 36% -98.3% 10836 ± 48% interrupts.CPU38.TLB:TLB_shootdowns 75.25 ± 3% +100.3% 150.75 interrupts.CPU39.84:PCI-MSI.1572903-edge.eth0-TxRx-39 598700 ± 42% -98.0% 11874 ± 46% interrupts.CPU39.CAL:Function_call_interrupts 279872 ± 9% +117.3% 608251 interrupts.CPU39.LOC:Local_timer_interrupts 9117 ± 38% -69.0% 2826 ± 26% interrupts.CPU39.RES:Rescheduling_interrupts 657735 ± 43% -98.4% 10793 ± 52% interrupts.CPU39.TLB:TLB_shootdowns 953724 ± 9% -98.1% 17800 ± 31% interrupts.CPU4.CAL:Function_call_interrupts 280572 ± 4% +116.5% 607494 interrupts.CPU4.LOC:Local_timer_interrupts 17128 ± 23% -77.3% 3896 ± 24% interrupts.CPU4.RES:Rescheduling_interrupts 1060132 ± 8% -98.4% 16760 ± 33% interrupts.CPU4.TLB:TLB_shootdowns 75.25 ± 3% +101.0% 151.25 interrupts.CPU40.85:PCI-MSI.1572904-edge.eth0-TxRx-40 603821 ± 36% -98.0% 11876 ± 45% interrupts.CPU40.CAL:Function_call_interrupts 282927 ± 8% +116.1% 611404 interrupts.CPU40.LOC:Local_timer_interrupts 1638 ± 60% -91.9% 132.75 ±173% interrupts.CPU40.NMI:Non-maskable_interrupts 1638 ± 60% -91.9% 132.75 ±173% interrupts.CPU40.PMI:Performance_monitoring_interrupts 8973 ± 45% -69.6% 2728 ± 31% interrupts.CPU40.RES:Rescheduling_interrupts 671599 ± 36% -98.4% 10786 ± 50% interrupts.CPU40.TLB:TLB_shootdowns 74.50 ± 2% +102.3% 150.75 interrupts.CPU41.86:PCI-MSI.1572905-edge.eth0-TxRx-41 578720 ± 35% -98.1% 10865 ± 48% interrupts.CPU41.CAL:Function_call_interrupts 287826 ± 8% +111.8% 609750 interrupts.CPU41.LOC:Local_timer_interrupts 1253 ± 20% -77.0% 288.00 ±173% interrupts.CPU41.NMI:Non-maskable_interrupts 1253 ± 20% -77.0% 288.00 ±173% interrupts.CPU41.PMI:Performance_monitoring_interrupts 8548 ± 37% -66.7% 2845 ± 31% interrupts.CPU41.RES:Rescheduling_interrupts 641704 ± 35% -98.5% 9786 ± 54% interrupts.CPU41.TLB:TLB_shootdowns 75.00 ± 2% +101.7% 151.25 interrupts.CPU42.87:PCI-MSI.1572906-edge.eth0-TxRx-42 540606 ± 34% -97.7% 12546 ± 32% interrupts.CPU42.CAL:Function_call_interrupts 282054 ± 11% +115.7% 608457 interrupts.CPU42.LOC:Local_timer_interrupts 8570 ± 35% -65.9% 2926 ± 28% interrupts.CPU42.RES:Rescheduling_interrupts 601634 ± 36% -98.1% 11456 ± 36% interrupts.CPU42.TLB:TLB_shootdowns 75.25 ± 3% +128.9% 172.25 ± 21% interrupts.CPU43.88:PCI-MSI.1572907-edge.eth0-TxRx-43 616654 ± 25% -97.6% 14630 ± 27% interrupts.CPU43.CAL:Function_call_interrupts 277903 ± 9% +119.7% 610610 interrupts.CPU43.LOC:Local_timer_interrupts 9259 ± 40% -67.3% 3031 ± 27% interrupts.CPU43.RES:Rescheduling_interrupts 680242 ± 25% -98.0% 13552 ± 29% interrupts.CPU43.TLB:TLB_shootdowns 77.75 ± 4% +95.2% 151.75 interrupts.CPU44.89:PCI-MSI.1572908-edge.eth0-TxRx-44 605519 ± 37% -98.1% 11416 ± 30% interrupts.CPU44.CAL:Function_call_interrupts 285149 ± 10% +113.7% 609312 interrupts.CPU44.LOC:Local_timer_interrupts 1236 ± 33% -83.4% 205.25 ±173% interrupts.CPU44.NMI:Non-maskable_interrupts 1236 ± 33% -83.4% 205.25 ±173% interrupts.CPU44.PMI:Performance_monitoring_interrupts 9232 ± 37% -69.7% 2796 ± 33% interrupts.CPU44.RES:Rescheduling_interrupts 669258 ± 38% -98.5% 10363 ± 34% interrupts.CPU44.TLB:TLB_shootdowns 74.50 ± 2% +137.6% 177.00 ± 19% interrupts.CPU45.90:PCI-MSI.1572909-edge.eth0-TxRx-45 611505 ± 42% -98.1% 11614 ± 25% interrupts.CPU45.CAL:Function_call_interrupts 278958 ± 9% +117.4% 606506 interrupts.CPU45.LOC:Local_timer_interrupts 1209 ± 54% -85.6% 174.00 ±173% interrupts.CPU45.NMI:Non-maskable_interrupts 1209 ± 54% -85.6% 174.00 ±173% interrupts.CPU45.PMI:Performance_monitoring_interrupts 9127 ± 40% -68.6% 2866 ± 25% interrupts.CPU45.RES:Rescheduling_interrupts 674116 ± 43% -98.4% 10521 ± 28% interrupts.CPU45.TLB:TLB_shootdowns 75.25 ± 3% +100.7% 151.00 interrupts.CPU46.91:PCI-MSI.1572910-edge.eth0-TxRx-46 604011 ± 33% -98.1% 11401 ± 34% interrupts.CPU46.CAL:Function_call_interrupts 289666 ± 7% +111.5% 612672 interrupts.CPU46.LOC:Local_timer_interrupts 9620 ± 35% -70.8% 2806 ± 30% interrupts.CPU46.RES:Rescheduling_interrupts 666771 ± 34% -98.5% 10313 ± 39% interrupts.CPU46.TLB:TLB_shootdowns 74.50 ± 2% +103.0% 151.25 interrupts.CPU47.92:PCI-MSI.1572911-edge.eth0-TxRx-47 592103 ± 38% -98.4% 9487 ± 39% interrupts.CPU47.CAL:Function_call_interrupts 292671 ± 5% +108.3% 609714 interrupts.CPU47.LOC:Local_timer_interrupts 1444 ± 25% -82.7% 250.25 ±173% interrupts.CPU47.NMI:Non-maskable_interrupts 1444 ± 25% -82.7% 250.25 ±173% interrupts.CPU47.PMI:Performance_monitoring_interrupts 9041 ± 40% -69.3% 2778 ± 31% interrupts.CPU47.RES:Rescheduling_interrupts 651013 ± 39% -98.7% 8392 ± 44% interrupts.CPU47.TLB:TLB_shootdowns 74.75 ± 2% +102.7% 151.50 interrupts.CPU48.93:PCI-MSI.1572912-edge.eth0-TxRx-48 554114 ± 41% -98.2% 10119 ± 45% interrupts.CPU48.CAL:Function_call_interrupts 284700 ± 9% +115.7% 614068 interrupts.CPU48.LOC:Local_timer_interrupts 1539 ± 61% -89.8% 157.25 ±173% interrupts.CPU48.NMI:Non-maskable_interrupts 1539 ± 61% -89.8% 157.25 ±173% interrupts.CPU48.PMI:Performance_monitoring_interrupts 8861 ± 48% -61.5% 3412 ± 23% interrupts.CPU48.RES:Rescheduling_interrupts 606269 ± 41% -98.5% 9011 ± 52% interrupts.CPU48.TLB:TLB_shootdowns 77.50 ± 6% +96.8% 152.50 interrupts.CPU49.94:PCI-MSI.1572913-edge.eth0-TxRx-49 577918 ± 32% -98.2% 10547 ± 41% interrupts.CPU49.CAL:Function_call_interrupts 278289 ± 10% +120.2% 612837 interrupts.CPU49.LOC:Local_timer_interrupts 1456 ± 71% -78.1% 319.75 ±173% interrupts.CPU49.NMI:Non-maskable_interrupts 1456 ± 71% -78.1% 319.75 ±173% interrupts.CPU49.PMI:Performance_monitoring_interrupts 8435 ± 48% -66.9% 2792 ± 32% interrupts.CPU49.RES:Rescheduling_interrupts 640512 ± 30% -98.5% 9472 ± 46% interrupts.CPU49.TLB:TLB_shootdowns 918859 ± 7% -98.1% 17263 ± 30% interrupts.CPU5.CAL:Function_call_interrupts 287835 ± 5% +112.1% 610589 interrupts.CPU5.LOC:Local_timer_interrupts 17179 ± 22% -77.0% 3950 ± 25% interrupts.CPU5.RES:Rescheduling_interrupts 1014576 ± 6% -98.4% 16226 ± 32% interrupts.CPU5.TLB:TLB_shootdowns 81.50 ± 9% +131.6% 188.75 ± 34% interrupts.CPU50.95:PCI-MSI.1572914-edge.eth0-TxRx-50 540125 ± 34% -98.1% 10216 ± 36% interrupts.CPU50.CAL:Function_call_interrupts 289163 ± 7% +111.3% 610942 interrupts.CPU50.LOC:Local_timer_interrupts 1808 ± 52% -91.0% 162.50 ±173% interrupts.CPU50.NMI:Non-maskable_interrupts 1808 ± 52% -91.0% 162.50 ±173% interrupts.CPU50.PMI:Performance_monitoring_interrupts 8609 ± 39% -65.0% 3010 ± 21% interrupts.CPU50.RES:Rescheduling_interrupts 591069 ± 35% -98.5% 9121 ± 41% interrupts.CPU50.TLB:TLB_shootdowns 80.75 ± 5% +86.7% 150.75 interrupts.CPU51.96:PCI-MSI.1572915-edge.eth0-TxRx-51 552961 ± 39% -98.1% 10256 ± 38% interrupts.CPU51.CAL:Function_call_interrupts 284759 ± 5% +115.1% 612655 interrupts.CPU51.LOC:Local_timer_interrupts 7988 ± 44% -65.4% 2764 ± 31% interrupts.CPU51.RES:Rescheduling_interrupts 616023 ± 40% -98.5% 9169 ± 43% interrupts.CPU51.TLB:TLB_shootdowns 74.50 ± 2% +105.0% 152.75 interrupts.CPU52.97:PCI-MSI.1572916-edge.eth0-TxRx-52 545175 ± 41% -97.5% 13766 ± 37% interrupts.CPU52.CAL:Function_call_interrupts 280809 ± 7% +117.6% 611036 interrupts.CPU52.LOC:Local_timer_interrupts 7862 ± 48% -65.9% 2681 ± 32% interrupts.CPU52.RES:Rescheduling_interrupts 603230 ± 41% -97.9% 12658 ± 40% interrupts.CPU52.TLB:TLB_shootdowns 75.25 ± 3% +100.7% 151.00 interrupts.CPU53.98:PCI-MSI.1572917-edge.eth0-TxRx-53 534168 ± 26% -98.1% 10303 ± 57% interrupts.CPU53.CAL:Function_call_interrupts 272007 ± 9% +124.3% 610206 interrupts.CPU53.LOC:Local_timer_interrupts 8567 ± 37% -65.8% 2933 ± 33% interrupts.CPU53.RES:Rescheduling_interrupts 589403 ± 26% -98.4% 9219 ± 64% interrupts.CPU53.TLB:TLB_shootdowns 74.50 ± 2% +105.0% 152.75 interrupts.CPU54.99:PCI-MSI.1572918-edge.eth0-TxRx-54 604818 ± 28% -98.2% 10869 ± 13% interrupts.CPU54.CAL:Function_call_interrupts 275029 ± 15% +122.9% 612975 interrupts.CPU54.LOC:Local_timer_interrupts 1375 ± 73% -84.9% 207.50 ±173% interrupts.CPU54.NMI:Non-maskable_interrupts 1375 ± 73% -84.9% 207.50 ±173% interrupts.CPU54.PMI:Performance_monitoring_interrupts 9396 ± 46% -73.1% 2525 ± 4% interrupts.CPU54.RES:Rescheduling_interrupts 671906 ± 28% -98.5% 9784 ± 15% interrupts.CPU54.TLB:TLB_shootdowns 75.00 +156.0% 192.00 ± 36% interrupts.CPU55.100:PCI-MSI.1572919-edge.eth0-TxRx-55 589116 ± 34% -98.4% 9579 ± 24% interrupts.CPU55.CAL:Function_call_interrupts 273467 ± 17% +123.2% 610320 interrupts.CPU55.LOC:Local_timer_interrupts 9561 ± 50% -76.8% 2219 ± 5% interrupts.CPU55.RES:Rescheduling_interrupts 650563 ± 33% -98.7% 8484 ± 27% interrupts.CPU55.TLB:TLB_shootdowns 74.50 ± 2% +104.0% 152.00 interrupts.CPU56.101:PCI-MSI.1572920-edge.eth0-TxRx-56 622630 ± 32% -98.1% 11790 ± 28% interrupts.CPU56.CAL:Function_call_interrupts 273117 ± 15% +124.1% 612148 interrupts.CPU56.LOC:Local_timer_interrupts 1066 ± 53% -79.4% 219.75 ±173% interrupts.CPU56.NMI:Non-maskable_interrupts 1066 ± 53% -79.4% 219.75 ±173% interrupts.CPU56.PMI:Performance_monitoring_interrupts 10161 ± 45% -77.4% 2295 ± 5% interrupts.CPU56.RES:Rescheduling_interrupts 684138 ± 31% -98.4% 10673 ± 31% interrupts.CPU56.TLB:TLB_shootdowns 74.50 ± 2% +103.0% 151.25 interrupts.CPU57.102:PCI-MSI.1572921-edge.eth0-TxRx-57 626725 ± 31% -98.7% 8124 ± 18% interrupts.CPU57.CAL:Function_call_interrupts 276783 ± 15% +121.8% 613820 interrupts.CPU57.LOC:Local_timer_interrupts 10089 ± 46% -78.1% 2213 ± 10% interrupts.CPU57.RES:Rescheduling_interrupts 685647 ± 31% -99.0% 7004 ± 21% interrupts.CPU57.TLB:TLB_shootdowns 75.00 ± 3% +101.0% 150.75 interrupts.CPU58.103:PCI-MSI.1572922-edge.eth0-TxRx-58 628212 ± 29% -98.6% 8874 ± 37% interrupts.CPU58.CAL:Function_call_interrupts 266509 ± 16% +129.6% 611829 interrupts.CPU58.LOC:Local_timer_interrupts 9521 ± 49% -77.0% 2188 ± 5% interrupts.CPU58.RES:Rescheduling_interrupts 698849 ± 28% -98.9% 7781 ± 42% interrupts.CPU58.TLB:TLB_shootdowns 595325 ± 25% -98.5% 8793 ± 36% interrupts.CPU59.CAL:Function_call_interrupts 272196 ± 15% +125.0% 612483 interrupts.CPU59.LOC:Local_timer_interrupts 9472 ± 47% -77.1% 2172 interrupts.CPU59.RES:Rescheduling_interrupts 653804 ± 26% -98.8% 7679 ± 40% interrupts.CPU59.TLB:TLB_shootdowns 80.75 ± 5% +203.7% 245.25 ± 53% interrupts.CPU6.51:PCI-MSI.1572870-edge.eth0-TxRx-6 904801 ± 12% -98.0% 18288 ± 41% interrupts.CPU6.CAL:Function_call_interrupts 276633 ± 6% +120.0% 608578 interrupts.CPU6.LOC:Local_timer_interrupts 16629 ± 23% -76.5% 3906 ± 22% interrupts.CPU6.RES:Rescheduling_interrupts 1003408 ± 12% -98.3% 17224 ± 43% interrupts.CPU6.TLB:TLB_shootdowns 75.00 ± 3% +102.0% 151.50 interrupts.CPU60.105:PCI-MSI.1572924-edge.eth0-TxRx-60 591936 ± 37% -98.3% 10246 ± 20% interrupts.CPU60.CAL:Function_call_interrupts 270957 ± 17% +126.6% 614004 interrupts.CPU60.LOC:Local_timer_interrupts 9243 ± 58% -75.4% 2274 ± 4% interrupts.CPU60.RES:Rescheduling_interrupts 652167 ± 37% -98.6% 9150 ± 22% interrupts.CPU60.TLB:TLB_shootdowns 76.50 ± 3% +98.4% 151.75 interrupts.CPU61.106:PCI-MSI.1572925-edge.eth0-TxRx-61 615054 ± 36% -98.2% 11117 ± 42% interrupts.CPU61.CAL:Function_call_interrupts 273847 ± 15% +123.3% 611369 interrupts.CPU61.LOC:Local_timer_interrupts 787.50 ± 45% -76.1% 188.00 ±173% interrupts.CPU61.NMI:Non-maskable_interrupts 787.50 ± 45% -76.1% 188.00 ±173% interrupts.CPU61.PMI:Performance_monitoring_interrupts 9607 ± 51% -76.6% 2245 ± 9% interrupts.CPU61.RES:Rescheduling_interrupts 674616 ± 36% -98.5% 10009 ± 46% interrupts.CPU61.TLB:TLB_shootdowns 77.25 ± 5% +95.8% 151.25 interrupts.CPU62.107:PCI-MSI.1572926-edge.eth0-TxRx-62 594390 ± 41% -98.4% 9601 ± 22% interrupts.CPU62.CAL:Function_call_interrupts 263541 ± 14% +132.9% 613769 interrupts.CPU62.LOC:Local_timer_interrupts 10900 ± 42% -80.7% 2100 ± 6% interrupts.CPU62.RES:Rescheduling_interrupts 659954 ± 41% -98.7% 8516 ± 26% interrupts.CPU62.TLB:TLB_shootdowns 576953 ± 40% -98.3% 9702 ± 35% interrupts.CPU63.CAL:Function_call_interrupts 272542 ± 16% +124.6% 612028 interrupts.CPU63.LOC:Local_timer_interrupts 9521 ± 47% -76.4% 2247 ± 5% interrupts.CPU63.RES:Rescheduling_interrupts 633974 ± 40% -98.6% 8589 ± 39% interrupts.CPU63.TLB:TLB_shootdowns 585690 ± 38% -98.2% 10319 ± 45% interrupts.CPU64.CAL:Function_call_interrupts 269339 ± 14% +127.4% 612393 interrupts.CPU64.LOC:Local_timer_interrupts 9503 ± 49% -77.1% 2179 ± 13% interrupts.CPU64.RES:Rescheduling_interrupts 641143 ± 38% -98.6% 9212 ± 51% interrupts.CPU64.TLB:TLB_shootdowns 534613 ± 32% -98.5% 8077 ± 29% interrupts.CPU65.CAL:Function_call_interrupts 270789 ± 18% +125.8% 611404 interrupts.CPU65.LOC:Local_timer_interrupts 1647 ± 21% -87.5% 206.50 ±173% interrupts.CPU65.NMI:Non-maskable_interrupts 1647 ± 21% -87.5% 206.50 ±173% interrupts.CPU65.PMI:Performance_monitoring_interrupts 8210 ± 49% -74.3% 2113 ± 7% interrupts.CPU65.RES:Rescheduling_interrupts 590235 ± 31% -98.8% 6981 ± 34% interrupts.CPU65.TLB:TLB_shootdowns 537019 ± 41% -97.9% 11099 ± 39% interrupts.CPU66.CAL:Function_call_interrupts 280077 ± 13% +119.2% 613899 interrupts.CPU66.LOC:Local_timer_interrupts 1361 ± 33% -85.0% 204.50 ±173% interrupts.CPU66.NMI:Non-maskable_interrupts 1361 ± 33% -85.0% 204.50 ±173% interrupts.CPU66.PMI:Performance_monitoring_interrupts 8435 ± 54% -74.1% 2184 ± 6% interrupts.CPU66.RES:Rescheduling_interrupts 587820 ± 41% -98.3% 10011 ± 43% interrupts.CPU66.TLB:TLB_shootdowns 563456 ± 41% -98.5% 8473 ± 29% interrupts.CPU67.CAL:Function_call_interrupts 273919 ± 17% +124.3% 614344 interrupts.CPU67.LOC:Local_timer_interrupts 1359 ± 37% -83.7% 221.00 ±173% interrupts.CPU67.NMI:Non-maskable_interrupts 1359 ± 37% -83.7% 221.00 ±173% interrupts.CPU67.PMI:Performance_monitoring_interrupts 9303 ± 52% -75.9% 2245 ± 10% interrupts.CPU67.RES:Rescheduling_interrupts 621279 ± 41% -98.8% 7372 ± 33% interrupts.CPU67.TLB:TLB_shootdowns 556297 ± 37% -98.3% 9711 ± 26% interrupts.CPU68.CAL:Function_call_interrupts 270481 ± 16% +125.8% 610830 interrupts.CPU68.LOC:Local_timer_interrupts 9370 ± 53% -75.6% 2286 ± 9% interrupts.CPU68.RES:Rescheduling_interrupts 608096 ± 37% -98.6% 8634 ± 30% interrupts.CPU68.TLB:TLB_shootdowns 547254 ± 35% -98.5% 8017 ± 28% interrupts.CPU69.CAL:Function_call_interrupts 264476 ± 17% +132.2% 614017 interrupts.CPU69.LOC:Local_timer_interrupts 8590 ± 54% -74.9% 2159 ± 15% interrupts.CPU69.RES:Rescheduling_interrupts 609526 ± 34% -98.9% 6926 ± 33% interrupts.CPU69.TLB:TLB_shootdowns 80.25 ± 3% +406.2% 406.25 ± 75% interrupts.CPU7.52:PCI-MSI.1572871-edge.eth0-TxRx-7 961374 ± 10% -98.2% 17601 ± 21% interrupts.CPU7.CAL:Function_call_interrupts 281703 ± 6% +115.9% 608138 interrupts.CPU7.LOC:Local_timer_interrupts 16700 ± 19% -75.3% 4130 ± 19% interrupts.CPU7.RES:Rescheduling_interrupts 1073837 ± 11% -98.5% 16549 ± 23% interrupts.CPU7.TLB:TLB_shootdowns 518881 ± 39% -97.7% 11814 ± 34% interrupts.CPU70.CAL:Function_call_interrupts 271884 ± 14% +124.8% 611226 interrupts.CPU70.LOC:Local_timer_interrupts 1667 ± 50% -75.8% 402.75 ±173% interrupts.CPU70.NMI:Non-maskable_interrupts 1667 ± 50% -75.8% 402.75 ±173% interrupts.CPU70.PMI:Performance_monitoring_interrupts 8469 ± 60% -74.0% 2205 ± 5% interrupts.CPU70.RES:Rescheduling_interrupts 569859 ± 38% -98.1% 10718 ± 37% interrupts.CPU70.TLB:TLB_shootdowns 527133 ± 34% -98.5% 7933 ± 35% interrupts.CPU71.CAL:Function_call_interrupts 277186 ± 14% +121.5% 613929 interrupts.CPU71.LOC:Local_timer_interrupts 1352 ± 72% -84.0% 216.00 ±173% interrupts.CPU71.NMI:Non-maskable_interrupts 1352 ± 72% -84.0% 216.00 ±173% interrupts.CPU71.PMI:Performance_monitoring_interrupts 9146 ± 53% -57.6% 3879 ± 8% interrupts.CPU71.RES:Rescheduling_interrupts 585937 ± 33% -98.8% 6859 ± 40% interrupts.CPU71.TLB:TLB_shootdowns 654632 ± 37% -97.3% 17562 ± 35% interrupts.CPU72.CAL:Function_call_interrupts 291651 ± 5% +109.0% 609650 interrupts.CPU72.LOC:Local_timer_interrupts 13458 ± 44% -72.9% 3650 ± 32% interrupts.CPU72.RES:Rescheduling_interrupts 722365 ± 36% -97.7% 16511 ± 37% interrupts.CPU72.TLB:TLB_shootdowns 667978 ± 38% -97.7% 15407 ± 40% interrupts.CPU73.CAL:Function_call_interrupts 281030 ± 7% +116.6% 608833 interrupts.CPU73.LOC:Local_timer_interrupts 11678 ± 45% -70.0% 3505 ± 29% interrupts.CPU73.RES:Rescheduling_interrupts 744888 ± 39% -98.1% 14379 ± 44% interrupts.CPU73.TLB:TLB_shootdowns 640668 ± 41% -97.4% 16536 ± 33% interrupts.CPU74.CAL:Function_call_interrupts 289715 ± 5% +110.3% 609245 interrupts.CPU74.LOC:Local_timer_interrupts 1871 ± 33% -86.0% 262.75 ±173% interrupts.CPU74.NMI:Non-maskable_interrupts 1871 ± 33% -86.0% 262.75 ±173% interrupts.CPU74.PMI:Performance_monitoring_interrupts 13712 ± 40% -74.6% 3478 ± 32% interrupts.CPU74.RES:Rescheduling_interrupts 705597 ± 41% -97.8% 15488 ± 36% interrupts.CPU74.TLB:TLB_shootdowns 696811 ± 39% -97.7% 16149 ± 20% interrupts.CPU75.CAL:Function_call_interrupts 275166 ± 8% +120.9% 607719 interrupts.CPU75.LOC:Local_timer_interrupts 2421 ± 13% -88.7% 274.00 ±173% interrupts.CPU75.NMI:Non-maskable_interrupts 2421 ± 13% -88.7% 274.00 ±173% interrupts.CPU75.PMI:Performance_monitoring_interrupts 12514 ± 41% -71.5% 3567 ± 28% interrupts.CPU75.RES:Rescheduling_interrupts 775982 ± 39% -98.1% 15095 ± 21% interrupts.CPU75.TLB:TLB_shootdowns 626250 ± 42% -97.5% 15467 ± 26% interrupts.CPU76.CAL:Function_call_interrupts 290528 ± 5% +110.3% 610896 interrupts.CPU76.LOC:Local_timer_interrupts 2829 ± 4% -89.9% 286.50 ±173% interrupts.CPU76.NMI:Non-maskable_interrupts 2829 ± 4% -89.9% 286.50 ±173% interrupts.CPU76.PMI:Performance_monitoring_interrupts 12229 ± 45% -71.3% 3508 ± 28% interrupts.CPU76.RES:Rescheduling_interrupts 690451 ± 42% -97.9% 14409 ± 28% interrupts.CPU76.TLB:TLB_shootdowns 638611 ± 43% -97.7% 14857 ± 22% interrupts.CPU77.CAL:Function_call_interrupts 287678 ± 6% +111.3% 607749 interrupts.CPU77.LOC:Local_timer_interrupts 12736 ± 42% -73.0% 3437 ± 29% interrupts.CPU77.RES:Rescheduling_interrupts 702382 ± 43% -98.0% 13828 ± 25% interrupts.CPU77.TLB:TLB_shootdowns 588064 ± 40% -96.9% 18002 ± 47% interrupts.CPU78.CAL:Function_call_interrupts 286610 ± 4% +113.5% 611772 interrupts.CPU78.LOC:Local_timer_interrupts 11932 ± 44% -71.0% 3463 ± 33% interrupts.CPU78.RES:Rescheduling_interrupts 649344 ± 40% -97.4% 16965 ± 49% interrupts.CPU78.TLB:TLB_shootdowns 604357 ± 44% -97.1% 17682 ± 27% interrupts.CPU79.CAL:Function_call_interrupts 289001 ± 6% +110.1% 607335 interrupts.CPU79.LOC:Local_timer_interrupts 11991 ± 46% -70.8% 3505 ± 30% interrupts.CPU79.RES:Rescheduling_interrupts 665022 ± 43% -97.5% 16642 ± 29% interrupts.CPU79.TLB:TLB_shootdowns 962075 ± 8% -97.9% 19819 ± 31% interrupts.CPU8.CAL:Function_call_interrupts 273781 ± 2% +121.6% 606707 interrupts.CPU8.LOC:Local_timer_interrupts 16561 ± 24% -76.8% 3848 ± 21% interrupts.CPU8.RES:Rescheduling_interrupts 1066718 ± 8% -98.2% 18793 ± 33% interrupts.CPU8.TLB:TLB_shootdowns 570144 ± 46% -97.0% 16951 ± 29% interrupts.CPU80.CAL:Function_call_interrupts 278401 ± 10% +120.0% 612455 interrupts.CPU80.LOC:Local_timer_interrupts 636618 ± 46% -97.5% 15911 ± 31% interrupts.CPU80.TLB:TLB_shootdowns 569041 ± 45% -97.3% 15328 ± 29% interrupts.CPU81.CAL:Function_call_interrupts 286046 ± 6% +113.2% 609890 interrupts.CPU81.LOC:Local_timer_interrupts 11659 ± 44% -70.3% 3457 ± 30% interrupts.CPU81.RES:Rescheduling_interrupts 625296 ± 45% -97.7% 14270 ± 31% interrupts.CPU81.TLB:TLB_shootdowns 558378 ± 46% -97.5% 13901 ± 35% interrupts.CPU82.CAL:Function_call_interrupts 284344 ± 5% +114.4% 609639 interrupts.CPU82.LOC:Local_timer_interrupts 2356 ± 22% -77.4% 531.75 ±173% interrupts.CPU82.NMI:Non-maskable_interrupts 2356 ± 22% -77.4% 531.75 ±173% interrupts.CPU82.PMI:Performance_monitoring_interrupts 11748 ± 48% -70.3% 3486 ± 36% interrupts.CPU82.RES:Rescheduling_interrupts 613989 ± 46% -97.9% 12842 ± 38% interrupts.CPU82.TLB:TLB_shootdowns 584944 ± 49% -97.5% 14462 ± 26% interrupts.CPU83.CAL:Function_call_interrupts 270881 ± 9% +124.8% 608906 interrupts.CPU83.LOC:Local_timer_interrupts 9747 ± 35% -64.1% 3495 ± 29% interrupts.CPU83.RES:Rescheduling_interrupts 662526 ± 50% -98.0% 13431 ± 29% interrupts.CPU83.TLB:TLB_shootdowns 570058 ± 44% -97.4% 14730 ± 39% interrupts.CPU84.CAL:Function_call_interrupts 270900 ± 3% +123.9% 606641 interrupts.CPU84.LOC:Local_timer_interrupts 1628 ± 41% -83.4% 271.00 ±173% interrupts.CPU84.NMI:Non-maskable_interrupts 1628 ± 41% -83.4% 271.00 ±173% interrupts.CPU84.PMI:Performance_monitoring_interrupts 639038 ± 44% -97.9% 13666 ± 42% interrupts.CPU84.TLB:TLB_shootdowns 583241 ± 32% -97.5% 14403 ± 29% interrupts.CPU85.CAL:Function_call_interrupts 285411 ± 7% +112.4% 606099 interrupts.CPU85.LOC:Local_timer_interrupts 2074 ± 33% -74.4% 530.00 ±173% interrupts.CPU85.NMI:Non-maskable_interrupts 2074 ± 33% -74.4% 530.00 ±173% interrupts.CPU85.PMI:Performance_monitoring_interrupts 11762 ± 45% -70.4% 3485 ± 30% interrupts.CPU85.RES:Rescheduling_interrupts 651834 ± 30% -98.0% 13348 ± 31% interrupts.CPU85.TLB:TLB_shootdowns 519447 ± 39% -97.1% 15180 ± 30% interrupts.CPU86.CAL:Function_call_interrupts 283913 ± 7% +114.1% 607963 interrupts.CPU86.LOC:Local_timer_interrupts 9967 ± 41% -65.9% 3397 ± 33% interrupts.CPU86.RES:Rescheduling_interrupts 577318 ± 39% -97.6% 14129 ± 33% interrupts.CPU86.TLB:TLB_shootdowns 605148 ± 49% -97.4% 15555 ± 29% interrupts.CPU87.CAL:Function_call_interrupts 278120 ± 6% +118.6% 608024 interrupts.CPU87.LOC:Local_timer_interrupts 1629 ± 23% -83.0% 277.75 ±173% interrupts.CPU87.NMI:Non-maskable_interrupts 1629 ± 23% -83.0% 277.75 ±173% interrupts.CPU87.PMI:Performance_monitoring_interrupts 11184 ± 42% -69.3% 3437 ± 30% interrupts.CPU87.RES:Rescheduling_interrupts 674556 ± 49% -97.8% 14513 ± 32% interrupts.CPU87.TLB:TLB_shootdowns 580830 ± 25% -97.5% 14593 ± 23% interrupts.CPU88.CAL:Function_call_interrupts 274738 +122.4% 611147 interrupts.CPU88.LOC:Local_timer_interrupts 1788 ± 38% -85.2% 264.50 ±173% interrupts.CPU88.NMI:Non-maskable_interrupts 1788 ± 38% -85.2% 264.50 ±173% interrupts.CPU88.PMI:Performance_monitoring_interrupts 658116 ± 22% -97.9% 13565 ± 25% interrupts.CPU88.TLB:TLB_shootdowns 570707 ± 42% -97.1% 16793 ± 23% interrupts.CPU89.CAL:Function_call_interrupts 289656 ± 3% +109.4% 606629 interrupts.CPU89.LOC:Local_timer_interrupts 2306 ± 17% -88.4% 267.50 ±173% interrupts.CPU89.NMI:Non-maskable_interrupts 2306 ± 17% -88.4% 267.50 ±173% interrupts.CPU89.PMI:Performance_monitoring_interrupts 10794 ± 50% -66.9% 3573 ± 29% interrupts.CPU89.RES:Rescheduling_interrupts 638975 ± 41% -97.5% 15767 ± 25% interrupts.CPU89.TLB:TLB_shootdowns 924319 ± 5% -98.1% 17790 ± 25% interrupts.CPU9.CAL:Function_call_interrupts 285670 ± 3% +112.3% 606480 interrupts.CPU9.LOC:Local_timer_interrupts 17214 ± 25% -77.4% 3889 ± 22% interrupts.CPU9.RES:Rescheduling_interrupts 1029542 ± 4% -98.4% 16728 ± 27% interrupts.CPU9.TLB:TLB_shootdowns 393870 ± 75% -97.3% 10668 ± 27% interrupts.CPU90.CAL:Function_call_interrupts 271294 ± 15% +124.6% 609395 interrupts.CPU90.LOC:Local_timer_interrupts 1789 ± 17% -75.2% 443.00 ±173% interrupts.CPU90.NMI:Non-maskable_interrupts 1789 ± 17% -75.2% 443.00 ±173% interrupts.CPU90.PMI:Performance_monitoring_interrupts 6046 ± 84% -60.9% 2364 ± 20% interrupts.CPU90.RES:Rescheduling_interrupts 447340 ± 75% -97.9% 9583 ± 31% interrupts.CPU90.TLB:TLB_shootdowns 339915 ± 74% -96.7% 11099 ± 20% interrupts.CPU91.CAL:Function_call_interrupts 273423 ± 11% +122.8% 609162 interrupts.CPU91.LOC:Local_timer_interrupts 7730 ± 83% -69.1% 2392 ± 21% interrupts.CPU91.RES:Rescheduling_interrupts 373043 ± 73% -97.3% 10039 ± 22% interrupts.CPU91.TLB:TLB_shootdowns 359581 ± 65% -97.5% 9131 ± 29% interrupts.CPU92.CAL:Function_call_interrupts 260859 ± 13% +133.9% 610075 interrupts.CPU92.LOC:Local_timer_interrupts 860.50 ± 27% -74.6% 218.25 ±173% interrupts.CPU92.NMI:Non-maskable_interrupts 860.50 ± 27% -74.6% 218.25 ±173% interrupts.CPU92.PMI:Performance_monitoring_interrupts 6781 ± 88% -65.7% 2328 ± 18% interrupts.CPU92.RES:Rescheduling_interrupts 401523 ± 63% -98.0% 8051 ± 34% interrupts.CPU92.TLB:TLB_shootdowns 349477 ± 81% -97.4% 9045 ± 36% interrupts.CPU93.CAL:Function_call_interrupts 268030 ± 14% +127.7% 610202 interrupts.CPU93.LOC:Local_timer_interrupts 7271 ± 93% -68.8% 2271 ± 24% interrupts.CPU93.RES:Rescheduling_interrupts 382395 ± 82% -97.9% 7906 ± 42% interrupts.CPU93.TLB:TLB_shootdowns 321246 ± 86% -96.9% 9875 ± 19% interrupts.CPU94.CAL:Function_call_interrupts 273328 ± 16% +124.4% 613382 interrupts.CPU94.LOC:Local_timer_interrupts 357093 ± 88% -97.5% 8765 ± 22% interrupts.CPU94.TLB:TLB_shootdowns 304313 ± 81% -97.3% 8200 ± 40% interrupts.CPU95.CAL:Function_call_interrupts 281165 ± 12% +117.7% 612236 interrupts.CPU95.LOC:Local_timer_interrupts 1173 ± 45% -82.0% 211.75 ±173% interrupts.CPU95.NMI:Non-maskable_interrupts 1173 ± 45% -82.0% 211.75 ±173% interrupts.CPU95.PMI:Performance_monitoring_interrupts 6966 ± 90% -67.7% 2247 ± 18% interrupts.CPU95.RES:Rescheduling_interrupts 330447 ± 81% -97.8% 7134 ± 46% interrupts.CPU95.TLB:TLB_shootdowns 295289 ± 83% -97.0% 8978 ± 34% interrupts.CPU96.CAL:Function_call_interrupts 276155 ± 14% +121.1% 610552 interrupts.CPU96.LOC:Local_timer_interrupts 7211 ± 84% -67.0% 2380 ± 11% interrupts.CPU96.RES:Rescheduling_interrupts 320702 ± 83% -97.5% 7915 ± 40% interrupts.CPU96.TLB:TLB_shootdowns 328715 ± 84% -96.8% 10481 ± 28% interrupts.CPU97.CAL:Function_call_interrupts 274690 ± 11% +122.1% 609974 interrupts.CPU97.LOC:Local_timer_interrupts 1407 ± 33% -85.6% 202.50 ±173% interrupts.CPU97.NMI:Non-maskable_interrupts 1407 ± 33% -85.6% 202.50 ±173% interrupts.CPU97.PMI:Performance_monitoring_interrupts 6028 ± 84% -62.2% 2278 ± 21% interrupts.CPU97.RES:Rescheduling_interrupts 368993 ± 86% -97.5% 9381 ± 32% interrupts.CPU97.TLB:TLB_shootdowns 310595 ± 90% -95.7% 13351 ± 31% interrupts.CPU98.CAL:Function_call_interrupts 275081 ± 16% +122.6% 612237 interrupts.CPU98.LOC:Local_timer_interrupts 339269 ± 88% -96.4% 12245 ± 34% interrupts.CPU98.TLB:TLB_shootdowns 328284 ± 76% -97.0% 9883 ± 19% interrupts.CPU99.CAL:Function_call_interrupts 269113 ± 17% +127.5% 612113 interrupts.CPU99.LOC:Local_timer_interrupts 359011 ± 74% -97.5% 8809 ± 23% interrupts.CPU99.TLB:TLB_shootdowns 40066695 ± 10% +119.6% 87976028 interrupts.LOC:Local_timer_interrupts 0.00 +1.4e+104% 144.00 interrupts.MCP:Machine_check_polls 1337223 ± 7% -70.5% 395127 ± 11% interrupts.RES:Rescheduling_interrupts 82621137 -98.1% 1538710 ± 10% interrupts.TLB:TLB_shootdowns *************************************************************************************************** lkp-bdw-ex2: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1SSD/ext4/sync/x86_64-rhel-7.6/100%/debian-x86_64-2019-09-23.cgz/300s/write/lkp-bdw-ex2/128G/fio-basic/0xb000038 commit: cba81e70bf (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") 3e05ad861b (" iQEzBAABCAAdFiEEghj4iEmqxSLpTPRwpekojE+kFfoFAl3cSywACgkQpekojE+k") cba81e70bf716d85 3e05ad861b9b2b61a1cbfd0d989 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x %stddev %change %stddev \ | \ 0.04 ± 18% -0.0 0.00 ±173% fio.latency_100ms% 49.09 ± 2% +15.7 64.80 ± 5% fio.latency_100us% 13.09 ± 6% -10.2 2.85 ± 17% fio.latency_10us% 6.11 ± 14% -3.8 2.29 ± 21% fio.latency_20us% 0.03 ± 9% -0.0 0.00 fio.latency_250ms% 13.09 ± 2% +6.0 19.13 ± 8% fio.latency_250us% 7.12 ± 8% -6.6 0.56 ± 17% fio.latency_4us% 0.26 ± 5% -0.2 0.01 fio.latency_50ms% 162.33 +2.9% 167.02 fio.time.elapsed_time 162.33 +2.9% 167.02 fio.time.elapsed_time.max 1278 +27.0% 1624 ± 2% fio.time.percent_of_cpu_this_job_got 1965 +30.7% 2567 ± 2% fio.time.system_time 111.25 ± 6% +30.8% 145.51 ± 6% fio.time.user_time 118928 ± 4% -88.1% 14135 ± 51% fio.time.voluntary_context_switches 2115 +228.8% 6953 fio.write_bw_MBps 110336 +10.7% 122112 ± 2% fio.write_clat_90%_us 140800 +9.8% 154624 ± 3% fio.write_clat_95%_us 228350 ± 3% -65.3% 79185 ± 2% fio.write_clat_mean_us 3178158 ± 2% -97.2% 89863 ± 66% fio.write_clat_stddev 541463 +228.8% 1780153 fio.write_iops 76.62 +19.2% 91.31 iostat.cpu.idle 17.09 ± 5% -98.3% 0.30 ± 8% iostat.cpu.iowait 5.95 ± 4% +33.5% 7.94 ± 3% iostat.cpu.system 76.36 +14.8 91.20 mpstat.cpu.all.idle% 17.29 ± 5% -17.0 0.30 ± 8% mpstat.cpu.all.iowait% 6.01 ± 4% +2.0 8.02 ± 3% mpstat.cpu.all.sys% 0.35 ± 7% +0.1 0.46 ± 6% mpstat.cpu.all.usr% 229.00 +24.1% 284.25 ± 2% turbostat.Avg_MHz 10.30 +2.4 12.68 turbostat.Busy% 32.59 ± 20% +28.4% 41.86 ± 8% turbostat.Pkg%pc2 24.86 ± 5% -13.4% 21.53 turbostat.RAMWatt 76.25 +19.3% 91.00 vmstat.cpu.id 16.50 ± 6% -100.0% 0.00 vmstat.cpu.wa 33.25 ± 6% -100.0% 0.00 vmstat.procs.b 11.75 ± 3% +34.0% 15.75 ± 2% vmstat.procs.r 193615 ± 5% -12.1% 170221 ± 4% meminfo.AnonHugePages 2536834 ± 3% -52.0% 1217182 ± 3% meminfo.Committed_AS 48784112 ± 3% +17.9% 57521626 meminfo.Dirty 180597 ± 3% -58.1% 75643 ± 2% meminfo.Inactive(anon) 180087 ± 3% -58.4% 74836 ± 2% meminfo.Mapped 8963 ± 4% -31.7% 6119 ± 2% meminfo.PageTables 183911 ± 2% -57.1% 78917 meminfo.Shmem 7474229 ± 11% +25.1% 9352532 ± 5% numa-numastat.node0.local_node 1267102 ± 48% -100.0% 0.00 numa-numastat.node0.numa_foreign 7485535 ± 11% +25.4% 9383721 ± 5% numa-numastat.node0.numa_hit 2191430 ± 26% -85.2% 323322 ± 66% numa-numastat.node0.numa_miss 2202741 ± 26% -83.9% 354512 ± 60% numa-numastat.node0.other_node 2143882 ± 51% -100.0% 0.00 numa-numastat.node1.numa_miss 2175164 ± 50% -98.9% 23603 ± 56% numa-numastat.node1.other_node 5596306 ± 8% +52.3% 8520908 ± 3% numa-numastat.node3.local_node 3315969 ± 28% -90.2% 323322 ± 66% numa-numastat.node3.numa_foreign 5619817 ± 8% +51.8% 8528961 ± 3% numa-numastat.node3.numa_hit 61467 ± 77% -86.9% 8053 ±167% numa-numastat.node3.other_node 4.398e+10 ± 2% +22.5% 5.388e+10 perf-stat.i.cpu-cycles 8.91 ± 5% -9.4% 8.07 ± 6% perf-stat.i.cpu-migrations 90.82 +1.5 92.34 perf-stat.i.node-load-miss-rate% 51.25 ± 6% -8.9 42.31 ± 9% perf-stat.i.node-store-miss-rate% 847672 ± 3% +23.2% 1044081 ± 4% perf-stat.i.node-stores 10.21 ± 3% +18.7% 12.12 ± 2% perf-stat.overall.cpi 0.10 ± 3% -15.8% 0.08 ± 2% perf-stat.overall.ipc 59.95 ± 2% -5.2 54.72 perf-stat.overall.node-store-miss-rate% 4.458e+10 ± 2% +22.6% 5.464e+10 perf-stat.ps.cpu-cycles 9.04 ± 4% -9.1% 8.22 ± 6% perf-stat.ps.cpu-migrations 858746 ± 2% +23.8% 1062755 ± 4% perf-stat.ps.node-stores 0.50 ± 14% -0.2 0.32 ± 22% perf-profile.children.cycles-pp.start_kernel 0.26 ± 23% -0.1 0.12 ± 46% perf-profile.children.cycles-pp.pm_qos_read_value 0.24 ± 9% -0.1 0.13 ± 32% perf-profile.children.cycles-pp.pm_qos_request 0.15 ± 14% +0.1 0.22 ± 19% perf-profile.children.cycles-pp.ret_from_fork 0.15 ± 16% +0.1 0.22 ± 18% perf-profile.children.cycles-pp.kthread 0.09 ± 21% +0.1 0.16 ± 19% perf-profile.children.cycles-pp.worker_thread 0.09 ± 24% +0.1 0.15 ± 22% perf-profile.children.cycles-pp.process_one_work 0.03 ±173% +1.3 1.30 ±150% perf-profile.children.cycles-pp.poll_idle 0.25 ± 24% -0.1 0.12 ± 46% perf-profile.self.cycles-pp.pm_qos_read_value 0.23 ± 8% -0.1 0.13 ± 30% perf-profile.self.cycles-pp.pm_qos_request 0.03 ±100% +0.0 0.06 perf-profile.self.cycles-pp.jbd2_journal_try_to_free_buffers 0.03 ±173% +1.2 1.25 ±150% perf-profile.self.cycles-pp.poll_idle 13157591 ± 2% +30.5% 17174615 ± 17% numa-meminfo.node0.Dirty 33997072 ± 13% -24.5% 25661111 ± 14% numa-meminfo.node1.FilePages 33729212 ± 13% -24.7% 25398197 ± 15% numa-meminfo.node1.Inactive 33725344 ± 13% -24.7% 25394580 ± 15% numa-meminfo.node1.Inactive(file) 35091038 ± 13% -23.8% 26728332 ± 14% numa-meminfo.node1.MemUsed 7031982 ± 6% +75.5% 12338670 ± 18% numa-meminfo.node3.Dirty 17700511 ± 8% +64.4% 29100305 ± 3% numa-meminfo.node3.FilePages 17413058 ± 8% +65.5% 28824338 ± 3% numa-meminfo.node3.Inactive 17335896 ± 8% +66.2% 28819271 ± 3% numa-meminfo.node3.Inactive(file) 47105126 ± 3% -24.2% 35711671 ± 3% numa-meminfo.node3.MemFree 18913416 ± 8% +60.2% 30306871 ± 4% numa-meminfo.node3.MemUsed 4179 ± 64% -81.1% 788.75 ± 66% numa-meminfo.node3.PageTables 19897 ± 6% +54.8% 30810 ± 12% numa-meminfo.node3.Writeback 5275 ± 8% -38.5% 3244 ± 16% slabinfo.dquot.active_objs 5275 ± 8% -38.5% 3244 ± 16% slabinfo.dquot.num_objs 1157 ± 2% +11.7% 1292 ± 5% slabinfo.khugepaged_mm_slot.active_objs 1157 ± 2% +11.7% 1292 ± 5% slabinfo.khugepaged_mm_slot.num_objs 17827 ± 2% -8.9% 16246 ± 6% slabinfo.kmalloc-2k.active_objs 1122 ± 2% -8.8% 1024 ± 6% slabinfo.kmalloc-2k.active_slabs 17971 ± 2% -8.7% 16402 ± 6% slabinfo.kmalloc-2k.num_objs 1122 ± 2% -8.8% 1024 ± 6% slabinfo.kmalloc-2k.num_slabs 43034 ± 3% -26.6% 31588 ± 2% slabinfo.kmalloc-512.active_objs 729.75 ± 2% -27.7% 527.25 slabinfo.kmalloc-512.active_slabs 46743 ± 2% -27.8% 33769 slabinfo.kmalloc-512.num_objs 729.75 ± 2% -27.7% 527.25 slabinfo.kmalloc-512.num_slabs 30130 ± 5% -15.9% 25354 ± 6% slabinfo.kmalloc-96.active_objs 719.75 ± 5% -15.8% 606.25 ± 6% slabinfo.kmalloc-96.active_slabs 30255 ± 5% -15.7% 25491 ± 6% slabinfo.kmalloc-96.num_objs 719.75 ± 5% -15.8% 606.25 ± 6% slabinfo.kmalloc-96.num_slabs 3616 +21.7% 4400 slabinfo.scsi_sense_cache.active_objs 3616 +21.7% 4400 slabinfo.scsi_sense_cache.num_objs 1173 ± 10% -12.1% 1032 ± 6% slabinfo.skbuff_fclone_cache.active_objs 1173 ± 10% -12.1% 1032 ± 6% slabinfo.skbuff_fclone_cache.num_objs 4712 ± 8% -25.8% 3498 ± 11% slabinfo.task_group.active_objs 4712 ± 8% -25.8% 3498 ± 11% slabinfo.task_group.num_objs 68815 -3.0% 66722 proc-vmstat.nr_active_anon 68152 -3.2% 65945 proc-vmstat.nr_anon_pages 94.00 ± 5% -12.0% 82.75 ± 5% proc-vmstat.nr_anon_transparent_hugepages 12197181 ± 4% +17.9% 14378493 proc-vmstat.nr_dirty 12170201 +100.2% 24363760 proc-vmstat.nr_dirty_background_threshold 24370159 +150.0% 60924277 proc-vmstat.nr_dirty_threshold 27819578 ± 4% +6.8% 29722573 proc-vmstat.nr_file_pages 94790571 -2.1% 92828291 proc-vmstat.nr_free_pages 45205 ± 3% -58.1% 18931 ± 2% proc-vmstat.nr_inactive_anon 27497853 ± 4% +7.0% 29428032 proc-vmstat.nr_inactive_file 27694 -1.2% 27364 proc-vmstat.nr_kernel_stack 45215 ± 3% -58.3% 18869 ± 2% proc-vmstat.nr_mapped 2239 ± 3% -31.6% 1532 proc-vmstat.nr_page_table_pages 46034 ± 2% -57.1% 19745 proc-vmstat.nr_shmem 792760 ± 3% +6.8% 846419 proc-vmstat.nr_slab_reclaimable 75722 -5.0% 71926 proc-vmstat.nr_slab_unreclaimable 68815 -3.0% 66722 proc-vmstat.nr_zone_active_anon 45205 ± 3% -58.1% 18931 ± 2% proc-vmstat.nr_zone_inactive_anon 27497852 ± 4% +7.0% 29428032 proc-vmstat.nr_zone_inactive_file 12230867 ± 4% +17.8% 14411084 proc-vmstat.nr_zone_write_pending 4808083 ± 30% -93.3% 323322 ± 66% proc-vmstat.numa_foreign 30361240 ± 4% +14.8% 34857288 proc-vmstat.numa_hit 30266897 ± 4% +14.9% 34763199 proc-vmstat.numa_local 4808083 ± 30% -93.3% 323322 ± 66% proc-vmstat.numa_miss 4902425 ± 29% -91.5% 417410 ± 51% proc-vmstat.numa_other 3349 ± 4% +26.6% 4241 ± 9% sched_debug.cfs_rq:/.exec_clock.stddev 506.67 ± 17% -26.7% 371.50 ± 6% sched_debug.cfs_rq:/.runnable_load_avg.max 138.66 ± 3% +13.8% 157.78 ± 4% sched_debug.cfs_rq:/.util_avg.stddev 945129 -46.8% 502972 sched_debug.cpu.avg_idle.avg 109022 ± 64% -75.6% 26619 ± 81% sched_debug.cpu.avg_idle.min 503386 -46.2% 270873 ± 2% sched_debug.cpu.max_idle_balance_cost.avg 500000 -50.0% 250000 sched_debug.cpu.max_idle_balance_cost.min 20130 ± 37% +186.1% 57597 ± 42% sched_debug.cpu.max_idle_balance_cost.stddev 1544 ± 8% -21.2% 1216 ± 14% sched_debug.cpu.nr_switches.min 6543 ± 15% +31.2% 8588 ± 6% sched_debug.cpu.nr_switches.stddev 0.08 ± 33% -82.7% 0.01 ± 14% sched_debug.cpu.nr_uninterruptible.avg 2659 ± 12% -24.1% 2017 ± 7% sched_debug.cpu.sched_count.avg 149.85 ± 29% -49.6% 75.50 ± 5% sched_debug.cpu.sched_count.min 6108 ± 16% +34.4% 8210 ± 5% sched_debug.cpu.sched_count.stddev 1245 ± 12% -25.6% 926.37 ± 8% sched_debug.cpu.sched_goidle.avg 29437 ± 26% +35.7% 39943 ± 7% sched_debug.cpu.sched_goidle.max 33.35 ± 20% -38.5% 20.50 ± 4% sched_debug.cpu.sched_goidle.min 2889 ± 16% +37.6% 3975 ± 6% sched_debug.cpu.sched_goidle.stddev 1337 ± 12% -24.3% 1012 ± 7% sched_debug.cpu.ttwu_count.avg 67654 ± 27% +45.6% 98484 ± 17% sched_debug.cpu.ttwu_count.max 60.54 ± 4% -36.3% 38.58 ± 2% sched_debug.cpu.ttwu_count.min 5284 ± 22% +40.6% 7432 ± 14% sched_debug.cpu.ttwu_count.stddev 691.05 ± 10% -71.7% 195.52 ± 17% sched_debug.cpu.ttwu_local.avg 24.00 -33.3% 16.00 sched_debug.sysctl_sched.sysctl_sched_latency 3.00 -46.7% 1.60 sched_debug.sysctl_sched.sysctl_sched_min_granularity 4.00 -50.0% 2.00 sched_debug.sysctl_sched.sysctl_sched_wakeup_granularity 3292949 ± 2% +30.5% 4296301 ± 17% numa-vmstat.node0.nr_dirty 3302011 ± 2% +30.4% 4305380 ± 17% numa-vmstat.node0.nr_zone_write_pending 1163801 ± 48% -100.0% 0.00 numa-vmstat.node0.numa_foreign 7192581 ± 12% +29.1% 9283819 ± 6% numa-vmstat.node0.numa_hit 7180573 ± 12% +28.8% 9251805 ± 7% numa-vmstat.node0.numa_local 1984427 ± 25% -85.6% 286712 ± 66% numa-vmstat.node0.numa_miss 1996442 ± 25% -84.0% 318738 ± 59% numa-vmstat.node0.numa_other 9281274 ± 9% -27.7% 6709488 ± 14% numa-vmstat.node1.nr_dirtied 8516765 ± 13% -24.6% 6422938 ± 14% numa-vmstat.node1.nr_file_pages 8448896 ± 13% -24.8% 6356318 ± 15% numa-vmstat.node1.nr_inactive_file 10161 ± 7% -32.8% 6831 ± 15% numa-vmstat.node1.nr_writeback 5292854 ± 11% -38.9% 3232303 ± 4% numa-vmstat.node1.nr_written 8448887 ± 13% -24.8% 6356310 ± 15% numa-vmstat.node1.nr_zone_inactive_file 1971155 ± 51% -100.0% 0.00 numa-vmstat.node1.numa_miss 2089389 ± 48% -94.7% 111231 ± 11% numa-vmstat.node1.numa_other 4971600 ± 6% +55.3% 7721225 ± 3% numa-vmstat.node3.nr_dirtied 1759273 ± 6% +75.4% 3085601 ± 18% numa-vmstat.node3.nr_dirty 4431518 ± 8% +64.4% 7286718 ± 3% numa-vmstat.node3.nr_file_pages 11769619 ± 3% -24.2% 8915950 ± 3% numa-vmstat.node3.nr_free_pages 4340312 ± 8% +66.3% 7216460 ± 3% numa-vmstat.node3.nr_inactive_file 1044 ± 64% -81.2% 196.75 ± 66% numa-vmstat.node3.nr_page_table_pages 5097 ± 17% +54.9% 7894 ± 5% numa-vmstat.node3.nr_writeback 3206942 ± 7% +44.3% 4627600 ± 12% numa-vmstat.node3.nr_written 4340309 ± 8% +66.3% 7216450 ± 3% numa-vmstat.node3.nr_zone_inactive_file 1764656 ± 6% +75.3% 3093612 ± 18% numa-vmstat.node3.nr_zone_write_pending 3016936 ± 27% -90.5% 286725 ± 66% numa-vmstat.node3.numa_foreign 5614717 ± 7% +47.6% 8285483 ± 4% numa-vmstat.node3.numa_hit 5504523 ± 7% +48.8% 8190007 ± 5% numa-vmstat.node3.numa_local 85.75 ± 6% +30.0% 111.50 ± 15% interrupts.57:PCI-MSI.1572870-edge.eth0-TxRx-6 86.25 ± 5% +784.1% 762.50 ± 87% interrupts.61:PCI-MSI.1572874-edge.eth0-TxRx-10 446.75 ± 49% -53.5% 207.75 ± 37% interrupts.CPU1.RES:Rescheduling_interrupts 86.25 ± 5% +784.1% 762.50 ± 87% interrupts.CPU10.61:PCI-MSI.1572874-edge.eth0-TxRx-10 76.50 ± 71% +140.2% 183.75 ± 32% interrupts.CPU105.TLB:TLB_shootdowns 68.50 ± 61% +156.9% 176.00 ± 39% interrupts.CPU106.TLB:TLB_shootdowns 77.50 ± 37% +107.4% 160.75 ± 45% interrupts.CPU108.TLB:TLB_shootdowns 79.50 ± 24% +128.9% 182.00 ± 35% interrupts.CPU109.TLB:TLB_shootdowns 92.75 ± 32% +101.1% 186.50 ± 38% interrupts.CPU110.TLB:TLB_shootdowns 100.50 ± 30% +80.6% 181.50 ± 35% interrupts.CPU111.TLB:TLB_shootdowns 82.25 ± 48% -90.0% 8.25 ± 60% interrupts.CPU128.RES:Rescheduling_interrupts 55.25 ± 80% -85.1% 8.25 ±125% interrupts.CPU130.RES:Rescheduling_interrupts 62.00 ± 77% -81.9% 11.25 ± 72% interrupts.CPU132.RES:Rescheduling_interrupts 50.00 ± 82% +160.5% 130.25 ± 80% interrupts.CPU14.TLB:TLB_shootdowns 76.25 ± 57% +80.3% 137.50 ± 22% interrupts.CPU159.RES:Rescheduling_interrupts 68.25 ± 63% +375.1% 324.25 ± 74% interrupts.CPU165.RES:Rescheduling_interrupts 75.50 ± 74% +468.2% 429.00 ± 91% interrupts.CPU167.RES:Rescheduling_interrupts 782.50 ± 11% -48.2% 405.50 ± 35% interrupts.CPU168.NMI:Non-maskable_interrupts 782.50 ± 11% -48.2% 405.50 ± 35% interrupts.CPU168.PMI:Performance_monitoring_interrupts 774.75 ± 15% -54.1% 355.25 ± 38% interrupts.CPU170.NMI:Non-maskable_interrupts 774.75 ± 15% -54.1% 355.25 ± 38% interrupts.CPU170.PMI:Performance_monitoring_interrupts 738.50 ± 9% -50.5% 365.75 ± 40% interrupts.CPU171.NMI:Non-maskable_interrupts 738.50 ± 9% -50.5% 365.75 ± 40% interrupts.CPU171.PMI:Performance_monitoring_interrupts 722.75 ± 15% -48.8% 370.25 ± 49% interrupts.CPU172.NMI:Non-maskable_interrupts 722.75 ± 15% -48.8% 370.25 ± 49% interrupts.CPU172.PMI:Performance_monitoring_interrupts 786.50 ± 17% -54.1% 361.00 ± 38% interrupts.CPU173.NMI:Non-maskable_interrupts 786.50 ± 17% -54.1% 361.00 ± 38% interrupts.CPU173.PMI:Performance_monitoring_interrupts 766.75 ± 15% -53.0% 360.75 ± 39% interrupts.CPU174.NMI:Non-maskable_interrupts 766.75 ± 15% -53.0% 360.75 ± 39% interrupts.CPU174.PMI:Performance_monitoring_interrupts 729.25 ± 11% -51.0% 357.00 ± 41% interrupts.CPU175.NMI:Non-maskable_interrupts 729.25 ± 11% -51.0% 357.00 ± 41% interrupts.CPU175.PMI:Performance_monitoring_interrupts 979.25 ± 54% -65.8% 334.50 ± 38% interrupts.CPU176.NMI:Non-maskable_interrupts 979.25 ± 54% -65.8% 334.50 ± 38% interrupts.CPU176.PMI:Performance_monitoring_interrupts 810.25 ± 23% -54.0% 372.75 ± 52% interrupts.CPU177.NMI:Non-maskable_interrupts 810.25 ± 23% -54.0% 372.75 ± 52% interrupts.CPU177.PMI:Performance_monitoring_interrupts 717.25 ± 15% -48.3% 370.75 ± 45% interrupts.CPU178.NMI:Non-maskable_interrupts 717.25 ± 15% -48.3% 370.75 ± 45% interrupts.CPU178.PMI:Performance_monitoring_interrupts 816.00 ± 23% -57.4% 347.75 ± 37% interrupts.CPU179.NMI:Non-maskable_interrupts 816.00 ± 23% -57.4% 347.75 ± 37% interrupts.CPU179.PMI:Performance_monitoring_interrupts 748.75 ± 10% -50.7% 369.00 ± 42% interrupts.CPU180.NMI:Non-maskable_interrupts 748.75 ± 10% -50.7% 369.00 ± 42% interrupts.CPU180.PMI:Performance_monitoring_interrupts 700.75 ± 14% -57.9% 295.00 ± 12% interrupts.CPU181.NMI:Non-maskable_interrupts 700.75 ± 14% -57.9% 295.00 ± 12% interrupts.CPU181.PMI:Performance_monitoring_interrupts 727.50 ± 15% -49.8% 365.50 ± 43% interrupts.CPU182.NMI:Non-maskable_interrupts 727.50 ± 15% -49.8% 365.50 ± 43% interrupts.CPU182.PMI:Performance_monitoring_interrupts 825.50 ± 13% -47.8% 430.75 ± 37% interrupts.CPU183.NMI:Non-maskable_interrupts 825.50 ± 13% -47.8% 430.75 ± 37% interrupts.CPU183.PMI:Performance_monitoring_interrupts 809.25 ± 15% -46.5% 433.25 ± 41% interrupts.CPU184.NMI:Non-maskable_interrupts 809.25 ± 15% -46.5% 433.25 ± 41% interrupts.CPU184.PMI:Performance_monitoring_interrupts 733.75 ± 10% -21.2% 578.25 ± 13% interrupts.CPU186.NMI:Non-maskable_interrupts 733.75 ± 10% -21.2% 578.25 ± 13% interrupts.CPU186.PMI:Performance_monitoring_interrupts 1346 ± 78% -58.2% 563.25 ± 7% interrupts.CPU187.NMI:Non-maskable_interrupts 1346 ± 78% -58.2% 563.25 ± 7% interrupts.CPU187.PMI:Performance_monitoring_interrupts 780.25 ± 12% -25.7% 580.00 ± 8% interrupts.CPU188.NMI:Non-maskable_interrupts 780.25 ± 12% -25.7% 580.00 ± 8% interrupts.CPU188.PMI:Performance_monitoring_interrupts 199.00 ± 53% -65.5% 68.75 ± 75% interrupts.CPU2.RES:Rescheduling_interrupts 114.50 ± 35% -77.5% 25.75 ± 47% interrupts.CPU24.RES:Rescheduling_interrupts 56.75 ± 94% +163.9% 149.75 ± 40% interrupts.CPU25.TLB:TLB_shootdowns 408.25 ± 31% +49.6% 610.75 ± 9% interrupts.CPU37.NMI:Non-maskable_interrupts 408.25 ± 31% +49.6% 610.75 ± 9% interrupts.CPU37.PMI:Performance_monitoring_interrupts 368.25 ± 42% +73.0% 637.25 ± 6% interrupts.CPU49.NMI:Non-maskable_interrupts 368.25 ± 42% +73.0% 637.25 ± 6% interrupts.CPU49.PMI:Performance_monitoring_interrupts 395.00 ± 35% +59.4% 629.50 ± 11% interrupts.CPU50.NMI:Non-maskable_interrupts 395.00 ± 35% +59.4% 629.50 ± 11% interrupts.CPU50.PMI:Performance_monitoring_interrupts 401.75 ± 40% +319.3% 1684 ±109% interrupts.CPU51.NMI:Non-maskable_interrupts 401.75 ± 40% +319.3% 1684 ±109% interrupts.CPU51.PMI:Performance_monitoring_interrupts 374.00 ± 37% +77.3% 663.00 ± 9% interrupts.CPU52.NMI:Non-maskable_interrupts 374.00 ± 37% +77.3% 663.00 ± 9% interrupts.CPU52.PMI:Performance_monitoring_interrupts 384.00 ± 34% +116.0% 829.50 ± 50% interrupts.CPU53.NMI:Non-maskable_interrupts 384.00 ± 34% +116.0% 829.50 ± 50% interrupts.CPU53.PMI:Performance_monitoring_interrupts 291.50 ± 9% +115.0% 626.75 ± 8% interrupts.CPU54.NMI:Non-maskable_interrupts 291.50 ± 9% +115.0% 626.75 ± 8% interrupts.CPU54.PMI:Performance_monitoring_interrupts 384.00 ± 46% +327.2% 1640 ± 57% interrupts.CPU57.NMI:Non-maskable_interrupts 384.00 ± 46% +327.2% 1640 ± 57% interrupts.CPU57.PMI:Performance_monitoring_interrupts 85.75 ± 6% +30.0% 111.50 ± 15% interrupts.CPU6.57:PCI-MSI.1572870-edge.eth0-TxRx-6 375.50 ± 38% +73.5% 651.50 ± 14% interrupts.CPU60.NMI:Non-maskable_interrupts 375.50 ± 38% +73.5% 651.50 ± 14% interrupts.CPU60.PMI:Performance_monitoring_interrupts 67.50 ± 57% +155.6% 172.50 ± 33% interrupts.CPU63.RES:Rescheduling_interrupts 72.25 ± 67% +154.7% 184.00 ± 35% interrupts.CPU64.RES:Rescheduling_interrupts 73.25 ± 84% +160.1% 190.50 ± 31% interrupts.CPU65.RES:Rescheduling_interrupts 57.75 ± 83% +297.8% 229.75 ± 63% interrupts.CPU69.RES:Rescheduling_interrupts 79.00 ± 55% +216.1% 249.75 ± 60% interrupts.CPU70.RES:Rescheduling_interrupts 53.00 ± 81% +423.1% 277.25 ± 53% interrupts.CPU71.RES:Rescheduling_interrupts 1192 ± 60% -48.8% 610.75 ± 14% interrupts.CPU89.NMI:Non-maskable_interrupts 1192 ± 60% -48.8% 610.75 ± 14% interrupts.CPU89.PMI:Performance_monitoring_interrupts 23200 ± 4% -31.1% 15987 ± 5% softirqs.CPU0.RCU 22518 ± 5% -15.8% 18969 ± 12% softirqs.CPU1.RCU 25629 ± 5% -36.1% 16373 ± 7% softirqs.CPU10.RCU 24471 ± 4% -34.0% 16153 ± 7% softirqs.CPU100.RCU 24082 ± 4% -35.6% 15518 ± 3% softirqs.CPU101.RCU 24381 ± 3% -35.3% 15787 ± 5% softirqs.CPU102.RCU 24282 ± 3% -32.0% 16517 ± 5% softirqs.CPU103.RCU 23824 -36.3% 15166 ± 3% softirqs.CPU104.RCU 23766 ± 9% -35.7% 15277 ± 16% softirqs.CPU105.RCU 24877 ± 6% -36.4% 15831 ± 6% softirqs.CPU106.RCU 24260 ± 5% -36.5% 15412 ± 2% softirqs.CPU107.RCU 23514 ± 4% -32.6% 15843 ± 4% softirqs.CPU108.RCU 23168 ± 11% -32.9% 15557 ± 4% softirqs.CPU109.RCU 24135 ± 3% -35.5% 15575 ± 4% softirqs.CPU11.RCU 23094 ± 11% -31.1% 15903 ± 6% softirqs.CPU110.RCU 24722 ± 7% -34.4% 16207 ± 6% softirqs.CPU111.RCU 22641 ± 7% -36.0% 14492 ± 6% softirqs.CPU112.RCU 22698 ± 6% -35.9% 14548 ± 5% softirqs.CPU113.RCU 20120 ± 3% -34.1% 13251 ± 2% softirqs.CPU114.RCU 20952 ± 3% -31.4% 14379 ± 6% softirqs.CPU115.RCU 20669 -32.4% 13982 ± 7% softirqs.CPU116.RCU 22652 ± 5% -34.8% 14758 ± 7% softirqs.CPU117.RCU 21951 ± 8% -32.7% 14770 ± 4% softirqs.CPU118.RCU 21279 ± 3% -34.5% 13946 ± 6% softirqs.CPU119.RCU 23698 ± 2% -31.7% 16175 ± 4% softirqs.CPU12.RCU 22161 ± 2% -35.0% 14413 ± 2% softirqs.CPU120.RCU 21560 ± 4% -38.3% 13307 ± 3% softirqs.CPU121.RCU 22145 ± 4% -39.0% 13518 ± 4% softirqs.CPU122.RCU 21705 ± 3% -38.1% 13425 ± 5% softirqs.CPU123.RCU 22025 ± 3% -39.1% 13416 ± 4% softirqs.CPU124.RCU 71707 ± 6% -12.6% 62663 ± 12% softirqs.CPU124.TIMER 21701 ± 3% -39.5% 13123 ± 3% softirqs.CPU125.RCU 21468 -37.3% 13469 ± 2% softirqs.CPU126.RCU 21540 ± 2% -38.6% 13228 ± 2% softirqs.CPU127.RCU 23737 ± 2% -35.6% 15282 ± 8% softirqs.CPU128.RCU 24378 ± 3% -44.5% 13528 ± 13% softirqs.CPU129.RCU 23915 ± 10% -33.2% 15987 ± 4% softirqs.CPU13.RCU 24830 ± 5% -40.7% 14724 ± 4% softirqs.CPU130.RCU 23809 ± 2% -39.3% 14455 ± 2% softirqs.CPU131.RCU 24304 ± 3% -38.7% 14892 ± 3% softirqs.CPU132.RCU 24225 -38.2% 14973 ± 4% softirqs.CPU133.RCU 24375 ± 3% -30.5% 16935 ± 13% softirqs.CPU134.RCU 26209 ± 4% -39.9% 15744 ± 7% softirqs.CPU135.RCU 26650 ± 4% -40.2% 15947 ± 6% softirqs.CPU137.RCU 23560 ± 2% -37.9% 14638 ± 4% softirqs.CPU138.RCU 23746 ± 9% -38.4% 14635 ± 13% softirqs.CPU139.RCU 23923 ± 8% -29.2% 16947 ± 4% softirqs.CPU14.RCU 22424 ± 17% -33.9% 14823 ± 6% softirqs.CPU140.RCU 26491 ± 3% -40.3% 15815 ± 7% softirqs.CPU141.RCU 26561 ± 4% -39.4% 16099 ± 5% softirqs.CPU142.RCU 24310 ± 6% -37.1% 15284 ± 5% softirqs.CPU143.RCU 24510 ± 5% -37.4% 15354 ± 3% softirqs.CPU144.RCU 23847 ± 5% -38.1% 14761 ± 3% softirqs.CPU145.RCU 25060 ± 6% -40.4% 14924 ± 3% softirqs.CPU146.RCU 23950 ± 11% -47.2% 12640 ± 14% softirqs.CPU147.RCU 24757 ± 7% -48.7% 12707 ± 18% softirqs.CPU148.RCU 23776 ± 2% -42.1% 13776 ± 11% softirqs.CPU149.RCU 25078 ± 7% -32.6% 16893 ± 6% softirqs.CPU15.RCU 23372 ± 14% -40.4% 13926 ± 16% softirqs.CPU150.RCU 25723 ± 8% -46.4% 13778 ± 16% softirqs.CPU151.RCU 24361 ± 6% -40.9% 14407 ± 10% softirqs.CPU152.RCU 24783 ± 3% -25.9% 18360 ± 22% softirqs.CPU153.RCU 20569 ± 3% -11.1% 18295 ± 7% softirqs.CPU153.SCHED 24343 ± 9% -47.7% 12731 ± 33% softirqs.CPU154.RCU 25566 ± 8% -41.9% 14842 ± 5% softirqs.CPU155.RCU 24887 ± 7% -39.1% 15167 ± 5% softirqs.CPU156.RCU 24441 ± 6% -37.2% 15340 ± 5% softirqs.CPU157.RCU 23774 ± 2% -36.0% 15226 ± 4% softirqs.CPU158.RCU 27239 ± 15% -44.0% 15263 ± 4% softirqs.CPU159.RCU 22945 ± 4% -35.0% 14923 ± 7% softirqs.CPU16.RCU 22114 -37.0% 13943 ± 6% softirqs.CPU160.RCU 23526 ± 7% -40.7% 13945 ± 2% softirqs.CPU161.RCU 21165 ± 6% -37.9% 13138 ± 3% softirqs.CPU162.RCU 22289 ± 9% -39.7% 13446 ± 5% softirqs.CPU163.RCU 21648 ± 9% -38.1% 13400 ± 4% softirqs.CPU164.RCU 23582 ± 8% -43.5% 13326 ± 11% softirqs.CPU165.RCU 24354 ± 8% -44.8% 13442 ± 5% softirqs.CPU166.RCU 21901 ± 9% -40.0% 13140 ± 4% softirqs.CPU167.RCU 22122 ± 11% -41.1% 13025 ± 3% softirqs.CPU168.RCU 20721 ± 16% -33.5% 13781 ± 9% softirqs.CPU169.RCU 21384 ± 4% -10.2% 19212 ± 4% softirqs.CPU169.SCHED 74084 ± 3% -22.6% 57373 ± 7% softirqs.CPU169.TIMER 23326 ± 3% -30.1% 16311 ± 15% softirqs.CPU17.RCU 22013 ± 9% -39.7% 13271 ± 3% softirqs.CPU170.RCU 69607 ± 9% -17.3% 57551 ± 6% softirqs.CPU170.TIMER 21648 ± 10% -40.4% 12903 ± 4% softirqs.CPU171.RCU 21544 ± 9% -42.0% 12491 ± 5% softirqs.CPU172.RCU 21436 ± 9% -40.3% 12803 ± 6% softirqs.CPU173.RCU 21782 ± 10% -39.3% 13224 ± 4% softirqs.CPU174.RCU 21609 ± 10% -40.8% 12798 ± 4% softirqs.CPU175.RCU 22224 ± 6% -36.9% 14026 ± 5% softirqs.CPU176.RCU 21640 ± 10% -36.0% 13851 ± 11% softirqs.CPU177.RCU 20431 ± 4% -8.2% 18755 ± 5% softirqs.CPU177.SCHED 22800 ± 4% -35.7% 14670 ± 5% softirqs.CPU178.RCU 22103 ± 7% -36.0% 14138 ± 4% softirqs.CPU179.RCU 21053 ± 5% -35.3% 13630 ± 2% softirqs.CPU18.RCU 22188 ± 7% -34.2% 14608 ± 4% softirqs.CPU180.RCU 22976 ± 5% -37.2% 14436 ± 4% softirqs.CPU181.RCU 22932 ± 9% -31.9% 15626 ± 11% softirqs.CPU182.RCU 24443 ± 8% -38.4% 15060 ± 5% softirqs.CPU183.RCU 25583 ± 8% -40.1% 15321 ± 4% softirqs.CPU184.RCU 70497 ± 6% -15.7% 59394 ± 5% softirqs.CPU184.TIMER 26175 ± 9% -41.6% 15275 ± 5% softirqs.CPU185.RCU 71005 ± 5% -17.8% 58334 ± 5% softirqs.CPU185.TIMER 23931 ± 3% -41.1% 14099 ± 4% softirqs.CPU186.RCU 26037 ± 12% -44.9% 14350 ± 8% softirqs.CPU187.RCU 23996 ± 5% -39.6% 14490 ± 4% softirqs.CPU188.RCU 27080 ± 11% -45.1% 14855 ± 10% softirqs.CPU189.RCU 20641 ± 5% -29.9% 14466 ± 6% softirqs.CPU19.RCU 25108 ± 7% -39.1% 15280 ± 5% softirqs.CPU190.RCU 70386 ± 8% -16.8% 58571 ± 4% softirqs.CPU190.TIMER 24651 ± 4% -41.6% 14404 ± 6% softirqs.CPU191.RCU 20449 ± 3% -7.5% 18912 ± 4% softirqs.CPU191.SCHED 69747 ± 8% -15.6% 58878 ± 5% softirqs.CPU191.TIMER 24459 ± 5% -32.2% 16575 ± 5% softirqs.CPU2.RCU 21287 ± 2% -29.9% 14911 ± 5% softirqs.CPU20.RCU 22989 ± 2% -31.2% 15815 ± 9% softirqs.CPU21.RCU 22529 ± 6% -32.1% 15308 ± 4% softirqs.CPU22.RCU 22249 ± 4% -34.1% 14666 ± 8% softirqs.CPU23.RCU 23833 ± 3% -34.1% 15702 ± 3% softirqs.CPU24.RCU 22340 ± 3% -37.3% 14004 ± 4% softirqs.CPU25.RCU 22821 ± 4% -37.6% 14234 ± 4% softirqs.CPU26.RCU 22365 ± 2% -37.9% 13888 ± 5% softirqs.CPU27.RCU 22753 ± 3% -48.4% 11736 ± 32% softirqs.CPU28.RCU 22303 ± 4% -39.2% 13560 ± 3% softirqs.CPU29.RCU 26947 ± 10% -38.6% 16549 ± 7% softirqs.CPU3.RCU 22344 ± 2% -37.4% 13996 ± 2% softirqs.CPU30.RCU 22285 ± 2% -45.5% 12154 ± 21% softirqs.CPU31.RCU 23877 ± 3% -37.2% 14996 ± 4% softirqs.CPU32.RCU 24827 ± 2% -43.3% 14078 ± 12% softirqs.CPU33.RCU 25217 ± 3% -38.8% 15427 ± 2% softirqs.CPU34.RCU 23942 ± 2% -38.2% 14798 ± 2% softirqs.CPU35.RCU 24326 ± 2% -37.0% 15331 ± 3% softirqs.CPU36.RCU 24743 ± 3% -36.9% 15610 ± 4% softirqs.CPU37.RCU 25003 -37.1% 15716 ± 7% softirqs.CPU38.RCU 26486 ± 3% -38.0% 16414 ± 7% softirqs.CPU39.RCU 25055 -34.4% 16424 ± 6% softirqs.CPU4.RCU 26355 ± 3% -37.4% 16504 ± 7% softirqs.CPU40.RCU 26796 ± 3% -39.5% 16219 ± 6% softirqs.CPU41.RCU 23944 -37.7% 14924 ± 4% softirqs.CPU42.RCU 24935 ± 2% -38.2% 15399 ± 9% softirqs.CPU43.RCU 25209 ± 4% -39.7% 15198 ± 7% softirqs.CPU44.RCU 26841 ± 3% -39.2% 16307 ± 7% softirqs.CPU45.RCU 26954 ± 5% -39.3% 16352 ± 6% softirqs.CPU46.RCU 72758 ± 6% -13.1% 63250 ± 12% softirqs.CPU46.TIMER 24713 ± 5% -36.4% 15719 ± 5% softirqs.CPU47.RCU 25005 ± 7% -35.4% 16153 ± 5% softirqs.CPU48.RCU 25086 ± 7% -36.6% 15905 ± 3% softirqs.CPU49.RCU 24352 ± 3% -35.1% 15802 ± 3% softirqs.CPU5.RCU 26029 ± 9% -39.9% 15634 ± 3% softirqs.CPU50.RCU 26066 ± 6% -42.7% 14933 ± 12% softirqs.CPU51.RCU 25792 ± 8% -47.6% 13517 ± 18% softirqs.CPU52.RCU 24668 ± 5% -41.3% 14477 ± 10% softirqs.CPU53.RCU 24755 ± 11% -40.6% 14711 ± 15% softirqs.CPU54.RCU 26696 ± 7% -43.4% 15108 ± 16% softirqs.CPU55.RCU 25310 ± 7% -39.1% 15405 ± 9% softirqs.CPU56.RCU 25491 ± 4% -40.8% 15103 ± 10% softirqs.CPU57.RCU 24493 ± 15% -36.5% 15564 ± 5% softirqs.CPU58.RCU 25247 ± 5% -38.9% 15414 ± 5% softirqs.CPU59.RCU 25244 ± 2% -35.4% 16309 ± 6% softirqs.CPU6.RCU 24964 ± 6% -37.1% 15694 ± 5% softirqs.CPU60.RCU 24997 ± 6% -37.0% 15758 ± 3% softirqs.CPU61.RCU 23131 ± 11% -31.2% 15923 ± 3% softirqs.CPU62.RCU 25362 ± 3% -37.9% 15751 ± 3% softirqs.CPU63.RCU 22262 ± 3% -35.3% 14396 ± 4% softirqs.CPU64.RCU 23891 ± 6% -40.2% 14285 softirqs.CPU65.RCU 21406 ± 7% -37.6% 13361 ± 3% softirqs.CPU66.RCU 22559 ± 8% -38.0% 13992 ± 5% softirqs.CPU67.RCU 21915 ± 9% -37.0% 13813 ± 3% softirqs.CPU68.RCU 24115 ± 7% -42.2% 13948 ± 10% softirqs.CPU69.RCU 26055 ± 9% -46.5% 13936 ± 34% softirqs.CPU7.RCU 23779 ± 8% -41.6% 13887 ± 8% softirqs.CPU70.RCU 22511 ± 9% -38.8% 13782 ± 4% softirqs.CPU71.RCU 23013 ± 11% -39.6% 13906 ± 3% softirqs.CPU72.RCU 21785 ± 13% -38.4% 13429 ± 6% softirqs.CPU73.RCU 75682 ± 4% -23.1% 58226 ± 6% softirqs.CPU73.TIMER 22508 ± 11% -39.5% 13627 ± 3% softirqs.CPU74.RCU 22691 ± 11% -38.5% 13960 ± 4% softirqs.CPU75.RCU 23127 ± 10% -41.7% 13480 ± 3% softirqs.CPU76.RCU 21909 ± 9% -38.0% 13580 ± 7% softirqs.CPU77.RCU 22584 ± 9% -40.4% 13460 ± 3% softirqs.CPU78.RCU 22352 ± 11% -40.0% 13420 ± 4% softirqs.CPU79.RCU 24521 -34.1% 16167 ± 4% softirqs.CPU8.RCU 23373 ± 11% -36.0% 14953 ± 4% softirqs.CPU80.RCU 23622 ± 4% -38.6% 14500 ± 9% softirqs.CPU81.RCU 20927 ± 4% -11.4% 18550 ± 2% softirqs.CPU81.SCHED 77620 ± 6% -27.0% 56642 ± 7% softirqs.CPU81.TIMER 24100 ± 4% -35.7% 15495 ± 6% softirqs.CPU82.RCU 23075 ± 8% -36.3% 14698 ± 4% softirqs.CPU83.RCU 23350 ± 9% -34.8% 15217 ± 3% softirqs.CPU84.RCU 23821 ± 6% -37.3% 14926 ± 2% softirqs.CPU85.RCU 23901 ± 9% -34.4% 15684 ± 4% softirqs.CPU86.RCU 25565 ± 7% -39.2% 15547 ± 4% softirqs.CPU87.RCU 26186 ± 8% -39.5% 15844 ± 3% softirqs.CPU88.RCU 71292 ± 6% -16.8% 59325 ± 5% softirqs.CPU88.TIMER 26995 ± 9% -42.1% 15632 ± 4% softirqs.CPU89.RCU 71906 ± 5% -18.9% 58342 ± 4% softirqs.CPU89.TIMER 24096 ± 9% -37.9% 14958 ± 14% softirqs.CPU9.RCU 24756 ± 4% -39.9% 14880 ± 3% softirqs.CPU90.RCU 27266 ± 18% -44.8% 15047 ± 4% softirqs.CPU91.RCU 24914 ± 7% -36.7% 15766 ± 8% softirqs.CPU92.RCU 26138 ± 7% -42.9% 14919 ± 10% softirqs.CPU93.RCU 26775 ± 6% -39.1% 16306 ± 5% softirqs.CPU94.RCU 71267 ± 8% -17.4% 58875 ± 4% softirqs.CPU94.TIMER 25074 ± 4% -40.2% 15004 ± 4% softirqs.CPU95.RCU 70923 ± 9% -17.2% 58728 ± 5% softirqs.CPU95.TIMER 24257 ± 12% -28.4% 17371 ± 16% softirqs.CPU96.RCU 22072 ± 6% -28.7% 15741 ± 5% softirqs.CPU97.RCU 23483 ± 5% -32.1% 15940 ± 6% softirqs.CPU98.RCU 23761 -32.8% 15969 ± 7% softirqs.CPU99.RCU 4572216 -37.7% 2847467 ± 3% softirqs.RCU Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen

1 year, 2 months

1
0
0 / 0

[debugfs] 5496197f9b: WARNING:at_fs/open.c:#do_dentry_open

by kernel test robot

FYI, we noticed the following commit (built with gcc-7): commit: 5496197f9b084f086cb410dd566648b0896fcc74 ("debugfs: Restrict debugfs when the kernel is locked down") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: boot on test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 8G caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace): +--------------------------------------+------------+------------+ | | 29d3c1c8df | 5496197f9b | +--------------------------------------+------------+------------+ | boot_successes | 12 | 1 | | boot_failures | 0 | 11 | | WARNING:at_fs/open.c:#do_dentry_open | 0 | 11 | | EIP:do_dentry_open | 0 | 11 | +--------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 6.694953] WARNING: CPU: 0 PID: 298 at fs/open.c:805 do_dentry_open+0x1ef/0x24b [ 6.696274] Modules linked in: [ 6.696719] CPU: 0 PID: 298 Comm: run-lkp Not tainted 5.2.0-00026-g5496197f9b084 #2 [ 6.697784] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 6.698966] EIP: do_dentry_open+0x1ef/0x24b [ 6.699554] Code: 40 8b 4d f0 74 78 8b 83 00 01 00 00 8b 80 84 00 00 00 85 c0 74 63 83 78 2c 00 b8 ea ff ff ff 0f 44 c8 eb 5a 89 c2 85 d2 7e 07 <0f> 0b ba ea ff ff ff f6 43 46 01 74 1b 89 55 f0 ff 8e 18 01 00 00 [ 6.702133] EAX: 00000001 EBX: f739e780 ECX: 00000001 EDX: 00000001 [ 6.703007] ESI: f0c47c70 EDI: c128e5ad EBP: f7283dec ESP: f7283ddc [ 6.703888] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 EFLAGS: 00010202 [ 6.704860] CR0: 80050033 CR2: 09407008 CR3: 37288000 CR4: 000406d0 [ 6.705756] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 6.706634] DR6: fffe0ff0 DR7: 00000400 [ 6.707175] Call Trace: [ 6.707531] vfs_open+0x23/0x27 [ 6.707985] path_openat+0x8f5/0xa3d [ 6.708502] do_filp_open+0x50/0xa8 [ 6.709005] ? _raw_spin_unlock+0x27/0x38 [ 6.709574] ? __alloc_fd+0x10f/0x119 [ 6.710102] do_sys_open+0x61/0xd3 [ 6.710591] sys_open+0x18/0x1a [ 6.711041] do_fast_syscall_32+0x90/0xcf [ 6.711612] entry_SYSENTER_32+0x79/0xde [ 6.712168] EIP: 0xb7efa795 [ 6.712570] Code: cd ff ff 85 d2 89 c8 74 02 89 0a 5b 5d c3 8b 04 24 c3 8b 14 24 c3 8b 1c 24 c3 8b 3c 24 c3 90 90 90 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76 [ 6.715182] EAX: ffffffda EBX: 09456168 ECX: 00008241 EDX: 000001b6 [ 6.716072] ESI: 00000000 EDI: b7ec8ff4 EBP: bf8c8378 ESP: bf8c8210 [ 6.716956] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000286 [ 6.717908] ---[ end trace d1d982a595e22ad7 ]--- To reproduce: # build kernel cd linux cp config-5.2.0-00026-g5496197f9b084 .config make HOSTCC=gcc-7 CC=gcc-7 ARCH=i386 olddefconfig prepare modules_prepare bzImage git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp qemu -k <bzImage> job-script # job-script is attached in this email Thanks, lkp

1 year, 2 months

1
0
0 / 0

86dc301f7b ("drivers/base/memory.c: cache blocks in radix tree .."): [ 1.341517] WARNING: suspicious RCU usage

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux/commits/Scott-Cheloha/drivers-base-memor... commit 86dc301f7b4815d90e3a7843ffed655d64efe445 Author: Scott Cheloha <cheloha(a)linux.vnet.ibm.com> AuthorDate: Thu Nov 21 13:59:52 2019 -0600 Commit: 0day robot <lkp(a)intel.com> CommitDate: Sun Nov 24 10:45:59 2019 +0800 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup Searching for a particular memory block by id is slow because each block device is kept in an unsorted linked list on the subsystem bus. Lookup is much faster if we cache the blocks in a radix tree. Memory subsystem initialization and hotplug/hotunplug is at least a little faster for any machine with more than ~100 blocks, and the speedup grows with the block count. Signed-off-by: Scott Cheloha <cheloha(a)linux.vnet.ibm.com> Acked-by: David Hildenbrand <david(a)redhat.com> 0e4a459f56 tracing: Remove unnecessary DEBUG_FS dependency 86dc301f7b drivers/base/memory.c: cache blocks in radix tree to accelerate lookup +---------------------------------------------------------------------+------------+------------+ | | 0e4a459f56 | 86dc301f7b | +---------------------------------------------------------------------+------------+------------+ | boot_successes | 17 | 0 | | boot_failures | 40 | 11 | | WARNING:possible_circular_locking_dependency_detected | 40 | 8 | | WARNING:suspicious_RCU_usage | 0 | 11 | | include/linux/radix-tree.h:#suspicious_rcu_dereference_check()usage | 0 | 11 | +---------------------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 1.335279] random: get_random_bytes called from kcmp_cookies_init+0x29/0x4c with crng_init=0 [ 1.336883] ACPI: bus type PCI registered [ 1.338295] PCI: Using configuration type 1 for base access [ 1.340735] [ 1.341049] ============================= [ 1.341517] WARNING: suspicious RCU usage [ 1.342266] 5.4.0-rc5-00070-g86dc301f7b481 #1 Tainted: G T [ 1.343494] ----------------------------- [ 1.344226] include/linux/radix-tree.h:167 suspicious rcu_dereference_check() usage! [ 1.345516] [ 1.345516] other info that might help us debug this: [ 1.345516] [ 1.346962] [ 1.346962] rcu_scheduler_active = 2, debug_locks = 1 [ 1.348134] no locks held by swapper/0/1. [ 1.348866] [ 1.348866] stack backtrace: [ 1.349525] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G T 5.4.0-rc5-00070-g86dc301f7b481 #1 [ 1.351230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 1.352720] Call Trace: [ 1.353187] ? dump_stack+0x9a/0xde [ 1.353507] ? node_access_release+0x19/0x19 [ 1.353507] ? walk_memory_blocks+0xe6/0x184 [ 1.353507] ? set_debug_rodata+0x20/0x20 [ 1.353507] ? link_mem_sections+0x39/0x3d [ 1.353507] ? topology_init+0x74/0xc8 [ 1.353507] ? enable_cpu0_hotplug+0x15/0x15 [ 1.353507] ? do_one_initcall+0x13d/0x30a [ 1.353507] ? kernel_init_freeable+0x18e/0x23b [ 1.353507] ? rest_init+0x173/0x173 [ 1.353507] ? kernel_init+0x10/0x151 [ 1.353507] ? rest_init+0x173/0x173 [ 1.353507] ? ret_from_fork+0x3a/0x50 [ 1.410829] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 1.427389] cryptd: max_cpu_qlen set to 1000 [ 1.457792] ACPI: Added _OSI(Module Device) [ 1.458615] ACPI: Added _OSI(Processor Device) [ 1.459428] ACPI: Added _OSI(3.0 _SCP Extensions) # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 1d5f18079cb10420c4e4b67f571b998ec861a20c 219d54332a09e8d8741c1e1982f5eae56099de85 -- git bisect good 00e5729fc7309fd1da26659b7dce0fcc0b46ab7e # 12:06 G 10 0 5 14 Merge 'linux-review/Daniel-W-S-Almeida/media-dummy_dvb_fe-register-adapter-frontend/20191127-043813' into devel-hourly-2019113015 git bisect bad da795b1ad310518d9100105235c9ba3ff4ee2524 # 12:07 B 0 11 27 0 Merge 'linux-review/Krzysztof-Kozlowski/media-Fix-Kconfig-indentation/20191122-055026' into devel-hourly-2019113015 git bisect bad 53f2832a6bf7103957815cef4ad58be22255ea7a # 12:08 B 0 11 27 0 Merge 'linux-review/Luc-Van-Oostenryck/misc-xilinx_sdfec-add-missing-__user-annotation/20191124-052749' into devel-hourly-2019113015 git bisect good 03a04e32f16a2c4bd267b48e8ab4aaf71c507050 # 12:13 G 10 0 5 13 Merge 'linux-review/Helmut-Grohne/mdio_bus-revert-inadvertent-error-code-change/20191124-171049' into devel-hourly-2019113015 git bisect bad 03403c59a591b12556bd0db1243eb0503112a0df # 12:13 B 0 11 27 0 Merge 'linux-review/Navid-Emamdoost/EDAC-Fix-memory-leak-in-i5100_init_one/20191124-103621' into devel-hourly-2019113015 git bisect good fae5c4b30fda35cbcc8389f432dcaac20f9b3a12 # 12:17 G 10 0 4 11 Merge 'linux-review/David-Gow/kunit-Always-print-actual-pointer-values-in-asserts/20191124-131742' into devel-hourly-2019113015 git bisect bad 1e0c725d2bb21789330a2fe1a360c37ae753eb18 # 12:17 B 0 11 27 0 Merge 'linux-review/Scott-Cheloha/drivers-base-memory-c-cache-blocks-in-radix-tree-to-accelerate-lookup/20191124-104557' into devel-hourly-2019113015 git bisect good 0902b87758e830e857e11f1538e50b218743f4b0 # 12:21 G 10 0 6 13 Merge 'linux-review/Julio-Faracco/drivers-net-virtio_net-Implement-a-dev_watchdog-handler/20191124-135051' into devel-hourly-2019113015 git bisect good 2d0c894673b55b51915ea858eabcf7c97c9b8ccb # 12:25 G 10 0 5 11 Merge 'linux-review/Maciej-enczykowski/net-ipv6-IPV6_TRANSPARENT-check-NET_RAW-prior-to-NET_ADMIN/20191124-120121' into devel-hourly-2019113015 git bisect good 3c095c856fb84271e56e1869aface37af1b078f1 # 12:29 G 10 0 6 16 Merge 'linux-review/Navid-Emamdoost/Bluetooth-Fix-memory-leak-in-hci_connect_le_scan/20191124-111255' into devel-hourly-2019113015 git bisect bad 86dc301f7b4815d90e3a7843ffed655d64efe445 # 12:58 B 0 11 41 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # first bad commit: [86dc301f7b4815d90e3a7843ffed655d64efe445] drivers/base/memory.c: cache blocks in radix tree to accelerate lookup git bisect good 0e4a459f56c32d3e52ae69a4b447db2f48a65f44 # 13:24 G 30 0 15 41 tracing: Remove unnecessary DEBUG_FS dependency # extra tests with debug options git bisect good 86dc301f7b4815d90e3a7843ffed655d64efe445 # 13:25 G 11 0 0 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # extra tests on head commit of linux-review/Scott-Cheloha/drivers-base-memory-c-cache-blocks-in-radix-tree-to-accelerate-lookup/20191124-104557 git bisect bad 86dc301f7b4815d90e3a7843ffed655d64efe445 # 13:31 B 0 11 41 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # bad: [86dc301f7b4815d90e3a7843ffed655d64efe445] drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # extra tests on revert first bad commit git bisect good 65bba365eabe04a22e5d68012a45b92eee26860c # 13:38 G 10 0 6 16 Revert "drivers/base/memory.c: cache blocks in radix tree to accelerate lookup" # good: [65bba365eabe04a22e5d68012a45b92eee26860c] Revert "drivers/base/memory.c: cache blocks in radix tree to accelerate lookup" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

1 year, 2 months

1
0
0 / 0

9dc510db74 ("perf: Sharing PMU counters across compatible .."): [ 15.895385] WARNING: CPU: 0 PID: 648 at kernel/events/core.c:1755 perf_event_remove_dup

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux/commits/Song-Liu/perf-Sharing-PMU-counte... commit 9dc510db74b1b873853747098e1cd66b71e63210 Author: Song Liu <songliubraving(a)fb.com> AuthorDate: Fri Nov 22 19:50:06 2019 +0000 Commit: 0day robot <lkp(a)intel.com> CommitDate: Mon Nov 25 02:13:06 2019 +0800 perf: Sharing PMU counters across compatible events > On Nov 22, 2019, at 11:33 AM, Jiri Olsa <jolsa(a)redhat.com> wrote: > > On Fri, Nov 15, 2019 at 03:55:04PM -0800, Song Liu wrote: >> This patch tries to enable PMU sharing. When multiple perf_events are >> counting the same metric, they can share the hardware PMU counter. We >> call these events as "compatible events". >> >> The PMU sharing are limited to events within the same perf_event_context >> (ctx). When a event is installed or enabled, search the ctx for compatible >> events. This is implemented in perf_event_setup_dup(). One of these >> compatible events are picked as the master (stored in event->dup_master). >> Similarly, when the event is removed or disabled, perf_event_remove_dup() >> is used to clean up sharing. >> >> A new state PERF_EVENT_STATE_ENABLED is introduced for the master event. >> This state is used when the slave event is ACTIVE, but the master event >> is not. >> >> On the critical paths (add, del read), sharing PMU counters doesn't >> increase the complexity. Helper functions event_pmu_[add|del|read]() are >> introduced to cover these cases. All these functions have O(1) time >> complexity. >> >> Cc: Peter Zijlstra <peterz(a)infradead.org> >> Cc: Arnaldo Carvalho de Melo <acme(a)redhat.com> >> Cc: Jiri Olsa <jolsa(a)kernel.org> >> Cc: Alexey Budankov <alexey.budankov(a)linux.intel.com> >> Cc: Namhyung Kim <namhyung(a)kernel.org> >> Cc: Tejun Heo <tj(a)kernel.org> >> Signed-off-by: Song Liu <songliubraving(a)fb.com> >> >> --- >> Changes in v7: >> Major rewrite to avoid allocating extra master event. > > hi, > what is this based on? I can't apply it on tip/master: > > Applying: perf: Sharing PMU counters across compatible events > error: patch failed: include/linux/perf_event.h:722 > error: include/linux/perf_event.h: patch does not apply > Patch failed at 0001 perf: Sharing PMU counters across compatible events > hint: Use 'git am --show-current-patch' to see the failed patch > When you have resolved this problem, run "git am --continue". > If you prefer to skip this patch, run "git am --skip" instead. > To restore the original branch and stop patching, run "git am --abort". I was using Linus's master branch. This one is specifically based on commit 96b95eff4a591dbac582c2590d067e356a18aacb Merge: 4e84608c7836 80591e61a0f7 Author: Linus Torvalds <torvalds(a)linux-foundation.org> Date: 8 days ago Merge tag 'kbuild-fixes-v5.4-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild Pull Kbuild fixes from Masahiro Yamada: - fix build error when compiling SPARC VDSO with CONFIG_COMPAT=y - pass correct --arch option to Sparse * tag 'kbuild-fixes-v5.4-3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild: kbuild: tell sparse about the $ARCH sparc: vdso: fix build error of vdso32 > > also I'm getting this when trying to see/apply plain text patch: > > [jolsa@dell-r440-01 linux-perf]$ git am --show-current-patch | tail > =09=09for_each_sibling_event(sibling, group_leader) { > =09=09=09perf_remove_from_context(sibling, 0); > =09=09=09put_ctx(gctx); > +=09=09=09WARN_ON_ONCE(sibling->dup_master); > =09=09} > =20 > =09=09/* > --=20 I also get these =09, =20 issues. I am not sure how to fix them. Attaching the patch here to see whether it fixes it. Thanks! Song From bb83b28deca52a2376d8b9d0b1d54e7fec797aa9 Mon Sep 17 00:00:00 2001 From: Song Liu <songliubraving(a)fb.com> Date: Wed, 6 Jun 2018 23:24:10 -0700 Subject: [PATCH v7] perf: Sharing PMU counters across compatible events This patch tries to enable PMU sharing. When multiple perf_events are counting the same metric, they can share the hardware PMU counter. We call these events as "compatible events". The PMU sharing are limited to events within the same perf_event_context (ctx). When a event is installed or enabled, search the ctx for compatible events. This is implemented in perf_event_setup_dup(). One of these compatible events are picked as the master (stored in event->dup_master). Similarly, when the event is removed or disabled, perf_event_remove_dup() is used to clean up sharing. A new state PERF_EVENT_STATE_ENABLED is introduced for the master event. This state is used when the slave event is ACTIVE, but the master event is not. On the critical paths (add, del read), sharing PMU counters doesn't increase the complexity. Helper functions event_pmu_[add|del|read]() are introduced to cover these cases. All these functions have O(1) time complexity. Cc: Peter Zijlstra <peterz(a)infradead.org> Cc: Arnaldo Carvalho de Melo <acme(a)redhat.com> Cc: Jiri Olsa <jolsa(a)kernel.org> Cc: Alexey Budankov <alexey.budankov(a)linux.intel.com> Cc: Namhyung Kim <namhyung(a)kernel.org> Cc: Tejun Heo <tj(a)kernel.org> Signed-off-by: Song Liu <songliubraving(a)fb.com> 6b8a794678 Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost 9dc510db74 perf: Sharing PMU counters across compatible events +--------------------------------------------------------------------------------------+------------+------------+ | | 6b8a794678 | 9dc510db74 | +--------------------------------------------------------------------------------------+------------+------------+ | boot_successes | 841 | 251 | | boot_failures | 2 | 37 | | BUG:kernel_hang_in_test_stage | 1 | | | Mem-Info | 1 | 2 | | WARNING:at_kernel/events/core.c:#event_function | 0 | 22 | | EIP:event_function | 0 | 22 | | WARNING:at_kernel/events/core.c:#event_sched_out | 0 | 21 | | EIP:event_sched_out | 0 | 21 | | WARNING:at_kernel/events/core.c:#perf_group_detach | 0 | 21 | | EIP:perf_group_detach | 0 | 21 | | WARNING:at_kernel/events/core.c:#list_del_event | 0 | 21 | | EIP:list_del_event | 0 | 21 | | BUG:kernel_NULL_pointer_dereference,address | 0 | 10 | | Oops:#[##] | 0 | 10 | | EIP:ctx_sched_out | 0 | 10 | | Kernel_panic-not_syncing:Fatal_exception | 0 | 10 | | WARNING:at_kernel/events/core.c:#perf_event_remove_dup | 0 | 1 | | EIP:perf_event_remove_dup | 0 | 1 | | WARNING:possible_circular_locking_dependency_detected | 0 | 3 | | BUG:kernel_hang_in_early-boot_stage,last_printk:Probing_EDD(edd=off_to_disable)...ok | 0 | 2 | | invoked_oom-killer:gfp_mask=0x | 0 | 1 | +--------------------------------------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [child0:625] quotactl (131) returned ENOSYS, mar . [child0:625] getresuid16 (165) returned ENOSYS, marking as inactive. [child0:625] setresuid16 (164) returned ENOSYS, marking as inactive. [ 15.893642] ------------[ cut here ]------------ [ 15.895385] WARNING: CPU: 0 PID: 648 at kernel/events/core.c:1755 perf_event_remove_dup+0x180/0x1b0 [ 15.898064] Modules linked in: [ 15.898830] CPU: 0 PID: 648 Comm: trinity-subchil Not tainted 5.4.0-rc8-00109-g9dc510db74b1b #1 [ 15.900469] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 15.901575] EIP: perf_event_remove_dup+0x180/0x1b0 [ 15.902213] Code: db fe ff ff 3b b8 fc 03 00 00 75 ee 89 88 fc 03 00 00 eb e6 8d b6 00 00 00 00 0f 0b e9 42 ff ff ff 8d b4 26 00 00 00 00 66 90 <0f> 0b e9 a0 fe ff ff 8d b4 26 00 00 00 00 66 90 c7 83 fc 03 00 00 [ 15.904798] EAX: ffffffff EBX: f12a9008 ECX: ffffffff EDX: f113d8b8 [ 15.905764] ESI: f16d5a08 EDI: f12a9008 EBP: f11c3e4c ESP: f11c3e34 [ 15.906664] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010082 [ 15.907645] CR0: 80050033 CR2: 08063c76 CR3: 17a98000 CR4: 00340690 [ 15.908548] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 15.909457] DR6: fffe0ff0 DR7: 00000600 [ 15.910030] Call Trace: [ 15.910458] list_del_event+0x39/0x100 [ 15.911107] perf_event_exit_task+0x199/0x3d0 [ 15.911716] do_exit+0x2bb/0xb70 [ 15.912160] ? find_held_lock+0x27/0xa0 [ 15.912649] ? get_signal+0x100/0xb90 [ 15.913117] do_group_exit+0x35/0x90 [ 15.913539] get_signal+0x149/0xb90 [ 15.913954] do_signal+0x24/0x540 [ 15.914373] ? finish_task_switch+0x69/0x220 [ 15.914890] ? kvm_cpu_online+0x20/0x20 [ 15.915343] exit_to_usermode_loop+0x5d/0xa0 [ 15.915980] prepare_exit_to_usermode+0xd7/0xe0 [ 15.916514] resume_userspace+0xd/0x12 [ 15.916956] EIP: 0x8063c80 [ 15.917292] Code: Bad RIP value. [ 15.917656] EAX: b7697000 EBX: 0813d000 ECX: bf904294 EDX: 00000003 [ 15.918469] ESI: b7697000 EDI: bf9042a5 EBP: 00000003 ESP: bf90423c [ 15.919318] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00010282 [ 15.920252] irq event stamp: 664 [ 15.920671] hardirqs last enabled at (663): [<d6ae619b>] __call_rcu+0x14b/0x410 [ 15.921613] hardirqs last disabled at (664): [<d746c0a1>] _raw_spin_lock_irq+0x11/0x40 [ 15.922788] softirqs last enabled at (0): [<d6a598b8>] copy_process+0x758/0x1fe0 [ 15.923842] softirqs last disabled at (0): [<00000000>] 0x0 [ 15.924701] ---[ end trace 74f78136c8701f9c ]--- [ 15.925237] # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 43a130289bccadcd56d4d500da6bc73ba4e2f653 219d54332a09e8d8741c1e1982f5eae56099de85 -- git bisect good 460c303a1b37f3aa67e4b08816f038c33a44271c # 22:30 G 307 0 23 28 Merge 'linux-review/Vladimir-Oltean/Configure-the-MTU-on-DSA-switches/20191125-041645' into devel-hourly-2019113011 git bisect bad 81902474a18eb750d77579ffc736a8f26661ca15 # 23:02 B 275 4 42 42 Merge 'linux-review/Darrick-J-Wong/iomap-trace-iomap_appply-results/20191124-135146' into devel-hourly-2019113011 git bisect bad 055037050e99520ad07793fbdf04eb319142447a # 23:45 B 129 1 22 22 Merge 'linux-review/Christophe-Kerello/mtd-Use-mtd-device-name-instead-of-mtd-name-when-registering-nvmem-device/20191125-011852' into devel-hourly-2019113011 git bisect good 5b9c62240d60e33f18af9a35495dcfd79820f17b # 00:37 G 304 0 20 20 Merge 'linux-review/Olga-Kornievskaia/NFS-allow-deprecation-of-NFS-UDP-protocol/20191127-084736' into devel-hourly-2019113011 git bisect good d87824744f4d98da8ea7b862ab9871e0c872e75f # 01:18 G 306 0 18 18 Merge 'linux-review/Charles-Keepax/spi-cadence-Correct-handling-of-native-chipselect/20191127-063016' into devel-hourly-2019113011 git bisect bad cf3ff810e66ff6b5d3b96aecea83224fae6242f6 # 01:51 B 302 2 54 54 Merge 'linux-review/Nick-Desaulniers/arm-explicitly-place-fixup-in-text/20191125-015858' into devel-hourly-2019113011 git bisect good f05394a456722fe19a68315a36e773c4df9a4459 # 02:45 G 309 0 15 15 Merge 'linux-review/Marc-Gonzalez/clk-Convert-managed-get-functions-to-devm_add_action-API/20191127-045647' into devel-hourly-2019113011 git bisect good 2604a2741f43dcbe5f180a4242c9ec9c9a984633 # 04:41 G 310 0 23 23 Merge 'userns/for-next' into devel-hourly-2019113011 git bisect bad 86762966e08c6cc38c5bfdede56a6660ab315d33 # 05:17 B 95 1 16 17 Merge 'linux-review/Song-Liu/perf-Sharing-PMU-counters-across-compatible-events/20191125-021305' into devel-hourly-2019113011 git bisect bad 9dc510db74b1b873853747098e1cd66b71e63210 # 06:41 B 309 1 58 58 perf: Sharing PMU counters across compatible events # first bad commit: [9dc510db74b1b873853747098e1cd66b71e63210] perf: Sharing PMU counters across compatible events git bisect good 6b8a794678763130b7e7d049985008641dc494e8 # 09:12 G 910 0 69 69 Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost # extra tests with debug options git bisect bad 9dc510db74b1b873853747098e1cd66b71e63210 # 11:18 B 17 1 2 2 perf: Sharing PMU counters across compatible events # extra tests on head commit of linux-review/Song-Liu/perf-Sharing-PMU-counters-across-compatible-events/20191125-021305 git bisect bad 9dc510db74b1b873853747098e1cd66b71e63210 # 11:21 B 251 1 0 58 perf: Sharing PMU counters across compatible events # bad: [9dc510db74b1b873853747098e1cd66b71e63210] perf: Sharing PMU counters across compatible events # extra tests on revert first bad commit git bisect good ff9b7027a44b53349b76d666b6e6f1d226433fa7 # 12:40 G 302 0 26 26 Revert "perf: Sharing PMU counters across compatible events" # good: [ff9b7027a44b53349b76d666b6e6f1d226433fa7] Revert "perf: Sharing PMU counters across compatible events" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

1 year, 2 months

1
0
0 / 0

86dc301f7b ("drivers/base/memory.c: cache blocks in radix tree .."): [ 1.341517] WARNING: suspicious RCU usage

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://github.com/0day-ci/linux/commits/Scott-Cheloha/drivers-base-memor... commit 86dc301f7b4815d90e3a7843ffed655d64efe445 Author: Scott Cheloha <cheloha(a)linux.vnet.ibm.com> AuthorDate: Thu Nov 21 13:59:52 2019 -0600 Commit: 0day robot <lkp(a)intel.com> CommitDate: Sun Nov 24 10:45:59 2019 +0800 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup Searching for a particular memory block by id is slow because each block device is kept in an unsorted linked list on the subsystem bus. Lookup is much faster if we cache the blocks in a radix tree. Memory subsystem initialization and hotplug/hotunplug is at least a little faster for any machine with more than ~100 blocks, and the speedup grows with the block count. Signed-off-by: Scott Cheloha <cheloha(a)linux.vnet.ibm.com> Acked-by: David Hildenbrand <david(a)redhat.com> 0e4a459f56 tracing: Remove unnecessary DEBUG_FS dependency 86dc301f7b drivers/base/memory.c: cache blocks in radix tree to accelerate lookup +---------------------------------------------------------------------+------------+------------+ | | 0e4a459f56 | 86dc301f7b | +---------------------------------------------------------------------+------------+------------+ | boot_successes | 8 | 0 | | boot_failures | 25 | 11 | | WARNING:possible_circular_locking_dependency_detected | 25 | 8 | | WARNING:suspicious_RCU_usage | 0 | 11 | | include/linux/radix-tree.h:#suspicious_rcu_dereference_check()usage | 0 | 11 | +---------------------------------------------------------------------+------------+------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 1.335279] random: get_random_bytes called from kcmp_cookies_init+0x29/0x4c with crng_init=0 [ 1.336883] ACPI: bus type PCI registered [ 1.338295] PCI: Using configuration type 1 for base access [ 1.340735] [ 1.341049] ============================= [ 1.341517] WARNING: suspicious RCU usage [ 1.342266] 5.4.0-rc5-00070-g86dc301f7b481 #1 Tainted: G T [ 1.343494] ----------------------------- [ 1.344226] include/linux/radix-tree.h:167 suspicious rcu_dereference_check() usage! [ 1.345516] [ 1.345516] other info that might help us debug this: [ 1.345516] [ 1.346962] [ 1.346962] rcu_scheduler_active = 2, debug_locks = 1 [ 1.348134] no locks held by swapper/0/1. [ 1.348866] [ 1.348866] stack backtrace: [ 1.349525] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G T 5.4.0-rc5-00070-g86dc301f7b481 #1 [ 1.351230] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 1.352720] Call Trace: [ 1.353187] ? dump_stack+0x9a/0xde [ 1.353507] ? node_access_release+0x19/0x19 [ 1.353507] ? walk_memory_blocks+0xe6/0x184 [ 1.353507] ? set_debug_rodata+0x20/0x20 [ 1.353507] ? link_mem_sections+0x39/0x3d [ 1.353507] ? topology_init+0x74/0xc8 [ 1.353507] ? enable_cpu0_hotplug+0x15/0x15 [ 1.353507] ? do_one_initcall+0x13d/0x30a [ 1.353507] ? kernel_init_freeable+0x18e/0x23b [ 1.353507] ? rest_init+0x173/0x173 [ 1.353507] ? kernel_init+0x10/0x151 [ 1.353507] ? rest_init+0x173/0x173 [ 1.353507] ? ret_from_fork+0x3a/0x50 [ 1.410829] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 1.427389] cryptd: max_cpu_qlen set to 1000 [ 1.457792] ACPI: Added _OSI(Module Device) [ 1.458615] ACPI: Added _OSI(Processor Device) [ 1.459428] ACPI: Added _OSI(3.0 _SCP Extensions) # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start 1d5f18079cb10420c4e4b67f571b998ec861a20c 219d54332a09e8d8741c1e1982f5eae56099de85 -- git bisect good 00e5729fc7309fd1da26659b7dce0fcc0b46ab7e # 23:10 G 11 0 9 9 Merge 'linux-review/Daniel-W-S-Almeida/media-dummy_dvb_fe-register-adapter-frontend/20191127-043813' into devel-hourly-2019113015 git bisect bad da795b1ad310518d9100105235c9ba3ff4ee2524 # 23:32 B 0 11 27 0 Merge 'linux-review/Krzysztof-Kozlowski/media-Fix-Kconfig-indentation/20191122-055026' into devel-hourly-2019113015 git bisect bad 53f2832a6bf7103957815cef4ad58be22255ea7a # 00:21 B 0 7 24 0 Merge 'linux-review/Luc-Van-Oostenryck/misc-xilinx_sdfec-add-missing-__user-annotation/20191124-052749' into devel-hourly-2019113015 git bisect good 03a04e32f16a2c4bd267b48e8ab4aaf71c507050 # 00:55 G 11 0 8 8 Merge 'linux-review/Helmut-Grohne/mdio_bus-revert-inadvertent-error-code-change/20191124-171049' into devel-hourly-2019113015 git bisect bad 03403c59a591b12556bd0db1243eb0503112a0df # 02:22 B 0 10 26 0 Merge 'linux-review/Navid-Emamdoost/EDAC-Fix-memory-leak-in-i5100_init_one/20191124-103621' into devel-hourly-2019113015 git bisect good fae5c4b30fda35cbcc8389f432dcaac20f9b3a12 # 03:02 G 11 0 7 7 Merge 'linux-review/David-Gow/kunit-Always-print-actual-pointer-values-in-asserts/20191124-131742' into devel-hourly-2019113015 git bisect bad 1e0c725d2bb21789330a2fe1a360c37ae753eb18 # 04:01 B 0 10 26 0 Merge 'linux-review/Scott-Cheloha/drivers-base-memory-c-cache-blocks-in-radix-tree-to-accelerate-lookup/20191124-104557' into devel-hourly-2019113015 git bisect good 0902b87758e830e857e11f1538e50b218743f4b0 # 04:29 G 10 0 7 7 Merge 'linux-review/Julio-Faracco/drivers-net-virtio_net-Implement-a-dev_watchdog-handler/20191124-135051' into devel-hourly-2019113015 git bisect good 2d0c894673b55b51915ea858eabcf7c97c9b8ccb # 05:33 G 10 0 6 6 Merge 'linux-review/Maciej-enczykowski/net-ipv6-IPV6_TRANSPARENT-check-NET_RAW-prior-to-NET_ADMIN/20191124-120121' into devel-hourly-2019113015 git bisect good 3c095c856fb84271e56e1869aface37af1b078f1 # 05:58 G 11 0 10 10 Merge 'linux-review/Navid-Emamdoost/Bluetooth-Fix-memory-leak-in-hci_connect_le_scan/20191124-111255' into devel-hourly-2019113015 git bisect bad 86dc301f7b4815d90e3a7843ffed655d64efe445 # 06:39 B 0 11 27 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # first bad commit: [86dc301f7b4815d90e3a7843ffed655d64efe445] drivers/base/memory.c: cache blocks in radix tree to accelerate lookup git bisect good 0e4a459f56c32d3e52ae69a4b447db2f48a65f44 # 07:25 G 33 0 25 25 tracing: Remove unnecessary DEBUG_FS dependency # extra tests with debug options git bisect good 86dc301f7b4815d90e3a7843ffed655d64efe445 # 09:13 G 11 0 0 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # extra tests on head commit of linux-review/Scott-Cheloha/drivers-base-memory-c-cache-blocks-in-radix-tree-to-accelerate-lookup/20191124-104557 git bisect bad 86dc301f7b4815d90e3a7843ffed655d64efe445 # 09:16 B 0 11 27 0 drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # bad: [86dc301f7b4815d90e3a7843ffed655d64efe445] drivers/base/memory.c: cache blocks in radix tree to accelerate lookup # extra tests on revert first bad commit git bisect good 65bba365eabe04a22e5d68012a45b92eee26860c # 10:18 G 10 0 9 9 Revert "drivers/base/memory.c: cache blocks in radix tree to accelerate lookup" # good: [65bba365eabe04a22e5d68012a45b92eee26860c] Revert "drivers/base/memory.c: cache blocks in radix tree to accelerate lookup" --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

1 year, 2 months

1
0
0 / 0

bf8e602186 ("tracing: Do not create tracefs files if tracefs .."): [ 2.789121] WARNING: CPU: 1 PID: 1 at kernel/trace/ftrace.c:989 ftrace_init_tracefs_toplevel

by kernel test robot

Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit bf8e602186ec402ed937b2cbd6c39a34c0029757 Author: Steven Rostedt (VMware) <rostedt(a)goodmis.org> AuthorDate: Fri Oct 11 20:41:41 2019 -0400 Commit: Steven Rostedt (VMware) <rostedt(a)goodmis.org> CommitDate: Sat Oct 12 20:49:07 2019 -0400 tracing: Do not create tracefs files if tracefs lockdown is in effect If on boot up, lockdown is activated for tracefs, don't even bother creating the files. This can also prevent instances from being created if lockdown is in effect. Link: http://lkml.kernel.org/r/CAHk-=whC6Ji=fWnjh2+eS4b15TnbsS4VPVtvBOwCy1jjEG_... Suggested-by: Linus Torvalds <torvalds(a)linux-foundation.org> Signed-off-by: Steven Rostedt (VMware) <rostedt(a)goodmis.org> 17911ff38a tracing: Add locked_down checks to the open calls of files created for tracefs bf8e602186 tracing: Do not create tracefs files if tracefs lockdown is in effect 81b6b96475 Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux; tag 'dma-mapping-5.5' of git://git.infradead.org/users/hch/dma-mapping 419593dad8 Add linux-next specific files for 20191129 +----------------------------------------------------------------+------------+------------+------------+---------------+ | | 17911ff38a | bf8e602186 | 81b6b96475 | next-20191129 | +----------------------------------------------------------------+------------+------------+------------+---------------+ | boot_successes | 31 | 0 | 0 | 0 | | boot_failures | 0 | 11 | 11 | 11 | | WARNING:at_kernel/trace/ftrace.c:#ftrace_init_tracefs_toplevel | 0 | 11 | 11 | 11 | | EIP:ftrace_init_tracefs_toplevel | 0 | 11 | 11 | 11 | | WARNING:at_kernel/trace/trace.c:#create_trace_option_files | 0 | 11 | 11 | 11 | | EIP:create_trace_option_files | 0 | 11 | 11 | 11 | | Mem-Info | 0 | 1 | | | +----------------------------------------------------------------+------------+------------+------------+---------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot <lkp(a)intel.com> [ 2.780405] Lockdown: swapper/0: use of tracefs is restricted; see man kernel_lockdown.7 [ 2.782605] Could not create tracefs 'set_graph_notrace' entry [ 2.784297] Lockdown: swapper/0: use of tracefs is restricted; see man kernel_lockdown.7 [ 2.786368] ------------[ cut here ]------------ [ 2.787647] Could not register function stat for cpu 0 [ 2.789121] WARNING: CPU: 1 PID: 1 at kernel/trace/ftrace.c:989 ftrace_init_tracefs_toplevel+0x184/0x1d8 [ 2.792367] Modules linked in: [ 2.793074] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G T 5.4.0-rc2-00007-gbf8e602186ec4 #1 [ 2.793074] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 [ 2.793074] EIP: ftrace_init_tracefs_toplevel+0x184/0x1d8 [ 2.793074] Code: fe 85 c0 0f 84 58 ff ff ff 31 c9 ba 01 00 00 00 8b 5d f0 b8 e0 8a c7 42 6a 01 e8 b8 23 42 fe 53 68 94 67 8f 42 e8 7d f5 2f fe <0f> 0b 31 c9 ba 01 00 00 00 6a 01 b8 c8 8a c7 42 e8 98 23 42 fe 8b [ 2.793074] EAX: 0000002a EBX: 00000000 ECX: 00000000 EDX: 00000000 [ 2.793074] ESI: 42dde7fc EDI: f55eab2c EBP: 402cbf00 ESP: 402cbedc [ 2.793074] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010202 [ 2.793074] CR0: 80050033 CR2: ffffffff CR3: 02fd9000 CR4: 00040690 [ 2.793074] Call Trace: [ 2.793074] ? register_tracer+0x170/0x170 [ 2.793074] ? tracer_init_tracefs+0x91/0x1e7 [ 2.793074] ? do_one_initcall+0x75/0x3b8 [ 2.793074] ? trace_initcall_level+0xe1/0x101 [ 2.793074] ? kernel_init_freeable+0xf5/0x192 [ 2.793074] ? kernel_init_freeable+0x114/0x192 [ 2.793074] ? rest_init+0x140/0x140 [ 2.793074] ? kernel_init+0x10/0x110 [ 2.793074] ? schedule_tail_wrapper+0x9/0xc [ 2.793074] ? ret_from_fork+0x33/0x40 [ 2.793074] _warn_unseeded_randomness: 123 callbacks suppressed [ 2.793074] random: get_random_bytes called from print_oops_end_marker+0x57/0x70 with crng_init=0 [ 2.793074] ---[ end trace f0d030514b1a50e3 ]--- [ 2.827821] Lockdown: swapper/0: use of tracefs is restricted; see man kernel_lockdown.7 # HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD git bisect start d196292990fce11fd7bb7585a782b3f4b34429e1 54ecb8f7028c5eb3d740bb82b0f1d90f2df63c5c -- git bisect bad 5d682fa3d8943c19a632ebeaf70e8b9e41c78a5b # 13:43 B 0 3 19 0 gpio: xgs-iproc: Fix section mismatch on device tree match table git bisect good 698b8eeaed7287970fc2b6d322618850fd1b1e6c # 14:31 G 11 0 0 0 gpio/mpc8xxx: change irq handler from chained to normal git bisect good 921d6c32b6f86c48e06667ce2f8c50ca45bfa212 # 15:03 G 10 0 0 0 MAINTAINERS: Add entry for RDA Micro GPIO driver and binding git bisect good 6a41b6c5fc20abced88fa0eed42ae5e5cb70b280 # 15:31 G 10 0 0 0 gpio: Add xgs-iproc driver git bisect bad c196924277ea82200d4c4fd9537c71390b96f247 # 16:04 B 0 1 18 1 Merge tag 'v5.4-rc6' into devel git bisect bad 998d75510e373aab5644d777d3b058312d550159 # 16:49 B 0 3 19 0 Merge branch 'akpm' (patches from Andrew) git bisect good c6ad7c3ce9800e91d6cc6d2f6f566339ebac5656 # 17:28 G 10 0 0 0 Merge tag '5.4-rc2-smb3' of git://git.samba.org/sfrench/cifs-2.6 git bisect good 71b1b5532b9c58f260911ee59c7b3007d6d673a5 # 17:48 G 11 0 0 0 Merge tag 'fixes-for-5.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux git bisect bad ad32fd7426e192cdf5368eda23a6482ff83c2022 # 18:16 B 0 8 24 0 Merge tag 'xtensa-20191017' of git://github.com/jcmvbkbc/linux-xtensa git bisect bad 8625732e7712882bd14e1fce962bdc3c315acd41 # 18:36 B 0 5 21 0 Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi git bisect bad 3c52b0af059e11a063970aed1ad143b9284a79c7 # 19:00 B 0 1 17 0 lib/generic-radix-tree.c: add kmemleak annotations git bisect bad d303de1fcf344ff7c15ed64c3f48a991c9958775 # 19:23 B 0 1 17 0 tracing: Initialize iter->seq after zeroing in tracing_read_pipe() git bisect good 8530dec63e7b486e3761cc3d74a22de301845ff5 # 20:15 G 11 0 0 0 tracing: Add tracing_check_open_get_tr() git bisect bad 7f8557b88d6aa5bf31f25f6013d81355a1b1d48d # 21:01 B 0 11 27 0 recordmcount: Fix nop_mcount() function git bisect bad bf8e602186ec402ed937b2cbd6c39a34c0029757 # 22:05 B 0 4 20 0 tracing: Do not create tracefs files if tracefs lockdown is in effect git bisect good 17911ff38aa58d3c95c07589dbf5d3564c4cf3c5 # 22:36 G 11 0 0 0 tracing: Add locked_down checks to the open calls of files created for tracefs # first bad commit: [bf8e602186ec402ed937b2cbd6c39a34c0029757] tracing: Do not create tracefs files if tracefs lockdown is in effect git bisect good 17911ff38aa58d3c95c07589dbf5d3564c4cf3c5 # 22:41 G 31 0 0 0 tracing: Add locked_down checks to the open calls of files created for tracefs # extra tests with debug options git bisect good bf8e602186ec402ed937b2cbd6c39a34c0029757 # 23:07 G 11 0 0 0 tracing: Do not create tracefs files if tracefs lockdown is in effect # extra tests on head commit of linus/master git bisect bad 81b6b96475ac7a4ebfceae9f16fb3758327adbfe # 23:57 B 0 2 18 0 Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux; tag 'dma-mapping-5.5' of git://git.infradead.org/users/hch/dma-mapping # bad: [81b6b96475ac7a4ebfceae9f16fb3758327adbfe] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux; tag 'dma-mapping-5.5' of git://git.infradead.org/users/hch/dma-mapping # extra tests on revert first bad commit git bisect good 38cce32718f755a5a296c21f7ee4e20a981a8a82 # 00:41 G 10 0 0 0 Revert "tracing: Do not create tracefs files if tracefs lockdown is in effect" # good: [38cce32718f755a5a296c21f7ee4e20a981a8a82] Revert "tracing: Do not create tracefs files if tracefs lockdown is in effect" # extra tests on linus/master # duplicated: [81b6b96475ac7a4ebfceae9f16fb3758327adbfe] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux; tag 'dma-mapping-5.5' of git://git.infradead.org/users/hch/dma-mapping # extra tests on linux-next/master git bisect bad 419593dad8439007452bb6f267861863b572c520 # 00:46 B 0 11 27 0 Add linux-next specific files for 20191129 # bad: [419593dad8439007452bb6f267861863b572c520] Add linux-next specific files for 20191129 --- 0-DAY kernel test infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation

1 year, 2 months

1
0
0 / 0

2021

2020

2019

2018

2017

2016

2015

2014

2013

LKP December 2019