[sched, rcu] 9234566d3a3: +1.6% will-it-scale.scalability, +1302.6% time.involuntary_context_switches
by Jet Chen
Hi Paul,
we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git torture.2014.04.18a
commit 9234566d3a36c0aead8852e3c2ca94cd8ebfe219 ("sched,rcu: Make cond_resched() report RCU quiescent states")
Comparison 1 - parent commit of 9234566d3a36c0aead8852e3c2ca94cd8ebfe219 vs 9234566d3a36c0aead8852e3c2ca94cd8ebfe219
e119454e74a852f 9234566d3a36c0aead8852e3c
--------------- -------------------------
1035948 ~ 0% +1.6% 1052990 ~ 0% TOTAL will-it-scale.per_thread_ops
1271322 ~ 0% +1.8% 1294004 ~ 0% TOTAL will-it-scale.per_process_ops
0.63 ~ 0% -5.2% 0.60 ~ 0% TOTAL will-it-scale.scalability
22470 ~ 2% +1302.6% 315168 ~ 2% TOTAL time.involuntary_context_switches
84265 ~ 5% +1047.1% 966581 ~ 1% TOTAL interrupts.IWI
1828 ~44% +189.6% 5295 ~13% TOTAL time.voluntary_context_switches
5337 ~ 1% +82.1% 9720 ~ 1% TOTAL vmstat.system.cs
118599 ~ 0% -30.4% 82545 ~ 0% TOTAL interrupts.0:IO-APIC-edge.timer
224021 ~ 4% +34.7% 301858 ~ 2% TOTAL interrupts.RES
25148 ~ 0% +7.0% 26917 ~ 0% TOTAL vmstat.system.in
7063439 ~ 0% -5.2% 6694536 ~ 0% TOTAL interrupts.LOC
188866 ~ 0% -3.1% 183008 ~ 0% TOTAL interrupts.NMI
188866 ~ 0% -3.1% 183008 ~ 0% TOTAL interrupts.PMI
3720 ~ 0% -1.5% 3665 ~ 0% TOTAL time.system_time
1215 ~ 0% -1.4% 1198 ~ 0% TOTAL time.percent_of_cpu_this_job_got
Comparison 2 - b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 vs 9234566d3a36c0aead8852e3c2ca94cd8ebfe219
Fengguang has reported stats changes about b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 to you days ago.
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")
Let's have a compare here.
b84c4e08143c98d 9234566d3a36c0aead8852e3c
--------------- -------------------------
809309 ~ 0% -2.6% 788400 ~ 0% TOTAL will-it-scale.per_process_ops
0.61 ~ 0% -1.9% 0.60 ~ 0% TOTAL will-it-scale.scalability
434080 ~ 0% -1.5% 427643 ~ 0% TOTAL will-it-scale.per_thread_ops
4 ~11% +1.2e+05% 5249 ~ 2% TOTAL interrupts.IWI
607 ~ 7% +28.0% 778 ~14% TOTAL interrupts.47:PCI-MSI-edge.eth0
12349 ~ 2% -14.6% 10548 ~ 1% TOTAL interrupts.0:IO-APIC-edge.timer
3078 ~ 3% +20.9% 3722 ~ 6% TOTAL interrupts.RES
Comparison 3 - parent commit of b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 vs b84c4e08143c98dad4b4d139f08db0b98b0d3ec4
Duplicated with Fengguang's report. Help you to see these info in a page :)
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
676264 ~ 0% +3.3% 698461 ~ 0% TOTAL will-it-scale.per_thread_ops
1174547 ~ 0% +3.0% 1209307 ~ 0% TOTAL will-it-scale.per_process_ops
1.67 ~ 0% -2.3% 1.63 ~ 0% TOTAL will-it-scale.scalability
10522 ~ 2% +921.2% 107463 ~ 1% TOTAL time.involuntary_context_switches
77671 ~ 3% +67.0% 129688 ~ 3% TOTAL interrupts.RES
99502 ~ 0% -27.8% 71813 ~ 0% TOTAL interrupts.0:IO-APIC-edge.timer
2554 ~ 0% +49.1% 3809 ~ 1% TOTAL vmstat.system.cs
11524 ~ 0% -2.3% 11259 ~ 0% TOTAL vmstat.system.in
213 ~ 0% -4.3% 204 ~ 0% TOTAL time.system_time
74 ~ 0% -4.1% 71 ~ 0% TOTAL time.percent_of_cpu_this_job_got
3495099 ~ 0% -3.1% 3387173 ~ 0% TOTAL interrupts.LOC
Thanks,
Jet
7 years
[KVM] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0
by Jet Chen
Hi Paolo,
We noticed the below kernel BUG on
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 93c4adc7afedf9b0ec190066d45b6d67db5270da
Author: Paolo Bonzini <pbonzini(a)redhat.com>
AuthorDate: Wed Mar 5 23:19:52 2014 +0100
Commit: Paolo Bonzini <pbonzini(a)redhat.com>
CommitDate: Mon Mar 17 12:21:39 2014 +0100
KVM: x86: handle missing MPX in nested virtualization
[ 15.785160] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0
[ 15.794972] IP: [<ffffffff81017fc9>] kvm_arch_init+0x110/0x22f
[ 15.802077] PGD 0
[ 15.804890] Oops: 0000 [#1] SMP
[ 15.809153] Modules linked in:
[ 15.813132] CPU: 12 PID: 1 Comm: swapper/0 Not tainted 3.14.0-rc7-03441-gad1b51f #1
[ 15.822566] Hardware name: Intel Corporation S2600CP/S2600CP, BIOS RMLSDP.86I.R2.28.D690.1306271008 06/27/2013
[ 15.834635] task: ffff880813da8000 ti: ffff880813db0000 task.ti: ffff880813db0000
[ 15.843883] RIP: 0010:[<ffffffff81017fc9>] [<ffffffff81017fc9>] kvm_arch_init+0x110/0x22f
[ 15.854097] RSP: 0000:ffff880813db1df0 EFLAGS: 00010246
[ 15.860514] RAX: 0000000000000000 RBX: 000000000000000c RCX: 0000000000000000
[ 15.868973] RDX: 0000000000000000 RSI: ffff880813db1dfc RDI: 0000000000000d90
[ 15.877436] RBP: ffff880813db1e28 R08: 0000000000015f40 R09: ffff88081f003e00
[ 15.885895] R10: ffffffff811bd5ec R11: 0000000000000000 R12: 0000000000000017
[ 15.894353] R13: 0000000000000018 R14: ffff880813db1dfc R15: ffffffff82218f20
[ 15.902813] FS: 0000000000000000(0000) GS:ffff88101ee40000(0000) knlGS:0000000000000000
[ 15.922590] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 15.929492] CR2: 00000000000002b0 CR3: 000000000220f000 CR4: 00000000001407e0
[ 15.937957] Stack:
[ 15.940680] 0000000013db1e10 00000000811f0f59 0000000000000006 ffffffff8236743a
[ 15.950159] 0000000000000040 0000000000003f40 0000000000000000 ffff880813db1e68
[ 15.959665] ffffffff81003d72 ffffffff8236743a 0000000000000006 ffffffff8236743a
[ 15.969172] Call Trace:
[ 15.972399] [<ffffffff8236743a>] ? vmx_check_processor_compat+0x61/0x61
[ 15.980376] [<ffffffff81003d72>] kvm_init+0x24/0x27b
[ 15.986504] [<ffffffff8236743a>] ? vmx_check_processor_compat+0x61/0x61
[ 15.994478] [<ffffffff8236743a>] ? vmx_check_processor_compat+0x61/0x61
[ 16.002452] [<ffffffff82367729>] vmx_init+0x2ef/0x579
[ 16.008675] [<ffffffff8236743a>] ? vmx_check_processor_compat+0x61/0x61
[ 16.016648] [<ffffffff81000309>] do_one_initcall+0xd1/0x175
[ 16.023462] [<ffffffff8110f5d2>] ? parse_args+0x20a/0x3bf
[ 16.030083] [<ffffffff823620ca>] kernel_init_freeable+0x1a6/0x22b
[ 16.037477] [<ffffffff823618bc>] ? do_early_param+0x88/0x88
[ 16.044287] [<ffffffff81ae99c9>] ? rest_init+0x89/0x89
[ 16.050612] [<ffffffff81ae99d7>] kernel_init+0xe/0xdf
[ 16.056839] [<ffffffff81b07b7c>] ret_from_fork+0x7c/0xb0
[ 16.063355] [<ffffffff81ae99c9>] ? rest_init+0x89/0x89
[ 16.069676] Code: 3c 9d 10 6a 21 82 4c 89 f6 e8 89 ef 05 00 66 90 8b 45 d4 85 c0 78 da 81 3c 9d 10 6a 21 82 90 0d 00 00 75 b3 48 8b 05 87 91 4a 01 <ff> 90 b0 02 00 00 84 c0 75 a2 48 83 c3 01 48 83 fb 0d 75 bc 45
[ 16.097850] RIP [<ffffffff81017fc9>] kvm_arch_init+0x110/0x22f
[ 16.105043] RSP <ffff880813db1df0>
[ 16.109417] CR2: 00000000000002b0
[ 16.113611] ---[ end trace 740886b77a583740 ]---
[ 16.119251] Kernel panic - not syncing: Fatal exception
Full dmesg & Kconifg are attached, and more details can be provided on your request.
BTW, same issue has been reported by Fengguang last month.
https://lkml.org/lkml/2014/3/26/200
Thanks,
Jet
7 years
[libata/ahci] 8a4aeec8d2d: +138.4% perf-stat.dTLB-store-misses, +37.2% perf-stat.dTLB-load-misses
by Jet Chen
HI Dan,
we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata for-next
commit 8a4aeec8d2d6a3edeffbdfae451cdf05cbf0fefd ("libata/ahci: accommodate tag ordered controllers")
2cf532f5e67c0cf 8a4aeec8d2d6a3edeffbdfae4
--------------- -------------------------
88694337 ~39% +138.4% 2.115e+08 ~46% TOTAL perf-stat.dTLB-store-misses
217057 ~ 0% -31.3% 149221 ~ 3% TOTAL interrupts.46:PCI-MSI-edge.ahci
6.995e+08 ~20% +37.2% 9.598e+08 ~25% TOTAL perf-stat.dTLB-load-misses
110302 ~ 0% -28.9% 78402 ~ 2% TOTAL interrupts.CAL
3.168e+08 ~ 9% +14.5% 3.627e+08 ~10% TOTAL perf-stat.L1-dcache-prefetches
2.553e+09 ~12% +26.5% 3.228e+09 ~11% TOTAL perf-stat.LLC-loads
5.815e+08 ~ 6% +27.3% 7.403e+08 ~11% TOTAL perf-stat.LLC-stores
3.662e+09 ~11% +22.9% 4.501e+09 ~10% TOTAL perf-stat.L1-dcache-load-misses
2.155e+10 ~ 1% +8.3% 2.333e+10 ~ 1% TOTAL perf-stat.L1-dcache-store-misses
3.619e+10 ~ 1% +5.9% 3.832e+10 ~ 2% TOTAL perf-stat.cache-references
1.605e+10 ~ 1% +4.3% 1.674e+10 ~ 1% TOTAL perf-stat.L1-icache-load-misses
239691 ~ 7% -8.4% 219537 ~ 1% TOTAL interrupts.RES
3483 ~ 0% -5.4% 3297 ~ 0% TOTAL vmstat.system.in
2.748e+08 ~ 1% +4.3% 2.865e+08 ~ 0% TOTAL perf-stat.cache-misses
98935369 ~ 0% +4.9% 1.038e+08 ~ 0% TOTAL perf-stat.LLC-store-misses
699 ~ 1% -3.7% 673 ~ 1% TOTAL iostat.sda.w_await
698 ~ 1% -3.7% 672 ~ 1% TOTAL iostat.sda.await
203893 ~ 0% +3.7% 211474 ~ 0% TOTAL iostat.sda.wkB/s
203972 ~ 0% +3.7% 211488 ~ 0% TOTAL vmstat.io.bo
618082 ~ 4% -4.6% 589619 ~ 1% TOTAL perf-stat.context-switches
1.432e+12 ~ 1% +3.0% 1.475e+12 ~ 0% TOTAL perf-stat.L1-icache-loads
3.35e+11 ~ 0% +3.2% 3.456e+11 ~ 0% TOTAL perf-stat.L1-dcache-stores
1.486e+12 ~ 0% +2.8% 1.527e+12 ~ 0% TOTAL perf-stat.iTLB-loads
3.006e+11 ~ 0% +2.6% 3.084e+11 ~ 0% TOTAL perf-stat.branch-instructions
1.793e+12 ~ 0% +2.8% 1.843e+12 ~ 0% TOTAL perf-stat.cpu-cycles
3.352e+11 ~ 1% +2.9% 3.451e+11 ~ 0% TOTAL perf-stat.dTLB-stores
2.994e+11 ~ 1% +3.1% 3.087e+11 ~ 0% TOTAL perf-stat.branch-loads
1.49e+12 ~ 0% +2.9% 1.533e+12 ~ 0% TOTAL perf-stat.instructions
5.48e+11 ~ 0% +2.8% 5.633e+11 ~ 0% TOTAL perf-stat.dTLB-loads
2.028e+11 ~ 1% +2.9% 2.086e+11 ~ 1% TOTAL perf-stat.bus-cycles
5.484e+11 ~ 0% +2.9% 5.644e+11 ~ 0% TOTAL perf-stat.L1-dcache-loads
1.829e+12 ~ 0% +2.7% 1.877e+12 ~ 1% TOTAL perf-stat.ref-cycles
Legend:
~XX% - stddev percent
[+-]XX% - change percent
Attach full stats changes entries for reference.
Thanks,
Jet
7 years
[rcu] 10a94227ba2: -2.0% will-it-scale.per_process_ops
by Fengguang Wu
Paul,
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git next.2014.04.16b
commit 10a94227ba229f1b05672754dc318a8fe7982c95 ("rcu: Update cpu_needs_another_gp() for futures from non-NOCB CPUs")
test case: nhm4/micro/will-it-scale/lseek1
11ba5ab363b9359 10a94227ba229f1b05672754d
--------------- -------------------------
11210675 ~ 0% -2.0% 10985451 ~ 0% TOTAL will-it-scale.per_process_ops
1.24 ~ 5% -33.4% 0.83 ~ 5% TOTAL perf-profile.cpu-cycles.trace_hardirqs_off_caller.lseek64
3.88 ~ 2% +49.0% 5.79 ~ 0% TOTAL perf-profile.cpu-cycles.trace_hardirqs_on_thunk.lseek64
295 ~16% +27.0% 375 ~ 8% TOTAL cpuidle.C1E-NHM.usage
45061 ~ 2% +16.7% 52590 ~ 2% TOTAL cpuidle.C6-NHM.usage
1.21 ~ 4% +5.8% 1.28 ~ 4% TOTAL perf-profile.cpu-cycles.shmem_file_llseek.sys_lseek.system_call_fastpath.lseek64
4206 ~ 1% -78.6% 900 ~ 8% TOTAL interrupts.IWI
14303 ~ 1% +26.7% 18120 ~ 1% TOTAL interrupts.0:IO-APIC-edge.timer
3228 ~ 4% -17.2% 2672 ~ 6% TOTAL interrupts.RES
182 ~ 2% -8.1% 167 ~ 3% TOTAL time.user_time
235 ~ 2% +6.2% 250 ~ 2% TOTAL time.system_time
379471 ~ 0% +1.2% 384127 ~ 0% TOTAL interrupts.LOC
Legend:
~XX% - stddev percent
[+-]XX% - change percent
It does effectively eliminate interrupts.IWI:
interrupts.IWI
4500 ++-*-----*-------*--------------------------------------------------+
|.. *. + .*. .*..*. .*..*..*.. .*..*..*..*.*..*..*
4000 *+ *. *. *. *.. .*..*. |
3500 ++ * |
| |
3000 ++ |
2500 ++ |
| |
2000 ++ |
1500 ++ |
| |
1000 ++ O O O O O O O
500 ++ O |
| |
0 O+-O--O--O-O--O--O--O--O--O-O--O--O--O--O--O-O----------------------+
[*] bisect-good sample
[O] bisect-bad sample
Thanks,
Fengguang
7 years
[sched,rcu] b84c4e08143: +3.1% will-it-scale.per_thread_ops
by Fengguang Wu
Hi Paul,
FYI, this improves will-it-scale/open1 throughput.
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git dev.2014.04.14a
commit b84c4e08143c98dad4b4d139f08db0b98b0d3ec4 ("sched,rcu: Make cond_resched() report RCU quiescent states")
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
563496 ~ 0% +3.1% 581059 ~ 0% nhm4/micro/will-it-scale/open1
563496 ~ 0% +3.1% 581059 ~ 0% TOTAL will-it-scale.per_thread_ops
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
756894 ~ 0% +2.8% 778452 ~ 0% nhm4/micro/will-it-scale/open1
756894 ~ 0% +2.8% 778452 ~ 0% TOTAL will-it-scale.per_process_ops
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
0.57 ~ 0% -2.7% 0.55 ~ 0% nhm4/micro/will-it-scale/open1
0.57 ~ 0% -2.7% 0.55 ~ 0% TOTAL will-it-scale.scalability
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
346764 ~ 2% -74.0% 90164 ~ 1% nhm4/micro/will-it-scale/open1
346764 ~ 2% -74.0% 90164 ~ 1% TOTAL slabinfo.kmalloc-256.active_objs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
10837 ~ 2% -73.9% 2824 ~ 1% nhm4/micro/will-it-scale/open1
10837 ~ 2% -73.9% 2824 ~ 1% TOTAL slabinfo.kmalloc-256.active_slabs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
10837 ~ 2% -73.9% 2824 ~ 1% nhm4/micro/will-it-scale/open1
10837 ~ 2% -73.9% 2824 ~ 1% TOTAL slabinfo.kmalloc-256.num_slabs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
346821 ~ 2% -73.9% 90393 ~ 1% nhm4/micro/will-it-scale/open1
346821 ~ 2% -73.9% 90393 ~ 1% TOTAL slabinfo.kmalloc-256.num_objs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
105961 ~ 1% -63.0% 39153 ~ 1% nhm4/micro/will-it-scale/open1
105961 ~ 1% -63.0% 39153 ~ 1% TOTAL meminfo.SUnreclaim
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
26432 ~ 1% -62.9% 9814 ~ 1% nhm4/micro/will-it-scale/open1
26432 ~ 1% -62.9% 9814 ~ 1% TOTAL proc-vmstat.nr_slab_unreclaimable
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
50298 ~ 0% +194.3% 148011 ~ 0% nhm4/micro/will-it-scale/open1
37020 ~ 0% +42.6% 52798 ~ 1% nhm4/micro/will-it-scale/signal1
87318 ~ 0% +130.0% 200809 ~ 0% TOTAL softirqs.RCU
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
140354 ~ 1% -47.6% 73490 ~ 0% nhm4/micro/will-it-scale/open1
140354 ~ 1% -47.6% 73490 ~ 0% TOTAL meminfo.Slab
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
77391 ~ 1% -46.7% 41235 ~ 2% nhm4/micro/will-it-scale/signal1
77391 ~ 1% -46.7% 41235 ~ 2% TOTAL cpuidle.C6-NHM.usage
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
19871 ~ 2% -37.6% 12397 ~ 2% nhm4/micro/will-it-scale/open1
18497 ~ 1% -37.5% 11556 ~ 1% nhm4/micro/will-it-scale/signal1
38368 ~ 2% -37.6% 23954 ~ 2% TOTAL softirqs.SCHED
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1.24 ~ 4% -35.4% 0.80 ~ 3% nhm4/micro/will-it-scale/open1
1.24 ~ 4% -35.4% 0.80 ~ 3% TOTAL perf-profile.cpu-cycles.do_notify_resume.int_signal.close
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1.43 ~ 4% +41.9% 2.03 ~ 4% nhm4/micro/will-it-scale/open1
1.43 ~ 4% +41.9% 2.03 ~ 4% TOTAL perf-profile.cpu-cycles.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1.27 ~ 3% -30.0% 0.89 ~ 6% nhm4/micro/will-it-scale/open1
1.27 ~ 3% -30.0% 0.89 ~ 6% TOTAL perf-profile.cpu-cycles.setup_object.isra.46.new_slab.__slab_alloc.kmem_cache_alloc.get_empty_filp
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1.54 ~ 7% +35.6% 2.09 ~ 8% nhm4/micro/will-it-scale/open1
1.54 ~ 7% +35.6% 2.09 ~ 8% TOTAL perf-profile.cpu-cycles.kmem_cache_alloc.getname_flags.getname.do_sys_open.sys_open
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
4.21 ~ 2% -29.1% 2.98 ~ 3% nhm4/micro/will-it-scale/open1
4.21 ~ 2% -29.1% 2.98 ~ 3% TOTAL perf-profile.cpu-cycles.link_path_walk.path_openat.do_filp_open.do_sys_open.sys_open
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1.37 ~ 4% -23.1% 1.05 ~ 7% nhm4/micro/will-it-scale/open1
1.37 ~ 4% -23.1% 1.05 ~ 7% TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_openat.do_filp_open
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
0.88 ~17% +29.1% 1.14 ~ 9% nhm4/micro/will-it-scale/open1
0.88 ~17% +29.1% 1.14 ~ 9% TOTAL perf-profile.cpu-cycles.path_init.path_openat.do_filp_open.do_sys_open.sys_open
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
0.67 ~16% +33.6% 0.90 ~10% nhm4/micro/will-it-scale/signal1
0.67 ~16% +33.6% 0.90 ~10% TOTAL perf-profile.cpu-cycles.restore_sigcontext.sys_rt_sigreturn.stub_rt_sigreturn.raise
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
3.19 ~ 1% +17.4% 3.74 ~ 5% nhm4/micro/will-it-scale/open1
3.19 ~ 1% +17.4% 3.74 ~ 5% TOTAL perf-profile.cpu-cycles.file_free_rcu.rcu_process_callbacks.__do_softirq.irq_exit.smp_apic_timer_interrupt
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
4329 ~ 7% +15.2% 4986 ~ 5% nhm4/micro/will-it-scale/open1
4329 ~ 7% +15.2% 4986 ~ 5% TOTAL slabinfo.vm_area_struct.active_objs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
2536 ~ 1% -75.8% 614 ~ 9% nhm4/micro/will-it-scale/open1
2536 ~ 1% -75.8% 614 ~ 9% TOTAL time.involuntary_context_switches
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
32593 ~ 1% -62.1% 12349 ~ 2% nhm4/micro/will-it-scale/signal1
32593 ~ 1% -62.1% 12349 ~ 2% TOTAL interrupts.0:IO-APIC-edge.timer
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
4374 ~14% +124.7% 9830 ~ 8% nhm4/micro/will-it-scale/open1
2559 ~ 2% +20.3% 3078 ~ 3% nhm4/micro/will-it-scale/signal1
6934 ~ 9% +86.2% 12908 ~ 7% TOTAL interrupts.RES
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
490 ~ 1% -37.3% 307 ~ 1% nhm4/micro/will-it-scale/signal1
490 ~ 1% -37.3% 307 ~ 1% TOTAL vmstat.system.cs
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
1639 ~ 0% -8.8% 1495 ~ 0% nhm4/micro/will-it-scale/signal1
1639 ~ 0% -8.8% 1495 ~ 0% TOTAL vmstat.system.in
ad86a04266f9b49 b84c4e08143c98dad4b4d139f
--------------- -------------------------
419046 ~ 0% -1.5% 412661 ~ 0% nhm4/micro/will-it-scale/open1
400635 ~ 0% -5.9% 376866 ~ 0% nhm4/micro/will-it-scale/signal1
819681 ~ 0% -3.7% 789527 ~ 0% TOTAL interrupts.LOC
Legend:
~XX% - stddev percent
[+-]XX% - change percent
time.involuntary_context_switches
3500 ++------------------------------------------------------------------+
| .*.. |
3000 ++ .*. *..*.. .*..*.. .*.. |
*..*..*..*. *. * |
| *..*.. .*.. .*..* |
2500 ++ *..*. *..*. |
| |
2000 ++ |
| |
1500 ++ |
| |
| |
1000 ++ |
| O O O O O O O O O
500 O+-O-----------O--O--O--O--------O-O--O--O--O--O--------O-----O--O--+
[*] bisect-good sample
[O] bisect-bad sample
Thanks,
Fengguang
7 years
[ext4] BUG: sleeping function called from invalid context at fs/ext4/ext4_jbd2.c:252
by Fengguang Wu
Hi Ted,
FYI, we noticed that simple dd write can trigger the below BUG in
commit 208b9cc3a32f29b8a9bba682b5bfbb962bc9892d ("ext4: add a new
spinlock i_raw_lock to protect the ext4's raw inode")
[ 20.909408] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 21.932207] BUG: sleeping function called from invalid context at fs/ext4/ext4_jbd2.c:252
[ 21.933168] in_atomic(): 1, irqs_disabled(): 0, pid: 3280, name: dd
[ 21.933709] CPU: 1 PID: 3280 Comm: dd Not tainted 3.14.0-01910-gdeb73c7 #1
[ 21.934278] Hardware name: / , BIOS VVRBLI9J.86A.2891.2007.0511.1144 05/11/2007
[ 21.935205] 0000000000000000 ffff88007de99aa8 ffffffff819b8b88 ffff880038f10000
[ 21.936226] ffff88007de99ab8 ffffffff811030c2 ffff88007de99b18 ffffffff812bba58
[ 21.937231] 000000000000130c ffffffff812b5e7f ffff88007de99b00 ffffffff812d8046
[ 21.938231] Call Trace:
[ 21.938627] [<ffffffff819b8b88>] dump_stack+0x4d/0x66
[ 21.939140] [<ffffffff811030c2>] __might_sleep+0x10a/0x10c
[ 21.939656] [<ffffffff812bba58>] __ext4_handle_dirty_metadata+0x39/0x1b5
[ 21.940237] [<ffffffff812b5e7f>] ? ext4_ext_tree_init+0x32/0x36
[ 21.940780] [<ffffffff812d8046>] ? jbd2_journal_get_write_access+0x33/0x3c
[ 21.941362] [<ffffffff812974e6>] ext4_mark_iloc_dirty+0x577/0x630
[ 21.941915] [<ffffffff812b5e7f>] ? ext4_ext_tree_init+0x32/0x36
[ 21.941916] [<ffffffff812976c7>] ext4_mark_inode_dirty+0xa6/0x1f0
[ 21.941920] [<ffffffff812b5e7f>] ext4_ext_tree_init+0x32/0x36
[ 21.941922] [<ffffffff81292988>] __ext4_new_inode+0xf29/0x10c9
[ 21.941925] [<ffffffff8129f5bb>] ext4_create+0xac/0x156
[ 21.941930] [<ffffffff811e8bdf>] vfs_create+0x68/0xa1
[ 21.941932] [<ffffffff811eaf1b>] do_last.isra.52+0x518/0xa23
[ 21.941933] [<ffffffff811e8db9>] ? link_path_walk+0x69/0x71f
[ 21.941935] [<ffffffff811eb64b>] path_openat+0x225/0x591
[ 21.941937] [<ffffffff811ec7f8>] do_filp_open+0x3a/0x7f
[ 21.941939] [<ffffffff811f72f5>] ? __alloc_fd+0x5e/0xef
[ 21.941942] [<ffffffff811deec2>] do_sys_open+0x147/0x1d6
[ 21.941943] [<ffffffff811f6d7b>] ? __fdget+0x13/0x15
[ 21.941945] [<ffffffff811def6f>] SyS_open+0x1e/0x20
[ 21.941947] [<ffffffff819c7ae9>] system_call_fastpath+0x16/0x1b
Thanks,
Fengguang
_______________________________________________
LKP mailing list
LKP(a)linux.intel.com
7 years
[sched] INFO: rcu_sched self-detected stall on CPU { 3}
by Jet Chen
Hi Alex
We noticed the below kernel BUG on
https://github.com/alexshi/power-scheduling.git noload
commit 6b74b2031e15ae58470fd8dde7438df35e358c62
Author: Alex Shi <alex.shi(a)linaro.org>
AuthorDate: Fri Apr 4 17:49:30 2014 +0800
Commit: Alex Shi <alex.shi(a)linaro.org>
CommitDate: Fri Apr 4 17:49:30 2014 +0800
sched: let task moving destination cpu do active balance
Now we let the task source cpu do the active balance, while the
destination cpu maybe idle. At that time the task will be stopped
on resource cpu and wait the destination cpu up. That hurt the
performace. Let destination cpu do active balance will give task
<3>[ 614.504149] INFO: rcu_sched self-detected stall on CPU { 3} (t=100007 jiffies g=1455 c=1454 q=87882)
<6>[ 614.504731] sending NMI to all CPUs:
<4>[ 614.505003] NMI backtrace for cpu 0
<4>[ 614.505228] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.0-01205-g0e2d6b2 #1
<4>[ 614.505671] Hardware name: /DX58SO, BIOS SOX5810J.86A.4196.2009.0715.1958 07/15/2009
<4>[ 614.506185] task: ffffffff82011440 ti: ffffffff82000000 task.ti: ffffffff82000000
<4>[ 614.506637] RIP: 0010:[<ffffffff814c7599>] [<ffffffff814c7599>] intel_idle+0xdc/0x132
<4>[ 614.507116] RSP: 0018:ffffffff82001e48 EFLAGS: 00000046
<4>[ 614.507401] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
<4>[ 614.507750] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000046
<4>[ 614.508100] RBP: ffffffff82001e70 R08: ffff8800bf213ebc R09: 00000000000000ca
<4>[ 614.508449] R10: 0000000000000006 R11: 000000000000049a R12: 0000000000000004
<4>[ 614.508799] R13: 0000000000000020 R14: 0000000000000003 R15: 0000000000000000
<4>[ 614.509148] FS: 0000000000000000(0000) GS:ffff8800bf200000(0000) knlGS:0000000000000000
<4>[ 614.509622] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>[ 614.509922] CR2: 00000000025ae424 CR3: 000000000200c000 CR4: 00000000000007f0
<4>[ 614.510271] Stack:
<4>[ 614.510440] 0000000000000018 ffff8800bf21dd00 ffffffff820a2a18 0000008f0b6dd4cf
<4>[ 614.510918] 0000000008004000 ffffffff82001eb0 ffffffff81866cb1 0000000400000006
<4>[ 614.511396] ffffffff820a28a0 ffff8800bf21dd00 0000000000000004 ffffffff820a28a0
<4>[ 614.511874] Call Trace:
<4>[ 614.512061] [<ffffffff81866cb1>] cpuidle_enter_state+0x45/0xb5
<4>[ 614.512369] [<ffffffff81866e2c>] cpuidle_idle_call+0x10b/0x1db
<4>[ 614.512678] [<ffffffff8104241b>] arch_cpu_idle+0xe/0x28
<4>[ 614.512965] [<ffffffff8112452b>] cpu_startup_entry+0x131/0x20a
<4>[ 614.513273] [<ffffffff819aae53>] rest_init+0x87/0x89
<4>[ 614.513550] [<ffffffff8214fde0>] start_kernel+0x407/0x412
<4>[ 614.513842] [<ffffffff8214f7e7>] ? repair_env_string+0x58/0x58
<4>[ 614.514150] [<ffffffff8214f120>] ? early_idt_handlers+0x120/0x120
<4>[ 614.514466] [<ffffffff8214f4a2>] x86_64_start_reservations+0x2a/0x2c
<4>[ 614.514792] [<ffffffff8214f5df>] x86_64_start_kernel+0x13b/0x148
<4>[ 614.515104] Code: b9 00 00 48 89 d1 48 2d c8 1f 00 00 0f 01 c8 65 48 8b 04 25 60 b9 00 00 48 8b 80 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 04 25 60 b9 00 00 83 a0 3c e0 ff ff fb 0f ae f0 65 48
<4>[ 614.519105] NMI backtrace for cpu 1
Full dmesg & Kconifg are attached, and more details can be provided on your request.
Thanks,
Jet
7 years
[raid5] kernel BUG at drivers/md/raid5.c:4374!
by Fengguang Wu
Shaohua,
We noticed the below BUG on
commit e240c1839d11152b0355442f8ac6d2d2d921be36 ("raid5: get_active_stripe avoids device_lock")
test case: lkp-ws02/micro/dd-write/11HDD-RAID5-cfq-ext4-10dd
27c0f68f0745218 e240c1839d11152b0355442f8
--------------- -------------------------
0 +Inf% 1 ~ 0% TOTAL dmesg.kernel_BUG_at_drivers/md/raid5.c
0 +Inf% 1 ~ 0% TOTAL dmesg.invalid_opcode
0 +Inf% 1 ~ 0% TOTAL dmesg.RIP:handle_active_stripes
0 +Inf% 1 ~ 0% TOTAL dmesg.Kernel_panic-not_syncing:Fatal_exception
Legend:
~XX% - stddev percent
[+-]XX% - change percent
[ 264.260444] kernel BUG at drivers/md/raid5.c:4374!
[ 264.267590] invalid opcode: 0000 [#1] SMP
[ 264.272076] Modules linked in: btrfs microcode ipmi_si ipmi_msghandler acpi_cpufreq processor
[ 264.281514] CPU: 0 PID: 4005 Comm: md0_raid5 Not tainted 3.15.0-rc1-00611-g2e76799 #1
[ 264.289823] Hardware name: Supermicro X8DTN/X8DTN, BIOS 4.6.3 01/06/2010
[ 264.296789] task: ffff8804151e41a0 ti: ffff88041672c000 task.ti: ffff88041672c000
[ 264.304750] RIP: 0010:[<ffffffff81804ef1>] [<ffffffff81804ef1>] handle_active_stripes.isra.24+0x254/0x360
[ 264.314951] RSP: 0018:ffff88041672dd10 EFLAGS: 00010002
[ 264.320527] RAX: ffff88021dc46000 RBX: ffff880220a4e000 RCX: ffff880220a4e080
[ 264.327926] RDX: 0000000000000001 RSI: ffff88021e0d7010 RDI: ffff880220a4e000
[ 264.335325] RBP: ffff88041672dda8 R08: 0000000000000000 R09: 0000000000000000
[ 264.342724] R10: 0000000000000000 R11: 000000000000ef58 R12: 0000000000000000
[ 264.350131] R13: ffff880220a4e080 R14: 0000000000000000 R15: ffff880220a4e268
[ 264.357530] FS: 0000000000000000(0000) GS:ffff880237c00000(0000) knlGS:0000000000000000
[ 264.366099] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 264.372111] CR2: 0000000001e21d64 CR3: 000000000200f000 CR4: 00000000000007f0
[ 264.379509] Stack:
[ 264.381780] ffff8804173d7de0 ffff880220a4e000 ffff88041672dd38 ffffffffffffffd8
[ 264.389907] ffffffff1e1bd000 ffff880220a4e090 ffffffff817fdba2 ffff880220a4e000
[ 264.398043] 0000000000000021 ffff880220a4e268 0000000000000000 ffff88041672dd78
[ 264.406172] Call Trace:
[ 264.408893] [<ffffffff817fdba2>] ? do_release_stripe+0xdf/0x158
[ 264.415168] [<ffffffff817fdc30>] ? __release_stripe+0x15/0x17
[ 264.421266] [<ffffffff818054f2>] raid5d+0x3e2/0x4f2
[ 264.426497] [<ffffffff819bcacf>] ? schedule_timeout+0x2f/0x19f
[ 264.432681] [<ffffffff81826151>] md_thread+0x123/0x139
[ 264.438171] [<ffffffff811148b6>] ? __wake_up_sync+0x12/0x12
[ 264.444096] [<ffffffff8182602e>] ? md_register_thread+0xd5/0xd5
[ 264.450368] [<ffffffff810fa267>] kthread+0xdb/0xe3
[ 264.455511] [<ffffffff810fa18c>] ? kthread_create_on_node+0x16f/0x16f
[ 264.462310] [<ffffffff819c7d7c>] ret_from_fork+0x7c/0xb0
[ 264.467973] [<ffffffff810fa18c>] ? kthread_create_on_node+0x16f/0x16f
[ 264.474764] Code: 60 00 00 00 00 48 8b 70 10 48 8b 48 18 48 8d 50 10 48 89 4e 08 48 89 31 48 89 50 10 48 89 50 18 f0 ff 40 50 8b 50 50
ff ca 74 02 <0f> 0b 4a 89 44 d5 98 49 ff c2 49 83 fa 08 0f 85 d7 fd ff ff 41
[ 264.498252] RIP [<ffffffff81804ef1>] handle_active_stripes.isra.24+0x254/0x360
[ 264.506112] RSP <ffff88041672dd10>
[ 264.509869] ---[ end trace 58f3875ff7b4e923 ]---
Thanks,
Fengguang
7 years
[net/sctp] 362d52040c7: +99.0% netperf.Throughput_Mbps
by Fengguang Wu
Hi Daniel,
We noticed the same improvements in netperf SCTP_STREAM test case
as described in your patch changelog.
commit 362d52040c71f6e8d8158be48c812d7729cb8df1 ("Revert "net: sctp: Fix a_rwnd/rwnd management to reflect real state of the receiver's buffer"")
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
2.14 ~ 0% +98.9% 4.25 ~ 0% kbuildx/micro/netperf/300s-200%-SCTP_STREAM
2.14 ~ 0% +99.1% 4.25 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
2.14 ~ 0% +99.0% 4.25 ~ 0% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
6.41 ~ 0% +99.0% 12.76 ~ 0% TOTAL netperf.Throughput_Mbps
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
2.67 ~26% -100.0% 0.00 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
0.92 ~27% -100.0% 0.00 ~ 0% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
3.59 ~26% -100.0% 0.00 ~ 0% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.skb_copy_datagram_iovec.skb_copy_datagram_iovec.sctp_recvmsg.sock_common_recvmsg
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
8.75 ~32% -80.4% 1.72 ~21% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
8.75 ~32% -80.4% 1.72 ~21% TOTAL perf-profile.cpu-cycles.sctp_packet_transmit.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter.sctp_do_sm
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.00 +Inf% 1.22 ~22% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
0.00 +Inf% 1.43 ~22% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
0.00 +Inf% 2.65 ~22% TOTAL perf-profile.cpu-cycles.sctp_packet_transmit.sctp_packet_transmit_chunk.sctp_outq_flush.sctp_outq_uncork.sctp_cmd_interpreter
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
6.26 ~13% +306.9% 25.46 ~26% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
6.26 ~13% +306.9% 25.46 ~26% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.clockevents_notify.intel_idle.cpuidle_enter_state.cpuidle_enter
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
966 ~45% -44.6% 535 ~29% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
966 ~45% -44.6% 535 ~29% TOTAL cpuidle.C1-NHM.time
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.45 ~33% +165.0% 1.20 ~14% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
0.48 ~37% +235.3% 1.62 ~12% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
0.93 ~35% +201.3% 2.81 ~13% TOTAL perf-profile.cpu-cycles.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.80 ~32% -56.1% 0.35 ~38% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
0.80 ~32% -56.1% 0.35 ~38% TOTAL perf-profile.cpu-cycles.menu_select.cpuidle_select.cpu_startup_entry.start_secondary
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
5.08 ~26% -58.3% 2.12 ~21% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1.34 ~27% +112.1% 2.85 ~ 9% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
6.43 ~26% -22.7% 4.97 ~14% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.skb_copy_datagram_iovec.sctp_recvmsg.sock_common_recvmsg.sock_recvmsg
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.72 ~41% +69.4% 1.22 ~25% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
0.72 ~41% +69.4% 1.22 ~25% TOTAL perf-profile.cpu-cycles.unmap_single_vma.unmap_vmas.exit_mmap.mmput.do_exit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
916917 ~ 0% +79.9% 1649300 ~ 0% kbuildx/micro/netperf/300s-200%-SCTP_STREAM
12832444 ~ 1% +115.7% 27683375 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
13749362 ~ 0% +113.3% 29332676 ~ 0% TOTAL proc-vmstat.pgalloc_normal
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
1175423 ~ 0% +79.7% 2111948 ~ 0% kbuildx/micro/netperf/300s-200%-SCTP_STREAM
12997556 ~ 0% +115.4% 27996526 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1181153 ~ 0% +79.3% 2118015 ~ 0% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
15354133 ~ 0% +109.9% 32226491 ~ 0% TOTAL proc-vmstat.pgfree
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
257597 ~ 0% +79.3% 461743 ~ 0% kbuildx/micro/netperf/300s-200%-SCTP_STREAM
1179621 ~ 0% +79.4% 2116528 ~ 0% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
1437218 ~ 0% +79.4% 2578271 ~ 0% TOTAL proc-vmstat.pgalloc_dma32
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
138767 ~ 3% +102.3% 280793 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
12422 ~ 1% +46.0% 18142 ~ 2% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
151189 ~ 3% +97.7% 298935 ~ 0% TOTAL softirqs.NET_RX
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
2.37 ~18% +78.7% 4.23 ~14% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
2.37 ~18% +78.7% 4.23 ~14% TOTAL perf-profile.cpu-cycles._raw_spin_lock_irqsave.tick_broadcast_oneshot_control.clockevents_notify.intel_idle.cpuidle_enter_state
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
386789 ~27% +43.0% 553161 ~ 6% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
386789 ~27% +43.0% 553161 ~ 6% TOTAL numa-numastat.node0.local_node
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
388443 ~28% +42.8% 554814 ~ 6% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
388443 ~28% +42.8% 554814 ~ 6% TOTAL numa-numastat.node0.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
14.18 ~24% -32.9% 9.52 ~16% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
3.43 ~26% +54.6% 5.30 ~20% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
17.60 ~24% -15.8% 14.82 ~17% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg.inet_sendmsg
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
393446 ~24% +40.0% 550772 ~ 3% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
393446 ~24% +40.0% 550772 ~ 3% TOTAL numa-numastat.node2.local_node
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
397581 ~23% +39.6% 554907 ~ 3% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
397581 ~23% +39.6% 554907 ~ 3% TOTAL numa-numastat.node2.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.79 ~ 4% +42.7% 1.13 ~ 3% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
0.79 ~ 4% +42.7% 1.13 ~ 3% TOTAL turbostat.%c1
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
357508 ~ 9% +42.9% 510964 ~ 8% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
357508 ~ 9% +42.9% 510964 ~ 8% TOTAL numa-numastat.node1.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
355027 ~ 9% +43.0% 507656 ~ 7% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
355027 ~ 9% +43.0% 507656 ~ 7% TOTAL numa-numastat.node1.local_node
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
298703 ~12% +27.8% 381632 ~ 4% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
298703 ~12% +27.8% 381632 ~ 4% TOTAL numa-vmstat.node2.numa_local
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
1513199 ~ 0% +39.1% 2105308 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1513199 ~ 0% +39.1% 2105308 ~ 0% TOTAL proc-vmstat.numa_local
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
1525603 ~ 0% +38.8% 2117713 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1525603 ~ 0% +38.8% 2117713 ~ 0% TOTAL proc-vmstat.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
320874 ~14% +24.8% 400369 ~ 4% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
320874 ~14% +24.8% 400369 ~ 4% TOTAL numa-vmstat.node0.numa_local
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
323017 ~ 1% +32.8% 428933 ~ 1% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
323017 ~ 1% +32.8% 428933 ~ 1% TOTAL cpuidle.C3-NHM.usage
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
386570 ~ 9% +21.4% 469471 ~ 3% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
386570 ~ 9% +21.4% 469471 ~ 3% TOTAL numa-vmstat.node2.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
380244 ~12% +20.9% 459829 ~ 3% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
380244 ~12% +20.9% 459829 ~ 3% TOTAL numa-vmstat.node0.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
352316 ~ 4% +19.9% 422485 ~ 5% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
352316 ~ 4% +19.9% 422485 ~ 5% TOTAL numa-vmstat.node1.numa_local
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
380978 ~ 4% +18.6% 451706 ~ 5% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
380978 ~ 4% +18.6% 451706 ~ 5% TOTAL numa-vmstat.node1.numa_hit
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
119700 ~ 2% +18.6% 141993 ~ 1% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
119700 ~ 2% +18.6% 141993 ~ 1% TOTAL softirqs.SCHED
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
45876 ~ 4% -7.9% 42242 ~ 7% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
45876 ~ 4% -7.9% 42242 ~ 7% TOTAL numa-meminfo.node0.AnonPages
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
11467 ~ 4% -7.9% 10559 ~ 7% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
11467 ~ 4% -7.9% 10559 ~ 7% TOTAL numa-vmstat.node0.nr_anon_pages
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
184229 ~ 1% +14.4% 210833 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
184229 ~ 1% +14.4% 210833 ~ 0% TOTAL softirqs.TIMER
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
347 ~ 5% +140.0% 834 ~10% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
347 ~ 5% +140.0% 834 ~10% TOTAL interrupts.98:PCI-MSI-edge.eth0-TxRx-0
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
9125 ~ 4% +59.8% 14585 ~13% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
9125 ~ 4% +59.8% 14585 ~13% TOTAL interrupts.RES
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
338 ~ 6% +15.8% 392 ~ 4% kbuildx/micro/netperf/300s-200%-SCTP_STREAM
1830 ~ 0% +71.2% 3135 ~ 0% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
352 ~ 0% +20.5% 425 ~ 1% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
2522 ~ 1% +56.7% 3952 ~ 1% TOTAL vmstat.system.cs
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
171971 ~ 3% +29.8% 223248 ~ 1% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
171971 ~ 3% +29.8% 223248 ~ 1% TOTAL interrupts.LOC
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
839 ~ 3% +24.4% 1044 ~ 1% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
839 ~ 3% +24.4% 1044 ~ 1% TOTAL vmstat.system.in
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
8197 ~11% +32.2% 10835 ~ 7% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1043 ~ 0% +15.2% 1202 ~ 3% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
9241 ~ 9% +30.3% 12037 ~ 7% TOTAL interrupts.PMI
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
8197 ~11% +32.2% 10835 ~ 7% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
1043 ~ 0% +15.2% 1202 ~ 3% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
9241 ~ 9% +30.3% 12037 ~ 7% TOTAL interrupts.NMI
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
0.73 ~ 4% +20.5% 0.88 ~ 6% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
0.94 ~ 0% +14.4% 1.08 ~ 1% lkp-t410/micro/netperf/300s-200%-SCTP_STREAM
1.67 ~ 2% +17.1% 1.96 ~ 3% TOTAL turbostat.%c0
bfae23249955819 362d52040c71f6e8d8158be48
--------------- -------------------------
17709 ~ 2% -4.1% 16985 ~ 1% lkp-nex04/micro/netperf/300s-200%-SCTP_STREAM
17709 ~ 2% -4.1% 16985 ~ 1% TOTAL interrupts.IWI
Legend:
~XX% - stddev percent
[+-]XX% - change percent
netperf.Throughput_Mbps
4.5 ++-------------------------------------------------------------------+
O O O O O O O O O O O O O O O O O O O O O O O O O O |
| |
4 ++ |
| |
| |
3.5 ++ |
| |
3 ++ |
| |
| |
2.5 ++ |
| |
*..*.*..*.*..*.*..*.*..*..*.*..*.*..*.*..*.*..*..*.*..*.*..*.*..*.*..*
2 ++-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Thanks,
Fengguang
7 years
[sched] 177672afd93: -40.9% dbench.throughput-MB/sec
by Fengguang Wu
Hi Tim,
Up to now, the system catches one regression in
git://bee.sh.intel.com/git/wfg/linux-devel.git tim-idle-load-balance
commit 177672afd93461854abe26c05bd549647efc09db ("sched: Reduce the rate of idle load balancing")
test case: nhm8/micro/dbench/100%
v3.14 177672afd93461854abe26c05
--------------- -------------------------
8214 ~ 0% -40.9% 4854 ~ 0% TOTAL dbench.throughput-MB/sec
443685 ~ 0% +156.5% 1138160 ~20% TOTAL softirqs.RCU
3.38 ~ 0% -52.1% 1.62 ~ 0% TOTAL turbostat.GHz
81260 ~ 1% -42.7% 46534 ~ 1% TOTAL slabinfo.kmalloc-256.active_objs
280 ~23% +85.4% 520 ~12% TOTAL cpuidle.C3-NHM.usage
2556 ~ 1% -42.6% 1467 ~ 1% TOTAL slabinfo.kmalloc-256.num_slabs
2556 ~ 1% -42.6% 1467 ~ 1% TOTAL slabinfo.kmalloc-256.active_slabs
81822 ~ 1% -42.6% 46982 ~ 1% TOTAL slabinfo.kmalloc-256.num_objs
8.34 ~ 0% -42.2% 4.82 ~ 1% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.generic_file_buffered_write.__generic_file_aio_write.generic_file_aio_write.do_sync_write
17.87 ~ 0% -42.6% 10.26 ~ 0% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.generic_file_aio_read.do_sync_read.vfs_read.sys_pread64
7029777 ~ 0% -40.8% 4164671 ~ 0% TOTAL proc-vmstat.pgactivate
0.78 ~ 1% +41.2% 1.10 ~ 1% TOTAL perf-profile.cpu-cycles.filldir.dcache_readdir.iterate_dir.sys_getdents.system_call_fastpath
0.75 ~ 2% +42.9% 1.07 ~ 1% TOTAL perf-profile.cpu-cycles.strchr@plt
91820 ~ 0% -25.1% 68743 ~ 0% TOTAL slabinfo.dentry.active_objs
2234 ~ 0% -25.0% 1675 ~ 0% TOTAL slabinfo.dentry.num_slabs
2234 ~ 0% -25.0% 1675 ~ 0% TOTAL slabinfo.dentry.active_slabs
93857 ~ 0% -25.0% 70409 ~ 0% TOTAL slabinfo.dentry.num_objs
1.48 ~ 1% +34.2% 1.98 ~ 0% TOTAL perf-profile.cpu-cycles.copy_user_generic_string.dcache_readdir.iterate_dir.sys_getdents.system_call_fastpath
0.82 ~ 5% +31.9% 1.08 ~ 4% TOTAL perf-profile.cpu-cycles.__d_lookup_rcu.lookup_fast.link_path_walk.path_lookupat.filename_lookup
9908 ~ 1% -21.3% 7798 ~ 0% TOTAL proc-vmstat.nr_slab_unreclaimable
39507 ~ 0% -21.4% 31047 ~ 0% TOTAL meminfo.SUnreclaim
61 ~ 1% -20.1% 49 ~ 2% TOTAL turbostat.CTMP
0.85 ~ 2% +22.9% 1.04 ~ 1% TOTAL perf-profile.cpu-cycles.close
1.31 ~ 1% +22.9% 1.61 ~ 1% TOTAL perf-profile.cpu-cycles.link_path_walk.path_lookupat.filename_lookup.user_path_at_empty.user_path_at
92159 ~ 0% -16.6% 76833 ~ 0% TOTAL meminfo.Slab
14061 ~ 4% +20.9% 16994 ~ 1% TOTAL interrupts.RES
1.45 ~ 1% +16.9% 1.69 ~ 0% TOTAL perf-profile.cpu-cycles._raw_spin_lock.dcache_readdir.iterate_dir.sys_getdents.system_call_fastpath
13180 ~ 0% -13.1% 11451 ~ 0% TOTAL proc-vmstat.nr_slab_reclaimable
52651 ~ 0% -13.0% 45786 ~ 0% TOTAL meminfo.SReclaimable
3096 ~ 0% +15.6% 3581 ~ 0% TOTAL time.user_time
5514 ~ 0% -8.9% 5023 ~ 0% TOTAL time.system_time
225997 ~ 0% +3.1% 233003 ~ 1% TOTAL time.involuntary_context_switches
Legend:
~XX% - stddev percent
[+-]XX% - change percent
time.user_time
3600 ++-------------O-------O------O-------------------------------------+
O O |
3500 ++ |
| |
| |
3400 ++ |
| |
3300 ++ |
| |
3200 ++ |
| |
*....... ....*...... ....*
3100 ++ *......*.......*......*... *.......*......*... |
| |
3000 ++------------------------------------------------------------------+
time.system_time
5600 ++------------------------------------------------------------------+
| |
5500 ++ ....*......*.......*......*.......*......*.......*......*.......*
*... |
| |
5400 ++ |
| |
5300 ++ |
| |
5200 ++ |
| |
| |
5100 ++ |
O O O |
5000 ++-------------O-------O--------------------------------------------+
dbench.throughput-MB/sec
8500 ++------------------------------------------------------------------+
*.......*......*.......*......*.......*......*.......*......*.......*
8000 ++ |
7500 ++ |
| |
7000 ++ |
| |
6500 ++ |
| |
6000 ++ |
5500 ++ |
| |
5000 ++ |
O O O O O |
4500 ++------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad sample
Thanks,
Fengguang
7 years