FYI, we noticed a -7.1% improvement of fio.latency_20us% due to commit: commit 62f37bbb539729c6b3182ba5345dccfc582ee6f2 ("remove mapping from balance_dirty_pages*()") https://github.com/kdave/btrfs-devel.git foreign/josef/no-btree-inode in testcase: fio-basic on test machine: 16 threads Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz with 8G memory with following parameters: runtime: 300s nr_task: 8 disk: 1SSD fs: btrfs rw: write bs: 4k ioengine: sync test_size: 400g cpufreq_governor: performance Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase: 4k/gcc-6/performance/1SSD/btrfs/sync/x86_64-rhel-7.2/8/debian-x86_64-2016-08-31.cgz/300s/write/lkp-bdw-de1/400g/fio-basic commit: v4.8-rc5 62f37bbb53 ("remove mapping from balance_dirty_pages*()") v4.8-rc5 62f37bbb539729c6b3182ba534 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | %stddev %change %stddev \ | \ 0.64 ± 0% -7.1% 0.59 ± 1% fio.latency_20us% 1584 ± 0% +2.5% 1624 ± 0% fio.write_clat_stddev 24776 ± 64% +114.1% 53052 ± 0% interrupts.CAL:Function_call_interrupts 150248 ± 14% +31.0% 196781 ± 4% cpuidle.C1-BDW.usage 1.138e+09 ± 12% -18.5% 9.274e+08 ± 8% cpuidle.C3-BDW.time 1365995 ± 10% -16.5% 1140547 ± 7% cpuidle.C3-BDW.usage 29.69 ± 0% +380.1% 142.52 ± 0% iostat.sda.avgqu-sz 66.80 ± 0% +377.3% 318.81 ± 0% iostat.sda.await 66.89 ± 0% +376.8% 318.94 ± 0% iostat.sda.w_await 3.16 ± 7% +52.5% 4.82 ± 2% iostat.sda.wrqm/s 8.036e+09 ± 1% +1.9% 8.188e+09 ± 0% perf-stat.cache-misses 8.036e+09 ± 1% +1.9% 8.188e+09 ± 0% perf-stat.cache-references 0.08 ± 1% -3.5% 0.08 ± 2% perf-stat.dTLB-load-miss-rate% 1.189e+11 ± 0% +2.1% 1.214e+11 ± 1% perf-stat.dTLB-stores 47.23 ± 4% +41.9% 67.00 ± 1% perf-stat.iTLB-load-miss-rate% 46522400 ± 2% +159.4% 1.207e+08 ± 2% perf-stat.iTLB-load-misses 14723 ± 2% -61.3% 5692 ± 2% perf-stat.instructions-per-iTLB-miss 36245 ± 0% +1605.2% 618076 ± 33% latency_stats.avg.max 0.00 ± -1% +Inf% 618076 ± 33% latency_stats.avg.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 55510 ± 0% +232.1% 184333 ± 9% latency_stats.max.balance_dirty_pages.balance_dirty_pages_ratelimited.__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 55510 ± 0% +1042.5% 634181 ± 33% latency_stats.max.max 0.00 ± -1% +Inf% 634181 ± 33% latency_stats.max.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 1070112 ± 57% -100.0% 0.00 ± -1% latency_stats.sum.msleep_interruptible.uart_wait_until_sent.tty_wait_until_sent.tty_port_close_start.tty_port_close_start.uart_close.tty_release.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath 3406813 ± 57% -100.0% 0.00 ± -1% latency_stats.sum.tty_wait_until_sent.tty_port_close_start.tty_port_close_start.uart_close.tty_release.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath 0.00 ± -1% +Inf% 4944613 ± 33% latency_stats.sum.wait_current_trans.[btrfs].start_transaction.[btrfs].btrfs_join_transaction.[btrfs].btrfs_alloc_data_chunk_ondemand.[btrfs].btrfs_check_data_free_space.[btrfs].__btrfs_buffered_write.[btrfs].btrfs_file_write_iter.[btrfs].__vfs_write.vfs_write.SyS_write.entry_SYSCALL_64_fastpath 48883 ± 51% +91.6% 93647 ± 0% meminfo.Active(anon) 23808 ± 98% +179.3% 66508 ± 1% meminfo.AnonHugePages 46431 ± 54% +96.4% 91173 ± 0% meminfo.AnonPages 1059 ± 0% -30.7% 733.40 ± 49% meminfo.Buffers 289278 ±100% +174.2% 793156 ± 0% meminfo.Committed_AS 108335 ±172% +325.4% 460910 ± 3% meminfo.Dirty 2023600 ±140% +244.0% 6961094 ± 0% meminfo.Inactive 45858 ±105% +183.1% 129843 ± 0% meminfo.Inactive(anon) 1977741 ±141% +245.4% 6831250 ± 0% meminfo.Inactive(file) 7500 ± 13% -22.5% 5814 ± 0% meminfo.KernelStack 51079 ± 66% +117.5% 111089 ± 0% meminfo.SReclaimable 37424 ± 13% +28.9% 48226 ± 0% meminfo.SUnreclaim 88503 ± 43% +80.0% 159316 ± 0% meminfo.Slab 12429 ± 50% +88.4% 23415 ± 0% proc-vmstat.nr_active_anon 11844 ± 52% +92.5% 22794 ± 0% proc-vmstat.nr_anon_pages 8569526 ±173% +300.5% 34317951 ± 0% proc-vmstat.nr_dirtied 27087 ±172% +325.2% 115182 ± 3% proc-vmstat.nr_dirty 11464 ±105% +183.1% 32459 ± 0% proc-vmstat.nr_inactive_anon 494358 ±141% +245.3% 1707266 ± 0% proc-vmstat.nr_inactive_file 7650 ± 13% -23.3% 5866 ± 0% proc-vmstat.nr_kernel_stack 12793 ± 65% +117.1% 27772 ± 0% proc-vmstat.nr_slab_reclaimable 9378 ± 13% +28.5% 12056 ± 0% proc-vmstat.nr_slab_unreclaimable 98.25 ±173% +582.3% 670.40 ± 4% proc-vmstat.nr_vmscan_immediate_reclaim 8488016 ±173% +300.5% 33992756 ± 0% proc-vmstat.nr_written 12429 ± 50% +88.4% 23415 ± 0% proc-vmstat.nr_zone_active_anon 11464 ±105% +183.1% 32459 ± 0% proc-vmstat.nr_zone_inactive_anon 494355 ±141% +245.3% 1707256 ± 0% proc-vmstat.nr_zone_inactive_file 8853611 ±168% +291.5% 34664544 ± 0% proc-vmstat.numa_hit 8853608 ±168% +291.5% 34664544 ± 0% proc-vmstat.numa_local 6742739 ±165% +287.5% 26131244 ± 0% proc-vmstat.pgalloc_normal 9938521 ±133% +232.2% 33013181 ± 0% proc-vmstat.pgfree 33960263 ±173% +300.8% 1.361e+08 ± 0% proc-vmstat.pgpgout 8138579 ±173% +300.6% 32604499 ± 0% proc-vmstat.pgscan_kswapd 8116173 ±173% +300.6% 32517422 ± 0% proc-vmstat.pgsteal_kswapd 102208 ±173% +303.2% 412083 ± 0% proc-vmstat.slabs_scanned 34553 ±173% +300.7% 138453 ± 0% proc-vmstat.workingset_nodereclaim 1401 ±173% +300.3% 5611 ± 0% sched_debug.cfs_rq:/.exec_clock.avg 63.00 ±173% +554.0% 412.03 ± 21% sched_debug.cfs_rq:/.exec_clock.min 36695 ± 89% +155.3% 93674 ± 15% sched_debug.cfs_rq:/.load.avg 175323 ±135% +360.9% 807984 ± 15% sched_debug.cfs_rq:/.load.max 56018 ±135% +302.8% 225649 ± 12% sched_debug.cfs_rq:/.load.stddev 1284 ± 41% -77.3% 291.42 ± 5% sched_debug.cfs_rq:/.load_avg.avg 1684 ± 29% -65.0% 589.23 ± 29% sched_debug.cfs_rq:/.load_avg.max 1176 ± 45% -79.7% 239.03 ± 4% sched_debug.cfs_rq:/.load_avg.min 0.62 ± 39% -63.0% 0.23 ± 5% sched_debug.cfs_rq:/.nr_running.avg 659.98 ± 38% -73.9% 172.16 ± 6% sched_debug.cfs_rq:/.util_avg.avg 818.29 ± 10% -53.1% 383.57 ± 30% sched_debug.cfs_rq:/.util_avg.max 530.92 ± 45% -80.0% 106.40 ± 5% sched_debug.cfs_rq:/.util_avg.min 506937 ± 33% +65.4% 838646 ± 1% sched_debug.cpu.avg_idle.avg 188707 ± 79% +154.6% 480359 ± 8% sched_debug.cpu.avg_idle.min 213657 ± 16% -32.4% 144386 ± 6% sched_debug.cpu.avg_idle.stddev 26.41 ±165% +528.9% 166.10 ± 25% sched_debug.cpu.clock.stddev 26.41 ±165% +528.9% 166.10 ± 25% sched_debug.cpu.clock_task.stddev 4.17 ± 54% -83.2% 0.70 ± 34% sched_debug.cpu.cpu_load[3].min 512.26 ± 33% -41.3% 300.60 ± 4% sched_debug.cpu.curr->pid.avg 175366 ±135% +346.1% 782313 ± 12% sched_debug.cpu.load.max 56184 ±135% +294.5% 221659 ± 11% sched_debug.cpu.load.stddev 0.00 ± 69% +246.9% 0.00 ± 19% sched_debug.cpu.next_balance.stddev 1529 ± 58% +116.2% 3306 ± 17% sched_debug.cpu.nr_load_updates.min 11523 ±173% +521.2% 71584 ± 4% sched_debug.cpu.sched_count.avg 32424 ±173% +527.3% 203408 ± 8% sched_debug.cpu.sched_count.max 9878 ±173% +552.0% 64407 ± 4% sched_debug.cpu.sched_count.stddev 78.71 ±173% +601.4% 552.03 ± 52% sched_debug.cpu.ttwu_local.min 1168 ± 0% -100.0% 0.00 ± -1% slabinfo.Acpi-Parse.active_objs 1168 ± 0% -100.0% 0.00 ± -1% slabinfo.Acpi-Parse.num_objs 34437 ± 2% +11.9% 38537 ± 1% slabinfo.Acpi-State.num_objs 379.00 ±127% +234.2% 1266 ± 2% slabinfo.bio-1.active_objs 392.25 ±129% +238.4% 1327 ± 2% slabinfo.bio-1.num_objs 800.50 ± 97% -100.0% 0.00 ± -1% slabinfo.blkdev_ioc.active_objs 803.50 ± 97% -100.0% 0.00 ± -1% slabinfo.blkdev_ioc.num_objs 1225 ±156% +274.6% 4589 ± 0% slabinfo.btrfs_extent_buffer.active_objs 43.00 ±157% +276.7% 162.00 ± 1% slabinfo.btrfs_extent_buffer.active_slabs 1225 ±156% +274.6% 4589 ± 0% slabinfo.btrfs_extent_buffer.num_objs 43.00 ±157% +276.7% 162.00 ± 1% slabinfo.btrfs_extent_buffer.num_slabs 2668 ± 36% -100.0% 0.00 ± -1% slabinfo.btrfs_extent_map.active_objs 2672 ± 37% -100.0% 0.00 ± -1% slabinfo.btrfs_extent_map.num_objs 442.75 ±132% +245.9% 1531 ± 15% slabinfo.btrfs_ordered_extent.active_objs 442.75 ±132% +245.9% 1531 ± 15% slabinfo.btrfs_ordered_extent.num_objs 118.25 ± 31% +91.0% 225.80 ± 8% slabinfo.btrfs_transaction.active_objs 118.25 ± 31% +91.0% 225.80 ± 8% slabinfo.btrfs_transaction.num_objs 9333 ± 4% -56.2% 4089 ± 12% slabinfo.cred_jar.active_objs 223.75 ± 5% -56.9% 96.40 ± 12% slabinfo.cred_jar.active_slabs 9398 ± 5% -56.5% 4089 ± 12% slabinfo.cred_jar.num_objs 223.75 ± 5% -56.9% 96.40 ± 12% slabinfo.cred_jar.num_slabs 881.50 ± 2% +12.9% 995.00 ± 1% slabinfo.dentry.active_slabs 37030 ± 2% +12.9% 41809 ± 1% slabinfo.dentry.num_objs 881.50 ± 2% +12.9% 995.00 ± 1% slabinfo.dentry.num_slabs 2118 ± 7% +57.8% 3343 ± 8% slabinfo.file_lock_cache.active_objs 2122 ± 7% +58.0% 3354 ± 8% slabinfo.file_lock_cache.num_objs 3463 ± 1% +22.7% 4250 ± 1% slabinfo.ftrace_event_field.active_objs 3463 ± 1% +22.7% 4250 ± 1% slabinfo.ftrace_event_field.num_objs 29613 ± 0% +14.8% 33999 ± 0% slabinfo.kernfs_node_cache.active_objs 870.75 ± 0% +14.7% 999.00 ± 0% slabinfo.kernfs_node_cache.active_slabs 29613 ± 0% +14.8% 33999 ± 0% slabinfo.kernfs_node_cache.num_objs 870.75 ± 0% +14.7% 999.00 ± 0% slabinfo.kernfs_node_cache.num_slabs 527.00 ± 74% +133.3% 1229 ± 1% slabinfo.kmalloc-4096.active_objs 67.25 ± 76% +139.1% 160.80 ± 0% slabinfo.kmalloc-4096.active_slabs 539.50 ± 76% +139.6% 1292 ± 0% slabinfo.kmalloc-4096.num_objs 67.25 ± 76% +139.1% 160.80 ± 0% slabinfo.kmalloc-4096.num_slabs 2708 ± 14% +21.8% 3299 ± 4% slabinfo.kmalloc-512.active_objs 514.50 ± 6% +41.0% 725.40 ± 7% slabinfo.mnt_cache.active_objs 514.50 ± 6% +41.0% 725.40 ± 7% slabinfo.mnt_cache.num_objs 1748 ± 1% +31.6% 2300 ± 2% slabinfo.trace_event_file.active_objs 1748 ± 1% +31.6% 2300 ± 2% slabinfo.trace_event_file.num_objs 1.90 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.__ata_qc_complete.ata_qc_complete.ata_qc_complete_multiple.ahci_handle_port_interrupt.ahci_handle_port_intr 1.87 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.__blk_mq_complete_request.blk_mq_complete_request.scsi_mq_done.ata_scsi_qc_complete.__ata_qc_complete 1.95 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.handle_irq 0.00 ± -1% +Inf% 2.28 ± 14% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter 1.92 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ahci_handle_port_interrupt.ahci_handle_port_intr.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event_percpu 1.93 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ahci_handle_port_intr.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event 1.94 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ahci_single_level_irq_intr.__handle_irq_event_percpu.handle_irq_event_percpu.handle_irq_event.handle_edge_irq 1.90 ± 2% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ata_qc_complete.ata_qc_complete_multiple.ahci_handle_port_interrupt.ahci_handle_port_intr.ahci_single_level_irq_intr 1.90 ± 2% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ata_qc_complete_multiple.ahci_handle_port_interrupt.ahci_handle_port_intr.ahci_single_level_irq_intr.__handle_irq_event_percpu 1.88 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.ata_scsi_qc_complete.__ata_qc_complete.ata_qc_complete.ata_qc_complete_multiple.ahci_handle_port_interrupt 0.00 ± -1% +Inf% 2.27 ± 14% perf-profile.calltrace.cycles-pp.blk_done_softirq.__softirqentry_text_start.irq_exit.do_IRQ.ret_from_intr 1.87 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.blk_mq_complete_request.scsi_mq_done.ata_scsi_qc_complete.__ata_qc_complete.ata_qc_complete 1.97 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.handle_edge_irq.handle_irq.do_IRQ.ret_from_intr.cpuidle_enter 1.97 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.handle_irq.do_IRQ.ret_from_intr.cpuidle_enter.call_cpuidle 1.97 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.handle_irq_event.handle_edge_irq.handle_irq.do_IRQ.ret_from_intr 1.97 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.handle_irq_event_percpu.handle_irq_event.handle_edge_irq.handle_irq.do_IRQ 0.00 ± -1% +Inf% 2.29 ± 14% perf-profile.calltrace.cycles-pp.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter.call_cpuidle 1.85 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.scsi_finish_command.scsi_softirq_done.__blk_mq_complete_request 0.00 ± -1% +Inf% 2.26 ± 14% perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.scsi_finish_command.scsi_softirq_done.blk_done_softirq 1.86 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.scsi_finish_command.scsi_softirq_done.__blk_mq_complete_request.blk_mq_complete_request.scsi_mq_done 0.00 ± -1% +Inf% 2.27 ± 14% perf-profile.calltrace.cycles-pp.scsi_finish_command.scsi_softirq_done.blk_done_softirq.__softirqentry_text_start.irq_exit 1.85 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.scsi_io_completion.scsi_finish_command.scsi_softirq_done.__blk_mq_complete_request.blk_mq_complete_request 0.00 ± -1% +Inf% 2.26 ± 14% perf-profile.calltrace.cycles-pp.scsi_io_completion.scsi_finish_command.scsi_softirq_done.blk_done_softirq.__softirqentry_text_start 1.87 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.scsi_mq_done.ata_scsi_qc_complete.__ata_qc_complete.ata_qc_complete.ata_qc_complete_multiple 1.87 ± 1% -100.0% 0.00 ± -1% perf-profile.calltrace.cycles-pp.scsi_softirq_done.__blk_mq_complete_request.blk_mq_complete_request.scsi_mq_done.ata_scsi_qc_complete 0.00 ± -1% +Inf% 2.27 ± 14% perf-profile.calltrace.cycles-pp.scsi_softirq_done.blk_done_softirq.__softirqentry_text_start.irq_exit.do_IRQ 2.13 ± 1% -99.3% 0.01 ±200% perf-profile.children.cycles-pp.__ata_qc_complete 2.10 ± 1% -100.0% 0.00 ± -1% perf-profile.children.cycles-pp.__blk_mq_complete_request 2.20 ± 1% -95.3% 0.10 ± 63% perf-profile.children.cycles-pp.__handle_irq_event_percpu 0.33 ± 50% +798.9% 2.94 ± 14% perf-profile.children.cycles-pp.__softirqentry_text_start 2.15 ± 1% -97.3% 0.06 ± 58% perf-profile.children.cycles-pp.ahci_handle_port_interrupt 2.16 ± 1% -96.3% 0.08 ± 61% perf-profile.children.cycles-pp.ahci_handle_port_intr 2.19 ± 1% -95.2% 0.10 ± 63% perf-profile.children.cycles-pp.ahci_single_level_irq_intr 2.13 ± 2% -99.3% 0.01 ±200% perf-profile.children.cycles-pp.ata_qc_complete 2.13 ± 2% -99.3% 0.01 ±200% perf-profile.children.cycles-pp.ata_qc_complete_multiple 2.12 ± 2% -99.5% 0.01 ±200% perf-profile.children.cycles-pp.ata_scsi_qc_complete 0.00 ± -1% +Inf% 2.62 ± 15% perf-profile.children.cycles-pp.blk_done_softirq 2.10 ± 1% -100.0% 0.00 ± -1% perf-profile.children.cycles-pp.blk_mq_complete_request 2.22 ± 1% -94.6% 0.12 ± 65% perf-profile.children.cycles-pp.handle_edge_irq 2.22 ± 1% -94.4% 0.12 ± 62% perf-profile.children.cycles-pp.handle_irq 2.21 ± 1% -94.9% 0.11 ± 61% perf-profile.children.cycles-pp.handle_irq_event 2.21 ± 1% -95.0% 0.11 ± 63% perf-profile.children.cycles-pp.handle_irq_event_percpu 1.38 ± 19% +177.4% 3.83 ± 12% perf-profile.children.cycles-pp.irq_exit 2.10 ± 1% -100.0% 0.00 ± -1% perf-profile.children.cycles-pp.scsi_mq_done perf-stat.iTLB-load-miss-rate_ 90 O+---------------------------------------------------------------------+ | O O O O | 80 ++ | 70 ++ O | | O O O 60 ++ | 50 ++ *........*........ | *. . * | 40 ++.. . | 30 ++ . . | | . .. | 20 ++ .. . | 10 ++ . . | | . | 0 ++-------*-------------------------------------------------------------+ perf-stat.instructions-per-iTLB-miss 16000 ++------------------------*-----------------------------------------+ * *.... * | 14000 ++ : | 12000 ++ : | | : : | 10000 ++ : : | | : : | 8000 ++ : : | | : : | 6000 ++ : : O O O O 4000 ++ : : | | : : | 2000 ++ :O: O O O | O : : | 0 ++-------*----------------------------------------------------------+ latency_stats.avg.max 800000 ++-----------------------------------------------------------------+ | O O 700000 O+ O | | | 600000 ++ | 500000 ++ | | O O O | 400000 ++ O | | | 300000 ++ | 200000 ++ | | O | 100000 ++ | *....... ....*.......*........* | 0 ++------*----------------------------------------------------------+ latency_stats.max.max 800000 ++-----------------------------------------------------------------+ O O O O 700000 ++ | | | 600000 ++ | 500000 ++ O O | | O O | 400000 ++ | | | 300000 ++ | 200000 ++ O | | | 100000 ++ | *....... ....*.......*........* | 0 ++------*----------------------------------------------------------+ iostat.sda.wrqm_s 5 ++---------------------------------O-----------------O-------O--------+ 4.5 ++ O O O O O O | 4 ++ | 3.5 ++ | | *.......*........* | 3 *+ + | 2.5 ++. + | 2 ++ . + | | .. + | 1.5 ++ . + | 1 ++ . + | | . + | 0.5 ++ + | 0 ++-------*------------------------------------------------------------+ iostat.sda.avgqu-sz 160 ++--------------------------------------------------------------------+ | O | 140 O+ O O O O O O O 120 ++ | | | 100 ++ | | | 80 ++ | | | 60 ++ | 40 ++ | *.... .*.......*........* | 20 ++ ... ... | | . ... | 0 ++-------*------------------------------------------------------------+ iostat.sda.await 350 ++--------------------------------------------------------------------+ O O O O O O O O O 300 ++ | | | 250 ++ | | | 200 ++ | | | 150 ++ | | | 100 ++ | *.... .*.......*........* | 50 ++ ... ... | | . ... | 0 ++-------*------------------------------------------------------------+ iostat.sda.w_await 350 ++--------------------------------------------------------------------+ O O O O O O O O O 300 ++ | | | 250 ++ | | | 200 ++ | | | 150 ++ | | | 100 ++ | *.... .*.......*........* | 50 ++ ... ... | | . ... | 0 ++-------*------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Thanks, Xiaolong