December 2017 - SPDK - Ml01.01.Org

Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

by Sreeni (Sreenivasa) Busam (Stellus)

Hello, I have configured the target and initiator for a subsystem with 1 NVMe device in target. Here are the errors I am getting on the initiator. I have a good NVMe device on the target side, but I am getting the error below. If you know why the initiator does not initialize the controller and reason for the error, please let me know. Target log: Starting DPDK 17.08.0 initialization... [ DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid27838 ] EAL: Detected 32 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... Total cores available: 1 Occupied cpu socket mask is 0x1 reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0 copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0 nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2017-06.io.spdk-MPcnode1 on lcore 0 on socket 0 rdma.c:1146:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init *** rdma.c:1353:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 172.17.2.175 port 11345 *** nvmf_tgt.c: 255:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 0 on socket 0 rdma.c:1515:spdk_nvmf_rdma_poll_group_create: *NOTICE*: Skipping unused RDMA device when creating poll group. Everything seems to be fine on the target application until the initiator connects to it and create a namespace. NVMF configuration file: [Nvmf] MaxQueuesPerSession 4 AcceptorPollRate 10000 [Subsystem1] NQN nqn.2017-06.io.spdk-MPcnode1 Core 1 SN SPDK0000000000000001 Listen RDMA 172.17.2.175:11345 AllowAnyHost Yes NVMe 0000:84:00.0 Initiator log: ./fio --name=nvme --numjobs=1 --filename="trtype=RDMA adrfam=IPV4 traddr=172.17.2.175 trsvcid=11345 subnqn=nqn.2017-06.io.spdk-MPcnode1 ns=1" --bs=4K --iodepth=1 --ioengine=/home.local/sfast/spdk20/spdk/examples/nvme/fio_plugin/fio_plugin --sync=0 --norandommap --group_reporting --size=12K --runtime=3 -rwmixwrite=30 --thread=1 --rw=rw nvme: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1 fio-3.3 Starting 1 thread Starting DPDK 17.11.0 initialization... [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid28214 ] EAL: Detected 32 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... nvme_ctrlr.c:1031:nvme_ctrlr_construct_namespaces: *ERROR*: controller has 0 namespaces fio_plugin.c: 298:spdk_fio_setup: *ERROR*: spdk_nvme_probe() Thanks for your suggestion Sreeni

3 years, 5 months

2
2
0 / 0

Re: [SPDK] Buffer I/O error on bigger block size running fio

by Harris, James R

Hi Victor, Could you provide a few more details? This will help the list to provide some ideas. 1) On the client, are you using the SPDK NVMe-oF initiator or the kernel initiator? 2) Can you provide the fio configuration file or command line? Just so we can have more specifics on “bigger block size”. 3) Any details on the HW setup – specifically details on the RDMA NIC (or if you’re using SW RoCE). Thanks, -Jim From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Victor Banh <victorb(a)mellanox.com> Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org> Date: Thursday, October 5, 2017 at 11:26 AM To: "spdk(a)lists.01.org" <spdk(a)lists.01.org> Subject: [SPDK] Buffer I/O error on bigger block size running fio Hi I have SPDK NVMeoF and keep getting error with bigger block size with fio on randwrite tests. I am using Ubuntu 16.04 with kernel version 4.12.0-041200-generic on target and client. The DPDK is 17.08 and SPDK is 17.07.1. Thanks Victor [46905.233553] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 [48285.159186] blk_update_request: I/O error, dev nvme1n1, sector 2507351968 [48285.159207] blk_update_request: I/O error, dev nvme1n1, sector 1301294496 [48285.159226] blk_update_request: I/O error, dev nvme1n1, sector 1947371168 [48285.159239] blk_update_request: I/O error, dev nvme1n1, sector 1891797568 [48285.159252] blk_update_request: I/O error, dev nvme1n1, sector 10833824 [48285.159265] blk_update_request: I/O error, dev nvme1n1, sector 614937152 [48285.159277] blk_update_request: I/O error, dev nvme1n1, sector 1872305088 [48285.159290] blk_update_request: I/O error, dev nvme1n1, sector 1504491040 [48285.159299] blk_update_request: I/O error, dev nvme1n1, sector 1182136128 [48285.159308] blk_update_request: I/O error, dev nvme1n1, sector 1662985792 [48285.191185] nvme nvme1: Reconnecting in 10 seconds... [48285.191254] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191291] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191305] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191314] ldm_validate_partition_table(): Disk read failed. [48285.191320] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191327] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191335] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191342] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191347] Dev nvme1n1: unable to read RDB block 0 [48285.191353] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191360] Buffer I/O error on dev nvme1n1, logical block 0, async page read [48285.191375] Buffer I/O error on dev nvme1n1, logical block 3, async page read [48285.191389] nvme1n1: unable to read partition table [48285.223197] nvme1n1: detected capacity change from 1600321314816 to 0 [48289.623192] nvme1n1: detected capacity change from 0 to -65647705833078784 [48289.623411] ldm_validate_partition_table(): Disk read failed. [48289.623447] Dev nvme1n1: unable to read RDB block 0 [48289.623486] nvme1n1: unable to read partition table [48289.643305] ldm_validate_partition_table(): Disk read failed. [48289.643328] Dev nvme1n1: unable to read RDB block 0 [48289.643373] nvme1n1: unable to read partition table

3 years, 6 months

4
22
0 / 0

SPDK FIO poor 4K read performance

by Mayank

Hi All, I have performed fio benchmark test with SPDK fio plugin for throughput test with iodepth = 32 and numjobs=4. Latency results are as below: rand read 11.2k 45.9MB/s 88.83 usec rand write 126k 518MB/s 7.56 usec Throughput results are as below: Throughput (io:32 thr:4) IOPS Throuhgput Latency (usec) seq read 406k 1665MB/s 314.71 seq write 455k 1865MB/s 280.87 rand read 478k 1959MB/s 267.29 rand write 442k 1812MB/s 289.03 In results, 'm getting sequential read IOPS lesser than sequential write IOPS. I am using NVME model number : INTEL SSDPEDMD020T4. kernel : 4.13.13 fio version : 2.21 can anybody tell me what could be the issue? Thanks, Mayank

3 years, 8 months

5
14
0 / 0

expose an SPDK bdev to linux kernel block device(/dev/xxx)

by 張安男

Dear All, Recently we have a requirement for access SPDK bdev from kernel block device. I had try two methods: (1) expose the SPDK block device from iscsi and connect the iscsi device from the same host. (2)Use SPDK provided the connection from /dev/nbd. These two methods can let us access the SPDK block device from /dev/xxx , but the performance is not well. The performance is only about 50K IOPS. So, does anyone can give us any suggestion of how to access SPDK block device from kernel block device with high performance??? Any suggestion is appreciated. -- Vincent Chang

3 years, 8 months

2
2
0 / 0

IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

by Li Wang

Dear all, I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 CPU cores). I noticed that the IOPS is around 1K when the queue depth is 1. By simply increase the queue length to 2, the IOPS grows to over 60K. May I ask if this is normal? $ perf -q 1 -s 512 -w read -t 5 ======================================================== Latency(us) Device Information : IOPS MB/s Average min max Amazon EC2 NVMe Inst (AWS6F57178D12C6C4DC1) from core 0: 1116.80 0.55 895.40 19.53 1969.98 ======================================================== Total : 1116.80 0.55 895.40 19.53 1969.98 $ perf -q 2 -s 512 -w read -t 5 ======================================================== Latency(us) Device Information : IOPS MB/s Average min max Amazon EC2 NVMe Inst (AWS6F57178D12C6C4DC1) from core 0: 61094.60 29.83 32.71 13.71 136.63 ======================================================== Total : 61094.60 29.83 32.71 13.71 136.63 Regards, Li Wang

3 years, 8 months

5
13
0 / 0

compiling spdk with clang

by liupan1234

Hi, I want to use clang to compile spdk, and also do some static code analysis. Could you tell me how to do that? Thanks Pan

3 years, 8 months

3
2
0 / 0

Core maintainers - new additions

by Harris, James R

The SPDK project has a team of core maintainers who are responsible for providing technical oversight for the SPDK project. I am pleased to announce that we are adding Shuhei Matsumoto and Changpeng Liu to the core maintainer team! Shuhei is from Hitachi and has been a key developer on the SPDK iSCSI and SCSI libraries, driving a huge number of improvements in these areas since starting with the SPDK project earlier this year. Changpeng is from Intel and has been contributing to SPDK since the project started in 2015. He has contributed to a wide variety of SPDK libraries and applications, including the nvme driver and extending SPDK vhost support for virtio-blk. To Shuhei and Changpeng, thank you for all of your focus and insight – and congratulations! Thanks, Jim Harris (representing the core maintainer team) P.S. Please visit http://www.spdk.io/development/#core for further details on the core maintainer team and their responsibilities.

3 years, 8 months

2
1
0 / 0

coredump in the memory malloc

by Huang Frank

hi,all I meet a strange problem. I first alloacte a block for some variable, I want 0x400000 align. In the following ways: spdk_dma_zmalloc(buf_size, 0x400000, &dma_addr); It return 0x7ffff2600000.And then the next malloc get coredump. the address of elem in malloc_elem_free_list_insert equal to the prev malloc address. (gdb) bt #0 0x0000000000431b70 in malloc_elem_free_list_insert (elem=0x7ffff2600000) at /home/xxx/spdk/dpdk/lib/librte_eal/common/malloc_elem.c:187 #1 0x0000000000431d94 in malloc_elem_alloc (elem=0x7ffff2600000, size=64, align=64, bound=0) at /home/xxx/spdk/dpdk/lib/librte_eal/common/malloc_elem.c:249 #2 0x00000000004323fe in malloc_heap_alloc (heap=0x7ffff7fefa1c, type=0x0, size=64, flags=0, align=64, bound=0) at /home/xxx/spdk/dpdk/lib/librte_eal/common/malloc_heap.c:168 #3 0x000000000043125f in rte_malloc_socket (type=0x0, size=48, align=0, socket_arg=-1) at /home/xxx/spdk/dpdk/lib/librte_eal/common/rte_malloc.c:91 #4 0x000000000041946f in spdk_dma_malloc_socket (size=48, align=0, phys_addr=0x0, socket_id=-1) at env.c:58 #5 0x00000000004194c8 in spdk_dma_zmalloc_socket (size=48, align=0, phys_addr=0x0, socket_id=-1) at env.c:68 #6 0x000000000041954a in spdk_dma_zmalloc (size=48, align=0, phys_addr=0x0) at env.c:84 Is there some limitation for using spdk_dma_zmalloc? ________________________________ kinzent(a)hotmail.com

3 years, 9 months

1
1
0 / 0

It looks that auto test always fail and hence please do not rush to queue your patch into GerritHub.

by 松本周平 / MATSUMOTO，SHUUHEI

Hi All, Now it looks that the auto test fails at the same location. https://github.com/spdk/spdk/blob/master/test/lvol/test_cases.py line 770 AddressSanitize detects heap-use-after-free. By skipping the code test passed but nothing changes yet. I think everyone is waiting for fixing this issue. My test pass is fake and please do not rush to queue your patch into the GerritHub. Of course I'm very glad if anyone is investingating the cause in parallel with me. Thank you for your understanding, Shuhei

3 years, 9 months

4
7
0 / 0

SPDK aware application

by Mayank

Hi All, I just want to check if there is any application or way through which i can export NVME disk (as a block device) via SPDK? Setup is as below : Currently, I am using NVME disk with my application in docker as a DAS (in respect of docker). I want to use NVME disk through SPDK with application in docker ( or let say, i can attach NVME disk to docker through SPDK). Regards, Mayank

3 years, 9 months

2
1
0 / 0

2021

2020

2019

2018

2017

2016

SPDK December 2017