NVMeF Target Error NVMe controller
by Ankur Srivastava
Hi All,
Here are some notes that I took on my system when starting the nvmf
target and faced an issue with NVMe Controller discovery
1. Mellanox NICs detected successfully.
2. Installed neccesary modules
a. modprobe nvme
b. modprobe nvmet
c. modprobe nvme-rdma
d. modprobe nvmet-rdma
3. Assign NICs ip addresses
a. ip addr add 192.168.25.3/24 dev nic0
b. ip addr add 192.168.25.31/24 dev nic1
4. My configuration file has the following Subsystems
# Direct controller
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 0
Mode Direct
Listen RDMA 192.168.25.3:4420
Host nqn.2016-06.io.spdk:init
NVMe 0000:06:00.0
# Multiple subsystems are allowed.
[Subsystem2]
NQN nqn.2016-06.io.spdk:cnode2
Core 0
Mode Direct
Listen RDMA 192.168.25.31:4420
Host nqn.2016-06.io.spdk:init
SN SPDK00000000000001
Namespace Malloc0
Namespace Malloc1
5. Start the nvmf target
cd ~/spdk/app/nvmf_tgt
sudo ./nvmf_tgt -c nvmf.conf.in
Getting the following error
[root@localhost spdk-master]# app/nvmf_tgt/nvmf_tgt -c etc/spdk/nvmf.conf.in
Starting Intel(R) DPDK initialization ...
[ DPDK EAL parameters: nvmf -c 1 -n 4 -m 2048 --master-lcore=0
--file-prefix=rte0 --proc-type=auto ]
EAL: Detected 12 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: Probing VFIO support...
done.
Occupied cpu core mask is 0x1
Occupied cpu socket mask is 0x1
Ioat Copy Engine Offload Enabled
Total cores available: 1
Reactor started on core 0 on socket 0
*** RDMA Transport Init ***
*** RDMA Transport Init ***
allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
*** NVMf Target Listening on 192.168.25.3 port 4420 ***
conf.c: 565:spdk_nvmf_parse_subsystem: ***ERROR*** Could not find NVMe
controller for Subsystem1
nvmf_tgt.c: 336:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf() failed
Best Reagrds
Ankur
4 years, 6 months
Error while compiling the spdk-16.12 for arm64 architecture
by Balaji G
Hi All,
I have downloaded the spdk-16.12.zip
and tried to compile for the arm64 architecture but i got the following
error
In file included from nvme_ctlr_cmd.c:30:0:
nvme_internal.h:49:23:fatal error:x86intrin.h: No such file or directory
Compilation terminated.
So, can anyone tell me does the spdk has the support for ARM architecture
Regards,
G.Balaji
4 years, 6 months
Re: [SPDK] 回复： NVMeF Target Error NVMe controller
by Ankur Srivastava
Hi Thomas,
No I am not having any nvme ssd drive in my system, I was using 'nullb0'
device installed through "modprobe null_blk nr_devices=1" for my NVMe over
Fabric bench-marking.
Few Question:
1) Since it is nullb0 device it can't be there in lspci, does this mean
this is an issue?
2) Also
# Direct controller
[Subsystem1]
NQN nqn.2016-06.io.spdk:cnode1
Core 0
Mode Direct
Listen RDMA 192.168.25.3:4420
Host nqn.2016-06.io.spdk:init
NVMe 0000:06:00.0
In the string "NVMe 0000:06:00.0" above, Do these numbers(
'domain:bus:func.dev') belong to the NVMe SSD drive attached via pci ??
Thanks in advance
Regards
Ankur
On Tue, Feb 28, 2017 at 8:30 PM, thomas.zhang.8288 <
thomas.zhang.8288(a)aliyun.com> wrote:
> Hi Ankur,
>
> Do you have any NVME SSD drive(s) in the system? You can assign one of
> nvme ssd bus info in the 'subsystem1', also you can dump you nvme ssd
> drive info and extract 'domain:bus:func.dev' info in your system.
> 'lspci | grep Non-volatile'
>
> Mode 'direct' means front nvmf directly control backend nvme ssd drive.
>
> Thomas
>
> ------------------------------------------------------------------
> 发件人：Ankur Srivastava <asrivastava014(a)gmail.com>
> 发送时间：2017年2月28日(星期二) 22:38
> 收件人：Storage Performance Development Kit <spdk(a)lists.01.org>;
> thomas.zhang.8288 <thomas.zhang.8288(a)aliyun.com>
> 主 题：Re: [SPDK] 回复： NVMeF Target Error NVMe controller
>
> Thanks Thomas for the work around I have made the subsystem2 as Virtual
> Mode, but still the error persists.
> Error I am getting is related to the subsystem1.
>
> Modified conf.in
> -------------------------------
> # Direct controller
> [Subsystem1]
> NQN nqn.2016-06.io.spdk:cnode1
> Core 0
> Mode Direct
> Listen RDMA 192.168.25.3:4420
> Host nqn.2016-06.io.spdk:init
> NVMe 0000:06:00.0
>
> # Multiple subsystems are allowed.
> # Virtual controller
> [Subsystem2]
> NQN nqn.2016-06.io.spdk:cnode2
> Core 0
> Mode Virtual
> Listen RDMA 192.168.25.31:4420
> Host nqn.2016-06.io.spdk:init
> SN SPDK00000000000001
> Namespace Malloc0
> Namespace Malloc1
>
>
>
> Error:
> --------------------------
> [root@localhost spdk-master]# app/nvmf_tgt/nvmf_tgt -c etc/spdk/
> nvmf.conf.in
> Starting Intel(R) DPDK initialization ...
> [ DPDK EAL parameters: nvmf -c 1 -n 4 -m 2048 --master-lcore=0
> --file-prefix=rte0 --proc-type=auto ]
> EAL: Detected 12 lcore(s)
> EAL: Auto-detected process type: PRIMARY
> EAL: Probing VFIO support...
> done.
> Occupied cpu core mask is 0x1
> Occupied cpu socket mask is 0x1
> Ioat Copy Engine Offload Enabled
> Total cores available: 1
> Reactor started on core 0 on socket 0
> *** RDMA Transport Init ***
> *** RDMA Transport Init ***
> allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on
> socket 0
> allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
> *** NVMf Target Listening on 192.168.25.3 port 4420 ***
> conf.c: 565:spdk_nvmf_parse_subsystem: ***ERROR*** Could not find NVMe
> controller for Subsystem1
> nvmf_tgt.c: 336:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf()
> failed
>
>
> Can you please tell me more about "domain:bus:func:dev", from where I can
> extract this data ?
>
>
> Thanks and Regards
> Ankur
>
>
> On Tue, Feb 28, 2017 at 7:56 PM, thomas.zhang.8288 <
> thomas.zhang.8288(a)aliyun.com> wrote:
> Hi Ankur,
>
> You configuration missing nvme 'domain:bus:func:dev' info in the second
> nvmf subsystem 'subsystem2'. If you choose mode 'Direct' , you have to
> specific 'nvme' bus info in the nvmf subsystem, otherwise you can choose
> 'virtual mode'.
>
> Best Reagrds
>
> Zhihao
>
>
> ------------------------------------------------------------------
> 发件人：Ankur Srivastava <asrivastava014(a)gmail.com>
> 发送时间：2017年2月28日(星期二) 21:57
> 收件人：spdk <spdk(a)lists.01.org>
> 主 题：[SPDK] NVMeF Target Error NVMe controller
>
> Hi All,
>
>
> Here are some notes that I took on my system when starting the nvmf target and faced an issue with NVMe Controller discovery
>
> 1. Mellanox NICs detected successfully.
>
> 2. Installed neccesary modules
>
> a. modprobe nvme
>
> b. modprobe nvmet
>
> c. modprobe nvme-rdma
>
> d. modprobe nvmet-rdma
>
> 3. Assign NICs ip addresses
>
> a. ip addr add 192.168.25.3/24 dev nic0
> b. ip addr add 192.168.25.31/24 dev nic1
>
>
> 4. My configuration file has the following Subsystems
>
> # Direct controller
> [Subsystem1]
> NQN nqn.2016-06.io.spdk:cnode1
> Core 0
> Mode Direct
> Listen RDMA 192.168.25.3:4420
> Host nqn.2016-06.io.spdk:init
> NVMe 0000:06:00.0
>
> # Multiple subsystems are allowed.
> [Subsystem2]
> NQN nqn.2016-06.io.spdk:cnode2
> Core 0
> Mode Direct
> Listen RDMA 192.168.25.31:4420
> Host nqn.2016-06.io.spdk:init
> SN SPDK00000000000001
> Namespace Malloc0
> Namespace Malloc1
>
>
> 5. Start the nvmf target
>
> cd ~/spdk/app/nvmf_tgt
> sudo ./nvmf_tgt -c nvmf.conf.in
>
>
> Getting the following error
>
> [root@localhost spdk-master]# app/nvmf_tgt/nvmf_tgt -c etc/spdk/nvmf.conf.in
> Starting Intel(R) DPDK initialization ...
> [ DPDK EAL parameters: nvmf -c 1 -n 4 -m 2048 --master-lcore=0 --file-prefix=rte0 --proc-type=auto ]
> EAL: Detected 12 lcore(s)
> EAL: Auto-detected process type: PRIMARY
> EAL: Probing VFIO support...
> done.
> Occupied cpu core mask is 0x1
> Occupied cpu socket mask is 0x1
> Ioat Copy Engine Offload Enabled
> Total cores available: 1
> Reactor started on core 0 on socket 0
> *** RDMA Transport Init ***
> *** RDMA Transport Init ***
> allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
> allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 0 on socket 0
> *** NVMf Target Listening on 192.168.25.3 port 4420 ***
> conf.c: 565:spdk_nvmf_parse_subsystem: ***ERROR*** Could not find NVMe controller for Subsystem1
> nvmf_tgt.c: 336:spdk_nvmf_startup: ***ERROR*** spdk_nvmf_parse_conf() failed
>
>
>
>
>
>
> Best Reagrds
>
> Ankur
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
4 years, 6 months
hotplug support and testing
by Will Del Genio
Hello,
I’m currently working on testing the hotplug support for local PCIe connected NVMe drives (P3700s and 3600s at the moment) on linux. (Linux sb1 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux)
Right now spdk requires the setup script to be manually run which unbinds the nvme driver and binds the uio_pci_generic driver. Is there a way to have that happen automatically on hotplugging of an nvme device? From what I understand it could be done via script called by a udev rule.
Is that the right way to handle it? Are there any other plans for hotplug support moving forward I should be aware of?
Thanks for the help,
Will Del Genio
4 years, 6 months
Adding Namespaces dynamically
by Sriram Popuri
Hi experts,
I am exploring spdk and particularly nvmf_tgt app. I would like to know
how we can add namespaces dynamically.
The app is reading from a conf file during initialization and doesn't seem
to have an option to add namespaces dynamically.
There is a way to add Namespace (spdk_nvmf_subsystem_add_ns), however that
seems to be only during initialization (either through conf or rpc).
Even if I invoke this api somehow, there are no AER commands implemented or
the relevant log pages (in this case: 04h - Changed namespace list) so the
host can discover the new namespace.
Is this in plan and if so when can we expect to see this coming?
Regards,
~Sriram
4 years, 6 months
Netlist storage card
by Dale Corlett
Hi,
I am new to SPDK, and driver writing in general. I am trying to get SPDK
working with a non NVMe storage card (Netlist Express Vault:
http://www.netlist.com/products/vault-memory-storage/expressvault-pcIe-ev...),
it shows up as a block device in Linux. Would anyone know if there are
already utilities in SPDK to configure new devices like this (I have been
looking at ioat, and the SPDK bdev libraries) or do I have to create some
sort of SPDK user space driver for the Neltist card? If I do have to create
and SPDK driver for the Netlist card, how would I go about that?
Thanks
4 years, 6 months
NVMf Target
by Kumaraparameshwaran Rathnavel
Hi All,
Why are we not using the same completion queue for multiple queue pairs. Whenever we create a queue pair , I see that a completion queue is also created. But completion queue can be shared between queue pairs. Will Using shared completion queue impact the performance?
Thanking you,
Param.
4 years, 7 months
How to contribute to SPDK
by Wenhua Liu
Hi,
While using SPDK NVMF target, I encountered 2 blocking issues at target side. I have fixed the problems. I'd like to contribute my changes to SPDK, how should I do that?
Thanks,
-Wenhua
4 years, 7 months
Isolation for applications
by Pramod Bhatotia
Hi everyone,
How could we support an isolation mechanism for competing applications for
the userspace NVMe over Intel SPDK?
Also, is there a file-system/OS support to enforce such isolation?
Thanks,
-Pramod
4 years, 7 months
Question for SGL Descriptor Programming with SPDK NVMF Target
by Wenhua Liu
Hi SPDK experts,
I'm using SPDK NVMF target to test my client code and run into an issue when transferring data described my multiple SGLs.
To better describe my issue, here I assume the data buffer is represented by variable dataSGL which containts 3 SGL elements:
dataSGL[0]: addr=0x1000, length=0x800
dataSGL[1]: addr=0x3000, length=0x2200
dataSGL[2]: addr=0x8000, length=0x1300
According to the NVMe over Fabrics speciciation (section 7.3.2), RDMA transport supports SGL Data Block, SGL Last Segment, and Keyed SGL Data Block descriptors only.
The first time I used SGL Last Segment descriptor. I created a buffer of size 3*sizeof(NVME_SGL_DESCRIPTOR) which is 48 bytes and put the above 3 dataSGLs in the buffer, set descriptor type to Keyed SGL Data Block for all and subtype to 0 (address), filled in key. Then I set SQE.dptr.sgl1's descriptor type to SGL Last Segment and subtype to 0 (address) and address field to the address of the buffer containing 3 dataSGLs. The size field is set to 48 indicating this Last SGL Segment contains 3 SGL descriptors. When doing RDMA send, the above buffer was also sent to the target side as part of RDMA send call. SPDK target received the command and failed it with error
message "Invalid NVMf I/O Command SGL: Type 0x3, Subtype 0x0". This is apparent if looking at SPDK source code as it does not support SGL Last Segment descriptor.
The second time I used SGL Data Block descriptor. Comparing to the first case, the only difference is in SQE.dptr.sgl1, I set descriptor type to SGL Data Block and subtype to 1 (address field contains offset) and address field to 0 (offset is 0). I was hoping the target would interpret the 48 byte in-capsule data as 3 Keyed SGL descriptors. This time, SPDK target said buffer too small. It expected a buffer with size (0x800+0x2200+0x1300), but SQE.dptr.sgl1 indicated the buffer size is 48.
The third time I used Keyed SGL Data Block descriptor. According to the NVMe over Fabrics spec (section 2.3.2), "In this case, there is one SGL descriptor that is contained within the Submission Queue Entry at CMD.SGL1. The SGL descriptor is a Keyed SGL Data Block descriptor. If more SGLs were needed to complete the command, the additional SGLs would be contained in the command capsule.". Following this example, I put dataSGL[0] in SQE.dptr.sgl1 by setting its descriptor type to Keyed SGL Data Block, subtype to 0 (address), address to 0x1000, length to 0x800 and the key to some value. I then created a buffer of size 2*sizeof(NVME_SGL_DESCRIPTOR) and put dataSGL[1] and dataSGL[2] into it the same way I put dataSGL[0] into SQE.dptr.sgl1. When doing RDMA send, this buffer was also sent to the target as in-capsule data of size 32. SPDK target received it and failed it again saying buffer too small. It expected a buffer with size(0x800+0x2200+0x1300), but SQE.dptr.sgl1 indicated the buffer size is 0x800.
As I still have trouble connecting to other targets, I do know how other targets will behave. I have a few questions hoping someone can answer/explain:
1. Of the above 3 approaches, is there one that is correct?
2. If none of them is correct, what is the right way prepare NVME SGL descriptors for the data described by multiple SGLs?
3. How should multiple SGLs be sent to target using RDMA transport? Using in-capsule data?
4. When receiving a NVME command (and maybe with some in-capsule data), how does target know that data is real data or SGLs?
5. Should a NVME target (RDMA based) support all 3 SGL descriptors or can target choose what it wants?
6. How does SPDK NVMF target handles multiple command containing data described by multiple SGLs? Which piece of code handles this?
Thanks,
-Wenhua Liu
4 years, 7 months
