Best practices on driver binding for SPDK in production environments
by Lance Hartmann ORACLE
This email to the SPDK list is a follow-on to a brief discussion held during a recent SPDK community meeting (Tue Jun 26 UTC 15:00).
Lifted and edited from the Trello agenda item (https://trello.com/c/U291IBYx/91-best-practices-on-driver-binding-for-spd... <https://trello.com/c/U291IBYx/91-best-practices-on-driver-binding-for-spd...>):
During development many (most?) people rely on the run of SPDK's scripts/setup.sh to perform a number of initializations, among them the unbinding of the Linux kernel nvme driver from NVMe controllers targeted for use by the SPDK and then binding them to either uio_pci_generic or vfio-pci. This script is applicable for development environments, but not targeted for use in productions systems employing the SPDK.
I'd like to confer with my fellow SPDK community members on ideas, suggestions and best practices for handling this driver unbinding/binding. I wrote some udev rules along with updates to some other Linux system conf files for automatically loading either the uio_pci_generic or vfio-pci modules. I also had to update my initramfs so that when the system comes all the way up, the desired NVMe controllers are already bound to the needed driver for SPDK operation. And, as a bonus, it should "just work" when a hotplug occurs as well. However, there may be additional considerations I might have overlooked on which I'd appreciate input. Further, there's the matter of how and whether to semi-automate this configuration via some kind of script and how that might vary according to Linux distro to say nothing of the determination of employing uio_pci_generic vs vfio-pci.
And, now some details:
1. I performed this on an Oracle Linux (OL) distro. I’m currently unaware how and what configuration files might be different depending on the distro. Oracle Linux is RedHat-compatible, so I’m confident my implementation should run similarly on RedHat-based systems, but I’ve yet to delve into other distro’s like Debian, SuSE, etc.
2. In preparation to writing my own udev rules, I unbound a specific NVMe controller from the Linux nvme driver by hand. Then, in another window I launched: "udevadm monitor -k -p” so that I could observe the usual udev events when a NVMe controller is bound to the nvme driver. On my system, I observed four (4) udev kernel events (abbreviated/edited output to avoid this become excessively long):
(Event 1)
KERNEL[382128.187273] add /devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0 (nvme)
ACTION=add
DEVNAME=/dev/nvme0
…
SUBSYSTEM=nvme
(Event 2)
KERNEL[382128.244658] bind /devices/pci0000:00/0000:00:02.2/0000:30:00.0 (pci)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:30:00.0
DRIVER=nvme
…
SUBSYSTEM=pci
(Event 3)
KERNEL[382130.697832] add /devices/virtual/bdi/259:0 (bdi)
ACTION=add
DEVPATH=/devices/virtual/bdi/259:0
...
SUBSYSTEM=bdi
(Event 4)
KERNEL[382130.698192] add /devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0/nvme0n1 (block)
ACTION=add
DEVNAME=/dev/nvme0n1
DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0/nvme0n1
DEVTYPE=disk
...
SUBSYSTEM=block
3. My udev rule triggers on (Event 2) above: the bind action. Upon this action, my udev rule appends operations to the special udev RUN variable such that udev will essentially mirror that which is done in the SPDK’s scripts/setup.sh for unbinding from the nvme driver and binding to, in my case, the vfio-pci driver.
4. With my new udev rules in place, I was successful getting specific NVMe controllers (based on bus-device-function) to unbind from the Linux nvme driver and bind to vfio-pci. However, I made a couple of observations in the kernel log (dmesg). In particular, I was drawn to the following for an NVMe controller at BDF: 0000:40:00.0 for which I had a udev rule to unbind from nvme and bind to vfio-pci:
[ 35.534279] nvme nvme1: pci function 0000:40:00.0
[ 37.964945] nvme nvme1: failed to mark controller live
[ 37.964947] nvme nvme1: Removing after probe failure status: 0
One theory I have for the above is that my udev RUN rule was invoked while the nvme driver’s probe() was still running on this controller, and perhaps the unbind request came in before the probe() completed hence this “name1: failed to mark controller live”. This has left lingering in my mind that maybe instead of triggering on (Event 2) when the bind occurs, that perhaps I should instead try to derive a trigger on the “last" udev event, an “add”, where the NVMe namespace’s are instantiated. Of course, I’d need to know ahead of time just how many namespaces exist on that controller if I were to do that so I’d trigger on the last one. I’m wondering if that may help to avoid what looks like a complaint during the middle of probe() of that particular controller. Then, again, maybe I can just safely ignore that and not worry about it at all? Thoughts?
I discovered another issue during this experimentation that is somewhat tangential to this task, but I’ll write a separate email on that topic.
thanks for any feedback,
--
Lance Hartmann
lance.hartmann(a)oracle.com
1 year, 10 months
Chandler Build Pool Test Failures
by Howell, Seth
Hi all,
There has been a rash of failures on the test pool starting last night. I was able to root cause the failures to a point in the NVMe-oF shutdown tests. The main substance of the failure is that QAT and the DPDK framework don't always play well with secondary dpdk processes. In the interest of avoiding these failures on future builds, please rebase your changes on the following patch series which includes the fix of not running bdevperf as a secondary process in the NVMe-oF shutdown tests.
https://review.gerrithub.io/c/spdk/spdk/+/435937/6
Thanks,
Seth Howell
1 year, 11 months
Topic from last week's community meeting
by Luse, Paul E
Hi Shuhei,
I was out of town last week and missed the meeting but saw on Trello you had the topic below:
"a few idea: log structured data store , data store with compression, and metadata replication of Blobstore"
Which I'd be pretty interested in working on with you or at least hearing more about it. When you get a chance, no hurry, can you please expand a little on how the conversation went and what you're looking at specifically?
Thanks!
Paul
1 year, 11 months
Add py-spdk client for SPDK
by We We
Hi, all
I have submitted the py-spdk code on https://review.gerrithub.io/#/c/379741/, please take some time to visit it, I will be very grateful to you.
The py-spdk is client which can help the upper-level app to communicate with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should I submit it into the other repo I rebuild rather than SPDK repo? Because I think it is a relatively independent kit upon the SPDK.
If you have some thoughts about the py-spdk, please share with me.
Regards,
Helloway
1 year, 11 months
Access control on the subsystem NQN
by Shahar Salzman
Hi,
On a previous thread, I got the great answer that a subsystem is actually an access control list.
We have been using it as such, but some of the behavior seems odd.
On host_remove the sessions to the hosts are not disconnected, this means that even though I removed the host from the ACL, it will have access, until it disconnects (although it cannot connect another session).
In addition, a host can see all the subsystems exposed on the IP, even though it has access to only some of them.
We would be happy to work on these issues, but I would like to understand the expected behavior.
Thanks,
Shahar
2 years, 8 months
Trouble Setting Up NVMeoF Target
by Gruher, Joseph R
Hi everyone. I'm running Ubuntu 16.04 with SPDK v18.10.1. I'm having trouble setting up the NVMeoF target. I can't seem to spot the problem, so maybe someone can help me out here. I'm pretty sure I'm either formatting the command incorrectly or I've screwed up the IB/RDMA related packages in my Ubuntu install.
I have a Mellanox CX5 NIC at 192.168.200.4, interface is up and linked:
rsd@nvme:~$ ifconfig enp6s0
enp6s0 Link encap:Ethernet HWaddr 98:03:9b:1e:de:ec
inet addr:192.168.200.4 Bcast:192.168.200.255 Mask:255.255.255.0
inet6 addr: fe80::9a03:9bff:fe1e:deec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:138 errors:0 dropped:0 overruns:0 frame:0
TX packets:135 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19029 (19.0 KB) TX bytes:24279 (24.2 KB)
rsd@nvme:~$ sudo ethtool enp6s0
Settings for enp6s0:
Supported ports: [ Backplane ]
Supported link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Advertised link modes: 1000baseKX/Full
10000baseKR/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Link partner advertised link modes: Not reported
Link partner advertised pause frame use: No
Link partner advertised auto-negotiation: Yes
Speed: 100000Mb/s
Duplex: Full
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000004 (4)
link
Link detected: yes
I ran setup and then started the target and gave it four cores:
rsd@nvme:~$ sudo ~/install/spdk/scripts/setup.sh
[sudo] password for rsd:
0000:d8:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:d9:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:da:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:db:00.0 (8086 0a54): nvme -> uio_pci_generic
0000:00:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:00:04.7 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.0 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.1 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.2 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.3 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.4 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.5 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.6 (8086 2021): ioatdma -> uio_pci_generic
0000:80:04.7 (8086 2021): ioatdma -> uio_pci_generic
rsd@nvme:~$ sudo /home/rsd/install/spdk/app/nvmf_tgt/nvmf_tgt -m 0xF
Starting SPDK v18.10.1 / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0xF --file-prefix=spdk_pid42493 ]
EAL: Detected 80 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 602:spdk_app_start: *NOTICE*: Total cores available: 4
reactor.c: 703:spdk_reactors_init: *NOTICE*: Occupied cpu socket mask is 0x1
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 1 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 2 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
reactor.c: 490:_spdk_reactor_run: *NOTICE*: Reactor started on core 3 on socket 0
I created my bdevs from my local NVMe devices:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d0 -t pcie -a 0000:d8:00.0
d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d1 -t pcie -a 0000:d9:00.0
d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d2 -t pcie -a 0000:da:00.0
d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvme_bdev -b d3 -t pcie -a 0000:db:00.0
d3n1
Matching prints from the target:
EAL: PCI device 0000:d8:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:d9:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:da:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
EAL: PCI device 0000:db:00.0 on NUMA socket 1
EAL: probe driver: 8086:a54 spdk_nvme
Then the problem starts when I try to create my NVMeoF subsystem. If I use this deprecated method I get an invalid parameters error:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py construct_nvmf_subsystem -a -n d0n1 -m 1 nqn.2018-12.io.spdk:nqn0 "trtype:RDMA traddr:192.168.200.4 trsvcid:4420" ""
Got JSON-RPC error response
request:
{
"jsonrpc": "2.0",
"params": {
"max_namespaces": 1,
"nqn": "nqn.2018-12.io.spdk:nqn0",
"allow_any_host": true,
"serial_number": "00000000000000000000",
"listen_addresses": [
{
"traddr": "192.168.200.4",
"trtype": "RDMA",
"trsvcid": "4420"
}
],
"namespaces": [
{
"bdev_name": "d0n1"
}
]
},
"method": "construct_nvmf_subsystem",
"id": 1
}
response:
{
"message": "Invalid parameters",
"code": -32602
}
Matching unhappy prints on the target:
nvmf_rpc_deprecated.c: 490:spdk_rpc_construct_nvmf_subsystem: *WARNING*: The construct_nvmf_subsystem RPC is deprecated. Use nvmf_subsystem_create instead.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such device
transport.c: 93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed
If I try to use the newer method, something similar happens, this part goes OK:
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create nqn.2018-12.io.spdk:nqn0
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create nqn.2018-12.io.spdk:nqn1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create nqn.2018-12.io.spdk:nqn2
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_create nqn.2018-12.io.spdk:nqn3
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2018-12.io.spdk:nqn0 d0n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2018-12.io.spdk:nqn1 d1n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2018-12.io.spdk:nqn2 d2n1
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_ns nqn.2018-12.io.spdk:nqn3 d3n1
But then I can't add the transport:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
"method": "nvmf_subsystem_add_listener",
"params": {
"nqn": "nqn.2018-12.io.spdk:nqn0",
"listen_address": {
"trtype": "rdma",
"traddr": "192.168.200.4",
"trsvcid": null
}
},
"jsonrpc": "2.0",
"id": 1
}
response:
{
"code": -32602,
"message": "Invalid parameters"
}
Which causes these prints on the target:
nvmf_rpc.c: 517:decode_rpc_listen_address: *ERROR*: spdk_json_decode_object failed
nvmf_rpc.c: 703:nvmf_rpc_subsystem_add_listener: *ERROR*: spdk_json_decode_object failed
Any ideas where I'm going wrong here?
I also tried providing more of the optional arguments:
rsd@nvme:~/install/nvme-cli-1.6$ sudo /home/rsd/install/spdk/scripts/rpc.py nvmf_subsystem_add_listener -t rdma -a 192.168.200.4 -f ipv4 -s 4420 nqn.2018-12.io.spdk:nqn0
Got JSON-RPC error response
request:
{
"params": {
"listen_address": {
"adrfam": "ipv4",
"trsvcid": "4420",
"traddr": "192.168.200.4",
"trtype": "rdma"
},
"nqn": "nqn.2018-12.io.spdk:nqn0"
},
"method": "nvmf_subsystem_add_listener",
"id": 1,
"jsonrpc": "2.0"
}
response:
{
"code": -32602,
"message": "Invalid parameters"
}
Matching prints on target:
libibverbs: Warning: couldn't load driver '/usr/lib/libibverbs/libmlx4': /usr/lib/libibverbs/libmlx4-rdmav2.so: cannot open shared object file: No such file or directory
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
rdma.c:1595:spdk_nvmf_rdma_create: *ERROR*: rdma_create_event_channel() failed, No such device
transport.c: 93:spdk_nvmf_transport_create: *ERROR*: Unable to create new transport of type RDMA
nvmf.c: 526:spdk_nvmf_tgt_listen: *ERROR*: Transport initialization failed
The no userspace driver message is certainly concerning. I did tinker about with the Ubuntu packages a bit trying to get ib_send_bw to work at one point and I wonder if I broke something along the way. Any ideas how to recover if that is the case (other than a clean new install of Ubuntu)?
The SPDK pkgdep script does run to completion without any obvious issues and doesn't resolve the problem.
rsd@nvme:~$ sudo /home/rsd/install/spdk/scripts/pkgdep.sh
Reading package lists... Done
Building dependency tree
Reading state information... Done
g++ is already the newest version (4:5.3.1-1ubuntu1).
gcc is already the newest version (4:5.3.1-1ubuntu1).
libaio-dev is already the newest version (0.3.110-2).
libiscsi-dev is already the newest version (1.12.0-2).
make is already the newest version (4.1-6).
sg3-utils is already the newest version (1.40-0ubuntu1).
astyle is already the newest version (2.05.1-0ubuntu1).
lcov is already the newest version (1.12-2).
libcunit1-dev is already the newest version (2.1-3-dfsg-2).
pep8 is already the newest version (1.7.0-2).
git is already the newest version (1:2.7.4-0ubuntu1.6).
libssl-dev is already the newest version (1.0.2g-1ubuntu4.14).
pciutils is already the newest version (1:3.3.1-1.1ubuntu1.2).
uuid-dev is already the newest version (2.27.1-6ubuntu3.6).
clang is already the newest version (1:3.8-33ubuntu3.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libunwind-dev is already the newest version (1.1-4.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libibverbs-dev is already the newest version (1.1.8-1.1ubuntu2).
librdmacm-dev is already the newest version (1.0.21-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
libnuma-dev is already the newest version (2.0.11-1ubuntu1.1).
nasm is already the newest version (2.11.08-1ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
doxygen is already the newest version (1.8.11-1).
mscgen is already the newest version (0.20-5).
graphviz is already the newest version (2.38.0-12ubuntu2.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-pip is already the newest version (8.1.1-2ubuntu0.4).
python3-pip is already the newest version (8.1.1-2ubuntu0.4).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in /usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python2.7/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in /usr/local/lib/python2.7/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The directory '/home/rsd/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/rsd/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied (use --upgrade to upgrade): configshell_fb in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pexpect in /usr/local/lib/python3.5/dist-packages
Requirement already satisfied (use --upgrade to upgrade): pyparsing in /usr/local/lib/python3.5/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): six in /usr/lib/python3/dist-packages (from configshell_fb)
Requirement already satisfied (use --upgrade to upgrade): ptyprocess>=0.5 in /usr/local/lib/python3.5/dist-packages (from pexpect)
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Crypto requires NASM version 2.12.02 or newer. Please install
or upgrade and re-run this script if you are going to use Crypto.
Thanks,
Joe
2 years, 8 months
Some thoughts for the code in rdma.c
by Yang, Ziye
Hi all,
I would like to discuss the following functions in /lib/nvmf/rdma.c
1 I do not think that the _nvmf_rdma_disconnect_retry function is necessary. This function is used in nvmf_rdma_disconnect. And there is a description as follows:
/* Read the group out of the qpair. This is normally set and accessed only from
* the thread that created the group. Here, we're not on that thread necessarily.
* The data member qpair->group begins it's life as NULL and then is assigned to
* a pointer and never changes. So fortunately reading this and checking for
* non-NULL is thread safe in the x86_64 memory model. */
But for group adding for the qpair, it is related with this function: spdk_nvmf_poll_group_add, if the group is not ready, we can just call spdk_nvmf_qpair_disconnect in that function. And the spdk_nvmf_qpair_disconnect should have the strategy to prevent the second time entering if the qpair is not destroyed due to async behavior.
2 If that, the following two functions in rdma.c are also unnecessary:
spdk_nvmf_rdma_qpair_inc_refcnt
spdk_nvmf_rdma_qpair_dec_refcnt
Generally, my idea is that, we should use spdk_nvmf_qpair_disconnect directly in every transport, and do not use two many async messaging for doing this because I do not think those are necessary.
Thanks.
Best Regards
Ziye Yang
2 years, 8 months
SPDKv18.10 & compatibility with DPDK versions
by Rajesh Ravi
Hi,
I noticed some compilation issues & then fio failure when I used SPDK
v18.10 along with DPDK v18.11 with some RDMA failures.
1) Is SPDKv18.10 compatible with DPDK v17.11 (LTS)
2) Is SPDK 18.10 supported for DPDK 18.11
Thanks in advance.
--
Regards,
Rajesh
2 years, 8 months
Re: [SPDK] FreeBSD CI environment
by Harris, James R
Hi Konrad,
I think we explored this a while back, and ran into the same issue as you. The problem is that FreeBSD in general has moved away from the GNU toolchain towards LLVM. I know that we already require GNU make, but I don't want to add yet another dependency on the GNU toolchain.
Can you try this with LLVM's lld linker instead? On systems where that is not available, I'd be open to skipping the unit test build on FreeBSD and/or adding lld to pkgdep.sh.
Thanks,
-Jim
On 12/13/18, 5:43 AM, "SPDK on behalf of Sztyber, Konrad" <spdk-bounces(a)lists.01.org on behalf of konrad.sztyber(a)intel.com> wrote:
Hi all,
I've pushed a patch recently (https://review.gerrithub.io/#/c/spdk/spdk/+/436319/) that adds parameters to compilation/linking of unit tests to remove unused functions from the final executables. It's rather useful, because it allows to omit mocking functions that aren't called by tested code. However, the problem with that is that it breaks the CI on the FreeBSD machine, because the linker hits some kind of assertion and crashes. The default linker used there is pretty old (BFD 2.17.50 [FreeBSD] 2007-07-03) and I've found that installing gcc and using it to build SPDK fixes it. Would it be possible to update the FreeBSD machines and force Jenkins to use gcc on FreeBSD?
Thanks,
Konrad
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
2 years, 8 months
change model number to be custom (from make file) instead define
by Gregory Shapiro
Hello, my name is Gregory and I am new to spdk project.
I now work on some branding feature and need to change MODEL_NUMBER to be
custom.
What do you think of defining it in makefile and reading the info from
there?
Proposed change look something like this:
diff --git a/lib/nvmf/ctrlr.c b/lib/nvmf/ctrlr.c
index ed5e68f0..bc80bf83 100644
--- a/lib/nvmf/ctrlr.c
+++ b/lib/nvmf/ctrlr.c
@@ -49,8 +49,9 @@
#define MIN_KEEP_ALIVE_TIMEOUT 10000
-#define MODEL_NUMBER "SPDK bdev Controller"
-
+#ifndef MODEL_NUMBER
+ #define MODEL_NUMBER "SPDK bdev Controller"
+#endif /* MODEL_NUMBER */
/*
* Report the SPDK version as the firmware revision.
* SPDK_VERSION_STRING won't fit into FR (only 8 bytes), so try to fit the
most important parts.
diff --git a/mk/spdk.common.mk b/mk/spdk.common.mk
index 3612551c..ff3e69cd 100644
--- a/mk/spdk.common.mk
+++ b/mk/spdk.common.mk
@@ -187,6 +187,7 @@ COMMON_CFLAGS += -pthread
LDFLAGS += -pthread
CFLAGS += $(COMMON_CFLAGS) -Wno-pointer-sign -Wstrict-prototypes
-Wold-style-definition -std=gnu99
+CFLAGS += -DMODEL_NUMBER=\"SPDK\ bdev\ Controller\"
CXXFLAGS += $(COMMON_CFLAGS) -std=c++0x
SYS_LIBS += -lrt
I will be happy to hear what do you think about such modification.
Best Regards,
Gregory
2 years, 8 months
