threads
by Trevor Kramer
Is it allowed to connect to a nvme in one thread (spdk_nvme_connect) and
use the controller handle returned in different threads to allocate and use
qpairs (spdk_nvme_ctrlr_alloc_io_qpair)? Or does everything have to be done
in a single thread? I'm seeing intermittent buffer corruption and I'm
wondering if this could be the cause?
Thanks,
Trevor
3 weeks, 6 days
[Release] 20.04: SPDK Top, IDXD, NVMe qpair groups
by Zawadzki, Tomasz
On behalf of the SPDK community I'm pleased to announce the release of SPDK 20.04!
This release contains the following new features:
- SPDK Top: Added application to allow users to monitor resource consumption by a running SPDK application.
- NVMe qpair groups: Added API to allow for pooling of NVMe qpairs across a single entity.
- OCF: Added support for OCF v20.03.
- Crypto bdev: Added support for AES_XTS for the QAT polled mode driver.
- IDXD: Added support for IDXD as an accel plug-in module allowing for use with the generic accel framework API. This feature is considered experimental.
Note: Legacy INI style configuration for SPDK applications has been deprecated and will be removed in a future release. Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration.
The full changelog for this release is available at:
https://github.com/spdk/spdk/releases/tag/v20.04
This release contains 989 commits from 43 authors with over 61k lines of code changed.
We'd especially like to recognize all of our first time contributors:
Allen Zhu
Charles Machalow
Maciej Szczepaniak
Michał Berger
Sudheer Mogilappagari
Sylvain Didelot
Xiaohui Zhu
Thanks to everyone for your contributions, participation, and effort!
Thanks,
Tomek
4 weeks
Long IO latency with NVMe/TCP target due to large low watermark socket setting
by Wenhua Liu
Hi,
We opened an issue https://github.com/spdk/spdk/issues/1377 but did not see response, so I thought I should bring it here.
In SPDK NVMe/TCP target, when initializing the socket, the low watermark is set to sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 24 bytes. In our testing, some times there might be very small data packet (as small as 16 bytes) be sent to wire. After this, if there is no more data sent to the same socket, this small data packet won’t be received by NVMe/TCP controller qpair thread because the size hasn’t reached the low watermark. Because of this, the qpair thread is waiting for more data come in and the initiator is waiting for the IO request to be completed. Hence the delay happens.
As the minimum data that allows target to determine the PDU type is sizeof(struct spdk_nvme_tcp_common_pdu_hdr), which is 8 bytes, we changed low watermark setting as below. With the change, the problem was gone immediately.
*** tcp.c.new 2020-04-30 05:31:37.196499792 +0000
--- tcp.c 2020-04-29 19:48:25.857651523 +0000
*************** spdk_nvmf_tcp_qpair_sock_init(struct spd
*** 911,917 ****
int rc;
/* set low water mark */
! rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_common_pdu_hdr));
if (rc != 0) {
SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");
return rc;
--- 911,917 ----
int rc;
/* set low water mark */
! rc = spdk_sock_set_recvlowat(tqpair->sock, sizeof(struct spdk_nvme_tcp_c2h_data_hdr));
if (rc != 0) {
SPDK_ERRLOG("spdk_sock_set_recvlowat() failed\n");
return rc;
I would suggest SPDK have this change included in future release.
Thanks,
-Wenhua Liu
4 weeks, 1 day
Re: The parameter max_qpairs_per_ctrlr is misleading
by Wenhua Liu
Thanks for suggestion. Created issue https://github.com/spdk/spdk/issues/1378.
Regards,
-Wenhua
On 4/29/20, 9:38 PM, "Yang, Ziye" <ziye.yang(a)intel.com> wrote:
Hi Wenhua,
Could you submit a github issue in : https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.... ?
We just had a community meeting and discussed this issue. And we would change it in SPDK code and target for 20.07 release.
Best Regards
Ziye Yang
-----Original Message-----
From: Wenhua Liu <liuw(a)vmware.com>
Sent: Thursday, April 30, 2020 11:59 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] The parameter max_qpairs_per_ctrlr is misleading
When creating nvmf transport using command “rpc.py nvmf_create_transport”, there is a parameter max_qpairs_per_ctrlr. It means number of IO queues plus admin queue per controller. But when people talk about number of queues, usually it means number of IO queues. When someone creates nvmf transport, if he/she knows what max_qpairs_per_ctrlr mean and wants to have 8 IO queues, he/she will set this parameter to 9. If given 8, he/she will get only 7 IO queues.
In the worst case, if someone gives one to this parameter, he won’t get any IO queue !!!
I would suggest SPDK NVMF target changes the definition of this parameter and treat it as number of IO queues. Internally, SPDK should allow one extra qpair be created for admin queue.
Thanks,
-Wenhua Liu
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
4 weeks, 1 day
The parameter max_qpairs_per_ctrlr is misleading
by Wenhua Liu
When creating nvmf transport using command “rpc.py nvmf_create_transport”, there is a parameter max_qpairs_per_ctrlr. It means number of IO queues plus admin queue per controller. But when people talk about number of queues, usually it means number of IO queues. When someone creates nvmf transport, if he/she knows what max_qpairs_per_ctrlr mean and wants to have 8 IO queues, he/she will set this parameter to 9. If given 8, he/she will get only 7 IO queues.
In the worst case, if someone gives one to this parameter, he won’t get any IO queue !!!
I would suggest SPDK NVMF target changes the definition of this parameter and treat it as number of IO queues. Internally, SPDK should allow one extra qpair be created for admin queue.
Thanks,
-Wenhua Liu
4 weeks, 1 day
Preparation for SPDK 20.04 release
by Zawadzki, Tomasz
Hello all,
The merge window for SPDK 20.04 release will close by April 24th.
Please ensure all patches you believe should be included in the release are merged to master branch by this date.
You can do it by adding a hashtag '20.04' in Gerrit on those patches.
The current set of patches that are tagged and need to be reviewed can be seen here:
https://review.spdk.io/gerrit/q/hashtag:%2220.04%22+status:open
On April 24th new branch 'v20.04.x' will be created, and a patch on it will be tagged as release candidate.
Then, on April 30th, a formal release will take place tagging the last patch on the branch as SPDK 20.04.
Between release candidate and formal release, only critical fixes shall be backported to the 'v20.04.x' branch.
Development can continue without disruptions on 'master' branch.
PS. Due to rescheduling of US SPDK Summit, date for code freeze was moved to the original date.
Thanks,
Tomek
1 month
vhost is destroyed many times when starting vhost device
by Li Feng
Hi,
In rte_vhost_compat.c, spdk_extern_vhost_pre_msg_handler will call
destroy_device when some conditions meet.
I tested and found when a vhost-blk setup, the vhost device will be
destroyed 9 times.
And the bdev(no matter which type) will be affected. I'm using the aio
bdev, and the io_setup/io_destroy will be called 9 times when a disk
is inserted to a VM.
My concern is:
Is there any solution to reduce the call times(at least: disks nums * 9)?
The call will slow down the VM boot time if VM has multiple disks.
There are some related logs:
Starting SPDK v20.01-pre git sha1 93eecf627 / DPDK 19.11.0 initialization...
[ DPDK EAL parameters: spdk_tgt --no-shconf -c 1 -m 1
--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
--iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid2004865 ]
EAL: No available hugepages reported in hugepages-1048576kB
app.c: 645:spdk_app_start: *NOTICE*: Total cores available: 1
reactor.c: 346:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 174:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 193:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
rte_vhost_compat.c: 225:spdk_extern_vhost_pre_msg_handler: *ERROR*:
destroy device
rte_vhost_compat.c: 130:stop_device: *ERROR*: call stop_device
bdev_aio.c: 581:bdev_aio_group_destroy_cb: *ERROR*: call io_destroy
bdev_aio.c: 563:bdev_aio_group_create_cb: *ERROR*: call io_setup
Thanks,
Feng Li
--
The SmartX email address is only for business purpose. Any sent message
that is not related to the business is not authorized or permitted by
SmartX.
本邮箱为北京志凌海纳科技有限公司（SmartX）工作邮箱. 如本邮箱发出的邮件与工作无关,该邮件未得到本公司任何的明示或默示的授权.
1 month
Can I use spdk_pci_hook_device to implement a virtual NVMe device?
by oscar.huang＠microchip.com
I'd like to create a virtual NVMe device and use SPDK NVMe APIs to operate it. It looks like spdk_pci_hook_device() can add a virtual PCI device to SPDK PCI system:
void spdk_pci_hook_device(struct spdk_pci_driver *drv, struct spdk_pci_device *dev);
But the struct spdk_pci_device only contains the following callbacks:
int (*map_bar)(struct spdk_pci_device *dev, uint32_t bar,
void **mapped_addr, uint64_t *phys_addr, uint64_t *size);
int (*unmap_bar)(struct spdk_pci_device *dev, uint32_t bar,
void *addr);
int (*cfg_read)(struct spdk_pci_device *dev, void *value,
uint32_t len, uint32_t offset);
int (*cfg_write)(struct spdk_pci_device *dev, void *value,
uint32_t len, uint32_t offset);
void (*detach)(struct spdk_pci_device *dev);
It does not have mmio callbacks. When applications write a device memory, sometimes it needs to trigger some actions on the device. Without mmio callbacks, how to make a useful virtual device with spdk_pci_hook_device? Pls suggest. Much appreciated!
1 month
spdk_nvme_ns_cmd_read returning -ENOMEM
by trevordkramer＠gmail.com
Hello,
I have a system with 8 1GB hugepages configured at boot time split between 2 sockets. I reserve all 8 for spdk with
struct spdk_env_opts opts{};
spdk_env_opts_init(&opts);
opts.mem_size = 1024 * 8;
When I call spdk_nvme_ns_cmd_read I get -ENOMEM if my buffer (which I allocated with spdk_zmalloc) is too large. The call to spdk_zmalloc succeeds so the buffer is created. I don't understand what the buffer size would have to do with spdk_nvme_ns_cmd_read failing. Can anyone shed any light? I've looked though the code a bit and it looks like nvme_allocate_request must be failing but I don't understand why.
Thanks,
Trevor
1 month
SPDK v19.10/DPDK v19.05.0 eal_adjust_config issue, Illegal instruction causing application to abort
by asprasannavenkatesh＠gmail.com
Hi guys,
I am running SPDK version 19.10 and DPDK version 19.05.0 in my initiator application and I am running into an issue that seems to be sporadic in nature. I am running into this issue, where the application aborts saying Illegal instruction, with Signal 4, SIGILL (code: illegal operand). Issue seems to be more of a library related/EAL config related issue.
I am using the following DPDK EAL parameters in the application. I am also attaching the topmost information from backtrace, as the other things are company confidential.
Starting SPDK v19.10 / DPDK 19.05.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x1 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid40756 ]
*** Aborted at 1587486078 (Unix time, try 'date -d @1587486078') ***
*** Signal 4 (SIGILL) (0x7f2e40ec986d) received by PID 40756 (pthread TID 0x7f2e17fff700) (linux TID 40767) (maybe from PID 1089247341, UID 32558) (code: illegal operand), stack trace: ***
@ 00007f2e3fb2938f (unknown)
@ 00007f2e40ec986d eal_adjust_config -> <wokspace>/spdk/spdk-src/lib/json/json_write.c
@ 00007f2e40eb03a5 rte_eal_init -> <wokspace>/spdk/spdk-src/lib/json/json_write.c
@ 00007f2e40e906e6 spdk_env_init <wokspace>/spdk/spdk-src/lib/env_dpdk/init.c:456
I am trying to debug this issue. But adding -g and -ggdb compilation flags to the cmake list file makes the issue to vanish. Have any of you seen this kind of crash and how should we go about fixing this? Is adding -g and -gdb a workaround or is that the required fix. Is this a DPDK issue or just a gcc issue?
Also here is the system information that I am using
Initiator_system:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 56
On-line CPU(s) list: 0-55
Thread(s) per core: 2
Core(s) per socket: 14
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
Stepping: 1
CPU MHz: 1236.281
CPU max MHz: 3300.0000
CPU min MHz: 1200.0000
BogoMIPS: 4801.15
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-13,28-41
NUMA node1 CPU(s): 14-27,42-55
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts
Kindly keep us posted if you guys have any inputs.
Thanks and Regards,
Prasanna
1 month
