December 2019 - SPDK - Ml01.01.Org

by Zawadzki, Tomasz

Hello everyone, Summary: With this message I wanted to update SPDK community on state of VPP socket abstraction as of SPDK 19.07 release. At this time there does not seem to be a clear efficiency improvements with VPP. There is no further work planned on SPDK and VPP integration. Details: As some of you may remember, SPDK 18.04 release introduced support for alternative socket types. Along with that release, Vector Packet Processing (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by expanding socket abstraction to use VPP Communications Library (VCL). TCP/IP stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early stages back then and has seen improvements throughout the last year. To better use VPP capabilities, following fruitful collaboration with VPP team, in SPDK 19.07, this implementation was changed from VCL to VPP Session API from VPP 19.04.2. VPP socket abstraction has met some challenges due to inherent design of both projects, in particular related to running separate processes and memory copies. Seeing improvements from original implementation was encouraging, yet measuring against posix socket abstraction (taking into consideration entire system, i.e. both processes), results are comparable. In other words, at this time there does not seem to be a clear benefit of either socket abstraction from standpoint of CPU efficiency or IOPS. With this message I just wanted to update SPDK community on state of socket abstraction layers as of SPDK 19.07 release. Each SPDK release always brings improvements to the abstraction and its implementations, with exciting work on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK 20.01. However there is no active involvement at this point around VPP implementation of socket abstraction in SPDK. Contributions in this area are always welcome. In case you're interested in implementing further enhancements of VPP and SPDK integration feel free to reply, or to use one of the many SPDK community communications channels<https://spdk.io/community/>. Thanks, Tomek

2 months

6
12
0 / 0

spdk_thread_{exit,destroy} safety

by Andrey Kuzmin

Hi guys, I'm using spdk_thread_create to provide single-threaded context for the operations that otherwise would require locking. Everything runs nice up to exiting/destroying such a thread. In particular, after calling spdk_thread_exit I'm witnessing thread being unable to process pending messages (such as _spdk_put_io_channel which are generated by spdk_put_io_channel issued on the exiting thread before calling spdk_thread_exit), and destroying thread at this doesn't seem safe. Is there any recommended procedure to be followed before calling thread_exit so that the thread consumes all outstanding messages and is safe to destroy? I would assume something like while (spdk_thread_poll(thread) > 0); spdk_thread_exit(thread); spdk_thread_destroy(thread); should do the trick, but I'd prefer to rely on a recommended procedure. Thanks, Andrey

6 months, 1 week

4
6
0 / 0

Gerrit code: sock_ut is reporting error

by anil.veerabhadrappa＠broadcom.com

Hi, I am trying to submit a couple of patches to fix broken FC NVMe compilation. However with the latest Gerrit code, unittest,sh (sock_ut) is failing and blocking my submission. Is anybody else also hitting this issue?. Thanks! --- snip --- spdk/test/unit/lib/sock/sock.c/sock_ut ==6027== CUnit - A unit testing framework for C - Version 2.1-3 http://cunit.sourceforge.net/ Suite: sock Test: posix_sock ...passed Test: ut_sock ...passed Test: posix_sock_group ...passed Test: ut_sock_group ...passed Test: posix_sock_group_fairness ...passed Test: posix_sock_close ...==6027== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s) ==6027== at 0x58C3C40: __sendmsg_nocancel (in /usr/lib64/libpthread-2.17.so) ==6027== by 0x402F1B: _sock_flush (posix.c:496) ==6027== by 0x40325F: spdk_posix_sock_group_impl_poll (posix.c:824) ==6027== by 0x40427D: spdk_sock_group_impl_poll_count (sock.c:507) ==6027== by 0x40427D: spdk_sock_group_poll_count (sock.c:541) ==6027== by 0x404E46: _sock_close (sock_ut.c:787) ==6027== by 0x404E46: posix_sock_close (sock_ut.c:810) ==6027== by 0x4E3D120: run_single_test (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3CD59: run_single_suite (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3AC84: CU_run_all_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3EE9F: basic_run_all_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3EC41: CU_basic_run_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x401732: main (sock_ut.c:842) ==6027== Address 0x1ffefff920 is on thread 1's stack ==6027== in frame #4, created by posix_sock_close (sock_ut.c:809) ==6027== ==6027== Syscall param sendmsg(msg.msg_iov[1]) points to uninitialised byte(s) ==6027== at 0x58C3C40: __sendmsg_nocancel (in /usr/lib64/libpthread-2.17.so) ==6027== by 0x402F1B: _sock_flush (posix.c:496) ==6027== by 0x40325F: spdk_posix_sock_group_impl_poll (posix.c:824) ==6027== by 0x40427D: spdk_sock_group_impl_poll_count (sock.c:507) ==6027== by 0x40427D: spdk_sock_group_poll_count (sock.c:541) ==6027== by 0x404E46: _sock_close (sock_ut.c:787) ==6027== by 0x404E46: posix_sock_close (sock_ut.c:810) ==6027== by 0x4E3D120: run_single_test (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3CD59: run_single_suite (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3AC84: CU_run_all_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3EE9F: basic_run_all_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x4E3EC41: CU_basic_run_tests (in /usr/local/lib/libcunit.so.1.0.1) ==6027== by 0x401732: main (sock_ut.c:842) ==6027== Address 0x1ffefff920 is on thread 1's stack ==6027== in frame #4, created by posix_sock_close (sock_ut.c:809) ==6027== passed Run Summary: Type Total Ran Passed Failed Inactive suites 1 1 n/a 0 0 tests 6 6 6 0 0 asserts 213 213 213 0 n/a Elapsed time = 0.050 seconds ==6027== ==6027== HEAP SUMMARY: ==6027== in use at exit: 0 bytes in 0 blocks ==6027== total heap usage: 63 allocs, 63 frees, 6,555 bytes allocated ==6027== ==6027== All heap blocks were freed -- no leaks are possible ==6027== ==6027== For counts of detected and suppressed errors, rerun with: -v ==6027== Use --track-origins=yes to see where uninitialised values come from ==6027== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0) #

7 months, 2 weeks

2
2
0 / 0

Lvol sync issues running SPDK on separate hosts connected via dual port drives

by jasonm＠ami.com

I'm trying to see logical volumes created by SPDK_A on SPDK_B, but the only way I can see them is if I reboot the NODE_B system running SPDK_B. What needs to be done in order to sync the logical volumes on a namespace between two SPDKs running on separate hosts (namespace sharing)? Is this even possible? My system setup is described below. I have confirmed namespace sharing works with the Linux kernel driver (not LVS just namespace). SPDK version is 19.04.00, Kernel 4.15.0-29, Distro Ubuntu 18.04.1 LTS. DUAL PORT CHASSIS - NODE_A --- SPDK_A (create volume store, create volumes) - NODE_B --- SPDK_B (must be rebooted to see logical volumes created)

7 months, 2 weeks

4
4
0 / 0

rdma-core

by Jonathan Richardson

Hi, We've been using OFED components (eg- libibverbs 1.2, librdmacm 1.1, etc) from www.openfabrics.org/downloads. These packages have been deprecated and replaced by rdma-core at https://github.com/linux-rdma/rdma-core. It looks like spdk is still using packages based on the openfabrics. I just want to stay in sync with what spdk is doing. Is there any plan to upgrade to rdma-core? Thanks.

7 months, 2 weeks

2
3
0 / 0

Preparation for SPDK 19.10.1 release

by Zawadzki, Tomasz

Hello all, Preparation for SPDK 19.10.1 release have started, goal is for it to contain fixes and improvements merged since release of SPDK 19.10. In order for patches to be backported to 19.10.x branch, they should be present in SPDK master first. Please mark any important patches with hashtag "19.10.1" or reply to this thread. Link for list of backporting candidates: https://review.gerrithub.io/#/q/status:merged+hashtag:"19.10.1" Thanks, Tomek

7 months, 3 weeks

3
2
0 / 0

Re: New core maintainer - Alexey Marchuk

by Luse, Paul E

Congrats Alexey!! Glad to have you on board! -Paul On 12/10/19, 4:51 PM, "Harris, James R" <james.r.harris(a)intel.com> wrote: The SPDK project has a team of core maintainers who are responsible for providing technical oversight for the SPDK project, including final review and merging of patches into the SPDK code base. As the SPDK project continues to grow, the core maintainer team also needs to grow by adding contributors who are already demonstrating technical skill and leadership within the project. So I am pleased to announce that we are adding Alexey Marchuk to the core maintainer team! Alexey has been a key contributor to SPDK with a special focus on the NVMe-oF target (including RDMA and TCP transports) and DIF. The core maintainers have increasingly relied on Alexey for his expert reviews as the SPDK NVMe-oF target gains broader adoption in the industry. Alexey has been active in resolving GitHub issues and is prominent in helping users on the mailing list and Slack. He is from Mellanox and based in Nizhniy Novgorod, Russia. Thanks, Jim Harris (representing the core maintainer team) Please visit Core Maintainers<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spdk...> for further details on the core maintainer team and their responsibilities. _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

7 months, 3 weeks

3
2
0 / 0

New core maintainer - Alexey Marchuk

by Harris, James R

The SPDK project has a team of core maintainers who are responsible for providing technical oversight for the SPDK project, including final review and merging of patches into the SPDK code base. As the SPDK project continues to grow, the core maintainer team also needs to grow by adding contributors who are already demonstrating technical skill and leadership within the project. So I am pleased to announce that we are adding Alexey Marchuk to the core maintainer team! Alexey has been a key contributor to SPDK with a special focus on the NVMe-oF target (including RDMA and TCP transports) and DIF. The core maintainers have increasingly relied on Alexey for his expert reviews as the SPDK NVMe-oF target gains broader adoption in the industry. Alexey has been active in resolving GitHub issues and is prominent in helping users on the mailing list and Slack. He is from Mellanox and based in Nizhniy Novgorod, Russia. Thanks, Jim Harris (representing the core maintainer team) Please visit Core Maintainers<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spdk...> for further details on the core maintainer team and their responsibilities.

7 months, 3 weeks

1
0
0 / 0

In spdk 19.10, Is there proper way to use spdk_thread_create with g_new_thread_fn == NULL?

by sejun.kwon＠samsung.com

Hello, All. Lately, I updated the spdk version from 19.01 to spdk 19.10. And I noticed that spdk_thread_create is slightly differnet. The scenario I want to implement is as below. 1) spdk_thread_lib_init -> reactor_start 2) (In another thread, ) spdk_thread_create, and this spdk_thread needs to be belong to the thread which is not a reactor (but to while loop) In 19.01, it is okay, but in 19.10 if I use spdk_thread_create, that spdk_thread is registered to one of reactors with given cpu_mask because g_new_thread_fn is registered in reactor_start step. If given cpu_mask does not contains cpus of reactor, that spdk_thread_create operation is failed. So, I put g_new_thread_fn == NULL between 1), 2) (because reactor_start() inits g_new_thread_fn as its own function) and it works in my environment, but, I'm not sure this is safe way to register thread without reactor. Please let me know the way how to create just thread not to a reactor, even after initializing reactors. Best, Sejun Kwon.

7 months, 3 weeks

2
2
0 / 0

nvmeof to localhost will hung forever

by peng yu

Below is the steps to reproduce the issue: (1) run a spdk applicatoin, e.g.: sudo ./app/spdk_tgt/spdk_tgt (2) run the nvmeof target part commands: sudo ./scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -p 8 -c 8192 sudo ./scripts/rpc.py bdev_malloc_create -b Malloc0 512 512 sudo ./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1 sudo ./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0 sudo ./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t tcp -a 127.0.0.1 -s 4420 (3) run the nvmeof initiator part command: sudo ./scripts/rpc.py bdev_nvme_attach_controller -b Nvme0 -t tcp -a 127.0.0.1 -f IPv4 -s 4420 -n nqn.2016-06.io.spdk:cnode1 The bdev_nvme_attach_controller command will hung forever. I found the problem is in the nvme_tcp_qpair_icreq_send function, the spdk will stuck on below code: while (tqpair->state == NVME_TCP_QPAIR_STATE_INVALID) { nvme_tcp_qpair_process_completions(&tqpair->qpair, 0); } The while loop won't finish. The nvme_tcp_qpair_process_completions function will try to receive response from the target. The target is the same spdk application, and as the application is spinning in the above while loop, the nvmeof target part code doesn't have a chance to send a response. Is it possible to use a poller to replace the while loop? We could add a callback function, and let the poller call it when the tqpair->state is not NVME_TCP_QPAIR_STATE_INVALID. Does it make sense?

8 months

2
6
0 / 0

2020

2019

2018

2017

2016

SPDK December 2019