On Wed, Aug 1, 2018, 20:55 Harris, James R <james.r.harris(a)intel.com> wrote:
Your gut is not fooling you. I think doing this in the context of an
existing polled mode thread is a great idea. (But I’m glad that the
spdk_allocate_thread() approach was able to work for you!)
Yeah, it works nice, albeit with some (anticipated) performance
On a separate note, I didn't notice spdk_mempool to provide an expected
miltiple consumers/single producer flavor of the dpdk mempool_create (which
I've utilized for inter-thread messages), and from my experiments with
replacing spdk _pool_create with dpdk call (with F_SP_MC set) I saw a few
percent improvement in IOPS against an malloc bdev.
You could just always pick the master thread. This is the thread that
calls your module’s init() function.
That's not necessarily the most convenient option as one might then end up
with all ST contexts being run by the same master thread (or that init
thread might turn out to be just a typical external init thread that brings
things up and then goes sleeping until needed).
Ideally, I'd like to be able to pick up a pinned thread on a chosen core,
and attach to that one w/o any dependency on what thread has brought things
up (unless explicitly told to pick up that init thread).
You can call spdk_poller_register() from your init function to start
pollers – and pick a polling frequency that makes sense for the
rate of synchronization messages. You can also save spdk_get_thread() to a
global variable so that you can do spdk_thread_send_msg() calls to it from
other threads – or maybe you create a multi-producer/single-consumer ring
that other threads can enqueue to, and then the poller picks up those
messages, executes them, and sends a message back to the calling thread.
There are a lot of cases in SPDK currently where we start up pollers on a
specific thread at application start – i.e. RPC handling (lib/event/rpc.c)
and iSCSI acceptors (lib/iscsi/acceptor.c) immediately come to mind. The
virtio bdev driver (lib/bdev/virtio/bdev_virtio_scsi.c) also uses this
poller + ring concept to synchronize reset operations.
Thanks for the pointers, that's enough to chew upon.
Would this be sufficient for your use case?
*From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Andrey Kuzmin <
*Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Date: *Tuesday, July 31, 2018 at 2:31 AM
*To: *Storage Performance Development Kit <spdk(a)lists.01.org>
*Subject: *[SPDK] Single-threaded semantics in vbdev
Working on a vbdev that maintains some global state and underlying data
structures, I decided to explore an SPDKish alternative to the traditional
synchronization, that of the single-threaded semantics where global data
structures are being maintained by a dedicated thread, with requests being
served by message passing to/from that thread.
Coding such a thread against the spdk_allocate_thread() guidelines proved
to be quite straightforward, and I've got it up and running. At the same
time, I'm still unclear as to why I should be essentially duplicating the
features of the pinned SPDK thread, with my gut feeling being that the very
same single-threaded semantics could have been easily achieved by pinning
my execution context to some randomly or round-robin chose pinned thread of
the configured allowed cores mask.
If that gut feeling doesn't fool me, I will appreciate a pointer to any
docs and/or examples showing how one can pin an arbitrary execution context
inside a vbdev to an SPDK native thread (similar to app_start and
friends, I believe), and do polling on that thread/ messaging to/from
that thread similar to what a POSIX thread equipped with spdk thread
envelope could do.
Thanks in advance,
SPDK mailing list