Reporting intermittent test failures
by Harris, James R
Hi all,
I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue.
Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked.
Thanks,
-Jim
2 months, 1 week
SPDK socket abstraction layer
by Zawadzki, Tomasz
Hello everyone,
Summary:
With this message I wanted to update SPDK community on state of VPP socket abstraction as of SPDK 19.07 release.
At this time there does not seem to be a clear efficiency improvements with VPP. There is no further work planned on SPDK and VPP integration.
Details:
As some of you may remember, SPDK 18.04 release introduced support for alternative socket types. Along with that release, Vector Packet Processing (VPP)<https://wiki.fd.io/view/VPP> 18.01 was integrated with SPDK, by expanding socket abstraction to use VPP Communications Library (VCL). TCP/IP stack in VPP<https://wiki.fd.io/view/VPP/HostStack> was in early stages back then and has seen improvements throughout the last year.
To better use VPP capabilities, following fruitful collaboration with VPP team, in SPDK 19.07, this implementation was changed from VCL to VPP Session API from VPP 19.04.2.
VPP socket abstraction has met some challenges due to inherent design of both projects, in particular related to running separate processes and memory copies.
Seeing improvements from original implementation was encouraging, yet measuring against posix socket abstraction (taking into consideration entire system, i.e. both processes), results are comparable. In other words, at this time there does not seem to be a clear benefit of either socket abstraction from standpoint of CPU efficiency or IOPS.
With this message I just wanted to update SPDK community on state of socket abstraction layers as of SPDK 19.07 release. Each SPDK release always brings improvements to the abstraction and its implementations, with exciting work on more efficient use of kernel TCP stack - changes in SPDK 19.10 and SPDK 20.01.
However there is no active involvement at this point around VPP implementation of socket abstraction in SPDK. Contributions in this area are always welcome. In case you're interested in implementing further enhancements of VPP and SPDK integration feel free to reply, or to use one of the many SPDK community communications channels<https://spdk.io/community/>.
Thanks,
Tomek
7 months, 3 weeks
add SP/SC message ring into the thread
by cranechu＠gmail.com
Hi, All,
In current SPDK thread, it has a single message ring, which is configured as MP/SC. It does work for both SP and MP rings, but it also introduces high overhead even when the application uses SP/SC only. For the example of the test bdevperf on one Malloc bdev, the cost of ring enqueue is as high as 15% of total CPU time.
My mitigation way is to add another SP/SC message ring into the thread. When one thread receives its first message, it reserves this SP/SC message ring to the first provider. Messages from other providers are still sent to the existed MP/SC message ring.
In this way, applications who take SP/SC into design considerations can get the benefits as expected. In the above example, we observe 15% performance improvement.
Here is the code diff: https://github.com/cranechu/spdk/commit/2704ab4902b2b7743a9b269350a9e0f95...
Any comments are welcome.
Thanks,
-Crane
1 year, 2 months
[Release] 19.10: Zoned bdev API, Opal bdev, NVMe character devices
by Zawadzki, Tomasz
On behalf of the SPDK community I'm pleased to announce the release of SPDK 19.10!
This release contains the following new features:
- Zoned bdev API: Zoned bdev is an extension of the bdev interface. Generic concept comes from ATA/SCSI and is also being worked as an NVMe TP. It will allow to use existing bdev infrastructure for new type zoned of devices.
- Opal bdev: A new opal bdev has been added to support management of NVMe self-encrypting drives through the Opal specification. This feature is considered as experimental.
- NVMe character device: Added NVMe character device implementation based on CUSE library. It allows creation of NVMe device nodes in Linux kernel for the controller and namespaces. This feature is considered as experimental.
- Blobfs mount via RPC: Added mounting blobfs to path on host using RPC.
- OCF: Added support for OCF v19.06. Added write-only, write-invalidate and write-around mode support.
- DPDK: Added support for DPDK 19.08.
- RPC names: Names of RPC were streamlined between different components. Previous RPC names are available, but will be deprecated.
The full changelog for this release is available at:
https://github.com/spdk/spdk/releases/tag/v19.10
This release contains 845 commits from 44 authors with over 44k lines of code changed.
We'd especially like to recognize all of our first time contributors:
Benjamin Saunders
Kay Bouché
Marcin Dzięgielewski
Michael Haeuptle
Michal Ben Haim
Tom Nabarro
Yang Fan
Yan Wang
Yao Liu
Thanks to everyone for your contributions, participation, and effort!
Thanks,
Tomek
1 year, 2 months
Re: Reporting intermittent test failures
by Harris, James R
The nvmf_tcp failure there looks Jenkins related. Karol/Seth - how do you think we should report these?
-Jim
On 10/29/19, 7:22 AM, "Wawryk, MaciejX" <maciejx.wawryk(a)intel.com> wrote:
Hi Jim,
Sometimes I have random fails like this:
https://dqtibwqq6s6ux.cloudfront.net/public_build/autotest-per-patch_4811...
https://10.102.17.104:8080/job/NVMeOF_tests/job/nvmf_autotest/41183/console
should I create issue for this random fails ?
-----Original Message-----
From: Harris, James R [mailto:james.r.harris@intel.com]
Sent: Friday, October 25, 2019 10:35 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Reporting intermittent test failures
Hi all,
I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue.
Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked.
Thanks,
-Jim
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
1 year, 2 months
Best practices on driver binding for SPDK in production environments
by Lance Hartmann ORACLE
This email to the SPDK list is a follow-on to a brief discussion held during a recent SPDK community meeting (Tue Jun 26 UTC 15:00).
Lifted and edited from the Trello agenda item (https://trello.com/c/U291IBYx/91-best-practices-on-driver-binding-for-spd... <https://trello.com/c/U291IBYx/91-best-practices-on-driver-binding-for-spd...>):
During development many (most?) people rely on the run of SPDK's scripts/setup.sh to perform a number of initializations, among them the unbinding of the Linux kernel nvme driver from NVMe controllers targeted for use by the SPDK and then binding them to either uio_pci_generic or vfio-pci. This script is applicable for development environments, but not targeted for use in productions systems employing the SPDK.
I'd like to confer with my fellow SPDK community members on ideas, suggestions and best practices for handling this driver unbinding/binding. I wrote some udev rules along with updates to some other Linux system conf files for automatically loading either the uio_pci_generic or vfio-pci modules. I also had to update my initramfs so that when the system comes all the way up, the desired NVMe controllers are already bound to the needed driver for SPDK operation. And, as a bonus, it should "just work" when a hotplug occurs as well. However, there may be additional considerations I might have overlooked on which I'd appreciate input. Further, there's the matter of how and whether to semi-automate this configuration via some kind of script and how that might vary according to Linux distro to say nothing of the determination of employing uio_pci_generic vs vfio-pci.
And, now some details:
1. I performed this on an Oracle Linux (OL) distro. I’m currently unaware how and what configuration files might be different depending on the distro. Oracle Linux is RedHat-compatible, so I’m confident my implementation should run similarly on RedHat-based systems, but I’ve yet to delve into other distro’s like Debian, SuSE, etc.
2. In preparation to writing my own udev rules, I unbound a specific NVMe controller from the Linux nvme driver by hand. Then, in another window I launched: "udevadm monitor -k -p” so that I could observe the usual udev events when a NVMe controller is bound to the nvme driver. On my system, I observed four (4) udev kernel events (abbreviated/edited output to avoid this become excessively long):
(Event 1)
KERNEL[382128.187273] add /devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0 (nvme)
ACTION=add
DEVNAME=/dev/nvme0
…
SUBSYSTEM=nvme
(Event 2)
KERNEL[382128.244658] bind /devices/pci0000:00/0000:00:02.2/0000:30:00.0 (pci)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:30:00.0
DRIVER=nvme
…
SUBSYSTEM=pci
(Event 3)
KERNEL[382130.697832] add /devices/virtual/bdi/259:0 (bdi)
ACTION=add
DEVPATH=/devices/virtual/bdi/259:0
...
SUBSYSTEM=bdi
(Event 4)
KERNEL[382130.698192] add /devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0/nvme0n1 (block)
ACTION=add
DEVNAME=/dev/nvme0n1
DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:30:00.0/nvme/nvme0/nvme0n1
DEVTYPE=disk
...
SUBSYSTEM=block
3. My udev rule triggers on (Event 2) above: the bind action. Upon this action, my udev rule appends operations to the special udev RUN variable such that udev will essentially mirror that which is done in the SPDK’s scripts/setup.sh for unbinding from the nvme driver and binding to, in my case, the vfio-pci driver.
4. With my new udev rules in place, I was successful getting specific NVMe controllers (based on bus-device-function) to unbind from the Linux nvme driver and bind to vfio-pci. However, I made a couple of observations in the kernel log (dmesg). In particular, I was drawn to the following for an NVMe controller at BDF: 0000:40:00.0 for which I had a udev rule to unbind from nvme and bind to vfio-pci:
[ 35.534279] nvme nvme1: pci function 0000:40:00.0
[ 37.964945] nvme nvme1: failed to mark controller live
[ 37.964947] nvme nvme1: Removing after probe failure status: 0
One theory I have for the above is that my udev RUN rule was invoked while the nvme driver’s probe() was still running on this controller, and perhaps the unbind request came in before the probe() completed hence this “name1: failed to mark controller live”. This has left lingering in my mind that maybe instead of triggering on (Event 2) when the bind occurs, that perhaps I should instead try to derive a trigger on the “last" udev event, an “add”, where the NVMe namespace’s are instantiated. Of course, I’d need to know ahead of time just how many namespaces exist on that controller if I were to do that so I’d trigger on the last one. I’m wondering if that may help to avoid what looks like a complaint during the middle of probe() of that particular controller. Then, again, maybe I can just safely ignore that and not worry about it at all? Thoughts?
I discovered another issue during this experimentation that is somewhat tangential to this task, but I’ll write a separate email on that topic.
thanks for any feedback,
--
Lance Hartmann
lance.hartmann(a)oracle.com
1 year, 2 months
About SPDK + Kubernetes
by Yibo Cai
Hello,
This is Yibo Cai from Arm. We are interested in leveraging SPDK to improve storage performance in Container Cloud.
At SPDK Beijing summit, we were glad to hear that Intel is developing similar features such as SPDK CSI in Kubernetes. We would like to know current status of the project and seek for possible collaboration opportunities. Any information is appreciated. Thanks.
Yibo
1 year, 2 months
Preparation for SPDK 19.10 release
by Zawadzki, Tomasz
Hello all,
The merge window for SPDK 19.10 release will close by October 24th.
Please ensure all patches you believe should be included in the release
are merged to master branch by this date.
You can do it by adding a hashtag '19.10' in GerritHub on those patches.
The current set of patches that are tagged and need to be reviewed can be seen here:
https://review.gerrithub.io/q/hashtag:%252219.10%2522+(status:open)
Starting with this release, process used so far for Maintenance releases (ex. 19.04.1)
will be applied to main releases. On October 24th new branch 'v19.10.x' will be created,
and a patch on it will be tagged as release candidate.
Then, on October 31th, a formal release will take place tagging the last patch
on the branch as SPDK 19.10.
Between release candidate and formal release,
only critical fixes shall be backported to the 'v19.10.x' branch.
Development can continue without disruptions on 'master' branch.
Thanks,
Tomek
1 year, 2 months
would spdk accept a linear vbdev module
by peng yu
I'd like to implement a linear vbdev module, like the linear target in
the linux kernel device mapper. MATSUMOTO SHUUHEI provided a very
useful email thread which discussed the same thing before:
https://lists.01.org/hyperkitty/list/spdk@lists.01.org/thread/GZQCP6HBLRW...
Some questions were asked in that email thread. I'd like to share my
answer here:
Q: Why a linear (or concatenate) vbdev is required
A: I hope the lvolstore could be extended. Currently, I could create a
thin provision lvs from a lvolstore. If I created too many snapshots,
the lvolstore might be out of space. So I want to find a method to
extend the lvolstore. A linear vbdev is a possible solution. Assume I
create a lvolstore from a linear vbdev, when I find the lvolstore will
be out of space, I could do following steps:
1 stop the spdk application
2 start the application again
3 re-create the linear vbdev, append a new physical device on it
4 re-create the lvolstore
Maybe we could let lvolstore support online extend in the further
(e.g. suspend lvolstore, let it release the underlying bdev, re-create
the linear bdev, and resume lvolstore).
Q: should I use raid0 instead of linear bdev
A: no, although raid0 could aggregate multiple bdev, it could not be
extended. If we add a new bdev to a raid0, the data layout will be
changed completed. So A linear bdev is what I required.
Q: how to handle bdev failure
A: a single bdev failure will be trade as the how linear vbdev
failure.
Q: should we split IO to multiple underlying vbdev
A: no, we could avoid to split IO. Below is an example:
bdev A:
total_size: 20k
io_boundary: 4k
bdev B:
total_size: 35k:
io_boundary: 8k
we use the max io_boundary of the underlying bdev's io_boundary, in
our example, it is 8k. Then we only use the first 16K of bdev A, and
the first 32k of bdev B. The linear bdev size is 16K+32K=48K. It will
waste some space, but we could avoid split IO.
Q: should we use (start, length) pair as parameter to create the
linear bdev
A: no. My current idea is group multiple bdev to a "pool". And let the
lvolstore use this "pool". Then create any size lvol from the
lvolstore. So the linear module don't need to do too many things. It
just concatenate all the underlying bdevs.
Is it acceptable by spdk upstream? Or do you have any suggestion?
1 year, 2 months
