Does BlobFS Asynchronous API support multi thread writing?
by chen.zhenghua＠zte.com.cn
Hi everyone,
I simply tested the BlobFS Asynchronous API by using SPDK events framework to execute multi tasks, each task writes one file.
But it doesn't work, the spdk_file_write_async() reported an error when resizing the file size.
The call stack looks like this:
spdk_file_write_async() -> __readwrite() -> spdk_file_truncate_async() -> spdk_blob_resize()
The resize operation must be done in the metadata thread which invoked the spdk_fs_load(), so only the task dispatched to the metadata CPU core works.
That's to say only one thread can be used to write files. It's hard to use, and performance issues may arise.
Does anyone knows further more about this?
thanks very much
3 months, 3 weeks
Reporting intermittent test failures
by Harris, James R
Hi all,
I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue.
Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked.
Thanks,
-Jim
1 year, 9 months
Why Open-CAS-Linux 's performance even surpasses SPDK + OCF ?
by Bob Chen
1. SPDK + OCF (Using a HDD as AIO bdev, a NVMe SSD as NVMe bdev)
LD_PRELOAD=/home/sankuai/spdk/examples/bdev/fio_plugin/fio_plugin
/usr/local/bin/fio --name=SPDK_OCF --filename=AIO0 --rwmixwrite=30
--rw=randrw --bs=4kb --direct=1 --time_based --size=100% --norandommap
--iodepth=128 --numjobs=4 --ioengine=spdk_bdev --runtime=300
--group_reporting --randrepeat=0 --thread --random_distribution=zipf:1.1
--spdk_conf=/home/sankuai/spdk.conf
Results:
read: IOPS=2129
write: IOPS=912
2. Open-CAS-Linux (process IO by kernel)
/usr/local/bin/fio --name=OPEN_CAS_LINUX --filename=/dev/cas1-1
--rwmixwrite=30 --rw=randrw --bs=4kb --direct=1 --time_based --size=100%
--norandommap --iodepth=128 --numjobs=4 --ioengine=libaio --runtime=3600
--group_reporting --randrepeat=0 --random_distribution=zipf:1.1
Results:
read: IOPS=2950
write: IOPS=1267
1 year, 9 months
gerrit error
by sunshihao＠huawei.com
Hey guys，i have meet a problem when i want to submit a patch to spdk gerrit, please give me some advice to solve it.
when verifi an email it occurs that
"An error occurred
Error 422 (Unprocessable Entity): invalid token
Endpoint: /gerrit/config/server/email.confirm"
-------------------------------------------------------------------------------
when generate a new password it occurs that
"An error occurred
Error 500 (Server Error): Internal server error
Endpoint: /gerrit/accounts/self/password.http"
This two problem bother me for a long time.
1 year, 9 months
[Release] 20.10: NVMe-oF multipath, NVMe ZNS, iSCSI login redirection
by Zawadzki, Tomasz
On behalf of the SPDK community I'm pleased to announce the release of SPDK 20.10!
This release contains the following new features:
- NVMe-oF Multipath: The NVMe-oF target now supports Asymmetric Namespace Access (ANA) reporting to provide multipath to NVMe-oF initiator.
- NVMe ZNS: Zoned Namespace Command Set support has been added to NVMe driver. See https://spdk.io/doc/nvme__zns_8h.html for details.
- iSCSI Redirection: The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance and load balancing particularly in scale out iSCSI target system.
- Interrupt mode: Added support for interrupt mode. It allows modules or libraries to selectively register file descriptors that the spdk_thread can wait on. This feature is considered experimental and enabled in a very small subset of SPDK libraries and modules.
- Scheduler and governor: Added thread scheduler and CPU frequency governor frameworks used to rebalance load among CPU cores. This feature is considered experimental.
- Legacy configuration removed: SPDK applications and bdev modules no longer support legacy INI style configuration. Only JSON-RPC configuration is supported, see https://spdk.io/doc/jsonrpc.html for details.
The full changelog for this release is available at:
https://github.com/spdk/spdk/releases/tag/v20.10
This release contains 805 commits from 43 authors with over 47k lines of code changed.
We'd especially like to recognize all of our first time contributors:
Jimbo Lu
Krzysztof Karas
Niklas Cassel
Pawel Piatek
Rafal Stefanowski
Vasuki Manikarnike
Wang Shilong
Thanks to everyone for your contributions, participation, and effort!
Thanks,
Tomek
1 year, 9 months
After spdk_bdev_reset() call, IO operations a still permitted on the same channel
by ssunder＠xilinx.com
I have performed some experiments to understand the behavior of IO operations after spdk_bdev_reset(). I used hello_bdev example for this purpose and I concluded the bdev reset has no impact on IO operation issued on the same channel. I thought once the reset is issued, the bdev layer should reject all new bdev read/write requests including inflight requests but it is not happening.
Can someone explain what is the expectation from spdk_bdev_reset() and how it supposed to work ?
1 year, 9 months
nvmf: Async event support for discovery log change
by Madhu Adav M J
Hi SPDK community,
I have submitted a patch for adding async event support for discovery log
change in nvmf targets.
4870: nvmf: Async event support for discovery log change |
https://review.spdk.io/gerrit/c/spdk/spdk/+/4870 .
According to nvme fabrics spec 1.1, if the connect command specifies a
non-zero keep alive timer value and the discovery controller does not
support asynchronous events then we need to return Connect Invalid. Since
SPDK currently does not implement this check I have instead added support
for asynchronous events for discovery log changes. This allows a host with
persistent connection to the discovery subsystem to automatically be
notified if any new subsystems are available.
Please check the patch when possible and let me know if this is the right
approach.. Also let me know if any issues need to be fixed.
Thanks.
--
*-----------------------------------------------------------------------------------------*
*This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error, please notify the
system manager. This message contains confidential information and is
intended only for the individual named. If you are not the named addressee,
you should not disseminate, distribute or copy this email. Please notify
the sender immediately by email if you have received this email by mistake
and delete this email from your system. If you are not the intended
recipient, you are notified that disclosing, copying, distributing or
taking any action in reliance on the contents of this information is
strictly prohibited.*****
****
*Any views or opinions presented in this
email are solely those of the author and do not necessarily represent those
of the organization. Any information on shares, debentures or similar
instruments, recommended product pricing, valuations and the like are for
information purposes only. It is not meant to be an instruction or
recommendation, as the case may be, to buy or to sell securities, products,
services nor an offer to buy or sell securities, products or services
unless specifically stated to be so on behalf of the Flipkart group.
Employees of the Flipkart group of companies are expressly required not to
make defamatory statements and not to infringe or authorise any
infringement of copyright or any other legal right by email communications.
Any such communication is contrary to organizational policy and outside the
scope of the employment of the individual concerned. The organization will
not accept any liability in respect of such communication, and the employee
responsible will be personally liable for any damages or other liability
arising.*****
****
*Our organization accepts no liability for the
content of this email, or for the consequences of any actions taken on the
basis of the information *provided,* unless that information is
subsequently confirmed in writing. If you are not the intended recipient,
you are notified that disclosing, copying, distributing or taking any
action in reliance on the contents of this information is strictly
prohibited.*
_-----------------------------------------------------------------------------------------_
1 year, 9 months
How to get transport ID from spdk_bdev
by Niu, Yawei
Hi,
Is there any way to get transport ID (and ns id) of an existing spdk_bdev? I think that could be useful for the applications built upon bdev layer and want to know the transport ID and namespace ID where the bdev is located, but I failed to find such kind of exported interfaces.
I’m wondering if this controller information is deliberately being hidden behind bdev layer? In our use case, we just want to get the association of bdev and ctrlr/ns for configuration management purpose.
Thanks
-Niu
1 year, 9 months
[SPDK]: Save the Date! November 14: 2020 SPDK PMDK & Intel VTune™ Profiler PRC Online Virtual Forum
by Cao, Gang
Hi all,
Save the Date for the upcoming SPDK, PMDK & Intel VTune™ Profiler PRC Online Virtual Forum (PRC time, November 14th).
Registration link will be sent out later.
Use the Attached Outlook invite to Save the Date!
This will be a great opportunity to meet with the SPDK, PMDK and VTune user communities and listen to a new series of talks from early adopters and long term users. Topics will include case studies, feature deep dives and technical tutorials.
Hope to see you online!
Thanks,
Gang
1 year, 9 months
Re: For Dummies guide to setting up ramdisk NVMe/TCP target
by Harris, James R
Hi Eric,
SPDK will start polled-mode threads on each of the cores specified in your coremask. So for coremask 0xF0, it would start polled-mode threads on cores 4, 5, 6 and 7. The number of cores in your coremask really depends on your system and what you want to test - if you give the application more cores to run on, you'll be able to get better overall performance and throughput.
If you don't specify a coremask, it will just use core 0 only. That's fine for just getting something up and running to play around with.
-Jim
On 10/16/20, 7:13 AM, "Curtin, Eric" <Eric.Curtin(a)dell.com> wrote:
Thanks guys, I got it working. You were right I was not running as root. The other steps I had to perform in case anyone else ever gets curious (I used 127.0.0.1). The only think I don't understand is the coremasks I used 0xf0 and 0x10, I just guessed those based on random examples I saw online. How do I know what coremasks I want to pick?
mkdir -p /dev/hugepages
mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
build/bin/nvmf_tgt -m 0xf
In another terminal:
scripts/rpc.py nvmf_create_transport -t TCP
scripts/rpc.py bdev_malloc_create -b Malloc0 200 512
scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 127.0.0.1 -s 4420
build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:127.0.0.1 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -q 128 -o 4096 -w write -t 100 -c 0x10
Mise le meas/Regards,
Eric Curtin
Software Engineer
Ovens Campus,
Cork,
Ireland
Dell EMC
Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Thursday 15 October 2020 02:49
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
[EXTERNAL EMAIL]
Yeah, it also works on my virtualbox 6.1 with Win10/X86_64 host and Ubuntu 20.4 guest.
Regards,
Rui
-----Original Message-----
From: Luse, Paul E <paul.e.luse(a)intel.com>
Sent: Thursday, October 15, 2020 07:48
To: Storage Performance Development Kit <spdk(a)lists.01.org>; Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target
Hi Eric,
Yeah, run with sudo. I get the same thing when I run w/o, it's DPDK unable to get the PA with privilege.
Thx
Paul
From: Luse, Paul E <paul.e.luse(a)intel.com>
Date: Wednesday, October 14, 2020 at 4:43 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>, Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: Re: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target Hi Eric,
Bummer that's not working. It works OK for my with vbox 6.1 on a mac and ubuntu 1901 and the output says iova-mode=pa. Did you change any settings on your VM and try simple stuf like adding memory and/or running with one core (how many cores did you give your VM?). Also are you running privelaged?
Thx
Paul
peluse@pels:~/spdk$ sudo ./build/bin/nvmf_tgt
[2020-10-14 23:38:50.539538] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 23:38:50.540128] [ DPDK EAL parameters: [2020-10-14 23:38:50.540346] nvmf [2020-10-14 23:38:50.540556] --no-shconf [2020-10-14 23:38:50.540755] -c 0x1 [2020-10-14 23:38:50.540957] --log-level=lib.eal:6 [2020-10-14 23:38:50.541153] --log-level=lib.cryptodev:5 [2020-10-14 23:38:50.541342] --log-level=user1:6 [2020-10-14 23:38:50.541566] --iova-mode=pa [2020-10-14 23:38:50.541763] --base-virtaddr=0x200000000000 [2020-10-14 23:38:50.541957] --match-allocations [2020-10-14 23:38:50.542290] --file-prefix=spdk_pid918 [2020-10-14 23:38:50.542544] ]
EAL: No legacy callbacks, legacy socket not created
[2020-10-14 23:38:50.552172] app.c: 667:spdk_app_start: *NOTICE*: Total cores available: 1
From: Curtin, Eric <Eric.Curtin(a)dell.com>
Date: Wednesday, October 14, 2020 at 1:56 AM
To: Rui Chang <Rui.Chang(a)arm.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Re: For Dummies guide to setting up ramdisk NVMe/TCP target How would I get my CPU mask on Linux? I'm running on a Virtualbox VM. I also used 0xf as a CPU mask and this was my output:
./build/bin/nvmf_tgt -m 0xf
[2020-10-14 09:47:07.650399] Starting SPDK v20.10-pre git sha1 779a6bdf8 / DPDK 20.08.0 initialization...
[2020-10-14 09:47:07.651075] [ DPDK EAL parameters: [2020-10-14 09:47:07.651092] nvmf [2020-10-14 09:47:07.651126] --no-shconf [2020-10-14 09:47:07.651132] -c 0xf [2020-10-14 09:47:07.651138] --log-level=lib.eal:6 [2020-10-14 09:47:07.651144] --log-level=lib.cryptodev:5 [2020-10-14 09:47:07.651150] --log-level=user1:6 [2020-10-14 09:47:07.651160] --iova-mode=pa [2020-10-14 09:47:07.651166] --base-virtaddr=0x200000000000 [2020-10-14 09:47:07.651172] --match-allocations [2020-10-14 09:47:07.651181] --file-prefix=spdk_pid702834 [2020-10-14 09:47:07.651187] ]
EAL: FATAL: Cannot use IOVA as 'PA' since physical addresses are not available
EAL: Cannot use IOVA as 'PA' since physical addresses are not available
[2020-10-14 09:47:07.659984] init.c: 577:spdk_env_init: *ERROR*: Failed to initialize DPDK
[2020-10-14 09:47:07.659999] app.c: 535:app_setup_env: *ERROR*: Unable to initialize SPDK env
Regards,
Eric Curtin
Software Engineer
Ovens Campus,
Cork,
Ireland
Dell EMC
Next planned absences: NA
________________________________
From: Rui Chang <Rui.Chang(a)arm.com>
Sent: Wednesday 14 October 2020 08:50
To: Curtin, Eric <Eric.Curtin(a)dell.com>
Cc: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: RE: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target
[EXTERNAL EMAIL]
Hi Eric
Refer to this page for HOWTO:
https://spdk.io/doc/nvmf.html
You said you want to run on one machine, suppose your machine's IP is 8.8.8.6
1. Build your SPDK
git clone https://github.com/spdk/spdk spdk
cd spdk
git submodule update --init
./configure
make
1. Run:
Start target:
./build/bin/nvmf_tgt -m 0xf (-m gives the CPU mask. Normally you can make it on the same CPU socket as network card to get better performance)
On another console, run rpc comamnds to do configuration:
./scripts/rpc.py nvmf_create_transport -t TCP (there are lots of additional options you can use if you wish)
./scripts/rpc.py bdev_malloc_create -b Malloc0 200 512
./scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1
./scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 Malloc0
./scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 8.8.8.6 -s 4420 (suppose 8.8.8.6 is the IP you want the target to listen on)
Run io:
./build/examples/perf -r 'trtype:TCP adrfam:IPv4 traddr:8.8.8.6 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -q 128 -o 4096 -w write -t 100 -c 0xf0 (-c give the cpu mask for perf tool, you'd better make it different with nvmf_tgt to get better performance)
Regards,
Rui
-----Original Message-----
From: eric.curtin(a)dell.com<mailto:eric.curtin@dell.com> <eric.curtin(a)dell.com<mailto:eric.curtin@dell.com>>
Sent: Thursday, October 1, 2020 3:23 PM
To: spdk(a)lists.01.org<mailto:spdk@lists.01.org>
Subject: [SPDK] For Dummies guide to setting up ramdisk NVMe/TCP target
Hi Guys,
I'm just getting started with SPDK and wanted to create a fake ramdisk NVMe/TCP target block device, just so I could simulate NVMe/TCP traffic on a single Linux instance for development purposes.
Anybody know how to set up a machine like this for NVMe/TCP development?
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org<mailto:spdk@lists.01.org>
To unsubscribe send an email to spdk-leave(a)lists.01.org<mailto:spdk-leave@lists.01.org>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org
1 year, 9 months
