Order of callback and free_request during NVMe completions
by Lance Hartmann ORACLE
Hello,
I just joined the SPDK email list after dabbling with the SPDK for a few weeks. I looked through the SPDK email list’s archives and didn’t see any subjects focused on the topic of order of operations in NVMe completions, hence my inaugural posting to this list begins with a question about this. I did see and read through the thread in July 2016 (Re: Callback passed to spdk_nvme_ns_cmd_read not being called sometimes) but it didn’t touch on the specific question I have. Please accept my apologies in advance if I missed something else related to NVMe completions when scanning through the SPDK email archives.
In the SPDK NVMe driver I noted that the order of operations in processing completions is as follows:
1. If a callback was provided, issue the callback.
2. Invoke nvme_free_request().
As the caller’s callback info was saved in the nvme_request struct allocated during the I/O request, it appears logical that one would reference those respective callback fields and invoke the callback, before freeing the request struct that contains them. However, I’m curious to know if there’s any risk or possible concern for diverging from the programming model if the following sequence is done instead:
1. Save off the req->cb_fn and req->cb_arg in new temporary pointers (local variables on the stack of the completion function).
2. Invoke nvme_free_request() for the req.
3. If a callback was provided (as saved in the new temporary pointer), issue the callback via the temporary saved function pointer and with the temporary ptr to the callback arg.
I’m inquiring about this alteration for the scenario where in the I/O completion callback code path, the application may (and likely will) wish to issue a new I/O. However, if this is the first completion invoked after a maximum number of requests (based on opts->io_queue_requests) have already been submitted, then the I/O callback’s attempt to issue a new I/O will fail because there are no free requests available based on the current implementation of the completion code. If on the other hand, the completion code is altered to invoke nvme_free_request() before calling the applications’s I/O callback, a request entry will be available to that application as it was just freed, thus enabling it to issue a new I/O successfully.
I made such a change in my sandbox and this appeared to operate, but as I’m new to the SPDK NVMe driver I’m eager to solicit feedback from the seasoned developers here to learn if I’m overlooking something.
regards,
3 years, 7 months
Failed to remap 2 MB pages
by Kumaraparameshwaran Rathnavel
Hi All,
I have a system with 256GB. Initially I was giving 16GB for the SPDK to work. Now I want to increase it to 32GB. I have given 16384 2MB hugepages. But I am getting the error Failed to remap 2 MB pages. Before the map_all_hugepages there is a call find_physaddrs and that passes. So can I assume that physical memory is available and failure is due to some other reason. Has anyone else observed it and aware of why this is happening.
Thanking You,
Param.
3 years, 7 months
SPDK v17.07 release announcement
by Verkamp, Daniel
The SPDK team is happy to announce the release of SPDK v17.07.
Highlights of this release include:
- New configure script and DPDK submodule
- Userspace vhost-blk target
- GPT partition table bdev
- Hotplug fixes throughout the stack
See more details and download the code on the GitHub release page:
https://github.com/spdk/spdk/releases/tag/v17.07
Bug reports and feature requests may be sent to the mailing list or filed on the GitHub issue tracker:
https://github.com/spdk/spdk/issues
Thanks to everyone who contributed to this release:
Ben Walker
Changpeng Liu
Cunyin Chang
Daniel Verkamp
Dariusz Stojaczyk
Ed Rodriguez
Edward Yang
Fam Zheng
Fan Lu
GangCao
Isaac Otsiabah
Jim Harris
Jing Xia
John Kariuki
John Meneghini
Karol Latecki
Liang Yan
Lu Fan
Maciej Szwed
Paul Luse
Pawel Niedzwiecki
Pawel Wodkowski
Piotr Pelplinski
Prashanth Nayak
Roman Sudarikov
Seth Howell
Tomasz Zawadzki
Vishal Verma
Wenzhong Wu
Xiaofan Yang
Yuliya Navrotskaya
Ziye Yang
3 years, 7 months
