nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization timed out in state 3
by Oza Oza
I have ported SPDK for ARMv8.
And DPDK is compiled with version: 16.11.1
init controller is failing.
root@ubuntu:/home/oza/SPDK/spdk#
odepth=128 --size=4G --readwrite=read --filename=0000.01.00.00/1 --bs=4096
--i
/home/oza/fio /home/oza/SPDK/spdk
EAL: pci driver is being registered 0x1nreadtest: (g=0): rw=read,
bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=spdk_fio, iodepth=128
fio-2.17-29-gf0ac1
Starting 1 process
Starting Intel(R) DPDK initialization ...
[ DPDK EAL parameters: fio -c 1 --file-prefix=spdk_pid6448
--base-virtaddr=0x1000000000 --proc-type=auto ]
EAL: Detected 8 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: cannot open /proc/self/numa_maps, consider that all memory is in
socket_id 0
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: using IOMMU type 1 (Type 1)
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=13
EAL: reg=0x2000 fd=13 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: Hotplug doesn't support vfio yet
spdk_fio_setup() is being called
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=16
EAL: reg=0x2000 fd=16 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: inside pci_vfio_write_config offset=4
nvme_ctrlr.c:1224:nvme_ctrlr_process_init: ***ERROR*** Initialization
timed out in state 3
nvme_ctrlr.c: 403:nvme_ctrlr_shutdown: ***ERROR*** did not shutdown within
5 seconds
EAL: Hotplug doesn't support vfio yet
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
EAL: vfio_group_fd=11 iommu_group_no=3 *vfio_dev_fd=18
EAL: reg=0x2000 fd=18 cap_offset=0x50
EAL: the msi-x bar number is 0 0x2000 0x200
EAL: Hotplug doesn't support vfio yet
EAL: Requested device 0000:01:00.0 cannot be used
spdk_nvme_probe() failed
Regards,
Oza.
4 years, 1 month
invalid usermode virtual address 0xffff8f600000
by Oza Oza
Hi,
Why do I get invalid user mode virtual address when I enabled DEBUG in
SPDK/DPDK ?
invalid usermode virtual address 0xffff8f600000
invalid usermode virtual address 0xffff0f800000
invalid usermode virtual address 0xffff0f400000
this is armv8.
Regards,
Oza.
4 years, 4 months
SPDK Dependencies
by Walker, Benjamin
Hi all,
There has been a lot of activity lately dealing with abstracting SPDK's dependencies to allow porting SPDK to additional platforms and frameworks. I think we'll end up talking about this extensively at the summit this week. The ability to use SPDK in a variety of environments is very important to us, so I look forward to the discussions. I wanted to write a basic summary of the current state of SPDK's dependencies for everyone to have, and so that those who are unable to attend the summit can provide feedback. I'll try to update this thread with the results of the discussions at the summit for everyone's benefit.
We've already eliminated all SPDK dependencies except for two:
1) POSIX
2) DPDK
SPDK already has a mechanism for abstracting away the "environment" it is running in. We do this by moving all calls to dependencies to the "env" library, whose interface is defined by include/spdk/env.h. The build system allows users to point to an alternate implementation of the env library if required. SPDK's default implementation of env is based on DPDK and runs in Linux and FreeBSD user space.
DPDK is a very large toolkit, of course, but until just recently SPDK only used DPDK's 'eal' module (environment abstraction library). The original goal of the SPDK env library was just to abstract all uses of DPDK's eal, but not all uses of POSIX APIs. We made some progress in this area, but there are a number of remaining calls as of this writing. Fortunately, the wider community has really come through. There are at least two pull requests out from two independent users that mostly finish the job. We'll work through those pull requests and try to get them merged shortly. The only part of the code that will continue to explicitly depend on DPDK will be the newly released vhost-scsi target because that relies on DPDK vhost infrastructure. This infrastructure is outside of DPDK's eal and is quite extensive. We feel this is acceptable because building vhost in SPDK is optional - you can just turn it off to eliminate the dependency.
There are a number of legitimate reasons to want to remove the DPDK dependency from SPDK. For instance, DPDK dictates a particular memory management model, requires a specific threading model, and performs PCI device management that may conflict with other libraries. However, DPDK provides tremendous value to SPDK in a number of areas, especially by implementing memory allocation suitable for DMA in user space. Memory management is a major challenge with all sorts of hidden traps and challenges, so we see significant value in using a production-tested library here. Re-implement the env without DPDK at your own risk!
Abstracting away our POSIX dependency is a similar but larger challenge. While we foresaw the need for people to replace DPDK, we didn't see any use case for dropping POSIX. We were definitely wrong though - people are using SPDK in all sorts of environments that we didn't predict, from embedded firmware to operating system kernels and beyond. So we'll need to address this as time goes on. I think this will result in further expansion of the 'env' library. Fortunately, we don't require too much from POSIX - mostly pthreads, the C standard library, and maybe a couple of other miscellaneous headers.
There are other less obvious dependencies in SPDK. For instance, it also depends on a relatively recent version of gcc or clang. This is mostly a practical concern and not so much by intention. Our continuous integration system (which I'll talk about at the summit - exciting developments here) currently consists of 4 machines, 3 of which are running Fedora Linux 25 while the other is running FreeBSD 11.0. These are very much up to date systems, so we tend not to catch bugs on older systems today. We also use cutting edge C features and tools, so we'll probably over time need to select a particular minimum version of the C standard that we're going to develop to.
I'm also concerned with the scope of the env library and I'd like to work as hard as possible to limit the size of the API. This is for two reasons - first and foremost to minimize the number of calls to these wrapper functions so that it doesn't impact performance. Performance really is everything and we can't sacrifice that. This isn't shaping up to be much of a problem so far because very few calls to our dependencies happen within the I/O path. Secondarily, I'd like to make it as straightforward as possible for users to re-implement the env. The best way to make it easy to do that is to keep the API small.
I look forward to the discussions at the summit this week. We'd love to hear about all of the different environments, platforms, and frameworks where SPDK is being used and how we can make it easier to integrate SPDK wherever it is valuable.
Thanks,
Ben
4 years, 4 months
Compile ERROR: ‘pci_device_list’ undeclared
by 邓林文
Hi all, there's a compile error when I tried to test the latest SPDK. Is that a code bug or a misconfigure in my environment?
Here is the log:
dpdk# make install T=x86_64-native-linuxapp-gcc DESTDIR=.
dpdk# cd ../spdk
spdk# ./configure --with-dpdk=../dpdk/x86_64-native-linuxapp-gcc
spdk# make
...
LIB libspdk_log_rpc.a
CC lib/env_dpdk/pci.o
In file included from /home/src/dpdk/x86_64-native-linuxapp-gcc/include/rte_pci.h:81:0,
from env_internal.h:55,
from pci.c:34:
pci.c: In function ‘spdk_pci_get_device’:
pci.c:173:22: error: ‘pci_device_list’ undeclared (first use in this function)
TAILQ_FOREACH(dev, &pci_device_list, next) {
^
pci.c:173:22: note: each undeclared identifier is reported only once for each function it appears in
/home/src/spdk/mk/spdk.common.mk:159: recipe for target 'pci.o' failed
make[2]: *** [pci.o] Error 1
/home/src/spdk/mk/spdk.subdirs.mk:35: recipe for target 'env_dpdk' failed
make[1]: *** [env_dpdk] Error 2
/home/src/spdk/mk/spdk.subdirs.mk:35: recipe for target 'lib' failed
make: *** [lib] Error 2
The codes are all latest:
dpdk commit eba33e87ad37626604be7186e746751f99691084
spdk commit 6c172475609120c293b265522c395c52fc5a3bb5
Regards,
Linwen Deng
4 years, 4 months
NVMe-oF Target Library
by Walker, Benjamin
Hi all,
I was hoping to start a bit of a design discussion about the future of the
NVMe-oF target library (lib/nvmf). The NVMe-oF target was originally created as
part of a skunkworks project and was very much an application. It wasn't
divided into a library and an app as it is today. Right before we released it,
I decided to attempt to break it up into a library and an application, but I
never really finished that task. I'd like to resume that work now, but let the
entire community weigh in on what the library looks like.
First, libraries in SPDK (most things that live in lib/) shouldn't enforce a
threading model. They should, as much as possible, be entirely passive C
libraries with as few dependencies as we can manage. Applications in SPDK
(things that live in app/), on the other hand, necessarily must choose a
particular threading model. We universally use our application/event framework
(lib/event) for apps, which spawns one thread per core, etc. We'll continue
this model for NVMe-oF where app/nvmf_tgt will be a full application with a
threading model dictated by the application/event framework, while lib/nvmf
will be a passive C library that will depend only on other passive C libraries.
I don't think this distinction is at all reality today, but let's work to make
it so.
The other major issue with the NVMe-oF target implementation is that it has
quite a few baked in assumptions about what the backing storage device looks
like. In particular, it was written assuming that it was talking directly to an
NVMe device (Direct mode), and the ability to route I/O to the bdev layer
(Virtual mode) was added much later and isn't entirely fleshed out yet. One of
these assumptions is that real NVMe devices don't benefit from multiple queues
- you can get the full performance from an NVMe device using just one queue
pair. That isn't necessarily true for bdevs, which may be arbitrarily
complex virtualized devices. Given that assumption, the NVMe-oF target
today only creates a single queue pair to the backing storage device and only
uses a single thread to route I/O to it. We're definitely going to need to
break that assumption.
The first discussion that I want to have is around what the high level concepts
should be. We clearly need to expose things like "subsystem", "queue
pair/connection", "namespace", and "port". We should probably have an object
that represents the entire target too, maybe "nvmf_tgt". However, in order to
separate the threading model from the library I think we'll need at least two
more concepts.
First, some thread has to be in charge of polling for new connections. We
typically refer to this as the "acceptor" thread today. Maybe the best way to
handle this is to add an "accept" function that takes the nvmf_tgt object as an
argument. This function can only be called one a single thread at a time and is
repeatedly called to discover new connections. I think the user will end up
passing a callback in to this function that will be called for each new
connection discovered.
Second, once a new connection is discovered, we need to hand it off to some
collection that a dedicated thread can poll. This collection of connections
would be tied specifically to that dedicated thread, but it wouldn't
necessarily be tied to a subsystem or a particular storage device. I don't
really know what to call this thing - right now I'm kind of thing "io_handler".
So the general flow for an application would be to construct a target, add
subsystems, namespaces, and ports as needed, and then poll the target for
incoming connections. For each new connection, the application would assign it
to an io_handler (using whatever algorithm it wanted) and then poll the
io_handlers to actually handle I/O on the connections. Does this seem like a
reasonable design at a very high level? Feedback is very much welcome and
encouraged.
If I don't hear back with a bunch of "you're wrong!" or "that's stupid!" type
replies over the next few days, the next step will be to write up a new header
file for the library that we can discuss in more detail.
Thanks,
Ben
4 years, 4 months
IRC: #spdk on FreeNode
by Walker, Benjamin
The SPDK community has created an IRC channel on FreeNode - #spdk -
where we'll all hang out. We'll use this channel for all real-time
discussion about the project including deep design discussions, day to
day operations, and Q&A. Eventually, we'd like to hold regular office
hours for questions, or even full status meetings in the channel. Come
join us and please participate!
Thanks,
Ben
4 years, 4 months
restricting SPDK/DPDK memory allcoation
by Oza Oza
Hi,
How do I ask SPDK/DPDK to restrict the iova and phys_addr allocation to
first 128 GB of its address space ?
In short I want to restrict
Function: vfio_type1_dma_map
dma_map.vaddr = ms[i].addr_64;
dma_map.iova = ms[i].phys_addr;
iova address which gets programmed in iommu restricted to first 128 GB of
space ?
I am using ARM64 port, but it should not have anything to do with
architecture anyway.
Regards,
Oza.
4 years, 4 months
Re: [SPDK] [spdk/spdk] Abstract and move rte_ functions into spdk functions that act ad pass-through functions in env_dpdk (#142)
by John Meneghini
Hi Ranjit.
I’m opening up this conversation to everyone by cc’ing the spdk mail list.
Yes, the upcoming changes we have will complete the separation of dpdk and spdk.
They will also abstract all of the posix dependencies (e.g. pthread_create becomes spdk_thread_create).
The question about how many components to refactor needs to be agreed upon.
At this point the feedback I’ve received from Ben and Daniel has been that they don’t want to convert all components at once. So I’ve been taking my pull requests apart into smaller pieces. Here is one of my older pull requests for reference.
https://github.com/spdk/spdk/pull/132/commits <https://github.com/spdk/spdk/pull/132/commits>
This can be compared with my current pull request:
SPDK EAL Abstractions <https://github.com/spdk/spdk/pull/150>
and with yours.
Abstract and move rte_ functions into spdk functions that act ad pass-through functions in env_dpdk <https://github.com/spdk/spdk/pull/142>
In terms of timeline, I’d like to complete these changes ASAP. If we can’t merge everything we have into master relatively soon (like this week) then I’d like to propose that a separate SPDK "platform_abstraction" branch be created (not a Github fork) so all of the environmental changes can be submitted there. This will allow contributors to develop these changes away from master more quickly, and the process of merging SHAs from the “platform_abstraction” into master can go more slowly.
As was discussed at the Intel Builders conference last week, we should set up a periodic conference call to others in the community can discuss this and other subjects. I’ll work with Jim, Ben, and Daniel to set that up.
John Meneghini
john.a.meneghini(a)gmail.com
> On Apr 26, 2017, at 2:30 AM, Ranjit Noronha <notifications(a)github.com> wrote:
>
> Hi John,
>
> Sure, lets work together on this.
>
> Will your upcoming changes complete the separation of dpdk and spdk? Or
> will there be components still left to refactor?
>
> Do you have a ballpark timeline on when these commits will complete?
>
> thanks,
>
> --ranjit
>
> On 4/26/2017 3:39 AM, John Meneghini wrote:
> >
> > *@johnmeneghini* commented on this pull request.
> >
> > Hi Ranjit.
> >
> > Some of these changes conflict with the changes in my pull requests
> > at: #136
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk...>
> > #148
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk...>
> > #150
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk...>
> >
> > As you can see, I've refactored my pull request several times, and I
> > have many more commits waiting to go in.
> >
> > I spoke to Ben about this last week and I think our changes
> > percipitated Ben's email on the reflector. Let's see if we can work
> > together to get these changes in.
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk...>,
> > or mute the thread
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notificat...>.
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub <https://github.com/spdk/spdk/pull/142#issuecomment-297252727>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMVL3Rosso6EsrPuJojLNae...>.
>
4 years, 4 months
spdk peer2peer dma fio latency
by PR PR
Hi, I am trying some experiments to evaluate performance of peer2peer dma.
I am using spdk to control the nvme drives and fio-plugin compiled with
spdk. I am seeing a weird behavior where when I run 4K IOs with IO-Depth of
1 peer2peer DMA from nvme drive to some pci device (which exposes memory
via Bar1) in a different numa node has a 50th percentile latency of 17
usecs. The same experiment but where nvme device and pcie device in same
numa node (node 0) has a latency of 38 usecs. In both cases fio was running
in node 0 cpu core and pci device (which exposes memory via Bar1) is
connected to node 1. DMA from nvme device to host memory also takes 38
usecs.
To summarize the cases below
1. nvme (numa node 0) - pci device (numa node 1) --- 18 usecs
2. nvme (numa node 1) - pci device (numa node 1) --- 38 usecs
3. nvme (numa node 0) - host memory --- 38 usecs
fio running in numa node 0 cpu core in all cases.
For higher IO Depth values cross numa case (case 1 above), latency
increases steeply and performs poorly than case 2 and case 3.
Any pointers on why this could be happening?
The nvme devices used are both identical intel datacenter ssd 400G.
Thanks
4 years, 4 months
NVMe device reinsertion not detected
by Baruch Even
Hi,
I'm using spdk 16.12 and dpdk 16.11 and am having an issue with NVMe device
reinsertion that is not getting detected and I wanted to check with you
about the current state of device hot plug and hot removal as I've seen at
ticket that mentions there are known issues with these.
Currently we have the devices working at startup by detecting the already
bound devices at startup and that works just fine. When I then go on to
test device removal the removal gets detected fine when I call the probe
function periodically[1]. If I reinsert the device and bind it to the
uio_pci_generic driver it is not being detected by spdk and is not added
back, at least there is no probe/attach callbacks getting called and I
can't see any traces from dpdk showing up.
If I restart the process the device will get detected and taken back as
usual and it can serve as a temporary workaround but it is not a good
overall approach.
Is this a known issue?
What can be done about this?
Baruch
[1] It would be nice to expose this file descriptor to allow
select/poll/epoll to be used to call an event instead of periodic polls for
this very rare event.
4 years, 4 months