Add py-spdk client for SPDK
by We We
Hi, all
I have submitted the py-spdk code on https://review.gerrithub.io/#/c/379741/, please take some time to visit it, I will be very grateful to you.
The py-spdk is client which can help the upper-level app to communicate with the SPDK-based app (such as: nvmf_tgt, vhost, iscsi_tgt, etc.). Should I submit it into the other repo I rebuild rather than SPDK repo? Because I think it is a relatively independent kit upon the SPDK.
If you have some thoughts about the py-spdk, please share with me.
Regards,
Helloway
2 years, 3 months
SPDK + user space appliance
by Shahar Salzman
Hi all,
Sorry for the delay, had to solve a quarantine issue in order to get access to the list.
Some clarifications regarding the user space application:
1. The application is not the nvmf_tgt, we have an entire applicance to which we are integrating spdk
2. We are currently using nvmf_tgt functions in order to activate spdk, and the bdev_user in order to handle IO
3. This is all in user space (I am used to the kernel/user distinction in order to separate protocol/appliance).
4. The bdev_user will also notify spdk of changes to namespaces (e.g. a new namespace has been added, and can be attached to the spdk subsystem)
I am glad that this is your intention, the question is, do you think that it would be useful to create such a bdev_user module which will allow other users to integrate spdk to their appliance using such a simple threading model? Perhaps such a module will allow easier integration of spdk.
I am attaching a reference application which is does NULL IO via bdev_user.
Regarding the RPC, we have an implementation of it, and will be happy to push it upstream.
I am not sure that using the RPC for this type of bdev_user namespaces is the correct approach in the long run, since the user appliance is the one adding/removing namespaces (like hot plugging of a new NVME device), so it can just call the "add_namespace_to_subsystem" interface directly, and does not need to use an RPC for it.
Thanks,
Shahar
3 years, 6 months
some question about SPDK iSCSI when evaluate performance with NULL/Malloc/NVMe device over 4*10 network.
by Lego Lin
Hi, all:
When I try to improve SPDK over 4*10G performance, I took some time to
study the codes. And I have 1 question. (My SPDK version is 17.10)
1. file iscsi_subsystem.c function:
spdk_iscsi_read_parameters_from_config_file
I saw there is a magic number: 256 for limit max qdepth of iSCSI target.
And this magic number also works for create iSCSI PDU_pool.
function: spdk_iscsi_initialize_pdu_pool
Is there any constraint if we want to enlarge this value?
Thanks
3 years, 9 months
SPDK performance over 4*10G nic cards
by Lego Lin
Hi @all:
Our team has setup a platform to measure SPDK performance. After an
evaluation, we found that SPDK read performance is not as good as write.
Following are the platform information
Server: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Client: CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz with 40 vcores
DRAM: 128G
nic card: 2*10G on board + 4*10G cards
NVMe SSD: Intel P4501 * 10
Software version: SPDK version 17.10
Network setting:
Server IP: 10.10.63.25, 10G ipaddr: 192.168.100 ~ 103.25
Client IP: 10.10.63.61, 10G ipaddr: 192.168.100 ~ 103.61
MTU 9000 for jumbo frames.
We test following backend devices
1. SPDK null bdev * 16
2. SPDK malloc bdev * 16
3. Intel P4501 bdev * 8 (We divide each P4501 into 2 partitions for get
enough bdev).
We will generate 16 target devices and use fio with 16 jobs, each job R/W 1
target devices.
FIO: qdepth = 1024
iSCSI initiator block device nr_requests = 1024, nomerge, noop scheduler
SPDK iSCSI target qdepth = 256
following is the performance data:
Null backend
Write: 1169 KIOPS
Read: 1008 KIOPS
Malloc backend
Write: 1168 KIOPS
Read: 975 KIOPS
NVMe backend
Write: 1069 KIOPS
Read: 799 KIOPS
Related configuration as following:
ReactorMask 0x3FF003F0 (16 vcores)
MinConnectionsPerCore 1
Can someone help that whether any other parameters for trying so that
overall read performance can hit 1M.
I already try increase HUGHMEM and different MinConnectionIdleInterval and
it won't works.
thanks
3 years, 9 months
SPDK Async I/O Thead
by Fenggang Wu
Hi All,
I have some questions about the SPDK async I/O thread. I know that such
async I/O thread is receiving I/O requests as events from other application
threads, and forward the requests to the I/O device then polling for the
completion.
My questions are:
1) When is the async I/O thread created? How many async I/O threads are
created? If more than one, is the number configurable? I know that for
blobstore there is only one such async I/O thread.
2) My intuition is that such Async I/O thread is crucial for the I/O
performance so we would prefer to prevent it from evicted from the core.
Does SPDK have any method that keeps such async I/O thread on the core?
With such method, is the async I/O thread free from eviction?
Thank you very much!
-Fenggang
3 years, 9 months
NVMe command ordering
by Jonas Pfefferle
Hi @all,
Can anyone clarify this for me:
NVMe spec 6.3 Command Ordering Requirements
"For all commands which are not part of a fused operation (refer to section
4.10), or for which the write size is greater than AWUN, each command is
processed as an independent entity without reference to other commands
submitted to the same I/O Submission Queue or to commands submitted to other
I/O Submission Queues"
So writes <= AWUN size are atomic but there is no ordering between the
commands, e.g. write command A issued before write command B to the same LBA
can be executed in order AB or BA, the only thing guaranteed is that they
are executed atomically? And there is no way to enforce ordering except on
the host side?
Regards,
Jonas
3 years, 9 months
SPDK Test Plans
by Luse, Paul E
Friendly reminder.... We are no longer merging test plans to the repo. We'll still use GerritHub to review test plans because it's convenient but once the doc has served its purpose between engineers, please abandon as opposed to merging it.
We're doing this for a few reasons, probably the biggest one is to avoid these docs from needing to be updated as the code evolves (old test docs do nothing but serve to confuse). Also, our test coverage is measured via the CI system as opposed to a paper trail of test plans and requirements.
Thanks!
Paul
3 years, 9 months
Lots of SSE4 Link Errors in Examples, But Only When Installing
by David Sweeris
Hello again,
SPDK will compile* and test ok, but when it comes time to actually install it, I get this error (solution and question at bottom):
LINK examples/blob/hello_world/hello_blob
/usr/local/lib/librte_eal.a(eal.o): In function `rte_eal_init':
eal.c:(.text+0x427): undefined reference to `__intel_sse4_strrchr'
eal.c:(.text+0x438): undefined reference to `__intel_sse4_strdup'
eal.c:(.text+0x68d): undefined reference to `__intel_sse4_strnlen'
eal.c:(.text+0x8b6): undefined reference to `__intel_sse4_strdup'
eal.c:(.text+0x8cf): undefined reference to `__intel_sse4_strdup'
eal.c:(.text+0x1810): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_eal.a(eal_hugepage_info.o): In function `eal_hugepage_info_init':
eal_hugepage_info.c:(.text+0xc7): undefined reference to `__intel_sse4_strncmp'
eal_hugepage_info.c:(.text+0x2f4): undefined reference to `__intel_sse4_strncmp'
eal_hugepage_info.c:(.text+0x5b7): undefined reference to `__intel_sse4_strdup'
eal_hugepage_info.c:(.text+0x9f6): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_memory.o): In function `rte_eal_hugepage_init':
eal_memory.c:(.text+0x1709): undefined reference to `_intel_fast_memset'
eal_memory.c:(.text+0x2dc3): undefined reference to `_intel_fast_memset'
eal_memory.c:(.text+0x2fd1): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_eal.a(eal_log.o): In function `console_log_write':
eal_log.c:(.text+0x5a): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_eal.a(eal_vfio.o): In function `rte_vfio_setup_device':
eal_vfio.c:(.text+0x1be): undefined reference to `_intel_fast_memset'
eal_vfio.c:(.text+0x1d2): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_eal.a(eal_vfio.o): In function `rte_vfio_release_device':
eal_vfio.c:(.text+0x13fa): undefined reference to `_intel_fast_memset'
eal_vfio.c:(.text+0x140e): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_eal.a(eal_vfio.o): In function `vfio_get_group_no':
eal_vfio.c:(.text+0x201b): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_eal.a(eal_vfio.o):eal_vfio.c:(.text+0x202f): more undefined references to `_intel_fast_memset' follow
/usr/local/lib/librte_eal.a(eal_timer.o): In function `rte_eal_timer_init':
eal_timer.c:(.text+0x21a): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_interrupts.o): In function `rte_intr_callback_register':
eal_interrupts.c:(.text+0x11d): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_eal.a(eal_interrupts.o): In function `rte_intr_enable':
eal_interrupts.c:(.text+0x461): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_eal.a(eal_common_memzone.o): In function `memzone_reserve_aligned_thread_unsafe':
eal_common_memzone.c:(.text+0x62): undefined reference to `__intel_sse4_strlen'
eal_common_memzone.c:(.text+0x9e): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_memzone.o): In function `rte_memzone_reserve_bounded':
eal_common_memzone.c:(.text+0x5d4): undefined reference to `__intel_sse4_strlen'
eal_common_memzone.c:(.text+0x610): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_memzone.o): In function `rte_memzone_lookup':
eal_common_memzone.c:(.text+0xcbe): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_memzone.o): In function `rte_eal_memzone_init':
eal_common_memzone.c:(.text+0xe0b): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_eal.a(eal_common_log.o): In function `rte_log_register':
eal_common_log.c:(.text+0x1f8): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_log.o): In function `rte_log_init':
eal_common_log.c:(.text+0x48f): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_tailqs.o): In function `rte_eal_tailq_lookup':
eal_common_tailqs.c:(.text+0x33): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_tailqs.o): In function `rte_eal_tailq_register':
eal_common_tailqs.c:(.text+0x88): undefined reference to `__intel_sse4_strncmp'
eal_common_tailqs.c:(.text+0x10e): undefined reference to `__intel_sse4_strncmp'
eal_common_tailqs.c:(.text+0x1b0): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_tailqs.o): In function `rte_eal_tailqs_init':
eal_common_tailqs.c:(.text+0x2cb): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_tailqs.o):eal_common_tailqs.c:(.text+0x372): more undefined references to `__intel_sse4_strncmp' follow
/usr/local/lib/librte_eal.a(eal_common_devargs.o): In function `rte_eal_parse_devargs_str':
eal_common_devargs.c:(.text+0x2b): undefined reference to `__intel_sse4_strdup'
eal_common_devargs.c:(.text+0x9d): undefined reference to `__intel_sse4_strchr'
eal_common_devargs.c:(.text+0xb1): undefined reference to `__intel_sse4_strdup'
eal_common_devargs.c:(.text+0xe8): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_devargs.o): In function `bus_name_cmp':
eal_common_devargs.c:(.text+0x12d): undefined reference to `__intel_sse4_strlen'
eal_common_devargs.c:(.text+0x13c): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_devargs.o): In function `rte_eal_devargs_add':
eal_common_devargs.c:(.text+0x293): undefined reference to `__intel_sse4_strdup'
eal_common_devargs.c:(.text+0x336): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_devargs.o): In function `rte_eal_devargs_parse':
eal_common_devargs.c:(.text.experimental+0x6f): undefined reference to `__intel_sse4_strlen'
eal_common_devargs.c:(.text.experimental+0xd8): undefined reference to `__intel_sse4_strdup'
eal_common_devargs.c:(.text.experimental+0x114): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_bus.o): In function `rte_bus_find_by_device_name':
eal_common_bus.c:(.text+0x491): undefined reference to `__intel_sse4_strchr'
/usr/local/lib/librte_eal.a(eal_common_options.o): In function `eal_parse_common_option':
eal_common_options.c:(.text+0x1a3): undefined reference to `__intel_sse4_strlen'
eal_common_options.c:(.text+0x302): undefined reference to `__intel_sse4_strlen'
eal_common_options.c:(.text+0x4c5): undefined reference to `__intel_sse4_atol'
eal_common_options.c:(.text+0x50e): undefined reference to `__intel_sse4_atol'
eal_common_options.c:(.text+0x54f): undefined reference to `__intel_sse4_atol'
eal_common_options.c:(.text+0x8bd): undefined reference to `_intel_fast_memset'
eal_common_options.c:(.text+0x8d0): undefined reference to `__intel_ssse3_strncpy'
eal_common_options.c:(.text+0x97d): undefined reference to `__intel_sse4_strlen'
eal_common_options.c:(.text+0xc74): undefined reference to `__intel_sse4_strlen'
eal_common_options.c:(.text+0xd33): undefined reference to `__intel_sse4_strlen'
eal_common_options.c:(.text+0xf4b): undefined reference to `__intel_sse4_strdup'
eal_common_options.c:(.text+0xfc4): undefined reference to `__intel_sse4_strchr'
/usr/local/lib/librte_eal.a(eal_common_options.o): In function `eal_parse_lcores':
eal_common_options.c:(.text+0x15c7): undefined reference to `__intel_sse4_strcspn'
eal_common_options.c:(.text+0x15e6): undefined reference to `__intel_sse4_strcspn'
eal_common_options.c:(.text+0x1889): undefined reference to `__intel_sse4_strcspn'
/usr/local/lib/librte_eal.a(eal_common_options.o): In function `eal_plugins_init':
eal_common_options.c:(.text+0x21ca): undefined reference to `_intel_fast_memset'
eal_common_options.c:(.text+0x21de): undefined reference to `__intel_ssse3_strncpy'
eal_common_options.c:(.text+0x2360): undefined reference to `_intel_fast_memset'
eal_common_options.c:(.text+0x2374): undefined reference to `__intel_ssse3_strncpy'
/usr/local/lib/librte_eal.a(eal_common_thread.o): In function `eal_thread_dump_affinity':
eal_common_thread.c:(.text+0x123): undefined reference to `_intel_fast_memmove'
/usr/local/lib/librte_eal.a(eal_common_thread.o): In function `rte_thread_get_affinity':
eal_common_thread.c:(.text+0x1bc): undefined reference to `_intel_fast_memmove'
/usr/local/lib/librte_eal.a(eal_common_thread.o): In function `rte_thread_set_affinity':
eal_common_thread.c:(.text+0x27c): undefined reference to `_intel_fast_memmove'
eal_common_thread.c:(.text+0x2b1): undefined reference to `_intel_fast_memmove'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `mp_handle':
eal_common_proc.c:(.text+0x224): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `send_msg':
eal_common_proc.c:(.text+0x5c3): undefined reference to `_intel_fast_memset'
eal_common_proc.c:(.text+0x63c): undefined reference to `_intel_fast_memcpy'
eal_common_proc.c:(.text+0x765): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `mp_request_one':
eal_common_proc.c:(.text+0x8db): undefined reference to `__intel_sse4_strcpy'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_channel_init':
eal_common_proc.c:(.text+0xcd0): undefined reference to `__intel_sse4_strdup'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_action_register':
eal_common_proc.c:(.text.experimental+0x58): undefined reference to `__intel_sse4_strnlen'
eal_common_proc.c:(.text.experimental+0xc1): undefined reference to `__intel_sse4_strcpy'
eal_common_proc.c:(.text.experimental+0xed): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_action_unregister':
eal_common_proc.c:(.text.experimental+0x214): undefined reference to `__intel_sse4_strnlen'
eal_common_proc.c:(.text.experimental+0x264): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_sendmsg':
eal_common_proc.c:(.text.experimental+0x369): undefined reference to `__intel_sse4_strnlen'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_request':
eal_common_proc.c:(.text.experimental+0x5fc): undefined reference to `__intel_sse4_strnlen'
/usr/local/lib/librte_eal.a(eal_common_proc.o): In function `rte_mp_reply':
eal_common_proc.c:(.text.experimental+0x91f): undefined reference to `__intel_sse4_strnlen'
/usr/local/lib/librte_eal.a(malloc_elem.o): In function `malloc_elem_free':
malloc_elem.c:(.text+0x4c8): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_mempool.a(rte_mempool.o): In function `rte_mempool_create_empty':
rte_mempool.c:(.text+0x26ac): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_mempool.a(rte_mempool.o): In function `rte_mempool_lookup':
rte_mempool.c:(.text+0x2da3): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_mempool.a(rte_mempool_ops.o): In function `rte_mempool_register_ops':
rte_mempool_ops.c:(.text+0x93): undefined reference to `__intel_sse4_strlen'
/usr/local/lib/librte_ring.a(rte_ring.o): In function `rte_ring_lookup':
rte_ring.c:(.text+0x503): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_mempool_ring.a(rte_mempool_ring.o): In function `common_ring_mp_enqueue':
rte_mempool_ring.c:(.text+0x1cd): undefined reference to `_intel_fast_memcpy'
rte_mempool_ring.c:(.text+0x233): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_mempool_ring.a(rte_mempool_ring.o): In function `common_ring_sp_enqueue':
rte_mempool_ring.c:(.text+0x4af): undefined reference to `_intel_fast_memcpy'
rte_mempool_ring.c:(.text+0x511): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_mempool_ring.a(rte_mempool_ring.o): In function `common_ring_mc_dequeue':
rte_mempool_ring.c:(.text+0x7d0): undefined reference to `_intel_fast_memcpy'
/usr/local/lib/librte_mempool_ring.a(rte_mempool_ring.o):rte_mempool_ring.c:(.text+0x837): more undefined references to `_intel_fast_memcpy' follow
/usr/local/lib/librte_bus_pci.a(pci.o): In function `pci_update_device':
pci.c:(.text+0x23b): undefined reference to `_intel_fast_memset'
pci.c:(.text+0x6ee): undefined reference to `__intel_sse4_strrchr'
pci.c:(.text+0x729): undefined reference to `__intel_sse4_strlen'
pci.c:(.text+0x739): undefined reference to `__intel_ssse3_strncpy'
pci.c:(.text+0x8f5): undefined reference to `_intel_fast_memmove'
/usr/local/lib/librte_bus_pci.a(pci.o): In function `rte_pci_scan':
pci.c:(.text+0xaa3): undefined reference to `__intel_sse4_strchr'
/usr/local/lib/librte_bus_pci.a(pci.o): In function `pci_scan_one':
pci.c:(.text+0xc54): undefined reference to `_intel_fast_memset'
pci.c:(.text+0x10de): undefined reference to `__intel_sse4_strrchr'
pci.c:(.text+0x1119): undefined reference to `__intel_sse4_strlen'
pci.c:(.text+0x1129): undefined reference to `__intel_ssse3_strncpy'
pci.c:(.text+0x130b): undefined reference to `_intel_fast_memmove'
/usr/local/lib/librte_bus_pci.a(pci.o): In function `rte_pci_ioport_map':
pci.c:(.text+0x1890): undefined reference to `__intel_sse4_strcspn'
pci.c:(.text+0x18f3): undefined reference to `__intel_sse4_strlen'
pci.c:(.text+0x1904): undefined reference to `__intel_sse4_strncmp'
pci.c:(.text+0x1a12): undefined reference to `__intel_sse4_strcspn'
pci.c:(.text+0x1a75): undefined reference to `__intel_sse4_strlen'
pci.c:(.text+0x1a86): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_bus_pci.a(pci_uio.o): In function `pci_get_uio_dev..0':
pci_uio.c:(.text+0x3d9): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_bus_pci.a(pci_uio.o): In function `pci_uio_map_resource_by_index':
pci_uio.c:(.text+0x77b): undefined reference to `__intel_sse4_strlen'
pci_uio.c:(.text+0x87a): undefined reference to `__intel_sse4_strcpy'
/usr/local/lib/librte_bus_pci.a(pci_uio.o): In function `pci_uio_ioport_map':
pci_uio.c:(.text+0x9c4): undefined reference to `__intel_sse4_strncmp'
/usr/local/lib/librte_bus_pci.a(pci_vfio.o): In function `pci_vfio_map_resource':
pci_vfio.c:(.text+0x84): undefined reference to `_intel_fast_memset'
pci_vfio.c:(.text+0x717): undefined reference to `_intel_fast_memset'
/usr/local/lib/librte_bus_pci.a(pci_vfio.o): In function `pci_vfio_unmap_resource':
pci_vfio.c:(.text+0xef3): undefined reference to `_intel_fast_memset'
collect2: error: ld returned 1 exit status
make[3]: *** [hello_blob] Error 1
make[2]: *** [hello_world] Error 2
make[1]: *** [blob] Error 2
make: *** [examples] Error 2
This happens regardless of whether I’ve got the compiler flag set to use SSE3/SSE4 (via ICC's “-xHost” flag) or not.
The fix is to su, run “export LDFLAGS=-lirc”, and then run “make install”. Setting LDFLAGS and then running “sudo make install” does not work.
Does this strike anyone else as a bit odd? I get needing to “sudo make install”, because non-root users don’t have write permissions in /usr/local. Seems strange that you’d need to be root just to link, though.
Thanks,
-Dave Sweeris
3 years, 10 months
