Enabling accelerated networking - seminar by Enea at the Embedded Conference Scandinavia

Enea Confidential – Under NDA

Embedded Conference Scandinavia 2014

2014-11-05

Enabling Accelerated Networking in

Embedded Systems

Ola Dahl

Product Management

Enea

Copyright © 2014 Enea AB


Why?


http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white_paper_c11-520862.html


http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white_paper_c11-520862.html

Enea Confidential – Under NDACopyright © 2013 Enea ABCopyright © 2014 Enea AB

Increasing amount of data

Increasing requirements on QoS

Hardware needs to

change

Software needs to

change

*

*Networking, data plane


What if

Copyright © 2013 Enea ABCopyright © 2014 Enea AB

Software was portable

Software could utilize (and

adapt to changes in)

hardware acceleration

and


Even better


What if software was open source

AND

It was not controlled by a single company



Enter ODP


OpenDataPlane – http://opendataplane.org

Linaro Networking Group

Open source

An API for portable, accelerated, multicore networking

Reference implementation on Linux

Open development of reference implementation(s)

ODP implementations for different SoCs, from different

vendors


http://opendataplane.org/


● Founded in 2010 with 6 members*, now 29

● >200 OSS engineers distributed globally, $50M annual revenue

Linaro Organization

Technical Steering Committee (TSC)

Off

ice o

f th

e C

TO

Enterprise Group (LEG)

16 companiesDigital Home Group (LHG)8 companies

Mobile Group (LMG)10 companies Networking Group (LNG)12 companies

Core Development

Kernel

Power Management

Security

Virtualization

Product Technology

LAVA Software Linaro Stable Kernel (LSK) Toolchain

Member

Services

Platform Engineering

Builds & Baselines

Systems

LAVA Lab

QA Services NDA Projects

Linaro Developer

Technical Support

(LDTS)

Field Engineering

*ARM, IBM, Freescale, Samsung, ST-Ericsson, TI


● Formed in February 2013

● Working on core open-source

software for networking● Real Time Support

● Virtualization for networking

● Core isolation

● Data Plane interfaces & frameworks

● Big-endian legacy code support

● Developing cross-platform

OpenDataPlane framework

Linaro Networking Group (LNG)

http://wiki.linaro.org/LNG

http://wiki.linaro.org/LNG


ODP – conceptual model*


* from http://www.opendataplane.org/wp-content/uploads/2014/01/ODPIntroductionandOverview-2014Jan29.pdf

http://www.opendataplane.org/wp-content/uploads/2014/01/ODPIntroductionandOverview-2014Jan29.pdf


Hello ODP


sudo apt-get install autoconf libtool libssl-dev

git clone https://git.linaro.org/lng/odp.git

cd odp

./bootstrap

./configure --prefix=$HOME/sw_install/odp

make

make install

cd example/odp_example/

./odp_example



Hello ODP – code snippets*


* SPDX-License-Identifier: BSD-3-Clause

/* ODP main header */

#include <odp.h>

#define MAX_WORKERS 32 /**< Max worker threads */

#define MSG_POOL_SIZE (4*1024*1024) /**< Message pool size */

/** @private Barrier for test synchronisation */

odp_barrier_t barrier;

buf = odp_queue_deq(queue);

buf = odp_schedule(&queue, ODP_SCHED_WAIT);


Fetch a buffer from

queue

Fetch a buffer from a

queue decided by the

scheduler

*source code at https://git.linaro.org/lng/odp.git/blob/HEAD:/example/odp_example/odp_example.c

https://git.linaro.org/lng/odp.git/blob/HEAD:/example/odp_example/odp_example.c


Hello ODP – code snippets*


* @internal Worker thread

static void *run_thread(void *arg)

thr = odp_thread_id();

printf("Thread %i starts on core %i\n", thr, odp_thread_core());

odp_barrier_sync(barrier);

int main(int argc, char *argv[])

/* Create and launch worker threads */

odph_linux_pthread_create(thread_tbl, num_workers, first_core,

run_thread, NULL);


Expected usage – one thread per

(data-plane) core

(spinning) wait for other

threads

Here we use threads – ODP can be

used also in ”bare-metal” mode (on

data-plane cores)

*source code at https://git.linaro.org/lng/odp.git/blob/HEAD:/example/odp_example/odp_example.c

https://git.linaro.org/lng/odp.git/blob/HEAD:/example/odp_example/odp_example.c


ODP – Key concepts


Packet

Buffer

Packet Buffer

Queue


Header Data

Header Packet

Buffer

Each element is a descriptor, referring to a buffer, like


ODP L2 forwarding example – code snippets*


#define MAX_WORKERS 32

#define SHM_PKT_POOL_SIZE (512*2048)

#define SHM_PKT_POOL_BUF_SIZE 1856

/* Use schedule to get buf from any input queue */

buf = odp_schedule(NULL, ODP_SCHED_WAIT);

pkt = odp_packet_from_buffer(buf);

/* Open a packet IO instance for this thread */

pktio = odp_pktio_open(args->srcif, pool);

pkts = odp_pktio_recv(thr_args->srcpktio, pkt_tbl, MAX_PKT_BURST);


Fetch a buffer from a

queue decided by the

scheduler – and get

packet stored in buffer

Communicate with the

outside world

*source code at https://git.linaro.org/lng/odp.git/blob/HEAD:/example/l2fwd/odp_l2fwd.c

https://git.linaro.org/lng/odp.git/blob/HEAD:/example/l2fwd/odp_l2fwd.c


ODP – Key concepts (cont.)


Packet I/0 – representation of (physical and logical) network interfaces

Timer – send timer event buffer (descriptor) to a queue at a specified time

Scheduler – distribute work (e.g. packets, timer events) to cores

Classification – packet analysis and pattern matching

Run-to-completion – threads* are assumed to consume packets and

process them in a run-to-completion manner

*The ODP threads are assumed to execute in isolation, on the data plane assigned cores (e.g. by

utilizing features available in Linux such as core isolation and NoHz Full, but also bare-metal

operation – without OS on the data plane cores – is possible)



ODP – Run-time visualization


Queue groups

Queues

Scheduler

Cores/threads associated

with queue groups

Control

plane core

Queues can be

dynamically created

and added to and

removed from

queue groups

Cores can be

dynamically added

to and removed

from queue groups

handlers associated

with queues

IP-fwd

NAT

GTP-U

DPI

RoHC


Can it be portable?


ODP

App

ODP

AppODP

App

ODP

App

SoC

A

ODP

Implementation

for SoC A*

SoC

B

SoC

C

ODP application

layer

ODP API

ODP port to

different

SoC/HW

architecture

ODP

Implementation

for SoC B*

ODP

Implementation

for SoC C*

* may be implemented with use of a vendor SDK

Portability vs Efficiency – is there a sweet spot?

ODP

App


Making an ODP implementation


• How to combine generic API with SoC-specific

implementation?

• How to utilize HW acceleration?

• How to synchronize development with development of

the ODP API?

• How to ensure compliance with the ODP API? (so that

an ODP application will run on top of the implementation)

We look at linux-keystone2 as an example


The different repos


git clone https://git.linaro.org/lng/odp.git

git clone https://git.linaro.org/lng/odp-keystone2.git

odp$ git tag

v0.3.0

odp$ git log `git rev-list v0.3.0 | head -n 1` -1

commit a25acf5f8565a14ead91946ef573d4558cf41c2d

odp-keystone2$ git log -1

commit 4b5063546ecec6f253267a96f6f0f706023cc012

Merge: 2c6c370 a25acf5

Merge tag 'v0.3.0' into keystone2/master

https://git.linaro.org/lng/odp.git

https://git.linaro.org/lng/odp-keystone2.git


The different repos


odp-keystone2.git odp.git

odp-keystone2 odp

platform

linux-generic linux-keystone2 linux-generic

platform

clone/fetch/push

(implementor)

merge*

* repeat when ODP API is updated – then modify code (if needed) and run compliance test

clone/fetch/push

(Linaro)


Packets and Buffers


odp-keystone2$ diff platform/linux-keystone2/include/api/odp_buffer.h ../odp/platform/linux-generic/include/api/odp_buffer.h | grep odp_buffer_t

< typedef Ti_Pkt * odp_buffer_t;

< static inline Ti_Pkt *_odp_buf_to_ti_pkt(odp_buffer_t buf)

< static inline odp_buffer_t _ti_pkt_to_odp_buf(Ti_Pkt *pkt)

> typedef uint32_t odp_buffer_t;

< static inline Cppi_HostDesc*_odp_buf_to_cppi_desc(odp_buffer_t buf)

< static inline void *odp_buffer_addr(odp_buffer_t buf)

> void *odp_buffer_addr(odp_buffer_t buf);


Packets and Buffers


http://www.ti.com/lit/ug/sprugr9f/sprugr9f.pdf

http://www.ti.com/lit/ug/sprugr9f/sprugr9f.pdf


Queues


http://www.ti.com/general/docs/datasheetdiagram.tsp?genericPartNumber=66AK2H12&diagramId=41464

http://www.ti.com/general/docs/datasheetdiagram.tsp?genericPartNumber=66AK2H12&diagramId=41464


Queues – linux-generic


int queue_enq(queue_entry_t *queue, odp_buffer_hdr_t

*buf_hdr)

{

int sched = 0;

LOCK(&queue->s.lock);

if (queue->s.head == NULL) {

/* Empty queue */

queue->s.head = buf_hdr;


Queues – linux-keystone2


int queue_enq(queue_entry_t *queue, odp_buffer_t buf)

{

odp_pr_vdbg("queue: %s, buf: %p, qmss_queue: %d\n",

queue->s.name, buf, queue->s.qmss_queue);

Qmss_queuePushDescSize(queue->s.qmss_queue,

_odp_buf_to_cppi_desc(buf),


Making an ODP implementation


The implementation also needs to handle (accelerate)

• Packet I/O

• Scheduling

• Classification

• Crypto

• CRC

• ...


An ODP implementation


IPSec with HW acceleration

Demonstrated at LCU 2014

TI Keystone II hardware (also on Cavium, Freescale, Avago)

http://www.slideshare.net/linaroorg/lcu14-310-cisco-odp-v2

http://www.slideshare.net/linaroorg/lcu14-310-cisco-odp-v2


Experimental setup



Several vendors are active



What’s in the pipeline


ODP 1.0 – end 2014

ODP 2.0 – end 2015


Good enough for

evaluation

Good enough for

production


Thank you


And here are some additional links

ODP software - http://www.opendataplane.org/downloads/

ODP docs - http://www.opendataplane.org/documentation/

ODP docs (alt.) - http://docs.opendataplane.org/

ODP arch doc git - https://git.linaro.org/lng/odp-architecture.git

Linaro Connect USA 2014 schedule (look for presentations about ODP –

and other things as well) –

http://lcu14.zerista.com/event?event_order=start&event_page=1&owner=ot

her&owner_id=739721

ODP mailing list - http://www.opendataplane.org/mailing-list/


http://www.opendataplane.org/downloads/

http://www.opendataplane.org/documentation/

http://docs.opendataplane.org/

https://git.linaro.org/lng/odp-architecture.git

http://lcu14.zerista.com/event?event_order=start&event_page=1&owner=other&owner_id=739721

http://www.opendataplane.org/mailing-list/

Enabling accelerated networking - seminar by Enea at the Embedded Conference Scandinavia

Software

Transcript of Enabling accelerated networking - seminar by Enea at the Embedded Conference Scandinavia