Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
-
Upload
cynthia-thomas -
Category
Technology
-
view
882 -
download
2
Transcript of Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF
Cilium – Kernel Native Security & DDOS
Mitigation for Microservices with BPF
Cynthia ThomasTechnology Evangelist, Covalent
@_techcet_
Gordon is back!And he got a job at Lego.
Robot Competition
● Local marketing launches an Robot competition for Danish residents
● Upcoming deadline means tons of uploaded media
Robot Competition
User Data MongoDB
Data store
Inventory
Image UploadService
Web Front-End
KafkaBrokerKafkaBrokerKafkaBroker Image
processingImage processing
Image processing
ZookeeperZookeeperZookeeper
End-User
Admintask
Audit
Gordon’s Task List● Deploy Cilium HTTP-aware security for microservices
● Isolate Kafka resources for old & new services
● DDOS mitigation via XDP/BPF
Gordon’s 1st TaskUpdate HTTP Security for Microservices
Application Architectures
Delivery Frequency
Operational Complexity
Single Server App
Yearly
Low
Evolution of Application Design & Delivery Frequency
Application Architectures
Delivery Frequency
Operational Complexity
Single Server App
Yearly
Low
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency
Application Architectures
Delivery Frequency
Operational Complexity
Single Server App
Yearly
Low
Distributed Microservices
10-100 x’s / day
Extreme
3-Tier App
Monthly
Moderate
Evolution of Application Design & Delivery Frequency
The world still runs on iptablesmatching IPs and ports:
$ iptables -A INPUT -p tcp -s 15.15.15.3 --dport 80 \-m conntrack --ctstate NEW -j ACCEPT
Robot Competition
User Data MongoDB
Data store
Inventory
Image UploadService
Web Front-End
KafkaBroker
Image processing
Zookeeper
End-User
Admintask
Audit
Example: HTTP calls
API
Web Front-End
Security for Microservices
Image UploadService
GET /image
POST /image
GET /image/flagged
PUT /image/id
APIPOST /image
Web Front-End
Security for Microservices
Image UploadService
GET /image
GET /image/flagged
PUT /image/id
POST /image
L3/L4
APIPOST /image
Web Front-End
Security for Microservices
iptables -s 10.1.1.1-p tcp --dport 80-j ACCEPT
Image UploadService
GET /image
POST /image
GET /image/flagged
PUT /image/id
L3/L4
APIPOST /image
Web Front-End
Security for Microservices
iptables -s 10.1.1.1-p tcp --dport 80-j ACCEPT
exposed
exposed
exposed
Image UploadService
GET /image
POST /image
GET /image/flagged
PUT /image/id
Network Security has not evolved for microservices…
…until now.
APIPOST /image
Web Front-End
Cilium API-Aware Security
Image UploadService
GET /image
GET /image/flagged
PUT /image/id
POST /image
L7
APIPOST /image
Web Front-End
Cilium API-Aware Security
Image UploadService
GET /image
POST /image
GET /image/flagged
PUT /image/id
FROM Web Front-EndALLOW POST /image
Under the Hood: BPFLinux
Superpowers
Unleashed
BPF instruction setstruct bpf_insn prog[] = {
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1), BPF_LD_ABS(BPF_B, ETH_HLEN + offsetof(struct iphdr, protocol) /* R0 =
ip->proto */), BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_0, -4), /* *(u32 *)(fp - 4) = r0
*/ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), /* r2 = fp - 4 */BPF_LD_MAP_FD(BPF_REG_1, map_fd), BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem), BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2), BPF_MOV64_IMM(BPF_REG_1, 1), /* r1 = 1 */BPF_RAW_INSN(BPF_STX | BPF_XADD | BPF_DW, BPF_REG_0, BPF_REG_1, 0, 0),
/* xadd r0 += r1 */BPF_MOV64_IMM(BPF_REG_0, 0), /* r0 = 0 */BPF_EXIT_INSN(),
};
Reference: https://git.kernel.org/pub/scm/linux/kernel/git/ast/bpf.git/tree/samples/bpf/sock_example.c
Cilium
CiliumAgent
Userspace
Kernelspace
Cilium Architecture
CiliumAgent
API Calls
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF
API Calls
BPF Code & Maps
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF
API Calls
BPF Code & Maps
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
Extremely scalable, highly customized
network filtering, load-balancing, and
monitoring
BPF BPF
API Calls
BPF Code & Maps
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
Plugins
Cilium Architecture
CiliumAgent
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
CLI,Monitoring,
Policies
Plugins
Cilium Architecture
CiliumAgent
Gordon’s 2nd TaskIsolate Kafka Topics
Kafka?Used for building real-time pipelines and streaming apps.
- Horizontally scalable
- Fault-tolerant
- “Wicked fast”
Defined by its own protocol.
1/3 of all Fortune 500 companies use Kafka[1]
Kafka ConceptsTopic 1 Producers
Kafka Broker
Topic 1
Topic N
Topic 1 Consumer Group A
Topic 1 Consumer Group B
Security for Microservices
Gordon worries about open Kafka topics
The API is too open!How do I bring on
new services?
Robot Competition
User Data MongoDB
Data store
Inventory
Image UploadService
Web Front-End
KafkaBrokerKafkaBrokerKafkaBroker Image
processingImage processing
Image processing
ZookeeperZookeeperZookeeper
End-User
Admintask
Audit
Example: Kafka calls
Kafka.API
Imageprocessing
Security for Microservices
Kafka Broker
Fetch inventory
Fetch image
Fetch userid
Fetch analytics
Produce inventory
Produce image
Produce userid
Produce analytics
Kafka.API
Fetch image
Imageprocessing
Security for Microservices
Kafka Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
L3/L4
Kafka.API
Fetch image
Imageprocessing
Security for Microservices
iptables -s 10.1.1.1-p tcp --dport 9092-j ACCEPT
Kafka Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
L3/L4
Kafka.API
Fetch image
Imageprocessing
Security for Microservices
iptables -s 10.1.1.1-p tcp --dport 9092-j ACCEPT
exposed
exposed
exposed
Kafka Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
Kafka.API
Fetch image
Imageprocessing
Security for Microservices
Kafka Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
L7
Kafka.API
Fetch image
Imageprocessing
Security for Microservices
Kafka Broker
Fetch inventory
Fetch image
Produce userid
Fetch userid
FROM Image processingALLOW Fetch image
Demo!
Get real with Gordon
Demo: Kafka API Filtering
app1-producer
app1-consumer
app2-producer
app2-consumer
kafka
What just happened?
L7
Kafka.API
Produce imageprocessing
app1-producer
Demo: Kafka API Filtering
Kafka Broker
Produce imageprocessing
Produce userdata
FROM app1-producerALLOW Produce imageprocessing
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
CLI,Monitoring,
Policies
Plugins
Sidecar Proxy
CiliumAgent
SidecarProxy
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
CLI,Monitoring,
Policies
Plugins
In-Kernel Kafka Parsing
CiliumAgent
kernel proxyBPF
Gordon’s 3rd TaskDDOS Mitigation
Robot Competition● Local marketing launches
competition for Danish residents only
● DDOS anticipated from other countries and anti-Lego activists
Robot Competition
User Data MongoDB
Data store
Inventory
Image UploadService
Web Front-End
KafkaBroker
Image processing
Zookeeper
End-User
Admintask
Audit
Example: DDOS attack!
BPF
BPF BPF
API Calls
BPF Code & Maps
Packets
Userspace
Kernelspace
CLI,Monitoring,
Policies
Plugins
XDP/BPF handled in NIC driver
CiliumAgent
XDP
Source: https://www.netdevconf.org/2.1/slides/apr6/zhou-netdev-xdp-2017.pdf
Facebook published BPF/XDP numbersfor L3/L4 LB at Netdev 2.1
BPF/XDP throughput
IPVS throughput
BPF with XDP Setup
pktgen attack:~11.6 Mpbsrandomly in 10.0.0.0/8
legit traffic:netperf tests on 10.192.1.0/24
Blacklist 16M rules
All /32s in 10.0.0.0/8
BPF with XDP for DDoS mitigationMetric iptables / ipset XDPDDoS rate [packets/s] 11.6M 11.6MDrop rate [packets/s] 7.1M 11.6MTime to load rules [time] 3 min 20 sec 31 secLatency under load [ms] 2.3ms 0.1msThroughput under DDoS [Gbit/s] 0.014 6.5
Requests/s under DDoS [kReq/s] 0.28 82.8
Sender: Send 64B packets as fast as possible è Receiver: Drop as fast as possible
Source: Daniel Borkmann’s presentation:http://schd.ws/hosted_files/ossna2017/da/BPFandXDP.pdf
● Cilium deployment for microservices successfully secured HTTP traffic
● Kafka resources were isolated to protect existing services from new ones
●Mitigated DDOS attacks via XDP/BPF
Gordon’s Summary
Cilium Project Status• Cilium v0.12 release in October
• Docker, Kubernetes, and Mesos integration
• Looking for feedback and contributions
Take Action!
• Getting Started Using Docker: docs.cilium.io/
• Join our Slack community!
• Check out the project website for more details: https://www.cilium.io/
Please ★us on
GitHub
Thank You!
github.com/cilium/cilium
cilium.io
@ciliumproject