Analysis big data by use php with storm
-
Upload
- -
Category
Engineering
-
view
174 -
download
2
Transcript of Analysis big data by use php with storm
PHP|
PaaS SAE
Top3 SuperAPP
Pla$orm Services Group
Sam
AWS S3
CDN ATS
Biwing
1k
30+10+
Monika
G o
Rambone
PHP Front-End
Services Panel
Skylee
C++ Go
Content Summary
"
# PHP
PHP
$ %
Case
Real 7me data collec7on
"
How to collect data in real 7me
?
…… ……
Ka<a —— a high-throughput distributed messaging system source from LinkedIn
Producer KaLa LinkedIn Apache
KaLa Producer
&
"
'
(
ModelTwo models: queuing and publish-subscribe. the la<er is more commonly used.
high-level abstracPonEach Topic corresponds to one or more log files. Each Topic has one or more par@@ons.
FeatureScalability & Durability & Reliability & Performance & Fault-tolerant
EcosystemStream processing systems, Hadoop integraPon, monitoring, and deployment tools.
Ka<a should be no7ced when using Ka<a
)
KaLa offsetConsumer
Buffer
KaLa gzip snappy lz4CPU
KaLa KaLa
KaLa
Topic Par@@onPar@@on
Topic Par@@on
ZookeeperKaLa Zookeeper
ZK ZK
Real-7me analysis
"
BackType nathanmarz Storm BackTypeTwiXer Storm TwiXer TwiXer
Apache
' Feature
Storm —— a distributed real-7me computa7on system source from TwiGer
Highly scalable
Fault-tolerant
Guarantees processing
Language agnosPc
* Topology of one Spout
* Topology of three Spout
* UI / Nimbus / ZK / Supervisor
Storm Storm cluster organiza7on structure
Storm UI ZookeeperStorm Nimbus
Storm Supervisor
Topology
Storm Storm mul7 language support
+
Python
PHP
JS
Shell
, Storm use STDIN & STDOUT
support multi language
PHPConnectPool
Storm PHPConnect Storm to PHP
+ PHP
STDIN
STDOUT
+PHP
PHP-FPMPHP
PHP
FastCGI
PHPphp analysis program
PHP
PHPPHP parse a real 7me log
*
-> Kafka -> Storm -> Spout Bolt -> Bolt FCGI
PHPPHP program for parse log
PHP
-
PHP-FPMdynamic PHP
pm.max_children
PHPDB Cache
Storm
web
RD
PHP
Input
PHP
PHP
Storm Bolt PHPStorm bolt send a request by FastCgi protocol
Architecture design
$
Overall architecture design
Performance data
CaseShow Case
%
Sta7s7cal service pla$orm
Sta7s7cal service pla$orm
Real-7me log analyse service
Q & A?