Project meniscus
-
Upload
kichul-jung -
Category
Technology
-
view
218 -
download
2
Transcript of Project meniscus
![Page 1: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/1.jpg)
Project Meniscus
Simplex Internet / Research Institute / Jeong, Ki-Chul
2013. 4. 29
![Page 2: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/2.jpg)
Meniscus?
• LAAS(Logging-as-a-Service) Backend API
• Python based system
• Event - collection, transit, processing
• Large-scale
• Cloud Logging
![Page 4: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/4.jpg)
APIS
![Page 5: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/5.jpg)
Tenant API
• Tenant = {Host, Event Producer, Profile}
• A Tenant = A Consumer
• Event Producer = APP or Service
• Profile = [Event Producer, …]
• Host = Node , link to a profile(1:1)
![Page 6: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/6.jpg)
Tenant API
• Requests to coordinator worker
• Tenant, Host, Event Producer, Profile
=> CRUD Available
• has a TOKEN
=> validate, reset a token
![Page 7: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/7.jpg)
Correlator API
• Requests
To a Correlator worker server
From a Consumer server
via http(s)
with a Message token(Tenant token)
• Stores a collection named “logs” in Mongo
![Page 8: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/8.jpg)
Coordinator API
• Pairing
- registers a worker
• Configuration
– requests a worker’s configuration
• Status
– updates a workers status
![Page 9: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/9.jpg)
Status API
• Worker Status API (PUT, Manually)
- System Load Average
- Disk Usage
• Coordinator Status API
- grid status
- individual worker status
![Page 10: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/10.jpg)
Broadcaster API
• What’s Brodcaster API?
I don’t know well!
• Communicates with coordinator worker
server.
• New Feature!
![Page 11: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/11.jpg)
GETTING STARTED
![Page 12: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/12.jpg)
Server deployment example
Host1
• Coordinator
• MongoDB
Host2
• Correlator
Host3
• Storage
![Page 13: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/13.jpg)
HOST1
• Creating a development environment
[LINK]
• Install MongoDB
[LINK]
![Page 14: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/14.jpg)
uWSGI.ini
![Page 15: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/15.jpg)
meniscus/etc/meniscus/meniscus.conf
![Page 16: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/16.jpg)
Run uwsgi daemon!!
# uwsgi uwsgi.ini
![Page 17: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/17.jpg)
HOST2
• Creating a development environment
[LINK]
• Run uwsgi daemon!!
Default Persona : pairing
![Page 18: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/18.jpg)
CURL Request
• Attempts pairing with coordinator
• Becomes correlation worker after automatically restart
![Page 19: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/19.jpg)
HOST3
• Creating a development environment
[LINK]
• Run uwsgi daemon!!
Default Persona : pairing
![Page 20: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/20.jpg)
CURL Request
• Attempts pairing with coordinator
• Becomes store worker after automatically restart
![Page 21: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/21.jpg)
COMPLETED FOR SETTING UP
![Page 22: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/22.jpg)
Logging a data
• Already MUST register a tenant
• Request to correlator server(host2)
![Page 23: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/23.jpg)
Flow
Correlator
(host2)
Storage
(host3)
MongoDB
(host1)
![Page 24: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/24.jpg)
Conclusion
• Well-made backend APIs for logging
(Good architecture)
• Distributed workers processing
• Diverse Persona(worker)
• In early development
• Nobody in its community
(Stack Overflow, Google Groups)
![Page 25: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/25.jpg)
Not yet
• Data Store – Only MongoDB
Hadoop / HDFS / Hbase
Elastic Search
• Perfect documentation
need more examples
need exact examples
![Page 26: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/26.jpg)
The thing we can now do with meniscus is
JUST to create log data into mongodb via
restful api and to fetch log data from
mongodb.
![Page 27: Project meniscus](https://reader033.fdocuments.in/reader033/viewer/2022042607/55a741121a28ab6e018b4702/html5/thumbnails/27.jpg)
THANKS