The analysis of data from small volumes and simple algorithms to large data and complex systems —...
-
Upload
black-sea-summit-it-conference-in-odessa -
Category
Technology
-
view
337 -
download
1
Transcript of The analysis of data from small volumes and simple algorithms to large data and complex systems —...
![Page 1: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/1.jpg)
Анализ данных: от малых объемов и простых алгоритмов до больших данных и сложных систем
Дмитрий Сподарец
![Page 2: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/2.jpg)
Обо мне
• Преподаватель ОНПУ кафедры Системного программного обеспечения
• Основатель FlyElephant и GeeksLab.
![Page 3: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/3.jpg)
FlyElephantPlatform for scientific computing and data management
![Page 4: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/4.jpg)
Данные
![Page 5: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/5.jpg)
Алгоритмы
![Page 6: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/6.jpg)
Инфраструктура
![Page 7: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/7.jpg)
Данные Алгоритмы
Инфраструктура
![Page 8: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/8.jpg)
Много данных - это сколько?
Данные
![Page 9: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/9.jpg)
~30 PB / день
~10 PB / год
LSST
~15 PB / год
![Page 10: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/10.jpg)
![Page 11: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/11.jpg)
Инфраструктура
![Page 12: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/12.jpg)
Данные Алгоритмы
Инфраструктура
![Page 13: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/13.jpg)
Сценарии
![Page 14: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/14.jpg)
Простые данные и простые алгоритмы
![Page 15: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/15.jpg)
Много данных и сложные алгоритмы
![Page 16: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/16.jpg)
Большие данные
![Page 17: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/17.jpg)
Комбинирование
![Page 18: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/18.jpg)
Простые данные и простые алгоритмыАлгоритмы
- Линейный поиск - Перемножение матриц- Поиска минимального пути- ….
Данные Инфраструктура
![Page 19: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/19.jpg)
Библиотеки и инструментыEigen
eigen.tuxfamily.org
intel-mklsoftware.intel.com/en-us/intel-mkl
SciPywww.scipy.org
ND4Jnd4j.org
MATLABwww.mathworks.com www.scilab.org
Scilab
Juliajulialang.orgOctave
octave.org
![Page 20: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/20.jpg)
Много данных и сложные алгоритмыАлгоритмы
- Data Mining- Machine Learning- Computer Vision- …
Данные Инфраструктура
MPI, OpenMP…
![Page 21: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/21.jpg)
![Page 22: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/22.jpg)
![Page 23: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/23.jpg)
![Page 24: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/24.jpg)
Message Passing Interface (MPI)
![Page 25: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/25.jpg)
OpenMP
![Page 26: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/26.jpg)
CUDA
![Page 27: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/27.jpg)
Intel Xeon Phi
![Page 28: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/28.jpg)
Большие данные
Данные Инфраструктура
NoSQL, MapReduce, Hadoop, Spark…
![Page 29: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/29.jpg)
NoSQL• Хранилище «ключ-значение»
Berkeley DB, MemcacheDB, Redis, Amazon DynamoDB.
• Хранилище семейств колонокHBase, Apache Cassandra, Apache Accumulo, Hypertable, SimpleDB (amazon.com)…
• Документо-ориентированная СУБД MongoDB, CouchDB, Couchbase, MarkLogic, eXist..
• Базы данных на основе графов Neo4j, OrientDB, AllegroGraph, InfiniteGraph…
![Page 30: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/30.jpg)
MapReduceМодель распределённых вычислений
• Map-шаг - предварительная обработка.
• Reduce-шаг - сверка результатов и формирование решения задачи.
![Page 31: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/31.jpg)
Hadoop и Spark
![Page 32: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/32.jpg)
https://aws.amazon.com/ru/elasticmapreduce/
http://azure.microsoft.com/ru-ru/services/hdinsight/
![Page 33: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/33.jpg)
Комбинирование
![Page 34: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/34.jpg)
![Page 35: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/35.jpg)
Анализ данных при помощи FlyElephant
Уже готово
C++OpenMP
![Page 36: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/36.jpg)
Анализ данных при помощи FlyElephant
Что ждать в ближайшем релизе
MPI
R Python
Java
![Page 37: The analysis of data from small volumes and simple algorithms to large data and complex systems — Dmitry Spodarets (FlyElephant, Tech Stage)](https://reader036.fdocuments.in/reader036/viewer/2022081604/5888b5d71a28ab80248b6169/html5/thumbnails/37.jpg)
http://flyelephant.net/
http://flyelephant.net/beta/