Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf ·...
Transcript of Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf ·...
![Page 1: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/1.jpg)
Desmistificando Big Data: é possível manipular grandes bases de dados em R?
Samuel Macêdo
![Page 2: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/2.jpg)
2
Definindo Big Data
- Volume- Velocidade- Variedade- Veracidade- Valor
![Page 3: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/3.jpg)
3
Definindo Big Data
- Veracidade- Valor- Volume
![Page 4: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/4.jpg)
4
Quem processa Big Data?
![Page 5: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/5.jpg)
5
Arquitetura
![Page 6: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/6.jpg)
6
Como funciona o R?
![Page 7: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/7.jpg)
7
R roda Big Data?
Claro que NÃO
![Page 8: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/8.jpg)
8
É possível trabalhar com Big Data em R?
DefinitivamenteSIM
![Page 9: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/9.jpg)
9
E qual mágica?
![Page 10: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/10.jpg)
10
Pacotes em R
- SparkR
![Page 11: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/11.jpg)
11
Pacotes em R
- sparklyr
![Page 12: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/12.jpg)
12
Invoke
\\ scala
package au.csiro.variantspark.api
class VSContext(val spark:SparkSession) {…}
\\ r
sparklyr::invoke_new(sc, "au.csiro.variantspark.api.VSContext", spark_session(sc))
![Page 13: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/13.jpg)
13
Configuração
sparklyr::spark_install()
![Page 14: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/14.jpg)
14
Eu preciso do R?
![Page 15: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/15.jpg)
15
E porque R?
- Fácil
- Concentrado
- Aprender uma linguagem
![Page 16: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/16.jpg)
16
Quando eu devo usar spark?
- Use o R sempre que possível!!!
- Spark em versão local
- Spark em cloud
![Page 17: Desmistificando Big Data: é possível manipular grandes ...rday.leg.ufpr.br/slides/samuel.pdf · The new book is now available on-line and in print! Visit: therinspark.com for more](https://reader033.fdocuments.in/reader033/viewer/2022050207/5f59ff698e22a471f95e2641/html5/thumbnails/17.jpg)
17
Resumo
- Trate bem seus dados
- R como interface para o spark
- Só use spark quando realmente precisar
- E o principal…
É possível trabalhar em Big Data com R !!!