ShareInsights on the Amazon Cloud...Amazon Athena is an interac?ve query service that makes it easy...
Transcript of ShareInsights on the Amazon Cloud...Amazon Athena is an interac?ve query service that makes it easy...
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage2of27
DocumentRevisions
Date Change Insec/ons
January2019 IniCalpublicaCon -
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage3of27
ContentsDocumentRevisions
Introduc<on
Architecture
Prerequisites
SpecializedKnowledge
DesignConsidera<ons
SampleDataset
IAMRolesandPolicies
DeploymentOp<ons
TechnicalRequirements
Duringdeployment
Postdeployment
SystemRequirements
II.PlanningGuidance
Security
Costs
Sizing
III.DeploymentSteps
Step1.PrepareYourAmazonAccount
Step2.LaunchusingCloudForma<on
Step3.TesttheDeployment
Op<onal:UsingYourOwnDataset
Troubleshoo<ng
IV.Opera@onalGuidance
HealthCheck
BackupandRecovery
Rou<neMaintenance
EmergencyMaintenance
Support
SupportCosts
Appendix
Namingconven<on
RoleandPolicyDetails
2
4
5
6
6
6
6
7
7
7
7
8
9
9
9
9
10
11
11
11
16
19
20
21
21
22
23
23
23
23
24
24
24
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage4of27
I.IntroductoryMaterial
Features
Usecases
DocumentRevisionsAccelerite presents ShareInsights, an end-to-end big data analy?cs pla@orm that unifies differentanaly?csopera?onslikedataprocessing,storage,andvisualiza?oninonetool.
This deployment guide provides step-by-step instruc?ons for deploying Accelerite ShareInsights in theAmazoncloud.YoucanusethisShareInsightsenvironmenttoprocessandanalyzeyourowndatasetsanddashboards. This deployment provides prepopulated sample Wikipedia open data and sampledashboardstoanalyzeusingAmazonAthenaandAmazonGlueservices.
ShareInsightsmakesiteasyforanyonetoexplore,transformandvisualizebigcomplexdataonAmazonorHadoopdatalakesinminutes.
ShareInsightsonAmazonisdesignedtoleveragethepowerofAmazonAnaly?csservicessuchasAthena,andGluewithasimpledrag-and-dropvisualdesignerwhichenablesdataanalystswithnoprogrammingexper?setoeasilybuildtheirownadvanceddatapipelinesthatincludepowerfulvisualiza?ons.
ShareInsightsusesAmazonS3asadatalakeandusesthepowerofAmazonGlueCatalog.
• Easytousedrag-and-dropvisualpipelinebuilder• Visualdatalakeexplora?onwithslicinganddicing• Integrateddatavisualiza?onenginefordrag-and-dropdesigningofdashboards• FullyIntegratedwithAmazonAthena,AmazonGlue,andAmazonS3.• SupportsAmazonServerlessanaly?cstechnologiesforincrediblecostsavings• Integrateseasilywithmostpopularanaly?csrepor?ngtools(suchasTableauandQlik)• Easy-to-usedata-as-a-servicecapabili?es(usingRESTfulAPIs)
• Projectmanagersinaconsul?ngfirmslookingtoimproveu?liza?on,meetdeadlines,mi?gaterisksproac?velytoimprovecustomersa?sfac?onandcapacityu?liza?on.
• EnterpriseIT&dataengineerslookingtoautomatemigra?onofon-premiseETL,ELTjobstoAWSdataservicesrapidly.
• Datascien?stslookingtopreparedata,trainanddeploymodelsreliablyonAWS.• ManageddatalakeserviceprovidersonAWSlookingtoenabletheirbusinessuserswiththe
abilitytoderiveinsightswithano-codeanaly?cspla@orm.• TVchannelsandadver?serscanprocessreal-?meviewershipdatafrommul?plesourcesand
createcustomizeddashboardstounderstandandtargettheiraudiencebeYer.• Marke?ngmanagerslookingtoenhancecustomersegmenta?onandtargetcustomersfor
higherconversionandruneffec?vecampaignswithuniqueinsightsfromallrelevantdatasources.
• Clinicalresearcherslookingtoimprovedrugtrialenrolmenteffec?venessbyanalyzingpa?entdatawithaself-serviceinterface.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage5of27
Architecture
This ShareInsights deployment provides no-code analy?cs for data lakes that seamlessly u?lizes na?veAmazonanaly?csservicessuchasAmazonAthenaandAmazonGlue.Thisdeploymentbuilds followingenvironment in the Amazon Cloud including ShareInsights server, Amazon Athena, Amazon Glue andotherAmazonservices
Figure1:ShareInsightsarchitectureontheAmazonCloud
Thearchitectureincludesthefollowingservicesandcomponents:
•Amazon S3. Amazon Simple Storage Service (Amazon S3) is an object store that provides ar?factsnecessaryforShareInsights,includingdatasetstoconfigureAmazonAthenaandAmazonGluedatabaseand to compute aggregates for the sample dataset. ShareInsights creates a S3 bucket to keep itsintermediateandfinaldatasets.
•AmazonAthena.AmazonAthenaisaninterac?vequeryservicethatmakesiteasytoanalyzedatainAmazon S3 using standard SQL. The deployment creates Athena tables for each sample datasets.ShareInsightsallowsuser to select theanaly?csengine–AmazonAthenaandAmazonGlue - toeasilybuildtheirownadvanceddatapipelinesthatincludepowerfulvisualiza?ons.
•Amazon Glue. The Amazon Glue is a fully managed ETL (extract, transform, and load) service thatmakes it simple and cost-effec?ve to categorize your data, clean it, enrich it, and move it reliablybetweenvariousdatastores.AmazonGlueconsistsofacentraldata repositoryknownas theAmazonGlue Data Catalog, an ETL engine, and a job scheduler. ShareInsights creates a database to keep itsmetadata.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage6of27
•ShareInsightsonAmazonEC2.Thedeploymentprovidesasingle-instancedeploymentofShareInsightswith theability forDataanalysisandvisualizaBonusingAmazonAnalyBcsservicessuchasAthenaandGlue. It also provides Trial userweb page for geDng started, sample dashboards, training videos andassignmentsonShareInsights.
•AmazonVPC. The Amazon Virtual Private Cloud (Amazon VPC) service lets you provision a logicallyisolated secBon of the Amazon Cloud where you can launch resources in a virtual network that youdefine. The VPC provides a network architecture with mulBple public and private subnets that spanmulBpleAvailabilityZones,sothatAmazonresourcescanbedeployedinhighlyavailableconfiguraBons.ThisdeploymentusesasingleAvailabilityZone.
•IAMroles.ThisdeploymentconfiguresAmazonIdenBtyandAccessManagement(IAM)rolestoprovidetherequiredaccessforAmazonresourcescreatedduringdeploymenttoaccessotherAmazonresourceswhenrequired.TheseIAMrolesenableaccesstodatainAmazonS3,enableAmazonAthenaandAmazonGluetoanalyzedataandsaveresultsonAmazonS3.
•Sampledataset.ThisShareInsightsdeploymentprovidesasampledataset,availableinAmazonS3,andcreatesAthenatablesundernewlycreatedGluedatabaseforShareInsights.
Prerequisites
Beforeyoubegin,reviewtheEndUserLicenseAgreement(EULA).
DesignConsidera@ons
SpecializedKnowledge
BeforeyoudeployAcceleriteShareInsights,werecommendthatyoubecomefamiliarwiththefollowingAmazonservices.(IfyouarenewtoAmazon,seeGe#ngStartedwithAmazon.)
SampleDataset
This ShareInsights deployment includes a dataset from aWikipedia open data. The dataset includesWikipediaopendata forWhatHappens inMonth,PageView,andMoviesBoxOfficeanalysis.TablesrelatedtothesedatasetsarecreatedinAmazonGluedatabase.Thisdeploymentallowsusertoanalyzethesesampledatasetsandviewsampledashboards.
•AmazonS3•AmazonEC2•AmazonAthena•AmazonGlue•AmazonVPC•AmazonIAM
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage7of27
IAMRolesandPolicies
Thisdeploymentcontainsnon-sensi>ve,sampledatasetthatdoesn’tneedtobesecured.However,youcanusealterna>vedatasetswithprivate, sensi>vedata.Thedeploymentcreates the following threeIAMrolestoensurethatprivatedataismanagedsecurely:
DeploymentOp8ons
ShareInsightsprovidesonlyonedeploymentop>onusingAmazonCloudForma>on.
Duringdeployment
AmazonCloudForma>oncreatesfollowingresourcesrequiredforShareInsightsenvironment:
• EC2AssumeRole. This IAM roleprovidesAmazonS3access to the referenceddataset fromEC2instances launchedwithin the deployment. It also enables launched EC2 instances to associate anIAMrolewithrequiredAmazonservicessuchasAmazonAthena,AmazonGlue.BecausethisroleisintendedtobeassociatedwithEC2 instances, thedeploymentalsocreatesan IAM instanceprofilethatincludesthisIAMrole.
• GlueServiceRole.ThisIAMroleprovidesAmazonGlueaccesstoAmazonEC2instance.
• LambdaExecu8onRole.This IAMroleprovidesAmazonLambdaaccess toAmazonEC2 instance,AmazonS3andotherAmazonservices.
• EC2InstancePolicy.Thispolicy isforAmazonAssumeroletobeusedbyAmazonEC2toconnectwithotherAmazonservices.
• IAMPassRolePolicy.ThisIAMpolicyallowAmazonEC2instancetopassaroletootherAmazonservices.
ReferRoleandPolicyDetailsunderAppendixforrole,policyandpermissiondetails.
TechnicalRequirements
This ShareInsights deployment includes one Amazon CloudForma>on templatewhich creates a virtualprivatecloud(VPC)withapublicsubnet,whichisusedtohousetheremainderofthecomponentsinthearchitecture.
1. AmazonVPC2. AmazonPublicsubnet3. AmazonVPCendpointforAmazonS34. AmazonInternetGateway5. AmazonPublicRouteTable6. AmazonSecurityGroup7. AmazonIAM
• ReferIAM Roles and Policiesformoredetails.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage8of27
8.AmazonS3•OneAmazonS3privatebucketforShareInsightsdatalake•SharedopendatafromAmazonS3publicbucket (hEps://s3.amazonaws.com/shareinsights-pub/)will be used to create Athena tables undernewlycreatedShareInsightsAmazonGluedatabase.Thisdataisusedforsampledashboards.
9.AmazonGlue•OneShareInsightsdatabasewillbecreated•CreateAmazonGlueconnecMon
10.AmazonEC2instance•Tocreateinstance•PermissiontoaEachAmazonEC2instanceprofileassumerole
11.AmazonLambdafuncMonsfor:•CreaMnguniqueresourcename:
i.IncludingwithMmestamptoallowusertocreatemulMpleinstancesinoneaccountii. NormalizenameforspecialcharactersandlengthforAmazonS3,AmazonIAMroles,AmazonGlueconnecMon
•ShareInsightsspecificAmazonS3bucketcleanup•GeXngmainAmazonRouteTableid
ReferappendixforresourceNamingconven,on.
1.AmazonEC2•Edge/GatewaynodehostsShareInsightswebserver• ShareInsights serves user request using Amazon services such as Athena, Glue, S3 andCloudWatch.•DuringrequestAmazonAssumeRoleisused
2.AmazonAthena•ShareInsightswillcreate/delete/modifyAmazonAthenatablesandviewsduringexecuMon
3.AmazonGlue•BasedondashboardsandanalyMcstransformaMonused:
1.Create,deleteandmodifyAmazonGluetables2.Create,deleteandmodifyAmazonGlueJobs
4.AmazonS3•ShareInsightsusesAmazonS3asDataLake•DuringexecuMonShareInsightsreadsandwritesAmazonS3datausingAmazonAthenatables
orviews• DuringexecuMonofdashboardtheintermediateandfinaldatasetsaresavedasAmazonS3 objects. ShareInsightsalsocreatesappropriateAmazonAthena tablesandviewson topofthesefiles.TheseAthenaobjectsarepartofShareInsightsGluedatabase.
5.CloudWatch•JobexecuMonlogginginformaMoniscollectedinAmazonCloudWatch
Postdeployment
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage9of27
SystemRequirements
Amazon private key file with appropriate permissions to Amazon resources men@oned in Role andPolicyDetailssec@onunderappendix.
II.PlanningGuidance
Security
Rolesandpoliciesarecreatedforsecuritypurpose.ReferIAMRolesandPoliciesforthelist.
Forpermissiondetails,referRoleandPolicyDetailsunderAppendix.
ShareInsights user login andpassword is authen@cated internally. Thepassword is stored in encryptedformatwithinShareInsightsdatabase.
AbovecostdoesNOTincludeusagecostofAmazonservicesusedbyShareInsights.
YouareresponsibleforthecostoftheAmazonservicesusedwhilerunningthisdeployment.TheAmazonCloudForma@on templates include configura@on parameters that you can customize. Some of theseseKngs,suchasinstancetype,willaffectthecostofdeployment.SeethepricingpagesforeachAmazonserviceyouwillbeusingforcostes@mates.
TheAcceleriteShareInsightsdeploymentallowsyoutobuildadvancedbigdatapipelinesanddashboardsusingAmazonAthena andAmazonGlue engine.However, you are responsible for the cost ofAmazonresources used by ShareInsights. Following is basic informa@on about the Amazon services used byShareInsights:
Costs
FollowingistheShareInsightscostbasedonEC2instancetype:
Instancetype So=ware/Hour(USD$)
m4.xlarge
r4.xlarge
m4.2xlarge
r4.2xlarge
m4.4xlarge
r4.2xlarge
$0.20
$0.27
$0.40
$0.53
$0.80
$1.06
$4.20
$4.27
$5.40
$5.53
$6.80
$7.06
$4
$5
$6
ApproximateEC2/Hour(USD$)
TotalCost/Hour(USD$)
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage10of27
•AmazonAthenaoYouarechargedbasedontheamountofdatascannedbyeachqueryroundeduptothenearest megabyte, with a 10MB minimum per query. Amazon Athena charges USD$5 per TB of datascanned.oFormoreinformaLonrefertoAmazonAthenapricinginforma1on
•AmazonGlueoYouarechargedbasedontheLmeyourETLjobtakestorunandthenumberofDataProcessingUnits(orDPUs)usedtorunyourETLjob.oAmazonGluecatalogchargeswillbebasedonnumberofobjects.AnobjectintheAmazonGlueDataCatalogisatable,tableversion,parLLon,ordatabase.oFormoreinformaLonrefertoAmazonGluepricinginforma1on
•AmazonS3oYouwillbechargedbasedonamountofdataandlocaLonofS3bucketoFormoreinformaLonrefertoAmazonS3pricinginforma1on
•AmazonCloudForma8on(duringdeploymentonly)oAsofnowthereisnoaddiLonalchargeforAmazonCloudFormaLonoFormoreinformaLonrefertoAmazonCloudForma1onpricinginforma1on
•AmazonCloudWatchoShareInsightsusesCloudWatchonlyforlogscollecLon.TheapproximatelogsrelatedcostisUSD$0.535/GB.oFormoreinformaLonrefertoAmazonCloudWatchpricinginforma1on
•AmazonLambda(duringdeploymentonly)o ShareInsights creates five Lambda funcLon and uses them only during stack creaLon. Each LambdafuncLoniscalledonlyonceduringcreaLon.OnlyoneLambdafuncLoniscalledduringstackterminaLon.o ApproximatecostwillbelessthanUSD$1o FormoreinformaLonrefertoAmazonLambdapricinginforma1on
Sizing
Usethefollowingguidelinestoselectyourinstancesize:
Instancetype
m4.xlarge/r4.xlarge
m4.2xlarge(recommended)/r4.2xlarge/c5.2xlarge/t2.2xlarge
m4.4xlarge/r4.4xlarge/c5.4xlarge
4GB-8GB
8GB-16GB
16GB-64GB
Averagedatasizeworkload
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage11of27
III.DeploymentStepsThisguidestepsthroughtherequirementsandprocessfordeployingShareInsightsthroughtheAmazonMarketplace
Step1.PrepareYourAmazonAccount
1.Ifyoudon’talreadyhaveanAmazonaccount,createoneathEps://aws.amazon.combyfollowingtheon-screeninstrucKons.
2.UsetheregionselectorinthenavigaKonbartochoosetheAmazonRegionwhereyouwanttodeployShareInsightsonAmazon.Refertosupportedregions.
3.Createakeypairinyourpreferredregion.
4. Ifnecessary,requestaservicelimit increase fortheAmazonEC2m4.xlargeinstancetype.Youmightneed todo this if youalreadyhaveanexisKngdeployment thatuses this instance type,andyouthinkyoumightexceedthedefaultlimitwiththisreferencedeployment.
5. Ifnecessary,requestaservicelimitincreaseforAmazonCloudFormaKonstacks.Thisdeploymentwillcreateupto1newstackwithinyouraccount.
Step2.LaunchusingCloudFormaAon
NoteYouareresponsibleforthecostoftheAmazonservicesusedwhilerunningthisdeployment.
1. Launch theAmazonCloudFormaKon template into yourAmazon account. SeeDeploymentOp7onsearlierinthisguide.
Deploymenttakesabout25minutestocomplete.
2. Check the region that’s displayed in the upper-right corner of the navigaKon bar and change it ifnecessary.CurrentlyShareInsightssupportsthreeregions:
a.USEast(N.Virginia)b.USEast(Ohio)c.USWest(Oregon)
3.OntheSelectTemplatepage,keepthedefaultse]ngforthetemplateURL,andthenchooseNext.
4.OntheSpecifyDetailspage,reviewtheparametersforthetemplate.Entervaluesfortheparametersthatrequireyourinput.Forallotherparameters,youcancustomizethedefaultse]ngsprovidedbythetemplate.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage12of27
StackConfigura1on:
Parameterlabel(name)
Stackname Requiresinput
Default
AmazonInfrastructureConfigura1on:
Parameterlabel(name)
Keypairname(KeyName)
Requiresinput
Default
Specifyastackname.
Instancetype(InstanceType) m4.xlarge EC2instancetypefortheShareInsightsserver.
Descrip1on
Public/privatekeypair,whichallowsyoutoconnectsecurelytoyourinstanceaLeritlaunches.WhenyoucreatedanAmazonaccount,thisisthekeypairyoucreatedinyourpreferredregion.
Descrip1on
ShareInsightsConfigura1ons:
Parameterlabel(name)
AdminPassword(Password) Requiresinput
Requiresinput
Default
ConfirmAdminPassword(PasswordConfirm)
ConfirmadministratorpasswordforShareInsights.
ThepasswordoftheadministratorforShareInsights.NOTE:Theusernameoftheadministratoris“shareinsights”.
Descrip1on
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage13of27
5. ChooseNext. 6. OntheOp,onspage,youcanspecifytags(key-valuepairs)forresourcesinyourstackandset
advanced opFons.Whenyou’redone,chooseNext.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage14of27
7. OntheReviewpage,reviewandconfirmthetemplateseCngs.UnderCapabili/es,selectthecheckboxtoacknowledgethatthetemplatewillcreateIAMresources.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage15of27
Figure2:CheckboxforIAMresources
8. ChooseCreatetodeploythestack.
9. Monitorthestatusofthestack.WhenthestatusisCREATE_COMPLETE,thedeploymentiscomplete.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage16of27
Step3.TesttheDeployment
1.LandingpageURL
2.ShareInsightsdocumentaBonURL
3.EC2details
a.SSHinformaBontoconnecttoEC2instance.
4.Trialuser(forgeIngstartedonShareInsights)
a.Logindetails•TrialuserIDandpasswordwithreadonlyaccess
b.URLdetails•ItincludesbasicinformaBonaboutShareInsightsandfewexercisestolearnShareInsights
AOertheinstallaBoniscomplete,theOutputsecBoninCloudFormaBonwillsharethefollowinginformaBon:
Figure3:Outputsec=oninCloudForma=onaAerinstalla=oniscomplete
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage17of27
Totestthedeployment:
1. NavigatetotheShareInsightsServerbypoinCngawebbrowsertotheURLforLandingPageUrlintheOutputstab.
2. ItshoulddisplaythreedashboardsunderDashboardExplorerforthesampleWikipediadata.
3. ViewtheDataLakeExplorerwithinShareInsightsServer.TheCloudFormaContemplateconfiguressampledatasetswhichwillbelistedunderPublishedDatasetssecCon.
Figure4:ShareInsightsLoginpage
Figure5:ShareInsightsHomepagedisplayingDashboardExplorer
4. ViewthedashboardsthathavebeenpublishedtoShareInsightsServerfromthesampledataset.ThedashboardsaredesignedtocombineAmazondatasourcesandusethearchitectureinpowerfulwayswhileretaininggoodperformanceasthevolumeofdata,numberofusers,andscaleofqueriesincreaseoverCme.
5. VisittheTrialUserURLforgeTngstartedonShareInsights,trysampledashboards,viewingtrainingvideosandtryingassignments:
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage18of27
Figure6:ShareInsightsTrialUserPage
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage19of27
Op'onal:UsingYourOwnDataset
FollowingarethestepstoingestdatainShareInsightsusingAthenaandGluetables:
1. Createanewdashboard.
2. FromthePublishConfigura'onopCons,select"ExecuConEngine"as"athena"or"gluespark".
3. SwitchtoAdvancedEditmode.Youcandoitbyremovingthetrailing"f"fromtheword"ediL"inthesampleURLbelow.h"ps://<hostname>/live/v1/dashboards/my_dashboard/edi8
4. AddaDataSec'on(i.e.DsecCon)toaddthedetailsofthesourcedata.Hereisanexamplewithquerybinding:
D: mytable_ds_1 : [language_code(number),greeting] # Add table columns mytable_ds_2 : [currentvalue(number),brand,sector] # Add table columns
D.mytable_ds_1 : # dataset name to be used protocol : hivequery source : select * from my_glue_db.mytable1 # mytable : Name of the Table. limit_rows : 1000 # This is optional publish : 'myNewDataset1' # Add this flag to publish the dataset in ShareInsights lake and to be consumed by other dashboards
D.mytable_ds_2 : # dataset name to be used protocol : hivequery source : select * from my_glue_db.mytable2 # mytable : Name of the Table. limit_rows : 1000 # This is optional publish : 'myNewDataset2' # Add this flag to publish the dataset in ShareInsights lake and to be consumed by other dashboards.
5. ToSavethedashboardclickthesaveicononupper-rightcornerorpressCtrl+Sonthekeyboard.
6. ToPublishthedashboardclicktheexecuteicononupper-rightcorner,orpressAlt+Ronthekeyboard.Youcanviewthelogsasthedashboardisbeingpublished.OncetheexecuConisoverthelastlinewillappearas:“Reloadingdataexplorer”.
ThedatasetswithpublishfieldswillbeavailableintheShareInsightsDataLakeExplorerforconsumpConinotherdashboardsforallShareInsightsusers.
TolearnmoreaboutLanguageReference,refertoShareInsightsHelponupper-rightcorner.
Figure7:HelpmenuunderUsernamemenu
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage20of27
Troubleshoo-ng
Q.IencounteredaROLLBACK_COMPLETEstatusaEerIlaunchedthedeployment.WhatshouldIdo?
A.ThisoccurswhenAmazonCloudFormaKonisnotabletocreateanAmazonresourcewhichisrequiredfordeployingShareInsights.FollowingarethestepstofindtheresourcewhichfailedduringcreaKon:
1.VisitAmazonCloudFormaKonConsole2.Selectthestackyouweretryingtocreate3.SelectEventstab4.SearchforeventwithStatusasCREATE_FAILED
5. LookforthereasonmenKonedintheStatusreasoncolumn Commonreasonsforfailureare:
1.InsufficientIAMPermissions(notauthorizedtoperform)•YoushouldhavepermissiontocreateresourcesrequiredforCloudFormaKontosuccessfullycreateShareInsightsenvironment.ForlistofresourcesreferDuringdeploymentsecKon.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage21of27
2. Exceededlimit
• Increasethelimitofiden?fiedresourcetypefromStatusreasoncolumn(men?onedabove)and relaunchtheAmazonCloudForma?ontemplate.
3. ResourceFailedtoStabilizeDuringaCreate,Update,orDeleteStackOpera>on
• Probablereasonforthisfailuretooccuristheresourcedoesnothavesufficientpermissionsand thecrea?onhasfailed.
Foraddi?onalinforma?on,seeTroubleshoo>ngAmazonCloudForma>onontheAmazonwebsite.
Q. AmazonCloudForma?onissuccessfulbuttheShareInsightsURLisnotresponding.WhatshouldIdo?
A. ReferEmergencyMaintenanceforinves?gate.
IV. Opera/onalGuidance
HealthCheck
AOerinstallingandloggingintoShareInsights,clickonthe“DataLakeExplorer”tabwhichshouldlistafew
datasetsunderthe“PublishedDatasets”sec?on.Youshouldseesomeofthedatasetsshownbelow.
Figure8:ShareInsightsDataLakeExplorertabwithPublishedDatasetssec/ononleJ
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage22of27
Alterna<vely,youcanautomatethisbyusingAmazonRoute53.FollowingarethestepstocreateHealthCheckusingAmazonRoute53:
1.VisitAmazonRoute53console
2.ChooseCreatehealthcheck
3.GivehealthcheckaName
4.SelectDomainnameunderSpecifyendpointby
5.SelectHTTPSProtocol
6.EntertheAmazonEC2PublicDNS(ShareInsightsserver)inDomainnamefield
7.SetPathfieldto"live/login"
8.UseAdvancedconfiguraAonseXngstocustomize,ifany
9.ChooseNext
10.Togetno<fiedonfailure,selectYestoCreatealarm.SelectNo,ifyouwishtocheckhealthonAmazonRoute53console.
11. ChooseCreatehealthcheck
12. YoucanseeShareInsightsserverhealthonAmazonRoute53consolea]erfewminutesofcrea<onundertheName(instep#3).
IncaseifShareInsightsisnotresponding,followstepsmen<onedinEmergencyMaintenancesec<on.
NoteYouareresponsibleforthecostoftheAmazonRoute53services.ReferAmazonRoute53pricing.
BackupandRecovery
ShareInsightsteamrecommendaweeklybackupofEBSvolumes.PleaseseetheAmazonrecommenda<onforbackupandrecoveryusingasnapshotoftheEBSvolumes.
Recovery:
1. Torecoveranydashboardsearchfordashboardnamefolderunder/data/context/shareinsights/dashboards/inthebackups.
2. Forcompleterecoverycopy/data/context/shareinsights/folderfromthebackup.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage23of27
Rou'neMaintenance
ShareInsightshasnorou<nemaintenance.ShareInsightsdoesnothaveanykeysandthereforedoes
notrequireanykeyrota<onasitworksontheAmazonAssumeRolepolicy.
AnewversionofShareInsightsAMIwillbeavailableeveryquarter.Upgradeinstruc<onswillbeshared
witheveryrelease.TheAmazonmarketplacewillupdateyouifthereisanupgradeavailable.Contact
ShareInsightssupportforhelp.
EmergencyMaintenance
ShareInsightsisavailableinlimitedAmazonregionsasitrequiresAmazonAthenaandGlueservices.
Refertosupportedregions.
IftheShareInsightsURLisnotresponding,thentrythefollowing:
1. CheckonAmazonEC2consoleiftheEC2instanceisupandrunning.Ifnot,thenstartitand
itwillrestarttheShareInsights.
2. IncaseifyouarenotabletostartEC2instanceoritisnotaccessiblethenpleasecheckif
Amazonservicesareupforyourregionandavailabilityzone.
3. IncaseifEC2isupandrunningbutShareInsightsisnotthenrestarttheEC2machine.This
willstartShareInsightsinapproximately2-3mins.AlternatelysshEC2machineandexecute
sudosystemctlrestartlivecommand.
4. IfnoneoftheabovestepsworkthensendallthefilesfromfollowingAmazonEC2instance
folderstoShareInsightssupport(referSupport):
• /data/contexts/shareinsights/logs/<<host-name>>/• /data/contexts/shareinsights/sta<c/context/live/
ThereshouldnotbeanylossofShareInsightsdashboardordata(whicharesavedonAmazonservices
suchasS3,AthenaandGlue).IncaseofanylossofShareInsightsdashboardpleaserecoverfromthe
lastbackuptakenasexplainedinBackupandRecoverysec<on.
Support
Pleasecontactusforsupport([email protected]).withyourqueryandcontact
details.
SupportCostsSupportcostswilldependonthecontract.
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage24of27
Appendix
Namingconven3on
Resource names are created using the “shareinsights-<<stack name>>-<<Bmestamp>>” template. This
ensuresproperfuncBoningofmulBpleShareInsightsinstancesinthesameAmazonregion.
Forexample,ifusergivenstacknameis“shareinsights-stack-1”thenresourcenameswillbe:
RoleandPolicyDetails
Rolescreated:
1. ShareInsightsEC2InstanceRole
2. ShareInsightsGlueServiceRole
3. ShareInsightsLambdaExecuBonRole(usedduringdeploymentonly)
Policiescreated:
1. ShareInsightsEC2InstancePolicy
2. ShareInsightsIAMPassRolePolicy
3. ShareInsightsGlueServicePolicy
RoletoPolicyMatrix:
First35charactersofstacknameisconsideredforcreaBngtheresourcenameduetoAmazonresource
namelimitaBons.
Resource
S3bucketname
Name
shareinsights-shareinsightsstack1-1548420158
AccessRole shareinsights_shareinsightsstack1_1548420158_AR
GlueServiceRole shareinsights_shareinsightsstack1_1548420158_GSR
Gluedatabasename shareinsights_shareinsightsstack1_1548420158
GlueconnecBon shareinsights_shareinsightsstack1_1548420158_redis_conn
Policyname
ShareInsightsEC2InstancePolicy
ShareInsightsEC2InstanceRole ShareInsightsGlueServiceRole
ShareInsightsIAMPassRolePolicy
ShareInsightsGlueServicePolicy
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage25of27
ShareInsightsEC2InstancePolicydetails:
Service
AmazonAthena
AmazonS3 arn:aws:s3:::<source_data_bucket>/*,arn:aws:s3:::<si_bucket_name>/*
StartQueryExecuJon
GetNamespace
GetQueryResults
GetCatalogs
GetNamespaces
CreateNamedQuery
GetExecuJonEngine
StopQueryExecuJon
PutObject
GetObject
ListBucket
DeleteObject
GetBucketLocaJon
GetLogEvents
Ac;on Resource
ShareInsightsIAMPassRolePolicydetails:
Service
GetRole
PassRole
Ac;on Resource
*
GetDatabase
GetDatabases
GetJobRun
GetTable
GetTables
CreateJob
CreateTable
DeleteJob
StartJobRun
*
AmazonCloudWatchLogs
AmazonGlue
IAM
arn:aws:logs:*:*:log-group:*
arn:aws:iam::<<account_id>>:role/ShareInsightsGlueServiceRole
ShareInsightsontheAmazonCloud
AcceleriteShareInsightsDeploymentGuidePage26of27
ShareInsightsGlueServicePolicydetails:
Service
CloudWatch PutMetricData
AssociateKmsKey
CreateLogStream
PutLogEvents
arn:aws:logs:::/aws-glue/*
arn:aws:logs:::/aws-glue/*
arn:aws:logs:::/aws-glue/*
Ac;on Resource
*
DescribeNetworkInterfaces
DescribeRouteTables
DescribeSecurityGroups
DescribeSubnets
DescribeVpcAMribute
DescribeVpcEndpoints
CreateNetworkInterface
ListAllMyBuckets
ListBucket
GetBucketAcl
GetBucketLocaOon
GetObject
DeleteObject
PutObject
DeleteNetworkInterface
DeleteTags
CreateTags
*
*
CloudWatchLogs
EC2
S3
"arn:aws:ec2:*:*:network-interface/*","arn:aws:ec2:*:*:security-group/*","arn:aws:ec2:*:*:instance/*"
"arn:aws:s3:::<bucket_name>""arn:aws:s3:::<bucket_name>/*"
GetConnecOon
GetConnecOons
GetDatabase
GetDatabases
GetParOOon
GetParOOons
GetTable
GetTables
*
Glue