SpringCloudDataFlowfor
PivotalCloudFoundry®
Documentation
Version1.1.4
Published:2April2019
©2019PivotalSoftware,Inc.AllRightsReserved.
SpringCloud®DataFlowforPCF
AboutSpringCloudDataFlowSpringCloudDataFlowisanopen-sourcetoolkitfordataintegrationandreal-timedataprocessing(seetheprojecthomepage ).TheSpringCloudDataFlow(SCDF)serverdeployspipelinescomposedofSpringCloudStream orSpringCloudTask applications.Adomain-specificlanguage(DSL)makesiteasytodescribetheapplicationsinapipelineandhowtheyareconnected.
TheSCDFserverexposesaRESTAPIforcomposinganddeployingdatapipelinesandadashboardwithagraphicalpipelineeditor.ItalsoincludesashellforworkingwiththeAPIfromthecommandline.
AboutSpringCloudDataFlowforPCFSpringCloudDataFlowforPivotalCloudFoundry(PCF)automatesthedeploymentofSCDFanditsdependentservicessothatdeveloperscanuseAppsManagertodeploytheirownSCDFinstances.TheycanthenaccesstheirDataFlowserver,eitherviatheSCDFdashboardUIorusingtheirPCFcredentialswiththeSCDFshell.
KeyFeaturesSpringCloudDataFlowforPCFincludesthefollowingkeyfeatures:
AdditionoftheSpringCloudDataFlowservertotheMarketplaceasamanagedservice
AutomaticintegrationwithdependentPCFservices:
MySQLforPCF(orauser-providedrelationaldatabaseservice)forapps,pipelines,andtaskhistoryRabbitMQforPCF(orauser-providedRabbitMQorKafkaservice)foreventmessagingRedisforPCF(orauser-providedRedisservice)forcapturinganalyticsdata
IntegrationofDataFlowserverwithPCF’sUAAsecuritymodel
ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutSpringCloudDataFlowforPCF.
Element Details
Version v1.1.4
Releasedate September7,2018
Softwarecomponentversion v1.5.1.RELEASE
CompatibleOpsManagerversion(s) v1.10.x,v1.11.x,v1.12.x,v2.0.x,andv2.1.x
CompatibleElasticRuntimeversion(s) v1.10.x,v1.11.x,v1.12.x,v2.0.x,andv2.1.x
IaaSsupport AllsupportedbyPCF
RequirementsSpringCloudDataFlowforPCFhasthefollowingrequirements:
JavaCloudFoundrybuildpack version3.8orlater(seethePrerequisitessectionofInstallationformoreinformation)
MySQLforPCFv2 ,oranalternativerelationaldatabaseservice
RabbitMQforPCF ,oranalternativeRabbitMQorKafkaservice
RedisforPCF ,oranalternativeRedisservice
CloudFoundryCommandLineInterface (cfCLI)
OptionalTousetheSpringCloudDataFlowshellinterfacewithSpringCloudDataFlowforPCFserviceinstances,installthefollowingcfCLIplugins:
SpringCloudDataFlowforPCFcfCLIplugin.Toinstalltheplugin,runthefollowingcommand:
$cfinstall-plugin-rCF-Community"spring-cloud-dataflow-for-pcf"
JavaRuntimeEnvironment (JRE).RequiredtoruntheDataFlowshell.YoucandownloadtheJREfromtheJavawebsite .
ServiceInstanceLoggingcfCLIplugin .Toinstalltheplugin,runthefollowingcommand:
Note:SpringCloudDataFlowforPCFv1.1isnolongersupported.Thesupportperiodforv1.1hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,upgradetoasupportedversion.
©CopyrightPivotalSoftwareInc,2013-2019 2 1.1
$cfinstall-plugin-rCF-Community"ServiceInstanceLogging"
Pleaseprovideanybugs,featurerequests,orquestionstothePivotalCloudFoundryFeedbacklist.
©CopyrightPivotalSoftwareInc,2013-2019 3 1.1
SpringCloudDataFlowReleaseNotesReleasenotesforSpringCloudDataFlowforPCF
v1.1.4ReleaseDate:September7,2018
Fixesincludedinthisrelease:
Fixedanissuethatpreventedthe“run-tests”errandfromcompletingwhentheSpringCloudDataFlowtilewasconfiguredtonotallowinternetaccess.
v1.1.3ReleaseDate:August6,2018
Enhancementsincludedinthisrelease:
Anew“DonotallowpublicInternetaccessfromserviceinstances”optionintheDataFlowtileconfigurationpreventsserviceinstancesfromaccessingtheInternet.
The“run-tests”errandnowcomplieswiththeUAApasswordpolicywhencreatingitstemporarytestuser.
Fixesincludedinthisrelease:
TheJavabuildpacksettingconfiguredintheDataFlowtileorforaparticularserviceinstanceisnowhonored.
v1.1.2ReleaseDate:July16,2018
Enhancementsincludedinthisrelease:
Serviceinstancesnowallow spring.cloud.dataflow.applicationProperties.* tobesetvia -c propertiesto cf createandupdatecommands
v1.1.1ReleaseDate:June25,2018
Fixesincludedinthisrelease:
Thebrokernowproceedswithanupdateoperationforaserviceinstancethatpreviouslyfailedtoupdate.
v1.1.0ReleaseDate:June18,2018
Featuresincludedinthisrelease:
ServiceinstancesarenowbasedonSpringCloudDataFlow1.5.1.RELEASE.
ServiceinstancesnowacceptMavenconfigurationpropertiesprovidedbySpringCloudDataFlowOSS.
The“run-tests”lifecycleerrandnolongerrequiresaccesstotheInternet.
©CopyrightPivotalSoftwareInc,2013-2019 4 1.1
InstallingandConfiguringSpringCloud®DataFlowforPCFFollowthebelowstepstoinstallSpringCloudDataFlowforPivotalCloudFoundry(PCF).
PrerequisitesSpringCloudDataFlowforPCFisbuiltusingSpringBoot1.5,whichrequiresversion3.8orlateroftheJavaCloudFoundrybuildpack.ThedefaultJavabuildpack—thebuildpackatthelowestpositionofallJavabuildpacks—onyourPCFinstallationmustthereforebeatversion3.8orlater.
YoucanusetheCloudFoundryCommandLineInterfacetool(cfCLI)toseetheversionoftheJavabuildpackthatiscurrentlyinstalled.
$cfbuildpacksGettingbuildpacks...
buildpackpositionenabledlockedfilenamejava_buildpack_offline1truefalsejava-buildpack-offline-v3.8.1.zipruby_buildpack2truefalseruby_buildpack-cached-v1.6.19.zipnodejs_buildpack3truefalsenodejs_buildpack-cached-v1.5.15.zipgo_buildpack4truefalsego_buildpack-cached-v1.7.10.zip
IfthedefaultJavabuildpackisolderthanversion3.8,youcandownloadanewerversionfromPivotalNetwork andupdatePivotalCloudFoundrybyfollowingtheinstructionsintheManagingCustomBuildpacks topic.ToensurethatthenewerbuildpackisthedefaultJavabuildpack,youmaydeleteordisabletheolderbuildpackormakesurethatthenewerbuildpackisinalowerposition.
DependentServicesSpringCloudDataFlowreliesonotherPCFservicesforitsservicebroker,fortheDataFlowservercreatedforeachserviceinstance,andfortheSpringCloudSkipperapplicationcreatedforeachserviceinstance.Youcanconfiguretheseinthetilesettings.Whenfollowingtheinstallationstepsbelow,reviewallofthesesettings,ensuringthatyouhavepreviouslyinstalledthespecifiedservicesandthatthespecifiedserviceplansareactive.
InstallationSteps1. DownloadtheproductfilefromPivotalNetwork .
2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.
3. UndertheImportaProductbutton,click+nexttotheversionnumberofSpringCloud®DataFlowforPCF.Thisaddsthetiletoyourstagingarea.
4. ClickthenewlyaddedSpringCloudDataFlowtile.IntheSettingstab,clickAssignAZsandNetworks.
Selecttheavailabilityzonesforthetiletouse.IntheNetworksection,selectthePAS(orERT)network.
5. IntheSettingstab,clickServiceBroker.
Important:IfanyoftheservicesorserviceplansconfiguredintheDataFlowtilesettingsarenotavailableinyourPCFdeployment,thetileinstallationmayfail.
©CopyrightPivotalSoftwareInc,2013-2019 5 1.1
EntertherelationaldatabaseservicenameandplannamefortheSpringCloudDataFlowtiletouseforstoringitsservicebroker’sserviceinstancedata.
YoucanalsoconfiguretheJavabuildpacktousefordeployingthebrokerandserviceinstanceapplications(bydefault,thetilewillusethehighest-priorityJavabuildpack).
6. IntheSettingstab,clickDataFlowServer.
ConfigurethedefaultservicesandserviceplansusedbytheSpringCloudDataFlowserverthatisdeployedforeachDataFlowserviceinstance.ThesevaluescanbeoverriddenwhencreatingaDataFlowserviceinstance.
7. StillintheSettingstab,clickSkipper.
ConfiguretherelationaldatabaseservicenameandplanusedbytheSpringCloudSkipperapplicationdeployedforeachDataFlowserviceinstance.
8. StillintheSettingstab,clickErrands.
©CopyrightPivotalSoftwareInc,2013-2019 6 1.1
SpringCloudDataFlowhasfourlifecycleerrands.InPCF2.2andlater,eacherrandcanbesettoalwaysrun(On)ortoneverrun(Off).InPCF2.1andearlier,errandscanalsobesettorunconditionally(WhenChanged).PivotalrecommendsthatallSpringCloudDataFlowlifecycleerrandsbesettoalwaysrun(On).
9. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoinstalltheSpringCloud®DataFlowtile.
©CopyrightPivotalSoftwareInc,2013-2019 7 1.1
TileConfigurationSeebelowforinformationaboutconfiguringoptionsintheSpringCloud®DataFlowforPCFtilesettings.
TileConfigurationOptionsTheSpringCloudDataFlowtileincludessettingsforvariousoptions.YoucanconfigurethesebyvisitingtheInstallationDashboardofPivotalCloudFoundry®OperationsManagerandclickingtheSpringCloudDataFlowtile.
ConfigureBrokerDatabaseServiceandPlanBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2productanditsdefault db-small serviceplantoprovisionadatabaseserviceinstancefortheDataFlowservicebroker.IfyouareusingtheMySQLforPCFv2productanditdoesnothaveanactive db-small planorifyouwishtouseanalternativeservice,youmustconfiguretheSpringCloudDataFlowproductfortheserviceandserviceplanyouwishtouse.YoucanconfigureanalternativerelationaldatabaseserviceandserviceplanintheServiceBrokerpaneoftheSpringCloudDataFlowsettings.
Thebroker’sdatabaseserviceshouldbeconfiguredonlyduringtheSpringCloudDataFlowtileinstallationprocess.Ifyouhavealreadycompletedthetileinstallationprocess,donotalterthatsetting(youmaystillconfigurethebroker’sdatabaseserviceplanafterthetileinstallation).
Seebelowformoreinformationabouttheresultsofconfiguringtheservicebroker’srelationaldatabase.
Ifyou andchange then
installthetile
theservicenameorplan
thetileusesthespecifiedserviceandplan.Youmayalsousethedefaultvaluesof p.mysql and db-small .
updatethetile
theserviceplan thetile’srelationaldatabaseserviceinstanceischangedtousethespecifiedplan.Nodataislost.
updatethetile
theservicename
theservicenamechangeisignored.Ifyouhavealsochangedtheserviceplan,thismayleavethebrokerinaninoperativestate.Toresolvethisissue,revertyourchanges.
ConfigureBuildpackforBrokerandServiceInstancesBydefault,theSpringCloudDataFlowservicebrokerapplicationandserviceinstancebackingapplicationsusethebuildpackchosenbytheplatform’sbuildpackdetection;normally,thiswillbethehighest-priorityJavabuildpack.TocausetheseapplicationstouseaparticularJavabuildpackregardlessofpriority,youcansetthenameofthebuildpacktouseintheServiceBrokerpaneoftheSpringCloudDataFlowtilesettings.
IntheBuildpackfield,enterthenameofthedesiredJavabuildpack.ClickSave,returntotheInstallationDashboard,andapplyyourchanges.Thebrokerapplicationandserviceinstanceapplicationswillnowusetheselectedbuildpack.
ConfigureDataFlowServerDependentServicesBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2,RabbitMQforPCF,andRedisforPCFproductstoprovisiondependentservice
Note:OpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .
warning:Configuringtheservicebroker’sdependentrelationaldatabaseserviceafterthetileinstallationhasfinishedcanresultinorphanedDataFlowserviceinstancesorinmultipledatasetsandcorruptionofthebrokerdatabase’sdata.
©CopyrightPivotalSoftwareInc,2013-2019 8 1.1
instancesfortheDataFlowserverapplicationdeployedforeachDataFlowserviceinstance.Ifyouwishtouseotherservicesortochangetheserviceplansused,youmustconfiguretheservicesandserviceplansyouwishtousefortheDataFlowserver.YoucanconfigurealternateservicesorserviceplansintheDataFlowServerpaneoftheSpringCloudDataFlowsettings.
DisableDataFlowServerInternetAccessYoucandisableinternetaccessforallSpringCloudDataFlowserviceinstances.Todothis,selectthe“DisablepublicInternetaccess”checkboxintheDataFlowServerpaneoftheSpringCloudDataFlowsettings.
ConfigureSkipperDatabaseServiceandPlanBydefault,theSpringCloudDataFlowproductusestheMySQLforPCFv2productanditsdefault db-small serviceplantoprovisionadatabaseserviceinstancefortheSpringCloudSkipperapplicationdeployedforeachDataFlowserviceinstance.IfyouareusingtheMySQLforPCFv2productanditdoesnothaveanactive db-small planorifyouwishtouseanalternativeservice,youmustconfiguretheserviceandserviceplanyouwishtouseforSkipper.YoucanconfigureanalternativerelationaldatabaseserviceandserviceplanintheSkipperpaneoftheSpringCloudDataFlowsettings.
EnableorDisableGlobalMarketplaceAccessBydefault,theSpringCloudDataFlowproductenablesaccesstoitsserviceoffering,called p-dataflow ,acrossallorgsinthePCFdeploymentaspartofthetileinstallationprocess.Ifyouwishtomanuallygrantserviceaccesstospecificorganizations,youcanconfigurethedefaultglobalaccessintheServiceAccesspaneoftheSpringCloudDataFlowsettings.
©CopyrightPivotalSoftwareInc,2013-2019 9 1.1
Todisablethedefaultofserviceaccessenabledforallorgs,cleartheEnableglobalaccesstoplansofservicep_dataflowcheckbox.ClickSave,returntotheInstallationDashboard,andapplyyourchanges.YoucannowenableordisableaccesstotheDataFlowserviceofferingforspecificorgs.
©CopyrightPivotalSoftwareInc,2013-2019 10 1.1
TheServiceBrokerandInstancesSeebelowforinformationaboutSpringCloud®DataFlow’sdeploymentmodelandotherinformationwhichmaybeusefulinadministeringDataFlowserviceinstancesordeployedapplications.
TheServiceBrokerSpringCloudDataFlowprovidesaSpringCloudDataFlowserverasaManagedService onPivotalCloudFoundry (PCF).ItusesCloudFoundry’sServiceBrokerAPI tomanagethisservice.SeebelowforinformationaboutSpringCloudDataFlow’sbrokerimplementation.
TheSpringCloudDataFlowservicebroker’sfunctionalityiscontainedinthefollowingSpringBootapplicationinstance,whichisdeployedinthe“system”organizationtothe“p-dataflow”space.
p-dataflow-[version]:ImplementstheServiceBrokerAPItoactonprovision,deprovision,bind,andunbindrequests.
ThebrokerreliesontheMySQLforPivotalCloudFoundryv2 productforthefollowingserviceinstance.
p-dataflow-db:AMySQLdatabaseusedasabackingstorefortheservicebroker.
ServiceBrokerUpgradesTheSpringCloudDataFlowproductupgradeprocesschecksbeforeredeployingtheservicebrokertoseewhethertheproduct’sversionhaschanged.Iftheversionhasnotchanged,theupgradeprocesswillcontinuewithoutredeployingtheservicebroker.
Theservicebrokerapplicationisdeployedusingablue-greendeploymentstrategy .Duringanupgradeoftheservicebroker,thebrokerwillcontinueprocessingrequeststoprovision,deprovision,bind,andunbindserviceinstances,withoutdowntime.
AccessViaAppsManagerToviewthebrokerapplicationinPivotalCloudFoundry®AppsManager,logintoAppsManagerasanadminuserandselectthe“system”org.
Note:OpsManageradministratorscanuseRole-BasedAccessControl(RBAC)tomanagewhichoperatorscanmakedeploymentchanges,viewcredentials,andmanageuserrolesinOpsManager.Therefore,yourrolepermissionsmightnotallowyoutoperformeveryprocedureinthisoperatorguide.FormoreinformationaboutrolesinOpsManager,seeUnderstandRolesinOpsManager .
Note:Youcanconfigureanalternaterelationaldatabaseserviceforthebrokertouse.SeetheConfigureBrokerDatabaseServiceandPlan
sectionoftheTileConfiguration topic.
©CopyrightPivotalSoftwareInc,2013-2019 11 1.1
Theapplicationisdeployedinthe“p-dataflow”space.
GetBrokerBuildInformationTheSpringCloudDataFlowbrokerprovidesbuildinformationusingtheSpringBootActuator info endpoint,whichismappedto /info .Youcanaccessthisendpointbyappending /info tothepathoftheSpringCloudDataFlowbroker.
IftheSpringCloudDataFlowservicebrokerapplicationislocatedatthefollowingURL:
https://p-dataflow.apps.wise.com
thenyoucanaccessthe info endpointbyvisiting:
https://p-dataflow.apps.wise.com/info
TheservicebrokerreturnsaJSONresponse,asinthefollowingexample.
{"git":{"commit":{"time":1526673193000,"id":"740fb80"},"branch":"740fb80a6e1d83de033a4b81bf29c8b4cfa83b74"},"build":{"version":"1.1.0-build.19","artifact":"scdf-for-pcf-service-broker","name":"scdf-for-pcf-service-broker","group":"io.pivotal.springcloud.dataflow","time":1526673593000}}
Theresponsecontainsinformationaboutthebuildoftheservicebrokerapplication,includingtheMavenprojectcoordinatesandbuildtime.ItalsocontainsinformationabouttheGitrepositoryforSpringCloudDataFlowforPCFatbuildtime.
ServiceInstanceArchitecture
Note:FieldssuchasthoseforGitrepositoryinformationarefordiagnosticpurposesandintendedtoprovidePivotalSupport withinformationtohelpintroubleshooting.
©CopyrightPivotalSoftwareInc,2013-2019 12 1.1
ForeachSpringCloudDataFlowserviceinstancecreated,theservicebrokerprovisionsthefollowingresources,allwithinthespacefromwhichtheserviceinstancewascreated(“theuserspace”)unlessnotedotherwise.
Anewspacewithinthe“p-dataflow”org,namedusingtheserviceinstanceGUIDandcontaining:
ADataFlowserverapplication.ADataFlowmetricsapplication.ASpringCloudSkipperpackagemanagementapplication.Arelationaldatabaseservice,boundtotheDataFlowserverapplication.
Arelationaldatabaseservice,boundtotheSkipperpackagemanagementapplication.Amessagingdataservice,boundtotheDataFlowmetricsapplication.
ARedisdatabaseservice,boundtotheDataFlowserverapplication.
Thefollowingresourcesarecreatedintheoriginatinguserspacewheretheserviceinstancecommandwastargetedat:
A“p-dataflow”serviceinstance.A“p-dataflow”relationaldatabaseserviceinstance(providingaccesstotherelationaldatabaseservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).A“p-dataflow”messagingserviceinstance(providingaccesstothemessagingdataservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).A“p-dataflow”analyticsserviceinstance(providingaccesstotheanalyticsservicecreatedintheserviceinstance’sspacewithinthe“p-dataflow”org).
CapacityRequirementsBelowaretheusagerequirementsoftheSpringCloudDataFlowservicebroker.
Application MemoryAllocated DiskAllocation
ServiceBroker 2GB 1GB
Theservicebrokerisboundtoarelationaldatabaseserviceinstance,whichstoresdatarelatingtothebroker’sserviceinstances.Therelationaldatabaseservicetouseisconfigurable,asdescribedinTheServiceBrokersectionabove.
BelowaretheusagerequirementsoftheDataFlowserverandmetricsapplicationsthatbackeachSpringCloudDataFlowserviceinstance.
BackingApplication MemoryAllocation/AppInstance DiskAllocation/AppInstance
DataFlowServer 2GB 2GB
DataFlowMetrics 1GB 1GB
Note:Thisrelationaldatabaseserviceisa“p.mysql”serviceinstancebydefault.Youcanconfigureanalternaterelationaldatabasewhenyoucreatetheserviceinstance.
Note:Thismessagingdataserviceisa“p-rabbitmq”serviceinstancebydefault.Youcanconfigureanalternatemessagingservicewhenyoucreatetheserviceinstance.
Note:ThisRedisdatabaseserviceisa“p-redis”serviceinstancebydefault.YoucanconfigureanalternateRedisdatabasewhenyoucreatetheserviceinstance.
©CopyrightPivotalSoftwareInc,2013-2019 13 1.1
SpringCloudSkipper 1GB 2GB
BackingApplication MemoryAllocation/AppInstance DiskAllocation/AppInstance
SpringCloudDataFlowserviceinstancesarealsobackedbyinstancesofotherPCFservices.TheseareeitherservicesfromPCFdataserviceproductsorcustomservicesprovidedtoaDataFlowserviceinstanceatcreatetime.Thesedataservicesincludearelationaldatabaseservice,amessagingservice,andananalyticsdatabaseserviceinstanceforeachDataFlowserviceinstancecreated.TheSkipperbackingapplicationusesaMySQLdatabaseserviceforitsbackingstore.
©CopyrightPivotalSoftwareInc,2013-2019 14 1.1
GettingStartedwithSpringCloud®DataFlowforPCF
SeebelowforstepstogetstartedusingSpringCloudDataFlowforPCF.TheexamplesbelowuseSpringCloudDataFlowforPCFtoquicklycreateadatapipeline.
CreatingaDataPipelineUsingtheShellCreateaSpringCloudDataFlowserviceinstance(seetheCreatinganInstancesectionoftheManagingServiceInstancestopic).IfyouusethedefaultbackingdataservicesofMySQLforPCFv2,RabbitMQforPCF,andRedisforPCF,youcanthenimporttheSpringCloudDataFlowOSS“RabbitMQ+Maven”streamappstarters .
StarttheDataFlowshellusingthe cfdataflow-shell commandaddedbytheSpringCloudDataFlowforPCFcfCLIplugin:
$cfdataflow-shelldata-flow...WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>
ImportthestreamappstartersusingtheDataFlowshell’s appimport command:
dataflow:>appimporthttps://bit.ly/Celsius-SR2-stream-applications-rabbit-mavenSuccessfullyregistered65applicationsfrom[source.sftp,source.mqtt.metadata,sink.mqtt.metadata,...processor.scriptable-transform.metadata]
Withtheappstartersimported,youcanusethree–the http source,the split processor,andthe log sink–tocreateastreamthatconsumesdataviaanHTTPPOSTrequest,processesitbysplittingitintowords,andoutputstheresultsinlogs.
CreatethestreamusingtheDataFlowshell’s streamcreate command:
dataflow:>streamcreate--namewords--definition"http|splitter--expression=payload.split('')|log"Creatednewstream'words'
Next,deploythestream,usingthe streamdeploy
command:
dataflow:>streamdeploywordsDeploymentrequesthasbeensentforstream'words'
CreatingaDataPipelineUsingtheDashboardCreateaSpringCloudDataFlowserviceinstance(seetheCreatinganInstancesectionoftheManagingServiceInstancestopic).IfyouusethedefaultbackingdataservicesofMySQLforPCFv2,RabbitMQforPCF,andRedisforPCF,youcanthenimporttheSpringCloudDataFlowOSS“RabbitMQ+Maven”streamappstarters .
InAppsManager,visittheSpringCloudDataFlowserviceinstance’spageandclickManagetoaccessitsdashboard.
Thiswilltakeyoutothedashboard’sAppstab,whereyoucanimportapplications.
ConsiderinstallingtheSpringCloudDataFlowforPCFandServiceInstanceLogscfCLIplugins.SeetheUsingtheShellandViewingServiceInstanceLogstopics.
TheexamplesinthistopicusetheSpringCloudDataFlowforPCFcfCLIplugin.
©CopyrightPivotalSoftwareInc,2013-2019 15 1.1
ClickBulkImportApplication(s)toimportthe“RabbitMQ+Maven”streamappstarters.IntheUrifield,enterhttps://bit.ly/Celsius-SR2-stream-applications-rabbit-maven .ThenclickImportTheApplication(s).
Withtheappstartersimported,visittheStreamstab.Youcanusethreeoftheimportedstarterapplications—the http source,the split processor,andthe log sink—tocreateastreamthatconsumesdataviaanHTTPPOSTrequest,processesitbysplittingitintowords,andoutputstheresultsinlogs.
ClickCreateAStreamtoenterthestreamcreationview.Intheleftsidebar,searchforthe http sourceapplication.Clickitanddragitontothecanvastobegindefiningastream.
©CopyrightPivotalSoftwareInc,2013-2019 16 1.1
Searchforandaddthe splitter processorapplicationand log sinkapplication.
Clickthe splitter application,thenclickthegeariconbesideittoedititsproperties.Intheexpressionfield,enter payload.split('') .ClickOK.
Clickanddragbetweentheoutputandinputportsontheapplicationstoconnectthemandcompletethestream.
©CopyrightPivotalSoftwareInc,2013-2019 17 1.1
ClicktheCreateStreambutton.Typethename“words”,thenclickCreateTheStream.
TheStreamstabnowdisplaysthenewstream.Clickthe▶buttontodeploythestream.
ClickDeployTheStream.
©CopyrightPivotalSoftwareInc,2013-2019 18 1.1
UsingtheDeployedDataPipelineYoucanrunthe cfapps commandtoseetheapplicationsdeployedaspartofthestream:
$cfappsGettingappsinorgmyorg/spacedevasuser...OK
namerequestedstateinstancesmemorydiskurlsRWSDZgk-words-http-v1started1/11G1GRWSDZgk-words-http-v1.apps.wise.comRWSDZgk-words-log-v1started1/11G1GRWSDZgk-words-log-v1.apps.wise.comRWSDZgk-words-splitter-v1started1/11G1GRWSDZgk-words-splitter-v1.apps.wise.com
Runthe cflogs commandonthe RWSDZgk-words-log-v1 application:
$cflogsRWSDZgk-words-log-v1RetrievinglogsforappRWSDZgk-words-log-v1inorgmyorg/spacedevasuser...
Then,inaseparatecommandline,usetheDataFlowshell(startedusingthe cfdataflow-shell command)tosendaPOSTrequesttotheRWSDZgk-words-http-v1 application:
$cfdataflow-shelldataflow...WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>httppost--targethttp://RWSDZgk-words-http-v1.apps.wise.com--data"Thisisatest">POST(text/plain)http://RWSDZgk-words-http-v1.apps.wise.comThisisatest>202ACCEPTED
Watchfortheprocesseddatainthe RWSDZgk-words-log-v1 application’slogs:
2018-06-07T16:47:08.80-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.808INFO16---[plitter.words-1]RWSDZgk-words-log-v1:This2018-06-07T16:47:08.81-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.810INFO16---[plitter.words-1]RWSDZgk-words-log-v1:is2018-06-07T16:47:08.82-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.820INFO16---[plitter.words-1]RWSDZgk-words-log-v1:a2018-06-07T16:47:08.82-0500[APP/PROC/WEB/0]OUT2018-06-0721:47:08.822INFO16---[plitter.words-1]RWSDZgk-words-log-v1:test
©CopyrightPivotalSoftwareInc,2013-2019 19 1.1
ManagingServiceInstances
SeebelowforinformationaboutmanagingDataFlowserviceinstancesusingtheCloudFoundryCommandLineInterfacetool(cfCLI).YoucanalsomanageDataFlowserviceinstancesusingPivotalCloudFoundry®AppsManager.
CreatinganInstance
Beginbytargetingthecorrectorgandspace.
$cftarget-omyorg-sdevelopmentapiendpoint:https://api.system.wise.comapiversion:2.75.0user:userorg:myorgspace:development
YoucanviewplandetailsfortheDataFlowproductusing cfmarketplace-s
.
$cfmarketplaceGettingservicesfrommarketplaceinorgmyorg/spacedevelopmentasuser...OK
serviceplansdescriptionp-dataflowstandardDeploysSpringCloudDataFlowserverstoorchestratedatapipelinesp-dataflow-mysqlproxyProxiestotheSpringCloudDataFlowMySQLserviceinstancep-dataflow-rabbitmqproxyProxiestotheSpringCloudDataFlowRabbitMQserviceinstancep-dataflow-redisproxyProxiestotheSpringCloudDataFlowRedisserviceinstance
TIP:Use'cfmarketplace-sSERVICE'toviewdescriptionsofindividualplansofagivenservice.
$cfmarketplace-sp-dataflowGettingserviceplaninformationforservicep-dataflowasuser...OK
serviceplandescriptionfreeorpaidstandardStandardPlanfree
SettingtheBuildpackEachDataFlowserviceinstancecanbegiventhenameofabuildpacktousefordeployingstreamandtaskapps.Youcansetthebuildpackfortheserviceinstanceusinga buildpack parametergivento cfcreate-
service.Tocreateaserviceinstancethatusesabuildpacknamed custom-java-buildpack todeploy
apps,youmightrun:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"buildpack":"custom-java-buildpack"}'
SettingDependentServicesEachDataFlowserviceinstanceusesthreedependentdataservices.Defaultsfortheseservicescanbeconfiguredinthetilesettings,andthesedefaultscanbeoverriddenforeachindividualserviceinstanceatcreatetime.
GeneralparametersusedtoconfiguredependentdataservicesforaDataFlowserviceinstancearelistedbelow.
Parameter Function
relational-data-service.name
ThenameoftheservicetouseforarelationaldatabasethatstoresSpringCloudDataFlowmetadataandtaskhistory.
relational-data-service.plan
Thenameoftheserviceplantousefortherelationaldatabaseservice.
messaging-data-service.name ThenameoftheservicetouseforaRabbitMQorKafkaserverthatfacilitateseventmessaging.
messaging-data-service.plan ThenameoftheserviceplantousefortheRabbitMQorKafkaservice.
analytics-data-service.name ThenameoftheservicetouseforaRedisserverthatstoresanalytics.
analytics-data-service.plan ThenameoftheserviceplantousefortheRedisserver.
skipper-relational.name ThenameoftheservicetouseforarelationaldatabaseusedbytheSkipperapplication.
skipper-relational.plan ThenameoftheserviceplantouseforarelationaldatabaseusedbytheSkipperapplication.
Important:IfyouareusingtheRedisforPCF productfortheSpringCloudDataFlowanalyticsstore,youcannotcreatemoreDataFlowserviceinstancesthanthesettingoftheRedisproduct’sServiceInstanceLimit(thedefaultServiceInstanceLimitforRedisforPCFis5).SeetheShared-VMPlan sectionoftheInstallingandUpgradingRedisforPCF topicintheRedisforPCFdocumentation forinformationaboutconfiguringthislimit.
Note:Theserviceofferingswiththeplan proxy areproxyservicesusedbySpringCloudDataFlowforPCFserviceinstances.TheSpringCloudDataFlowservicebrokercreatesanddeletesinstancesoftheseservicesautomaticallyalongwitheachSpringCloudDataFlowserviceinstance.Donotmanuallycreateordeleteinstancesoftheseservices.
©CopyrightPivotalSoftwareInc,2013-2019 20 1.1
SettingMavenPropertiesEachDataFlowserviceinstancecanoptionallyspecifyMavenconfigurationproperties.Forthecompletelistofpropertiesthatcanbespecified,seethe“Maven”sectionintheOSSSpringCloudDataFlowdocumentation .
MavenconfigurationpropertiescanbesetforeachDataFlowserviceinstanceusingparametersgivento cfcreate-service
.Tosetthe
maven.remote-repositories.repo1.url property,youmightuseacommandsuchasthefollowing:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"maven.remote-repositories.repo1.url":"https://repo.spring.io/libs-snapshot"}'
CreatingtheInstanceCreatetheserviceinstanceusing cfcreate-
service.TocreateaDataFlowservicethatusesaRedisCloudserviceavailablefromyourPCFmarketplaceand
setstheMaven maven.remote-repositories.repo1.url propertyto https://repo.spring.io/release ,youmightrun:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"analytics-data-service":{"name":"rediscloud","plan":"30mb"},"maven.remote-repositories.repo1.url":"https://repo.spring.io/libs-snapshot"}'Creatingserviceinstancedata-flowinorgmyorg/spacedevelopmentasuser...OK
Createinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.
Asthecommandoutputsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.Whentheserviceinstanceisready,the cfservice commandwillgiveastatusof create
succeeded:
$cfservicedata-flow
Serviceinstance:data-flowService:p-dataflowBoundapps:Tags:Plan:standardDescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesDocumentationurl:http://cloud.spring.io/spring-cloud-dataflow/Dashboard:https://p-dataflow.apps.wise.com/instances/f09e5c77-e526-4f49-86d6-721c6b8e2fd9/dashboard
LastOperationStatus:createsucceededMessage:CreatedStarted:2017-07-20T18:24:14ZUpdated:2017-07-20T18:26:17Z
UpgradinganInstanceAfteranupgradeoftheSpringCloudDataFlowforPCFproduct,youcanusethe cfupdate-service commandtoupgradeindividualDataFlowserviceinstances.
Beginbytargetingthecorrectorgandspace.
$cftarget-omyorg-sdevelopmentapiendpoint:https://api.system.wise.comapiversion:2.75.0user:userorg:myorgspace:development
Youcanviewallserviceinstancesinthespaceusing cfservices .
$cfservicesGettingservicesinorgmyorg/spacedevelopmentasuser...OK
nameserviceplanboundappslastoperationdata-flowp-dataflowstandardcreatesucceededmysql-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-mysqlproxycreatesucceededrabbitmq-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-rabbitmqproxycreatesucceededredis-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-redisproxycreatesucceeded
UpgradetheDataFlowserviceinstanceusing cfupdate-service ,passingthe -c flagtosetthe upgrade parameterto true .
$cfupdate-servicedata-flow-c'{"upgrade":true}'Updatingserviceinstancedata-flowasuser...OK
Updateinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.
Astheoutputfromthe cfupdate-service commandsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.WhentheDataFlowserviceinstancehasbeenupgraded,the cfservice commandwillgiveastatusof update
succeeded:
©CopyrightPivotalSoftwareInc,2013-2019 21 1.1
$cfservicedata-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...
name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/1cf8ff5b-4a65-469d-bee7-36e6541ac241/dashboard
Showingstatusoflastoperationfromservicedata-flow...
status:updatesucceededmessage:Updatedstarted:2018-06-19T19:26:09Zupdated:2018-06-19T19:29:17Z
DeletinganInstanceDeletingaDataFlowserviceinstancewillresultindeletionofallofitsdependentserviceinstances.
Beginbytargetingthecorrectorgandspace.
$cftarget-omyorg-sdevelopmentapiendpoint:https://api.system.wise.comapiversion:2.75.0user:userorg:myorgspace:development
Youcanviewallserviceinstancesinthespaceusing cfservices .
$cfservicesGettingservicesinorgmyorg/spacedevelopmentasuser...OK
nameserviceplanboundappslastoperationdata-flowp-dataflowstandardcreatesucceededmysql-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-mysqlproxycreatesucceededrabbitmq-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-rabbitmqproxycreatesucceededredis-b3e76c87-c5ae-47e4-a83c-5fabf2fc4f11p-dataflow-redisproxycreatesucceeded
DeletetheDataFlowserviceinstanceusing cfdelete-service
.Whenprompted,enter y toconfirmthedeletion.
$cfdelete-servicedata-flow
Reallydeletetheservicedata-flow?>yDeletingservicedata-flowinorgmyorg/spacedevelopmentasuser...OK
Deleteinprogress.Use'cfservices'or'cfservicedata-flow'tocheckoperationstatus.
ThedependentserviceinstancesfortheDataFlowserverserviceinstancearedeletedfirst,andthentheDataFlowserverserviceinstanceitselfisdeleted.
Astheoutputfromthe cfdelete-service
commandsuggests,youcanusethe cfservices or cfservice commandstocheckthestatusoftheserviceinstance.
WhentheDataFlowserviceinstanceanditsdependentserviceinstanceshavebeendeleted,the cfservices commandwillnolongerlisttheserviceinstance:
$cfservicesGettingservicesinorgmyorg/spacedevelopmentasuser...OK
Noservicesfound
©CopyrightPivotalSoftwareInc,2013-2019 22 1.1
UsingUser-ProvidedServiceInstances
Bydefault,aSpringCloudDataFlowserviceinstanceusesMySQLforPCFforitsbackingrelationaldatabaseservice,RabbitMQforPCFforitsbackingmessagingservice,andRedisforPCFforitsbackinganalyticsservice.WhencreatingaSpringCloudDataFlowserviceinstance,youcanconfigureittouseCloudFoundryuser-providedservices foritsdependentdataservicesinstead.SeebelowforinformationaboutconfiguringaDataFlowserviceinstancetouseanalternativedependentdataservice.
DependentServiceParametersYoucanspecifythataDataFlowserviceinstanceshoulduseauser-providedserviceinstancebypassingconfigurationparameterstothe cfcreate-
service
commandinJSONusingthe -c flag.TheparametersusedtoconfigurealternativedependentdataservicesforaDataFlowserviceinstancearelistedbelow.
Parameter Function
relational-data-service.user-provided
AJSONobjectcontainingconnectioninformationfortherelationaldatabaseserviceusedtostoretaskhistory.
messaging-data-service.user-provided
AJSONobjectcontainingconnectioninformationforthemessagingserviceusedforeventmessaging.
analytics-data-service.user-provided
AJSONobjectcontainingconnectioninformationfortheRedisserviceusedforananalyticsstore.
Seethefollowingsectionsforinformationaboutusingtheseparameters.
User-ProvidedRelationalDatabaseServiceYoucanusethe relational-data-service parametertosupplytheconfigurationforauser-providedrelationaldatabaseservice.ThisparametercontainsaJSONobject user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedrelationaldatabaseserviceinstance.
TherelationaldatabaseservicecanbethedefaultMySQLforPCFv2serviceoranyotherrelationaldatabaseservice.Itmustprovidearelationaldatabase.
Anexampleoffieldscontainedinthe user-provided objectforarelationaldatabaseserviceisshownbelow.
{"uri":"mysql://kempercrabb:[email protected]:3306/vigil","username":"kempercrabb","password":"fake-password","dbname":"vigil","host":"123.234.456.879","port":3306,"tags":["mysql"]}
TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’srelationaldatabaseserviceinstance:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"relational-data-service":{"user-provided":{"uri":"mysql://kempercrabb:[email protected]:3306/vigil","username":"kempercrabb","password":"fake-password","dbname":"vigil","host":"123.234.456.879","port":3306,"tags":["mysql"]}}}'
User-ProvidedMessagingServiceYoucanusethe messaging-data-service parametertosupplytheconfigurationforauser-providedmessagingservice.ThisparametercontainsaJSONobject
user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedmessagingserviceinstance.
ThemessagingdataservicecanbethedefaultRabbitMQforPCFserviceoranothermessagingservice.ItmustprovideeitheraRabbitMQserveroraKafkaserver.
Anexampleoffieldscontainedinthe user-provided objectforamessagingserviceisshownbelow.
{"dashboard_url":"https://api.cloudamqp.com/console/a-GUID-would-go-here/details","username":"buckstorm","vhost":"buckstorm","password":"fake-password","ssl":false,"hostname":"stonehill.rmq.cloudamqp.com","uri":"amqp://buckstorm:[email protected]/buckstorm","http_api_uri":"http://buckstorm:[email protected]:1883/api","tags":["rabbitmq"]}
TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’smessagingserviceinstance:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"messaging-data-service":{"user-provided":{"dashboard_url":"https://api.cloudamqp.com/console/a-GUID-would-go-here/details","username":"buckstorm","vhost":"buckstorm","password":"fake-password","ssl":false,"hostname":"stonehill.rmq.cloudamqp.com","uri":"amqp://buckstorm:[email protected]/buckstorm","http_api_uri":"http://buckstorm:[email protected]:1883/api","tags":["rabbitmq"]}}}'
©CopyrightPivotalSoftwareInc,2013-2019 23 1.1
User-ProvidedAnalyticsServiceYoucanusethe analytics-data-service parametertosupplytheconfigurationforauser-providedanalyticsservice.ThisparametercontainsaJSONobject
user-provided ,withfieldsforeachoftheconnectionvaluesneededtoprovisionaCloudFoundryuser-providedRedisserviceinstance.
TheanalyticsservicecanbethedefaultRedisforPCFserviceoranyotherRedisservice.ItmustprovideaRedisdatabase.
Anexampleoffieldscontainedinthe user-provided objectforananalyticsserviceisshownbelow.
{"host":"garrels.gce.cloud.redislabs.com","uri":"redis://:[email protected]:11781/boars-head","port":11781,"dbname":"boars-head","password":"fake-password","tags":["redis"]}
TocreateaSpringCloudDataFlowserviceinstanceusingtheseconnectionvaluesfortheserviceinstance’sanalyticsserviceinstance:
$cfcreate-servicep-dataflowstandarddata-flow-c'{"analytics-data-service":{"user-provided":{"host":"garrels.gce.cloud.redislabs.com","uri":"redis://:[email protected]:11781/boars-head","port":11781,"dbname":"boars-head","password":"fake-password","tags":["redis"]}}}'
©CopyrightPivotalSoftwareInc,2013-2019 24 1.1
ViewingServiceInstanceLogs
SpringCloudDataFlowforPCFprovidesaccesstothelogsgeneratedbyeachDataFlowserverserviceinstance,includinglogsforeachofthethreebackingapplications(DataFlowserverapplication,metricsapplication,andSkipperapplication)foreachinstance.YoucanviewtheselogseitherusingtheServiceInstanceLogscfCLIpluginorbyvisitingthedashboardoftheSpringCloudDataFlowservicebroker.
UsingthecfCLIPluginAfterinstallingtheServiceInstanceLogscfCLIplugin (seetheinstructionsintheInstalling sectionoftheplugin’sREADME ),youcanusethe
service-logs commandtotaillogsordumprecentlogsforaserviceinstance.
TotaillogsforaDataFlowserviceinstance,run cfservice-logsSERVICE_NAME ,where SERVICE_NAME isthenameoftheserviceinstance:
$cfservice-logsdata-flow
Todumprecentlogsfortheinstance,usethe --recent flag:
$cfservice-logs--recentdata-flow
IfyourPivotalCloudFoundrydeploymentusesaself-signedcertificate,youmustusethe --skip-ssl-validation flagtodisablethedefaultvalidationoftheplatform’sSSLcertificate:
$cfservice-logs--skip-ssl-validationdata-flow
UsingtheServiceBrokerDashboard
VisittheSpringCloudDataFlowservicebroker’sdashboard.YoucanaccessitatthefollowingURL,whereapps.wise.com istheapplicationdomainofyourPCFdeployment:
https://p-dataflow.apps.wise.com/
Thedashboardshowsthename,org,andspaceofeachserviceinstance,aswellasalinktoviewlogsfortheinstance.
ClicktheLogslinktoviewlogsforaparticularserviceinstance’sbackingapplication.
Youcanstreamcurrentlogsfortheinstancebyclickingthe▶button.
Note:Toaccesstheservicebrokerdashboard,youmustbeaSpaceDeveloperinthebrokerapplication’sspace(thisistypicallythe system organd p-dataflow space).
©CopyrightPivotalSoftwareInc,2013-2019 25 1.1
ReadingAggregatedLogsThelogsretrievedbytheServiceInstanceLogscfCLIpluginaggregatelogsfromthreebackingapplications:aSpringCloudDataFlowserverapplication,ametricsapplication,andaSpringCloudSkipperapplication.Thefollowingexcerptshowslogsafterdeployingastream:
2018-02-09T11:12:02.45-0600[RTR/dataflow0]OUTdataflow-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com-[2018-02-09T17:12:02.429+0000]"GET/metrics/streams?names=httptestHTTP/1.1"200014765"https://dataflow-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com/dashboard/""Mozilla/5.0(Macintosh;IntelMacOSX10_13_2)AppleWebKit/537.36(KHTML,likeGecko)Chrome/63.0.3239.132Safari/537.36""130.211.0.191:63089""10.0.4.8:61018"x_forwarded_for:"71.10.174.240,35.201.75.216,130.211.0.191"x_forwarded_proto:"https"vcap_request_id:"adc3ee08-a97a-41ae-71d4-5fc2dff4a82c"response_time:0.025969938app_id:"7e61bf00-2a8c-4cc9-833a-9dd92442ff47"app_index:"0"x_b3_traceid:"6862348a02fa27db"x_b3_spanid:"6862348a02fa27db"x_b3_parentspanid:"-"
2018-02-09T11:12:02.45-0600[RTR/metrics0]OUTdf-metrics-11f71dd3-f902-4e31-b631-fbbbfdb82459.apps.americancanyon.cf-app.com-[2018-02-09T17:12:02.444+0000]"GET/collector/metrics/streamsHTTP/1.1"200022261"-""Apache-HttpClient/4.5.3(Java/1.8.0_144)""130.211.3.158:53467""10.0.4.35:61004"x_forwarded_for:"35.184.235.101,35.201.75.216,130.211.3.158"x_forwarded_proto:"https"vcap_request_id:"87c87336-863e-424d-6ebe-3fa85f6545e0"response_time:0.007074433app_id:"369ae87c-b9a6-4801-a4c5-61b779ff4f71"app_index:"0"x_b3_traceid:"60c018a7f484a97e"x_b3_spanid:"60c018a7f484a97e"x_b3_parentspanid:"-"
2018-02-09T11:12:02.60-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:02.603INFO15---[ry-client-nio-3]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forhttptest-http-v12018-02-09T11:12:02.94-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:02.940INFO15---[ry-client-nio-3]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forastream-log-v42018-02-09T11:12:03.06-0600[APP/PROC/WEB/skipper0]OUT2018-02-0917:12:03.068INFO15---[ry-client-nio-1]o.s.c.d.s.c.CloudFoundryAppDeployer:Successfullycomputedstatus[deployed]forastream-time-v4
TheDataFlowserverapplication’slogsareidentifiedasbelongingtothe dataflow application.Themetricsapplicationsimilarlyiscalled metrics .TheSpringCloudSkipperapplicationiscalled skipper .
©CopyrightPivotalSoftwareInc,2013-2019 26 1.1
UsingtheDashboard
Tofindthedashboard,navigateinPivotalCloudFoundry®AppsManagertotheDataFlowserviceinstance’sspace,clickthelistingfortheserviceinstance,andthenclickManage.
Ifyouareusingversion6.8.0orlateroftheCloudFoundryCommandLineInterfacetool(cfCLI),youcanalsouse cfserviceSERVICE_NAME ,whereSERVICE_NAME isthenameoftheDataFlowserviceinstance:
$cfservicedata-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...
name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/2c9a66d3-99d8-458c-860d-01e7ac64eb1d/dashboard
Showingstatusoflastoperationfromservicedata-flow...
status:createsucceededmessage:Createdstarted:2018-05-17T19:02:04Zupdated:2018-05-17T19:08:14Z
VisittheURLgivenfor“Dashboard”.
DashboardInformationThedashboardprovidesanoverviewofregisteredapplications,streamandtaskdefinitions,andbatchjobs.Italsoprovidescontrolsfordeployingstreams,launchingtasks,andrestartingbatchjobexecutions.
Forcompleteinformationaboutthedashboardanditsprovidedfunctionality,seetheOSSSpringCloudDataFlowproject’sdocumentation .
©CopyrightPivotalSoftwareInc,2013-2019 27 1.1
UsingtheShell
Theopen-sourceSpringCloudDataFlow projectprovidesashell ,whichcanbeusedtointeractlocallywithaDataFlowserviceinstancedeployedonPivotalCloudFoundry(PCF).
YoucanusetheshellwithSpringCloudDataFlowforPCFserviceinstancesineitheroftwoways:
TheSpringCloudDataFlowforPCFCloudFoundryCLIplugin(recommended)
Theopen-sourceshellbinary,withmanually-configuredcommand-lineoptions
TheSpringCloudDataFlowforPCFCloudFoundryCLIpluginwascreatedtoeasetheuseoftheSpringCloudDataFlowshellwithSpringCloudDataFlowserviceinstancesonPCF.
UsingtheCloudFoundryCLIPluginThepluginaddsa dataflow-shell command,whichattachestheDataFlowshelltoaSpringCloudDataFlowforPCFserviceinstance.Givenanexistingserviceinstancenamed data-flow ,thefollowingcommandwilldownloadtheappropriateshellversionandattachittotheserviceinstance:
$cfdataflow-shelldata-flowAttachingshelltodataflowservicedata-flowinorgmyorg/spacedevasuser...LaunchingdataflowshellJAR___________/___|______(_)______/___||_______||\___\|'_\|'__||'_\/_`|||||/_\||||/_`|___)||_)|||||||(_||||___||(_)||_||(_|||____/|.__/|_||_|_||_|\__,|\____|_|\___/\__,_|\__,_|____|_|___|___/__________|_\___||____|___||_______\\\\\\||||/_`|__/_`|||_||/_\\/\//\\\\\\||_||(_||||(_|||_|||(_)\VV///////|____/\__,_|\__\__,_||_||_|\___/\_/\_//_/_/_/_/_/
1.5.0.RELEASE
WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>
FormoreinformationabouttheSpringCloudDataFlowshell,seeitsdocumentation .
UsingtheShellManuallyIfyouwouldliketodownloadandconfiguretheDataFlowshellmanually,downloadtheshellJARfilefromtheSpringReleasesMavenrepository.SpringCloudDataFlowforPCFcanbeusedwithversion1.5.0.RELEASEorlateroftheSpringCloudDataFlowshell.
TotargetaDataFlowserviceinstance’sserverwiththeshell,youmustobtaintheserver’sURL.Run cfserviceSERVICE_NAME ,where SERVICE_NAME isthenameoftheserviceinstance:
$cfservicedata-flow
Serviceinstance:data-flowShowinginfoofservicedata-flowinorgmyorg/spacedevasuser...
name:data-flowservice:p-dataflowboundapps:tags:plan:standarddescription:DeploysSpringCloudDataFlowserverstoorchestratedatapipelinesdocumentation:dashboard:https://p-dataflow.apps.wise.com/instances/2f6ec0c6-c828-45bb-905a-4779ce50552b/dashboard
Showingstatusoflastoperationfromservicedata-flow...
status:createsucceededmessage:Createdstarted:2018-05-17T14:43:38Zupdated:2018-05-17T14:49:43Z
VisittheURLgivenfor“Dashboard”andauthenticateusingyourPCFcredentials.Whenredirectedtotheserviceinstance’sdashboard,copythedashboard’sdomainname(theURLminusthepathfollowingthedomain).
Forexample,giventhefollowingURL:
https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com/dashboard/#/apps
Copythefollowing:
Note:ToruntheSpringCloudDataFlowshell,youmusthaveaJavaRuntimeEnvironment(JRE)installed.YoucandownloadtheJREfromtheJavawebsite .
Note:BeforeconnectingtoaDataFlowserviceinstancewiththeSpringCloudDataFlowshell,besuretologintothePCFdeploymentusingtheCloudFoundryCommandLineInterfacetool(cfCLI)andtargettheorgandspaceoftheDataFlowserviceinstance.TheDataFlowshellusesthecfCLItoauthenticatetoPCF.
©CopyrightPivotalSoftwareInc,2013-2019 28 1.1
https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com
FromthedirectorycontainingtheshellJARfile,runtheshellfromthecommandlineusingacommandasshownbelow.
$java-jarJAR_NAMESKIP_VALIDATION--dataflow.uri=SERVER_URL--dataflow.credentials-provider-command="cfoauth-token"--dataflow.mode=skipper
Inthiscommand,replacethefollowingplaceholdersasshownbelow.
JAR_NAME withthenameofthedownloadedSpringCloudDataFlowshellJARfile
SKIP_VALIDATION withtheflagandvalue --dataflow.skip-ssl-validation=true ifyourPCFinstallationisusingself-signedSSLcertificates;otherwiseremove
SERVER_URL withtheDataFlowserviceinstance’sserverURL(thedomaincopiedabove)
Thecompletecommandmaylooksomethinglikethefollowing:
$java-jarspring-cloud-dataflow-shell-1.5.0.RELEASE.jar\--dataflow.skip-ssl-validation=true\--dataflow.uri=https://dataflow-2f6ec0c6-c828-45bb-905a-4779ce50552b.apps.wise.com\--dataflow.credentials-provider-command="cfoauth-token"--dataflow.mode=skipper
Whenyourunthe java-jar
command,theshellwillinitializeandgivea dataflow:> prompt.
WelcometotheSpringCloudDataFlowshell.ForassistancehitTABortype"help".dataflow:>
Foralistofavailableshellcommands,run help fromwithintheshell.Forinformationabouttheflagswhichcanbeprovidedtotheshellonstartup,seetheOSSSpringCloudDataFlowproject’sdocumentation .
©CopyrightPivotalSoftwareInc,2013-2019 29 1.1
Top Related