Training AI/ML models using Digital Data Marketplaces
Transcript of Training AI/ML models using Digital Data Marketplaces
![Page 1: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/1.jpg)
TrainingAI/MLmodelsusingDigitalDataMarketplaces
DDSGteleconferenceNovember29th 2018
LeonGommans,AnneSavelkoul,Wouter Kalfsbeek,DirkvandenHerik,DavidLangerveld,ErikIJzermans,FlorisFreeman,Brend Dikkers,CeesdeLaat,TomvanEngers,Wouter Los,PaolaGrosso,JosephHill,ReggieCushing,Giovanni Sileno,LuZhang,Ameneh Deljoo,
ThomasBaeck,WillemKoeman,LaurieStrom,AxelBerg,Gerben vanMalenstein,Kaladhar Voruganti,RodneyWilson, PatriciaFlorissi
![Page 2: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/2.jpg)
BUSINESS CONTEXT
Companies increasinglyunderstandhowtoapplyAI technologiestoextractbusinessvaluefromdata.
Themoredatathe better:algorithmqualitydependsondataquantityandqualityKnowledge howtotranslatesuch dataintoreliable algorithms iscompetitive
Companies arereluctanttosharedatawhenconsidering involvedrisk.
Emergingplatformdominance: “Whilecreatingrealvalueforusers,thesecompaniesarealsocapturingadisproportionateandexpandingshareofthevalue,andthat‘sshapingourcollectiveeconomicfuture”.*
Sharingdataacrosscompaniesincreasesthepotentialof
creatingbusinessvaluenosingleorganizationcancreateonitsown.
* M. Iansiti, K.R. Lakhani, Managing our hub economy, Harvard Business Review, pg. 85-92, Sep/Oct 2017
![Page 3: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/3.jpg)
Consideringvalueexchangeandinvolvedriskraisesthemainresearchquestion:
Howcan(big)dataassetsbesharedbetweendatasuppliersandalgorithmsdevelopersin1) Afairandeconomicway,2) whilstprovidingadequatemeanstoreducerisk?
DATAISINCREASINGLYCONSIDEREDANASSET
![Page 4: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/4.jpg)
CURRENTALGORITHMDEVELOPMENTCONTEXT
DataLake/DataWarehouse
Periodicstorage:raworwithenhancedquality
(Near) Real TimeOperational Data
Decision Support Systems
Planning, Prediction,Prevention,
Effectiveness,Efficiency,
etc.
Algorithm
sAlgorithmSupplierRealTimeusingowndata
System EngineerMaintenance Planner
Howcanaircraftoperateatmaximumsafety- andreliability
levelsatminimalcost?
![Page 5: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/5.jpg)
RESEARCHCONTEXTARRANGEADDITIONALDATATOIMPROVEALGORITHMQUALITY&INNOVATION
AlgorithmDevelopersownorthirdparty
Historic(Big)Data
(Near)RealTimeOperationalData
Decision Support Systems
Planning, Prediction,Prevention,
Effectiveness,Efficiency,
etc.
Datasuppliedbyotherorganizations
Compe
titive
Algorithm
Choice
Competitive
OwnOrganizationData
OwnOrganization
Periodicstorage:raworwithenhancedquality
Dat
a Ex
chan
ge
![Page 6: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/6.jpg)
B2BDATASHARINGAPPROACHESANEUSTUDYBYEVERISJAN2018
Open vs Closed
DifferencewithDataMarketplaces:GovernancebyamembershiporganizationDifferencewithIndustrialDataPlatforms:Dataisstoredoutside dataplatformstoallowmultipleplatformstousesamedataContractsdetermineaccess/useMarketrulesarrangepre-contractualelements
![Page 7: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/7.jpg)
EXAMPLE USE:DEVELOP A DIGITAL TWIN TO ESTIMATE MAINTENANCE CREDITSDATASHARINGCHALLENGESWHENTRAININGMODELSWITHASMUCHDATAASPOSSIBLE
Manyorganizationswanttokeeptheirhistoricaldataintheirsovereigndatazones.
Manyimplicationsneedtobeconsidered:
Datalevel
ProcessingStorageManagementTransportTransformSecurity
Business level
ValueCostBenefitsAgreementsExchangeTrade
Legal level
OwnershipAccessUsageCompliancyLiabilityMarketRules
Worldwide Scale
![Page 8: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/8.jpg)
OVERCOMMINGCHALLENGESELEMENTSTOORGANIZETRUSTASMEANSTOREDUCERISK
COMMON BENEFIT
Defineandagreecommonbenefitnosingleorganizationcanachieveonitsown.
GROUP RULES
Define consortiumrulesconsideringdatause,accessandbenefitsharing
ORGANIZE TRUST
Organizepowerandtrustasameanstoreduceriskforparticipatingmembers
IMPLEMENT INFRASTRUCTURE
ResearchoperationalizationofDigitalDataMarketplace&DataExchangeconcepts
![Page 9: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/9.jpg)
![Page 10: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/10.jpg)
![Page 11: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/11.jpg)
![Page 12: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/12.jpg)
![Page 13: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/13.jpg)
![Page 14: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/14.jpg)
Example:enabledatasharingtoimprovequalityofAI/MLinnovations• Understandneed:themoredatathebetter• Expect:capabilitythatwillhelptransformtheMRObusinessinthedigitalera.
Innovationsthatwillimproveairsafety,passengerexperienceandadditionalcostreductionsby:§avoidingunplannedmaintenance§increasingmaintenanceplanningflexibility§movingfromfixedintervalplanningtomaintenancewhenindicated§lessnetworkdisruptionsbyavoiding‘AircraftOnGround’situations
DEFINEANDAGREECOMMONBENEFIT
![Page 15: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/15.jpg)
TrustisconsideredasameanstoreduceriskDefiningconsortiummembershiprulesisastartingpoint
Legalresearchtopic’sfordiscussion:- Dataassetownership- Dataaccess&usage- Liabilityofowner&user- Non-compliantbehavior- Marketrules- Purposebinding
CONSORTIUMMEMBERSHIPRULES:WHATKINDOFRULESDOWENEED?
![Page 16: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/16.jpg)
DIGITALDATAMARKETPLACECONCEPT:COMBINEDBUSINESS,LEGALANDCOMPUTERSCIENCERESEARCH
Market rules
National Law & Regulations
Member admission
DeploymentSpecification
InfrastructurePatterns
Agreement
Data Exchange Infrastructure
AlgorithmDeveloppers
Registry
DisputeResolution
Accounting & Auditing
Business & Legal ResearchComputer Science ResearchBlockchain/Finance Research
Data suppliers
Digital Data Marketplace Membership Organization
Future Internet Capabilities:Software Definable - No Bandwidth
Limitations, On demand, transaction based
![Page 17: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/17.jpg)
DATAEXCHANGECONCEPTENVISAGEDGLOBALEXCHANGEINFRASTRUCTURE
GlobalDataExchangeInfrastructure
AMDEX
BERDEX
LNDDEX
CHIDEX
NYDEX
SFODEX
SovereignDataOwners(e.g.Airlines)
AutonomousDataSciencePlatformsMarketplaceA MarketplaceB
![Page 18: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/18.jpg)
RESEARCHINGEXCHANGEARCHITECTURES
SovereignDataZones
DataOwnerLayer
AirlineA
DataTransfer&Processing
Node
AirlineB
DataTransfer&Processing
Node
DigitalDataMarketplaceInfrastructureprovidedbyDataExchange
DataTransfer&Processing
Node
DataSciencePlatformLayer
DataScienceDevelopmentPlatform
KLMAmsterdam SiliconValley
DataCage DataCage
Amsterdam
Publiccloud
TrustModelling:Whatistheoptimalinfrastructurearchetype,describingstorageandprocessinglocationsandtheirrelationships,whichbestsuitmemberrequirementswhenconsideringrisk?
See CIENA booth 2847 and demo
ProcessingModels:Whataretheimplicationsofdistributingdataprocessingacrossmembershiporganizationownedinfrastructuresintermsofachievablemodelaccuracyandprocessingperformanceusingfederated/distributedmodelsvscentralizedmodels
MarketplaceReferenceArchitecture:Whatconstitutesamarketplace?Researchingneededfunctions,personas,flows,credentials,contracts&rules,conflictresolution,andmuchmore…
![Page 19: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/19.jpg)
DataSharingInfrastructureModelResearchusing FutureInternetcapabilites
Researchcollaboration
GLOBAL RESEARCHINFRASTRUCTURES GLOBALDATACENTERINFRASTRUCTURES
How to create a Global Digital Data MarketEcosystem via Data Exchanges
AM3 and AM4DatacentersAmsterdamScience Park
SV10Datacenter
Silicon Valley
RESEARCHINGPHYSICALIMPLEMENTATIONINVOLVINGBOTHRESEARCHANDITINDUSTRY
![Page 20: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/20.jpg)
DC4
AP
Internet
DC1 DC2 DC3
Customer P Processing
Algorithm
Result
Dataset
Contract
Filetransfer
Resultoutput
Algorithmcopy
Remotefilesystemmount
ContainerorVM
ContractDrivenSlice
A
Traditionalmodel
DC4actsasplatform:1:createspotentialcompetitivebottleneck/lock-in.and2:raisesdataownerconcernsaboutrisk
![Page 21: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/21.jpg)
DatacenterEcosystem
DMP
DC1 DC2 DC3
DC4
A
PA
Customer P Processing
Algorithm
Result
Dataset
Contract
Filetransfer
Resultoutput
Algorithmcopy
Remotefilesystemmount
ContainerorVM
ContractDrivenSlice
A
DMPprovidesneutralprocessingcapabilities,whichdissolvesafterExecution.
![Page 22: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/22.jpg)
DatacenterEcosystem
DC1 DC2 DC3
DC4
Customer
DatacenterEcosystem
DC5
DMPP P P
A A AP
A
DataExchangeLocationB
DataExchangeLocationA
PA
PA
P Processing
Algorithm
Result
Dataset
Contract
Filetransfer
Resultoutput
Algorithmcopy
Remotefilesystemmount
ContainerorVM
ContractDrivenSlice
A
![Page 23: Training AI/ML models using Digital Data Marketplaces](https://reader033.fdocuments.in/reader033/viewer/2022051420/627e3093368aee0c9e220efd/html5/thumbnails/23.jpg)
Enterprisesjoinamembershiporganizationtoachieveacommongoalnosingleenterprisecanachieveonitsown
Membershiprulesaredefinedbyrulemaking&standardsprocesses,subsequentlyexecution,enforcementandjudgementisorganizedbymembershiporganizationasameanstoreducerisk.
Membersarrangedatasharingandprocessingviaagreementsdeployedinaninfrastructure,providedbyasecuredigitalmarketplaceownedbyitsmembers.
Membersachievecommonbenefitsinatransparentway.Memberstrustitsoperationbasedonuseofaccounting&auditingmechanisms,relyingonmarketdisputeresolutionmechanisms.
SUMMARY